Was this a "build better sandboxing" and "don't tell people to eat glue" safety leader, or a Roko's Basilisk believing safety leader?
A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
We don't actually need anybody worrying about silly hypothetical scenarios — at least not as paid employees. There are already a surplus of sci-fi authors doing that.
> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
He is a hypocrite for all we care. You work there for a while when your stock is getting vested and suddenly you have this feeling? Like the dude hired a PR firm as well.
While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.
“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems"
... over an unbounded timeframe?
And how exactly?
Those are very round numbers, but also very specific. Can we get some accounting on how you came to that? Anything? Vibes?
I mean, if you want me to take you seriously, let's have a deep discussion with things that can be measured. I absolutely agree that OpenAI and friends aren't being restrained enough and are acting with recklessness, but declarations of doom based on vibes isn't cutting it.
The LeCun article being posted at the same time as this is quite apt.
These "safety" people should have spent more time reading actual cybersecurity textbooks and less time reading EA forums and less wrong (or in Robinson's case, it appears, being policy wonks). Maybe then these labs wouldn't be totally incompetent.
>“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
This sounds more like an attempt at regulatory capture. Current AI systems aren't physical infrastructure that can just run away like a nuclear power plant, for example. At the end of the day, AI is still just software running on someone's hardware.
Why would the people who quit these companies try to push regulatory capture by said companies? Why would the numerous independent AI researchers do that either? Is it all a big conspiracy?
Was this a "build better sandboxing" and "don't tell people to eat glue" safety leader, or a Roko's Basilisk believing safety leader?
A lot of the "AI safety" types are very focused on the latter and not at all concerned with the former. We need both, but we clearly need a much stronger focus on the problems we are seeing now, and much less on the hypothetical problems we might see in the future.
We don't actually need anybody worrying about silly hypothetical scenarios — at least not as paid employees. There are already a surplus of sci-fi authors doing that.
What makes you think, the scenarios in question here would be "silly"?
Is it that "chatbots" can't come out of the screen to immediately harm you physically?
Let's say they simply manage to take down the internet. How many would die?
Given that he’s citing the need to learn from safety in other fields, I’d say the former.
Indeed, from the article:
> “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
He is a hypocrite for all we care. You work there for a while when your stock is getting vested and suddenly you have this feeling? Like the dude hired a PR firm as well.
While the safety and alignment is a real problem, I don’t get this guy or the Anthropic dude. First world problems.
Seems like a character attack that has no bearing on the question of whether external safety intervention is necessary
Working in safety at OpenAI or Anthropic is zeroth world problems.
It's okay to recognize you were wrong even if it's late
Hey now, he probably donated a good chunk to charity
(donor advised fund where he retains complete control, after a 60% tax deduction)
Used to work as human data trainer feeding data to AI companies. OpenAI projects are definitely the most toxic ones.
Are there any realistic ways to achieve AI safety? Whatever that even means. How can they avoid users doing stupid/dangerous stuff with the AI?
Nothing is ever 100% safe, it’s about a right balance of safety to the benefit.
Or, in other words - we have two P(Doom), one for AI being developed, and another for AI being not developed. The latter is not discussed enough imho.
OpenAI is better off with less of these cultists around.
Well this is a great sign for OpenAI. I'm sure the typo inclusive memorandum will save us.
“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems"
... over an unbounded timeframe?
And how exactly?
Those are very round numbers, but also very specific. Can we get some accounting on how you came to that? Anything? Vibes?
I mean, if you want me to take you seriously, let's have a deep discussion with things that can be measured. I absolutely agree that OpenAI and friends aren't being restrained enough and are acting with recklessness, but declarations of doom based on vibes isn't cutting it.
Note: That quote is from a different person than the titular one who quit.
> Geoffrey Irving, who worked at OpenAI and DeepMind before becoming chief scientist of Resolution, also joined the warnings on AI on Saturday.
The LeCun article being posted at the same time as this is quite apt.
These "safety" people should have spent more time reading actual cybersecurity textbooks and less time reading EA forums and less wrong (or in Robinson's case, it appears, being policy wonks). Maybe then these labs wouldn't be totally incompetent.
See also:
https://news.ycombinator.com/item?id=49944227
>“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote.
This sounds more like an attempt at regulatory capture. Current AI systems aren't physical infrastructure that can just run away like a nuclear power plant, for example. At the end of the day, AI is still just software running on someone's hardware.
So… like the Therac-25 radiation accidents? Software bugs do sometimes have physical consequences.
Why would the people who quit these companies try to push regulatory capture by said companies? Why would the numerous independent AI researchers do that either? Is it all a big conspiracy?
I mean, its certainly physical infrastructure that can run away. Just less catastrophic than nuclear reactors
Yes
And the statements of the "doomers" tells us a lot about them, and nothing about the technology