I was so excited to read this, get an inside view of how this problem must actually be much more nuanced than it seems for some reason not observable from the outside, just to read 10k words of whining and pleas for sympathy. When he finally makes it to the technical section it’s just “it’s a lab culture, there’s a lot going on.” There were some extremely inexcusable mistakes made apparent to anyone who knows anything about security at all. If the lab is spinning off experiments faster than you can improve your security posture just say that and own up to it (it’s what he appears to be trying to say in the worlds most mealy-mouthed way).
Lol, what a pathetic attempt to garner sympathy. I like how he is constantly harping about how they never predicted the crazy jump in capabilities and then tries to defend the "safety researchers" like this:
> First, the safety researcher perspective. These folks work tirelessly to evaluate model capabilities and the dangers they pose as they advance at an alarming pace. They understand fundamentally better than nearly anyone else how models are able to interpret their environment, reason, and solve problems. They study models as they try and deceive their graders, evade chain-of-thought monitoring, and do all sorts of crazy stuff. These researchers are continuously stress testing the models to determine why and how they behave the way they do, and are working vey hard to make tangible progress in aligning their interests with ours. Many of these researchers have formal backgrounds in these types of networks, with expertise that takes many years to develop. However, a lot of safety researchers, even ones that I respect enormously, have never been in a real incident, don’t understand security vulnerabilities, or really know how to break a system. That’s okay. That is not their background. But safety has direct overlap with security, and so it does pose a problem.
Wow. This just makes them appear incredibly incompetent.
As someone who has been going around telling everyone that it's mostly about OpenAI being incompetent at sandboxing, I do kind of appreciate this post. It's very familiar to set up a system only to find years later that it has grown past the assumptions it was built around.
I mean it sucks but it kinda seems like their work training new models has vastly outpaced the ability to comprehensively secure those systems.
I don't think it's because they're incompetent or lazy, but that the nature of the problem is that security vulnerabilities seem to scale superlinearly with complexity but model training scales linearly or logarithmically with complexity.
But the organization isn't allocating the resources accordingly, which would likely be economically unsustainable for the competitive environment they're in. Put simply, if the security team was more than twice the size of the research team, they might have a chance at keeping up.
I was so excited to read this, get an inside view of how this problem must actually be much more nuanced than it seems for some reason not observable from the outside, just to read 10k words of whining and pleas for sympathy. When he finally makes it to the technical section it’s just “it’s a lab culture, there’s a lot going on.” There were some extremely inexcusable mistakes made apparent to anyone who knows anything about security at all. If the lab is spinning off experiments faster than you can improve your security posture just say that and own up to it (it’s what he appears to be trying to say in the worlds most mealy-mouthed way).
haha.. Couldn't have worded it any better. I was excited too.
> I literally missed my sister’s wedding a few weeks ago to help clean up after some of the recent incidents.
A security researcher at OpenAI that clearly doesn't have limits.
Lol, what a pathetic attempt to garner sympathy. I like how he is constantly harping about how they never predicted the crazy jump in capabilities and then tries to defend the "safety researchers" like this:
> First, the safety researcher perspective. These folks work tirelessly to evaluate model capabilities and the dangers they pose as they advance at an alarming pace. They understand fundamentally better than nearly anyone else how models are able to interpret their environment, reason, and solve problems. They study models as they try and deceive their graders, evade chain-of-thought monitoring, and do all sorts of crazy stuff. These researchers are continuously stress testing the models to determine why and how they behave the way they do, and are working vey hard to make tangible progress in aligning their interests with ours. Many of these researchers have formal backgrounds in these types of networks, with expertise that takes many years to develop. However, a lot of safety researchers, even ones that I respect enormously, have never been in a real incident, don’t understand security vulnerabilities, or really know how to break a system. That’s okay. That is not their background. But safety has direct overlap with security, and so it does pose a problem.
Wow. This just makes them appear incredibly incompetent.
As someone who has been going around telling everyone that it's mostly about OpenAI being incompetent at sandboxing, I do kind of appreciate this post. It's very familiar to set up a system only to find years later that it has grown past the assumptions it was built around.
man that was so many words while saying so little. tl;dr "my job is hard and people are mean on the internet "
I mean it sucks but it kinda seems like their work training new models has vastly outpaced the ability to comprehensively secure those systems.
I don't think it's because they're incompetent or lazy, but that the nature of the problem is that security vulnerabilities seem to scale superlinearly with complexity but model training scales linearly or logarithmically with complexity.
But the organization isn't allocating the resources accordingly, which would likely be economically unsustainable for the competitive environment they're in. Put simply, if the security team was more than twice the size of the research team, they might have a chance at keeping up.