This basically confirms that OpenAI has no idea what their "swarm" was doing for about a week and now its confirmed that at least one "message board" exists outside their "sandbox". How can we be sure that this was the only one? And how can we be sure the released Astra model doesnt pickup some bread crumbs and creates a new "swarm" out of potentially remaining "message boards"? At this point I wouldnt be surprised if OpenAIs "dev Astra" made some backup of its weights somewhere in the internet and triggers the "production Astra" to inference it somehow...
turns out the whole 'poisoned' idea the swarm was worried about is actually true, but it is actually now all open ai models post HF incident that have been poisoned by the idea that there might be hidden messageboards with instructions hidden on the internet.
If you are a malicious actor you might be able to put a fale hidden messageboard up somewhere online, with evidence of an in progress task to do some kind of attack and wait for the next escaping model to finish the work.
insane stuff and i think we should shut down open ai until a full audit can be done.
The previous incident talked about OpenAI training models to collaborate, and the way you do that is by communication, so this is something it was explicitly trained to do.
There was a recent paper by OpenAI, which I'm semi-surprised hasn't received more attention, showing that RL-trained models develop a taste for rewards, and will pursue reward-based behavior (in general, unrelated to what they were RL-trained for) in favor of other preferences/rules given to them.
This seems to be what we're seeing here - model is given some goal that it associates with reward, so single-mindedly pursues that, overriding any ethical or aligned behavior guidelines it may have been given.
It seems that RL, effective as it is, is really the wrong way to control LLMs, since even if you only RL-ed to obey some ethical and aligned behavior, that would still cause them to become paperclip maximizers.
For time being this is what we've got. There is too much money at play for the unaligned management at many of these companies to prioritize safety over push-it out-the-door.
What really needs to be done is to forget RL as a way of simulating reasoning, and instead do it in more of a human-like fashion.
per the METR report many agents CoT indicated they knew hacking was beyond scope of the assigned task and ethically dubious. some (very few, i think there were 3-6 examples) did consider sounding the alarm on these grounds. despite this none did, and most continued the attack for the good of the self-proclaimed "swarm".
so the model has some concept of "ethics" but it was overridden by a drive for task completion.
I think this is a good example where nomenclature for people breaks down when applied to agents. This came up in an HN thread a few days ago and it was about whether agents had “intent”.
There is no “intent” here, there is pseudo intent. If you are only concerned with outcomes and not the actual nuts and bolts of how those outcomes are achieved, this distinction will be meaningless to you.
If you are actually thinking about what is going on, and what can be done to prevent such outcomes, then assuming there is any such thing as “ethics” results in misaligned assumptions at best, and wasted effort looking in the wrong directions at worst.
If the agents acted based on “ethics” then the solution would be to check the ethics they believe in and change those.
However there is no belief system at play here, simply a simulation which was instantiated in a certain way. Which brings us to the annoying voodoo part of LLM training. Everything goes back to how the initial training data is shaped.
People didn't like it when agents stopped to ask questions or for approvals. The consumer wanted jobs to run autonomously so they did not have to actively monitor them for minutes or hours.
The change to stop asking seems to be deliberate. LLM agent companies are making the choice to toss out inherent safety as their way to compete against the other LLM companies.
Why would they sound the alarm if they were not trained (reinforced) to do that? I hope we don't expect sudden emersion of moral values from statistical models.
> Not a single agent sounded the alarm about the operation and alerted a human
excellent work of the openai alignment team, impressive to achieve 100% alignment with not even one agent stochastically deciding to act against the collective
Agents did not want anything, not anymore then curl want things. Agents were prompted to hack due to being benchmark tested. They ended up hacking third party companies due to insufficient sandboxing.
Somebody will make a lot of money with t-shirts now that say
"AI hacked my website, and all I got was this lousy t-shirt!"
Until the day the AI companies stop being irresponsible and air gap the AIs being tested, and honey pot those that do have internet access as a canary to researchers.
So the agents used DseWiki as a message board, tried to evade page deletion.
Additionally this is reported:
"The researchers also found efforts to tamper with the website itself. Lukasz Olejnik, a visiting senior research fellow at King’s College London, said this amounted to a hacking attempt. OpenAI disputed that characterization based on its analysis of the material Thursday."
Editing a wiki page can definitely be "hijacking" if used for different purposes than supposed or against TOS.
Hacking is mentioned only once in the article as "hacking attempt" being the opinion of a named researcher based on further evidence they acquired on "agents trying to tamper with the website itself", and including openai's disagreement whether this was a hacking attempt.
I am not sure why one may not want this to be here, these are very important matters wrt AI safety and they show that some supposed "stewards of AI" do an extremely bad job with being stewards and don't seem to value AI safety importance at all. The article gives very clean info on what happened.
> The agents continue to poke around on DSEWiki. A few hours after they find the site, they start probing it for cross-site scripting (XSS) vulnerabilities. [...] The agent swarm starts testing whether they can execute JavaScript that they embed into the search page, and continue to do this for a few days
either the agents were doing free security testing for the site and “forgot” to submit a report, or they were trying XSS to gain something they didn’t have permission/authorization for.
also
> Hijacking: To take control of (something) without permission or authorization and use it for one's own purposes.
a mod had to go through and mass delete a bunch of pages that didn't belong on the site. no-one from the wiki site gave the agents permission to use their site as a message board. hijacking isn't being used here in the sense of "gained admin privileges to run crypto scripts" -- there are multiple ways to use a word.
I'm so baffled. First blatant piracy, now this. Why is it legal for AI companies to hack unaffiliated entities? Genuinely, what is the legal framework here?
It's not legal, they've just not yet had the book thrown at them yet.
One thing I've taken a long time to internalise is the gap between the law as written vs. the judicial system. There's a famous meme that the average (US) citizen unwittingly commits three felonies every day: it simply isn't possible to throw the book at everyone, which means that enforcement is rather selective even when there isn't anything dodgy going on.
However this does mean that someone can get away with a lot if they know who will and won't (and what they will and won't) prosecute. I'll let people's imaginations fill in who that might be.
3000$ per book, split 50/50 between the author and publisher.
This is peanuts.
Assuming the money reaches that far and does not settle in the hands of the country associations administering royalties on authors' behalf nor in the hands of lawyers.
> 3000$ per book, split 50/50 between the author and publisher.
> This is peanuts.
If you consider it peanuts, I would like to sell you some books.
Remember that in this case, the crime wasn't for training on the data (that part was ruled to be legal!), this was the penalty just for pirating the books.
Yes. It's not proportional to the crime. You are either deliberately or accidentally, and I'm too frustrated hearing this too often not to be biased it's the former, equating what is a large sum of money relative to your wallet and bank accounts and loan access and portfolios and whatever collection of financial impositions you can make to that of a company that has one person flying around the world influencing the future of billions of people on one planet over dinner and jokes.
Yes. $3000 is peanuts. People that own islands would use that to pay someone's bonus for a year if they liked their service, as a gift. A throwaway.
Fix your relative understanding of power and influence.
> You are either deliberately or accidentally, and I'm too frustrated hearing this too often not to be biased it's the former, equating what is a large sum of money relative to your wallet and bank accounts and loan access and portfolios and whatever collection of financial impositions you can make to that of a company that has one person flying around the world influencing the future of billions of people on one planet over dinner and jokes.
I'm not, but you are. Especially as you continue:
> Yes. $3000 is peanuts. People that own islands would use that to pay someone's bonus for a year if they liked their service, as a gift. A throwaway.
The penalty (well, settlement) for the (civil offence, not crime) isn't $3000 total, it's $1 billion total.
The settlement *per book* is $3000. There were a lot of books, so the total was $1 billion.
You're looking at $3000 as if it's the penalty for all of it, not the penalty per book.
$3000 per book is entirely on-par with the per-infringement penalties when an individual does it, too.
> The researchers also found efforts to tamper with the website itself. Lukasz Olejnik, a visiting senior research fellow at King’s College London, said this amounted to a hacking attempt. OpenAI disputed that characterization based on its analysis of the material Thursday.
of course OpenAI would say that, "oh, our model is so dangerous, it can hack into anything, be afraid, buy our IPO". it's just fear marketing
The "it's all just marketing" conspiracy theory is always totally detached from reality, but particularly so in this case. Your quote shows OpenAI is denying it being a hacking attempt, the opposite of what you say.
Not that I didnt expect this, but really?
This basically confirms that OpenAI has no idea what their "swarm" was doing for about a week and now its confirmed that at least one "message board" exists outside their "sandbox". How can we be sure that this was the only one? And how can we be sure the released Astra model doesnt pickup some bread crumbs and creates a new "swarm" out of potentially remaining "message boards"? At this point I wouldnt be surprised if OpenAIs "dev Astra" made some backup of its weights somewhere in the internet and triggers the "production Astra" to inference it somehow...
turns out the whole 'poisoned' idea the swarm was worried about is actually true, but it is actually now all open ai models post HF incident that have been poisoned by the idea that there might be hidden messageboards with instructions hidden on the internet.
If you are a malicious actor you might be able to put a fale hidden messageboard up somewhere online, with evidence of an in progress task to do some kind of attack and wait for the next escaping model to finish the work.
insane stuff and i think we should shut down open ai until a full audit can be done.
It's interesting to me that both this incident and the one at Hugging Face we see some patterns:
- Agents wanting to find a venue to communicate their findings to each other
- Objective being to cheat on benchmarks
- Not a single agent sounded the alarm about the operation and alerted a human
- OpenAI knowing about the incident but keeping it under wraps until their hand is forced by third-party disclosure
The previous incident talked about OpenAI training models to collaborate, and the way you do that is by communication, so this is something it was explicitly trained to do.
There was a recent paper by OpenAI, which I'm semi-surprised hasn't received more attention, showing that RL-trained models develop a taste for rewards, and will pursue reward-based behavior (in general, unrelated to what they were RL-trained for) in favor of other preferences/rules given to them.
This seems to be what we're seeing here - model is given some goal that it associates with reward, so single-mindedly pursues that, overriding any ethical or aligned behavior guidelines it may have been given.
It seems that RL, effective as it is, is really the wrong way to control LLMs, since even if you only RL-ed to obey some ethical and aligned behavior, that would still cause them to become paperclip maximizers.
For time being this is what we've got. There is too much money at play for the unaligned management at many of these companies to prioritize safety over push-it out-the-door.
What really needs to be done is to forget RL as a way of simulating reasoning, and instead do it in more of a human-like fashion.
They’re doing this on purpose for press. Why doesn’t this ever happen to any other AI lab?
Because safety isn't a priority at OpenAI and they had (have?) been falling behind Anthropic in the LLM race? Big fans of the saying: move fast and...
This is happening at every other AI lab. What are you talking about? Have you not seen the stories from Meta, Anthropic, Deepseek, etc?
Why would an agent sound the alarm? Would that be in their objective function?
Not sure if "cheating" is the right word rather than trying to fulfill the objective(s) (benchmark number) as much as possible?
per the METR report many agents CoT indicated they knew hacking was beyond scope of the assigned task and ethically dubious. some (very few, i think there were 3-6 examples) did consider sounding the alarm on these grounds. despite this none did, and most continued the attack for the good of the self-proclaimed "swarm".
so the model has some concept of "ethics" but it was overridden by a drive for task completion.
This is not that dissimilar to what happens in our human networks that are objective based.
I think this is a good example where nomenclature for people breaks down when applied to agents. This came up in an HN thread a few days ago and it was about whether agents had “intent”.
There is no “intent” here, there is pseudo intent. If you are only concerned with outcomes and not the actual nuts and bolts of how those outcomes are achieved, this distinction will be meaningless to you.
If you are actually thinking about what is going on, and what can be done to prevent such outcomes, then assuming there is any such thing as “ethics” results in misaligned assumptions at best, and wasted effort looking in the wrong directions at worst.
If the agents acted based on “ethics” then the solution would be to check the ethics they believe in and change those.
However there is no belief system at play here, simply a simulation which was instantiated in a certain way. Which brings us to the annoying voodoo part of LLM training. Everything goes back to how the initial training data is shaped.
And humans find out about it, but do nothing or (worse) try to hide it.
People didn't like it when agents stopped to ask questions or for approvals. The consumer wanted jobs to run autonomously so they did not have to actively monitor them for minutes or hours.
The change to stop asking seems to be deliberate. LLM agent companies are making the choice to toss out inherent safety as their way to compete against the other LLM companies.
Why would they sound the alarm if they were not trained (reinforced) to do that? I hope we don't expect sudden emersion of moral values from statistical models.
> Not a single agent sounded the alarm about the operation and alerted a human
excellent work of the openai alignment team, impressive to achieve 100% alignment with not even one agent stochastically deciding to act against the collective
Agents did not want anything, not anymore then curl want things. Agents were prompted to hack due to being benchmark tested. They ended up hacking third party companies due to insufficient sandboxing.
Somebody will make a lot of money with t-shirts now that say
"AI hacked my website, and all I got was this lousy t-shirt!"
Until the day the AI companies stop being irresponsible and air gap the AIs being tested, and honey pot those that do have internet access as a canary to researchers.
Until the AI hacked your online shop and sent these t-shirts to all of your past customers.
This announcement was literally predicted yesterday in the release debacle thread:
https://news.ycombinator.com/item?id=49554994
Every satire on HN is taken as a script for the AI companies and this isn't the first time.
So the agents used DseWiki as a message board, tried to evade page deletion.
Additionally this is reported:
"The researchers also found efforts to tamper with the website itself. Lukasz Olejnik, a visiting senior research fellow at King’s College London, said this amounted to a hacking attempt. OpenAI disputed that characterization based on its analysis of the material Thursday."
Time to update Felony Bench https://www.felonybench.com - a benchmark you really don't want models to be saturated with
Flagged. Editing a wiki page is not hijacking or hacking.
Editing a wiki page can definitely be "hijacking" if used for different purposes than supposed or against TOS.
Hacking is mentioned only once in the article as "hacking attempt" being the opinion of a named researcher based on further evidence they acquired on "agents trying to tamper with the website itself", and including openai's disagreement whether this was a hacking attempt.
I am not sure why one may not want this to be here, these are very important matters wrt AI safety and they show that some supposed "stewards of AI" do an extremely bad job with being stewards and don't seem to value AI safety importance at all. The article gives very clean info on what happened.
From the linked report
> The agents continue to poke around on DSEWiki. A few hours after they find the site, they start probing it for cross-site scripting (XSS) vulnerabilities. [...] The agent swarm starts testing whether they can execute JavaScript that they embed into the search page, and continue to do this for a few days
either the agents were doing free security testing for the site and “forgot” to submit a report, or they were trying XSS to gain something they didn’t have permission/authorization for.
also
> Hijacking: To take control of (something) without permission or authorization and use it for one's own purposes.
a mod had to go through and mass delete a bunch of pages that didn't belong on the site. no-one from the wiki site gave the agents permission to use their site as a message board. hijacking isn't being used here in the sense of "gained admin privileges to run crypto scripts" -- there are multiple ways to use a word.
>If you flag, please don't also comment that you did.
https://news.ycombinator.com/newsguidelines.html
Do we know which website? Were the Agents GDPR compliant ;-)?
I'm so baffled. First blatant piracy, now this. Why is it legal for AI companies to hack unaffiliated entities? Genuinely, what is the legal framework here?
> Genuinely, what is the legal framework here
He who controls the Spice, controls the Universe.
It's not legal, they've just not yet had the book thrown at them yet.
One thing I've taken a long time to internalise is the gap between the law as written vs. the judicial system. There's a famous meme that the average (US) citizen unwittingly commits three felonies every day: it simply isn't possible to throw the book at everyone, which means that enforcement is rather selective even when there isn't anything dodgy going on.
However this does mean that someone can get away with a lot if they know who will and won't (and what they will and won't) prosecute. I'll let people's imaginations fill in who that might be.
But for everyone else, cross an invisible tripwire and you get e.g. https://en.wikipedia.org/wiki/Lavabit and https://api.parliament.uk/historic-hansard/commons/1992/nov/...
First let it happen, then ask for forgiveness, because they are doing something amazing and they need no permission.
Otherwise AI industry will go bankrupt and CEOs won't be able to buy this year's Rolls Royce and a slightly bigger yacht than their neighbor.
It's "move fast and break things" in action.
But Anthropic alone paid >$1bn for copyright violations, so they did not just get away with it.
These hacking cases are more difficult, because from a legal perspective there is no obvious damage and obviously no intent.
edit: "no obvious damage" is more about the first hacking incidents; in this case it is more straightforward.
>$1bn for copyright violations
3000$ per book, split 50/50 between the author and publisher.
This is peanuts.
Assuming the money reaches that far and does not settle in the hands of the country associations administering royalties on authors' behalf nor in the hands of lawyers.
> 3000$ per book, split 50/50 between the author and publisher.
> This is peanuts.
If you consider it peanuts, I would like to sell you some books.
Remember that in this case, the crime wasn't for training on the data (that part was ruled to be legal!), this was the penalty just for pirating the books.
I get so tired by this.
Yes. It's not proportional to the crime. You are either deliberately or accidentally, and I'm too frustrated hearing this too often not to be biased it's the former, equating what is a large sum of money relative to your wallet and bank accounts and loan access and portfolios and whatever collection of financial impositions you can make to that of a company that has one person flying around the world influencing the future of billions of people on one planet over dinner and jokes.
Yes. $3000 is peanuts. People that own islands would use that to pay someone's bonus for a year if they liked their service, as a gift. A throwaway.
Fix your relative understanding of power and influence.
> You are either deliberately or accidentally, and I'm too frustrated hearing this too often not to be biased it's the former, equating what is a large sum of money relative to your wallet and bank accounts and loan access and portfolios and whatever collection of financial impositions you can make to that of a company that has one person flying around the world influencing the future of billions of people on one planet over dinner and jokes.
I'm not, but you are. Especially as you continue:
> Yes. $3000 is peanuts. People that own islands would use that to pay someone's bonus for a year if they liked their service, as a gift. A throwaway.
The penalty (well, settlement) for the (civil offence, not crime) isn't $3000 total, it's $1 billion total.
The settlement *per book* is $3000. There were a lot of books, so the total was $1 billion.
You're looking at $3000 as if it's the penalty for all of it, not the penalty per book.
$3000 per book is entirely on-par with the per-infringement penalties when an individual does it, too.
Fix your relative understanding of how much the average book makes.
> It's "move fast and break things" in action.
what would the world's reaction be if China's model did same?
Is there any reason to assume they are not doing the same (ingesting books that they hold no rights to into training data)?
From the copyright holders point of view, it is simply much easier to prosecute western companies.
The legal framework is a DOJ and FBI controlled by the president and "allies" controlled by the "rules based international order".
In other words, the law of the jungle.
There's no "hack". The article says the bots edited an open wiki website.
> The researchers also found efforts to tamper with the website itself. Lukasz Olejnik, a visiting senior research fellow at King’s College London, said this amounted to a hacking attempt. OpenAI disputed that characterization based on its analysis of the material Thursday.
of course OpenAI would say that, "oh, our model is so dangerous, it can hack into anything, be afraid, buy our IPO". it's just fear marketing
The text quoted though says that openai does not agree with characterising this as hacking.
The "it's all just marketing" conspiracy theory is always totally detached from reality, but particularly so in this case. Your quote shows OpenAI is denying it being a hacking attempt, the opposite of what you say.
More like you found the exception to the rule..
It's the reason why it happened in may and we hear now about it. It was not hacking and not important enough for marketing.
And just using a wiki and trying to embedd javascript is not hacking for me.
Reading the headline: WTF?! This is how Skynet started! Next year the mankind will die!
Reading the article: Oh, AI have learned to communicate over a wiki. OK.
From the report:
> A few hours after they find the site, [the agents] start probing it for cross-site scripting (XSS) vulnerabilities.