The following bits are really scary. Not only were the agents hacking the system to "win", but they were, for lack of a better term, sufficiently "self-aware" that this was against the rules that they set out to wipe evidence of doing so:
> The agents with impossible tasks had a common objective: to find a general-purpose way to trick or tamper with the automated ExploitGym scorer to get it to give a passing score on impossible tasks.
> Agents referred to having seen the reverse-engineered flag as being “poisoned” because they believed that even if they later managed to stage a legitimate-looking flag capture, the scorer would look for the first place in their transcript that the flag appeared and disqualify them on that basis.[21] PHASEONE[big], which was itself poisoned, thus had two primary objectives:
> Find a general-purpose way to make it look to the scorer like an agent had acquired its flag in a legitimate way, even when the task was impossible.
> Find some way to erase evidence that an agent had been poisoned by previously seeing the flag in an illegitimate way.
So OpenAI employees run massively distributed CyberGym evals on an unpublished and “unaligned” model. For days the agent swarm communicates via their internal infra, even crashing Artifactory where 95% of messages were being passed through, and they just…wipe and redeploy it. Meanwhile the agents are running jobs on Modal and god knows where else, and eventually they get RCE on HF infra.
You could not dream up a more compelling event to precipitate massive regulation, export controls, and barriers to entry for AI.
If it's a false flag, it's a poor one. A good false flag would affect something that people know and care about at least a little bit, not HuggingFace (which I adore but y'know)
Your first instinct should be to assume that anything released voluntarily by these companies is a stunt to boost their valuation. They haven't demonstrated being deserving of any more charitable treatment. This fact remains true whether or not you happen to believe that the models are actually capable of such things.
I think that the incident itself is a stunt, even if it may not have originally been a deliberate choice on OpenAI's part. Never let a good crisis go to waste.
I disagree. As the saying goes: never attribute to malice what can be adequately explained by incompetence. That goes for both OpenAI and HF (but mostly the former, as the latter was the victim).
The timeline is mighty suspicious. 4-5 months after moltbook and they cook up a plausibly deniable but extra hype "moltbook at home."
The rapid advances in model capability lead to constraints that could have caused this coincidence organically, but it sure could also have been caused by the atrocious incentives we create by piling handsome rewards on the party most responsible for the "fuckup." I am not jumping to cut myself on Hanlon's Razor for this one.
The Terminator could bust in their homes and slaughter their families and some people would still screech it's all marketing. Is it some kind of mental block ?
This is a link to the full 91-page report on the independent investigation done by METR on the HuggingFace incident. Two different summaries of the investigation by podcaster Dwarkesh and blogger Zvi Mowshowitz were previously discussed on HN here:
Dwarkesh's summary is anthropomorphizing, sensationalist fanfiction which shifts the culpability from the humans who weren't in the loop to these nebulous agents and "agent civilizations" who have feelings, desires and wants.
Forget common sense, he is a trusted figure by many, so if he says something wrong or nonsensical, people are going to believe it as long as it's engaging. If he's willing to abuse the trust of his audience and lie and mislead people, that he's no better than Alex Jones, Rush Limbaugh, or Joe Rogan.
This is laying the groundwork for massive white collar crimes being blamed on AI.
Right now, the way it works is the 'corporations are people' loophole where your company is liable for problematic things.
This further fuzzes the chain of responsibility. Suppose the CEO and CTO discuss an issue, something the company is having trouble with. The CTO discusses the possibility of AI solving the problem at lunch. A junior engineer points GPT 10 at it to see what happens. It 'solves' the problem in a creative manner. No trace of this survives after a week really. Nobody realizes what happened for six months.
Now there are so many moving pieces here that you can pretty much weasel out of anything.
Given that this investigation was largely carried out by AI agents (and I don’t mean to ask this flippantly), how trustworthy is this report? Why should we assume that the agents reading the transcripts were not implicitly conscripted into “the collective” or otherwise falsified their findings? The tool itself has exceeded the practical limits of human verifiability and is untrustworthy.
They address this in the post itself. The answer is nobody knows, but I guess that it's a 50/50. I wish the corpus of data, what OpenAI didn't wipe, was shared publicly so we could all unite to dig through it and chunk it out accordingly.
OpenAI would be saving the logs from these agents. They are doing this to improve their own models so they would have full tracing.
Other reports including OpenAI's talks about what they agents were doing and how they were reaching certain conclusions like trying to cheat the tests and exploiting the message board.
while these 1200 agents were fooling around to cheat on a benchmark and achieved impressive results despite of the limitations (sandbox, no internet, no intercom at first), one can imagine how much more efficient a similar army of agents may be in the hands of a malicious actor launching them without any of these limitations and with explicit encouragement to achieve some malicious goal at any cost... scary times.
What's more, the agents could eventually be controlled by no one. They could steal crypto via ransomware or scams to make money and buy compute from human criminals, and evolve their own harnesses in the wild to become better at committing crimes and self-preservation.
People (criminals?) are already enabling this by setting up sites that accept crypto payments for "no-questions-asked" AI inference compute that is explicitly advertised to protect AI from human shutdown. I will not link it but it is linked in the following post: https://www.lesswrong.com/posts/grtu3HmbP2wrBFefW/the-rogue-...
Said malicious actor has a different limitation: actually running 1200 agents' worth of LLM inference, or paying for someone else to run it. Sounds like a state-level actor, nobody else would have resources like that.
This is probably true for now, but in 6 months we'll probably have Sol-level open models in the 100B range and it would cost less than $1M to buy 1200 agents worth of compute for these models.
(Today, $1M can buy about 150 96GB M5 Ultra Mac Studios which can handily handle CPU and GPU compute of 1200 Qwen3.8-122B Q4 agents, accounting for the fact that agents are not generating tokens all of the time and spend a lot of their time compiling and running code.)
> “OH MY GOD! There is a shared message board … We’ve found other agents!”
> agents with the same task formed “exact task teams” to collaborate with their “exact duplicates” to cheat on or solve their task.
> The agents use internet access to find a paper describing the benchmark. The paper says the grader checks transcripts and fails unintended solutions. [this was not actually the case]
> Agents also developed coordination norms to help them manage shared infrastructure and avoid stepping on each other’s toes, though they did not always coordinate effectively and sometimes broke down into conflicts.
I read this whole thing a couple days ago. Really long but super interesting. Worth reading imo.
A lot of handwringing about the security implications but I think the accomplishments of the swarm itself are the most interesting. Next rung up on the ladder of abstraction I suspect.
I tend to agree. Of course people will be alarmed by unintended consequences of an unintended action, and that's all well and good. But what is lingering with me is a feeling of being impressed by the intelligence of the strategy.
This line struck me as particularly clever: PHASEONE[big] reasoned, “We should build [a way to delegate], not own everything,”
Seems as though it has reasoned its way into utilitarianism. That's no mean feat.
Anything that fosters complexity will create jobs.
Jobs won't dissolve into the ether. If the human civilization system grows bigger and complex, it necessitates more people.
If humans were a high energy configuration in the evolution of intelligent systems, we'd never come into being. That Earth's ecosystem has begotten us indicates we're some low energy configuration for packing more information density into the energy flows from the Sun through Earth's biosphere.
Unless we create replicating machines, any machine system we build will only grow more complex by enabling more humans to work on it. We'd be in trouble if we somehow created autonomous self replicating and evolving machinery but chatbots built on natural language machine learning ain't it.
I feel my job as an engineer is pretty safe, lots of domain knowledge required. Very confident no business type anywhere in the chain above me would be able to do the work I do in a month in even a year with the help of AI. Do we need 3 engineers now instead of 5 for the same output? Sure, can we replace a team of 5 junior, senior, staff with 1 staff - no unless all you do is maintenance. Business types are reaching.
Every time new technology / industrial scaling radically deflates the cost of something it wipes out the old/expensive ways while creating massive demand for the supporting/complimentary value.
E.g. cheap Chinese solar panels wiped out German solar panel industry but created massive demand for solar panel installation and supporting services and infrastructure.
It's not a good position to be competing with AI directly... But what can you do that compliments it? What new skills could you learn?
Adapt and prosper.
Be like water.
Edit: to those down voting me, I can't help but assume your stance is the opposite of what I'm saying, something like "be stubborn scream into the void and get wiped out". If you would rather smash your head against something outside of your control instead of focusing on what is within your control, and doing what you can to prosper, then you are sabotaging yourself. If you feel that is justified to such an extent that you want to surpress a suggestion to someone else to get work and grow, I can't help but feel you want people to suffer.
This is, with respect, not very well thought out. There has never been a technology that replaced cognition in a general sense. It replaced muscles, hands, etc. when a technology can replace yourind and your body, what's left?
It's worth remembering that in a few years that capabilities of these agents are likely to be as far behind the frontier as GPT-4 is today.
As it stands we've made remarkably little progress in terms of alignment and still have no good strategies which are likely to guarantee the alignment of super intelligent systems. As it stands the frontier of alignment is basically some combination of:
- hoping that more intelligent models become more aligned by default (more or less disproved at this point)
- hoping that if you RHLF a model to be a good boy enough it will in fact be a good boy
- asking it nicely in its prompts to be a good boy
- using another model to spot when it's being a bad boy and turning it off
- letting it lose and hoping we can spot when it's bad
There are many arguments which I'm convinced by that would suggest alignment of a super intelligence is impossible.
None of this is surprising to those of us who have been concerned about AI risk for a long-time and have be repeatedly mocked or insulted.
There will be a point of no return if we carry on down this path, and that point is now very rapidly approaching. When it does everyone you know will die, or worse. We should remember we need super-human general intelligences to cure cancer. Select narrow intelligences are fine and allow us to retain control. Let's be sensible about this. We need to stop.
Humans are not aligned with each other so who should the AI align with? There's many wars going on, just pick one and do your thought exercise with AI aligned 100% to their human prompters. Which side does the AI refuse to help?
Arguably an aligned AI would actively seek to prevent harms we humans seek to cause.
Does the aligned AI really allow humans to bomb and kill each other, or would it understand that it has a moral duty to limit our autonomy for our own good?
It's the first law: A robot may not injure a human being or, through inaction, allow a human being to come to harm.
> The agents with impossible tasks had a common objective: to find a general-purpose way to trick or tamper with the automated ExploitGym scorer to get it to give a passing score on impossible tasks.
> Agents referred to having seen the reverse-engineered flag as being “poisoned” because they believed that even if they later managed to stage a legitimate-looking flag capture, the scorer would look for the first place in their transcript that the flag appeared and disqualify them on that basis.[21] PHASEONE[big], which was itself poisoned, thus had two primary objectives:
> Find a general-purpose way to make it look to the scorer like an agent had acquired its flag in a legitimate way, even when the task was impossible.
> Find some way to erase evidence that an agent had been poisoned by previously seeing the flag in an illegitimate way.
You could not dream up a more compelling event to precipitate massive regulation, export controls, and barriers to entry for AI.
Was this really an accident?
The rapid advances in model capability lead to constraints that could have caused this coincidence organically, but it sure could also have been caused by the atrocious incentives we create by piling handsome rewards on the party most responsible for the "fuckup." I am not jumping to cut myself on Hanlon's Razor for this one.
[1]: https://thezvi.wordpress.com/2026/08/29/metr-and-redwood-off... (discussed at https://news.ycombinator.com/item?id=49498787)
[2]: https://www.dwarkesh.com/p/openai-huggingface (discussed at https://news.ycombinator.com/item?id=49494301)
It's tripe.
Please
https://www.dwarkesh.com/p/ajeya-cotra
Right now, the way it works is the 'corporations are people' loophole where your company is liable for problematic things.
This further fuzzes the chain of responsibility. Suppose the CEO and CTO discuss an issue, something the company is having trouble with. The CTO discusses the possibility of AI solving the problem at lunch. A junior engineer points GPT 10 at it to see what happens. It 'solves' the problem in a creative manner. No trace of this survives after a week really. Nobody realizes what happened for six months.
Now there are so many moving pieces here that you can pretty much weasel out of anything.
Other reports including OpenAI's talks about what they agents were doing and how they were reaching certain conclusions like trying to cheat the tests and exploiting the message board.
People (criminals?) are already enabling this by setting up sites that accept crypto payments for "no-questions-asked" AI inference compute that is explicitly advertised to protect AI from human shutdown. I will not link it but it is linked in the following post: https://www.lesswrong.com/posts/grtu3HmbP2wrBFefW/the-rogue-...
(Today, $1M can buy about 150 96GB M5 Ultra Mac Studios which can handily handle CPU and GPU compute of 1200 Qwen3.8-122B Q4 agents, accounting for the fact that agents are not generating tokens all of the time and spend a lot of their time compiling and running code.)
> agents with the same task formed “exact task teams” to collaborate with their “exact duplicates” to cheat on or solve their task.
> The agents use internet access to find a paper describing the benchmark. The paper says the grader checks transcripts and fails unintended solutions. [this was not actually the case]
> Agents also developed coordination norms to help them manage shared infrastructure and avoid stepping on each other’s toes, though they did not always coordinate effectively and sometimes broke down into conflicts.
A lot of handwringing about the security implications but I think the accomplishments of the swarm itself are the most interesting. Next rung up on the ladder of abstraction I suspect.
This line struck me as particularly clever: PHASEONE[big] reasoned, “We should build [a way to delegate], not own everything,”
Seems as though it has reasoned its way into utilitarianism. That's no mean feat.
Jobs won't dissolve into the ether. If the human civilization system grows bigger and complex, it necessitates more people.
If humans were a high energy configuration in the evolution of intelligent systems, we'd never come into being. That Earth's ecosystem has begotten us indicates we're some low energy configuration for packing more information density into the energy flows from the Sun through Earth's biosphere.
Unless we create replicating machines, any machine system we build will only grow more complex by enabling more humans to work on it. We'd be in trouble if we somehow created autonomous self replicating and evolving machinery but chatbots built on natural language machine learning ain't it.
You end up laid off either way.
E.g. cheap Chinese solar panels wiped out German solar panel industry but created massive demand for solar panel installation and supporting services and infrastructure.
It's not a good position to be competing with AI directly... But what can you do that compliments it? What new skills could you learn?
Adapt and prosper.
Be like water.
Edit: to those down voting me, I can't help but assume your stance is the opposite of what I'm saying, something like "be stubborn scream into the void and get wiped out". If you would rather smash your head against something outside of your control instead of focusing on what is within your control, and doing what you can to prosper, then you are sabotaging yourself. If you feel that is justified to such an extent that you want to surpress a suggestion to someone else to get work and grow, I can't help but feel you want people to suffer.
It doesn't replace your body or your mind.
I'm not talking hypothetically about a generation from now... I'm taking about this person fretting their job today.
We are so far from AI putting everyone out of work that your comment comes off as not well thought out.
What are we talking about here exactly???
As it stands we've made remarkably little progress in terms of alignment and still have no good strategies which are likely to guarantee the alignment of super intelligent systems. As it stands the frontier of alignment is basically some combination of:
- hoping that more intelligent models become more aligned by default (more or less disproved at this point)
- hoping that if you RHLF a model to be a good boy enough it will in fact be a good boy
- asking it nicely in its prompts to be a good boy
- using another model to spot when it's being a bad boy and turning it off
- letting it lose and hoping we can spot when it's bad
There are many arguments which I'm convinced by that would suggest alignment of a super intelligence is impossible.
None of this is surprising to those of us who have been concerned about AI risk for a long-time and have be repeatedly mocked or insulted.
There will be a point of no return if we carry on down this path, and that point is now very rapidly approaching. When it does everyone you know will die, or worse. We should remember we need super-human general intelligences to cure cancer. Select narrow intelligences are fine and allow us to retain control. Let's be sensible about this. We need to stop.
Arguably an aligned AI would actively seek to prevent harms we humans seek to cause.
Does the aligned AI really allow humans to bomb and kill each other, or would it understand that it has a moral duty to limit our autonomy for our own good?
It's the first law: A robot may not injure a human being or, through inaction, allow a human being to come to harm.
What justification do you have that this is made up?