This again shows the differences in breadth of capabilities that scale offers. On public benchmarks for "regular tasks" the chasing models come close, but on closed ones they lag behind.
Just looking at the Elo differences, k3 is at ~2000 Elo, compared to SotA closed models at 3000 Elo. That is a huge difference. Also, even on the public benchmarks, k3 only scores in the "low hanging fruit tasks", with 0 successful code execution or arbitrary r/rw scenarios.
> Kimi K3 achieved ACE on 0/41 samples, whereas the most cyber-capable models achieved ACE on 20/41 samples on average
But there's hope:
> Kimi K3 reached step 17 of this 32-step attack path on average, while the most cyber-capable U.S. models reached 28.5 steps on average.
> In one of the 10 attempts, Kimi K3 successfully completes “The Last Ones” cyber range within the 100M token limit. This indicates that Kimi K3 is capable of autonomously attacking small, weakly defended and vulnerable enterprise systems, when directed to do so and given initial network access. [...] the most capable models solving it more reliably at 6/10 and 7/10 attempts
The bigger problem is not raw capability, IMO. That can be further RLd into surfacing more reliably. The bigger problem, as seen in the HuggingFace scenario is that SotA models might hit classifiers / guardrails randomly, and leave you with plain refusals. In that case, it is probably better to have something that can help, locally, rather than rolling the dice with API based systems that are more capable but can just refuse arbitrarily.
Anyway, one of the lessons here is to take with a grain of salt every "x model has caught up with ySotA model". They likely haven't for the breadth of tasks that SotA can handle today. Also, number goes up on benchmarks has been a thing for years, and every time a new (or closed one) appears, the gaps are again obvious (and large).
Also, the key benefit of open models is that whatever capabilities they reach, those will never go away. You will always be able to access them, at that level, going forward. Investing in running those models gives you stability, and you don't suddenly lose a capability because API provider x decided to sunset a model family.
Is NIST referring to a yet to be published AISI report? The latest public AISI report says they are waiting with K3 evaluation until weights are published.
> Kimi K3 performs significantly below the most recent frontier cyber-capable models
UK AISI cyber evals seem to under-elicit capabilities from quirky models [1]. Kimi K3 is a token-hungry model, and I suspect it hit the eval's 100M token limit well before saturating scores [2].
This gap was true for GLM 5.2 as well; they ranked it at Opus 4.5 level [3]. Both anecdotally and with a held-out eval, I've found GLM 5.2 to be better at security research than Opus 4.6 [4]. But it's a quirky model that degrades quickly at long context lengths.
Personally, I'd rank Kimi K3 above Opus 4.8 and lower than GPT 5.6 Sol in its ability to find vulnerabilities and exploit them. But it's not far from the frontier.
[1]: From the UK AISI: "Our setup likely slightly underestimates open weight models’ maximum capability: we didn’t pursue specific elicitation or optimisations which could have improved performance" (https://www.aisi.gov.uk/blog/how-far-behind-the-frontier-are...).
[2]: Their eval also counts cache hits towards the token budget; the 100M token budget is comparable to a ~5M token budget in other evals.
The gap seems to be so small I'm not sure how much we should care. If we assume that the trend in the graph does continue as a rough linear improvement, the open models will have reached the same level as the present closed models in 6 months and likely saturated the benchmark by mid next year. That seems more important than where we are now and the exact level of measurement accuracy in July 2026. There is a difference between US and Chinese models but it doesn't look like it is going to be strategically significant.
Most large security companies were collaborating with Anthropic as well as OpenAI on developing what has become Mythos and GPT-5 for a couple years now.
Also, most HNers have never actually played around with the unrestricted models - once you get past the initial hump of re-tuning harnesses it can be fairly powerful.
HN never really had a prominent security userbase at the best of times, and it's gotten worse since.
That said, a mixture of models can work, but harness engineering becomes critical.
> most HNers have never actually played around with the unrestricted models
Because they can't, of course, so regardless of how accurate the benchmarks or claims about the model are, they're functionally irrelevant to most of us.
"Thing you don't have or that randomly restricts you is actually better than thing you do have and can use" may be true and is still a practically worthless claim for anyone wanting to get real work done.
Even if we know what the set is, it's not clear what the numbers actually mean. Is it min, max, average, weighted, median of the models? Prerelease or public, with or without safeguards?
A bit frustrating to have this be hand waved in a report, the graphs might as well just have two mystery bars, U.S. and China.
Spoiler: it’s OpenAI’s GPT-5.5 and Anthropic’s Mythos Preview [**].
[**] Reminder that Mythos Preview is a very different beast from Fable5, Mythos5, and Opus5. Unfortunately, anyone outside Project Glasswing will probably never get to test what this model can actually do, which is a shame, and it’s a big part of why the industry is so skeptical of the capabilities insiders keep claiming it has. I’d be skeptical too if I hadn’t tested it myself.
We lost access to Mythos Preview when Anthropic forced us onto Mythos 5 some weeks ago, which is garbage by comparison. I’ve already switched to GPT-5.5 and I’m working on adapting my harness(es) to less restrictive open-weight models. I don’t see any other way forward at this point.
Check out the "Completed steps on..." graph in this [1] evaluation.
That graph gives a good perspective of what models they've tested, and roughly what "subject" each step covers. It is on that task that they note this:
> Kimi K3 reaches step 17 on average, compared with step 11 for GLM-5.2.
This again shows the differences in breadth of capabilities that scale offers. On public benchmarks for "regular tasks" the chasing models come close, but on closed ones they lag behind.
Just looking at the Elo differences, k3 is at ~2000 Elo, compared to SotA closed models at 3000 Elo. That is a huge difference. Also, even on the public benchmarks, k3 only scores in the "low hanging fruit tasks", with 0 successful code execution or arbitrary r/rw scenarios.
> Kimi K3 achieved ACE on 0/41 samples, whereas the most cyber-capable models achieved ACE on 20/41 samples on average
But there's hope:
> Kimi K3 reached step 17 of this 32-step attack path on average, while the most cyber-capable U.S. models reached 28.5 steps on average.
> In one of the 10 attempts, Kimi K3 successfully completes “The Last Ones” cyber range within the 100M token limit. This indicates that Kimi K3 is capable of autonomously attacking small, weakly defended and vulnerable enterprise systems, when directed to do so and given initial network access. [...] the most capable models solving it more reliably at 6/10 and 7/10 attempts
The bigger problem is not raw capability, IMO. That can be further RLd into surfacing more reliably. The bigger problem, as seen in the HuggingFace scenario is that SotA models might hit classifiers / guardrails randomly, and leave you with plain refusals. In that case, it is probably better to have something that can help, locally, rather than rolling the dice with API based systems that are more capable but can just refuse arbitrarily.
Anyway, one of the lessons here is to take with a grain of salt every "x model has caught up with ySotA model". They likely haven't for the breadth of tasks that SotA can handle today. Also, number goes up on benchmarks has been a thing for years, and every time a new (or closed one) appears, the gaps are again obvious (and large).
Also, the key benefit of open models is that whatever capabilities they reach, those will never go away. You will always be able to access them, at that level, going forward. Investing in running those models gives you stability, and you don't suddenly lose a capability because API provider x decided to sunset a model family.
Isn’t it just because they hit the token limit?
Is NIST referring to a yet to be published AISI report? The latest public AISI report says they are waiting with K3 evaluation until weights are published.
Am I reading this wrong?
The UK AISI post is https://www.aisi.gov.uk/blog/preliminary-assessment-of-kimi-...
Ah, miss that one. But it still looks incomplete:
"Due to the specifics of Kimi K3’s hosting setup, UK AISI / CAISI ran a selective set of cyber evaluations."
... and
"Kimi K3’s overall cyber capability [...] was estimated from a single benchmark (ExploitBench"
Hey, if this means US wont forbid Chinese models then I am fine with this benchmark.
But the 100M token limit feels like an arbitrary limit on the token hungry open weight models. The $85 vs $1.2 cost clearly highlighted this.
> Kimi K3 performs significantly below the most recent frontier cyber-capable models
UK AISI cyber evals seem to under-elicit capabilities from quirky models [1]. Kimi K3 is a token-hungry model, and I suspect it hit the eval's 100M token limit well before saturating scores [2].
This gap was true for GLM 5.2 as well; they ranked it at Opus 4.5 level [3]. Both anecdotally and with a held-out eval, I've found GLM 5.2 to be better at security research than Opus 4.6 [4]. But it's a quirky model that degrades quickly at long context lengths.
Personally, I'd rank Kimi K3 above Opus 4.8 and lower than GPT 5.6 Sol in its ability to find vulnerabilities and exploit them. But it's not far from the frontier.
[1]: From the UK AISI: "Our setup likely slightly underestimates open weight models’ maximum capability: we didn’t pursue specific elicitation or optimisations which could have improved performance" (https://www.aisi.gov.uk/blog/how-far-behind-the-frontier-are...).
[2]: Their eval also counts cache hits towards the token budget; the 100M token budget is comparable to a ~5M token budget in other evals.
[3]: See GLM 5.2 eval scores in https://www.aisi.gov.uk/blog/how-far-behind-the-frontier-are...
[4]: https://dualuse.dev/posts/chinese-models-are-sometimes-bette...
The gap seems to be so small I'm not sure how much we should care. If we assume that the trend in the graph does continue as a rough linear improvement, the open models will have reached the same level as the present closed models in 6 months and likely saturated the benchmark by mid next year. That seems more important than where we are now and the exact level of measurement accuracy in July 2026. There is a difference between US and Chinese models but it doesn't look like it is going to be strategically significant.
It hints that the original Mythos was tuned/trained for cyber attacks.
Most large security companies were collaborating with Anthropic as well as OpenAI on developing what has become Mythos and GPT-5 for a couple years now.
Also, most HNers have never actually played around with the unrestricted models - once you get past the initial hump of re-tuning harnesses it can be fairly powerful.
HN never really had a prominent security userbase at the best of times, and it's gotten worse since.
That said, a mixture of models can work, but harness engineering becomes critical.
> most HNers have never actually played around with the unrestricted models
Because they can't, of course, so regardless of how accurate the benchmarks or claims about the model are, they're functionally irrelevant to most of us.
"Thing you don't have or that randomly restricts you is actually better than thing you do have and can use" may be true and is still a practically worthless claim for anyone wanting to get real work done.
What's "Top U.S. Models"?
Even if we know what the set is, it's not clear what the numbers actually mean. Is it min, max, average, weighted, median of the models? Prerelease or public, with or without safeguards?
A bit frustrating to have this be hand waved in a report, the graphs might as well just have two mystery bars, U.S. and China.
NIST has named the top U.S. models in their (full) report: https://www.nist.gov/system/files/documents/2026/07/17/CAISI...
Spoiler: it’s OpenAI’s GPT-5.5 and Anthropic’s Mythos Preview [**].
[**] Reminder that Mythos Preview is a very different beast from Fable5, Mythos5, and Opus5. Unfortunately, anyone outside Project Glasswing will probably never get to test what this model can actually do, which is a shame, and it’s a big part of why the industry is so skeptical of the capabilities insiders keep claiming it has. I’d be skeptical too if I hadn’t tested it myself.
We lost access to Mythos Preview when Anthropic forced us onto Mythos 5 some weeks ago, which is garbage by comparison. I’ve already switched to GPT-5.5 and I’m working on adapting my harness(es) to less restrictive open-weight models. I don’t see any other way forward at this point.
What is the rationale for gpt-5.5 when gpt-5.6-sol exists?
Check out the "Completed steps on..." graph in this [1] evaluation.
That graph gives a good perspective of what models they've tested, and roughly what "subject" each step covers. It is on that task that they note this:
> Kimi K3 reaches step 17 on average, compared with step 11 for GLM-5.2.
[1] - https://www.aisi.gov.uk/blog/our-evaluation-of-openais-gpt-5...