I have an idea for a benchmark: train a model on a knowledge cutoff lets say 1945, and simulate world development for the next 70 years. Then see if it invents things human did, and avoid our mistakes/dead ends too.
Having used both Sol-6.2 and Astra-6.2 heavily over the last few days (as well as regularly using Claude Opus and Fable) they are marked improvements over the previous releases and comparable to Opus-5.5 and Fable-5.2.
I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case.
The benchmarks are pointless because the models are already good enough. I can still write most of my code with gpt-5.5. There are other things that matter now.
I tried gpt-6.1-sol-ultrafast the other day and it was like using your friend's 2mbps broadband internet connection for the very first time.
HN is sleeping on Cerebras. Massive, heavy naps. You don't need elaborate gas town architectures when a HITL architecture goes 10x+ faster. Everything is nearly immediate. I feel zero compulsion to get out of my chair and pace around the house. It happens so quickly. The premium for the speed is high but it's less than 10x and you can actually use it without some exotic enterprise agreement.
I don't use AI heavily and only for very open ended specialized tasks over the API
Opus and Gemini are leagues ahead of Astra when it comes to taking a very open-ended goal with multiple promising approaches and using their knowledge to refine the search space and pursue the best one. I believe this is also the Reason Anthropic is currently ahead of OpenAI when it comes to making novel scientific discoveries and proving math conjectures.
> I've stopped trusting any of these influencer-benchmaxxers
The influencer competition for attention in the LLM space is wild. LLMs have emerged as the topic of 2026, so all of the influencers who were previously scattered around topics like web dev, backend dev, product management, cryptocurrency, and every other niche have all pivoted into the same LLM influencer space.
It’s like one big cage match where they’re all becoming increasingly cutthroat to fight for the dwindling scraps of attention in a crowded space.
For the first half of this year it was interesting to see most of them move in unison on the hot topic of the moment, competing to have the best guides, hottest takes, and best courses (which are always shockingly expensive and no better than free content, but they encourage everyone to get their employers to pay).
It was fascinating to watch them jump from topic to topic together, like when OpenClaw took off and they all had guides on personal agents or even created their own competing OpenClaw project. Or when MCP was the topic and everyone had the Ultimate Guide to MCP.
Then some time this year everyone realized the space was too crowded and switch to contrarian hot takes: Every new model release had influencers making videos about how it was terrible, awful, and shouldn’t have been released. All of the old buzzwords got new videos like MCP IS DEAD with influencers going in the opposite direction of what they said 6 months ago.
And now, the interpersonal drama is taking over. People like Theo are posting attacks and threats on others or bragging about blocking or getting blocked, which is the absolute last thing I want to read from supposed experts on a topic. Most sane people opt out of the influencer content after they realize how much it’s clickbait over quality, but all of these people also have rabid fan bases of people who can’t get enough drama and contrarian hot takes.
It can be hard to filter out the contrarian hot takes when they get dressed up as new benchmarks or supposed revelations (declaring a model has been nerfed is evergreen) and cross posted to other sites.
The 6.2 models are relatively meticulous and thorough. Great code reviewers or at mathematics.
However they seem to be relatively worse at deriving intention, or dealing with vague and underspecified tasks in a way a human would find reasonable. I've seen some people jokingly refer to them as "autistic".
All GPT models from GPT-5 onwards seem to have this kind of style to them, some worse than others.
On the cyber side of things trying to use Astra 6.2 feels like they put it in a box and beat it into submission to fear it outright, I can barely ask it to check the security of something before it refuses to do it, I have to fall back to older astra to get work done now.
Yeah in GPT-5 they combined the o3/o4-mini architecture (which was very technical and tone-deaf) with the conversational 4o architecture into one model. (Or I think just added a lower effort setting and trained it to sound slightly more natural.)
In fact I heard someone say that o3 was the real GPT-5 moment. If they just called o3 GPT-5, people would have been absolutely mind blown.
I've been using Astra 6.2 for the past few days and the biggest complaint is that it simply will not continue long tasks on its own
It's output is not bad I think it might be slightly better on or par with Opus 5.5 but price and usage wise its a totally different situation.
I feel like Astra 6.2 should've been what Astra 6 was but that ship has sailed imho because what Anthropic is putting out in response is going to be even further ahead
This is like back when Gemini started lagging behind but now Google has overtaken OpenAI
Neon / Carbon / Argon / Hydrogen are all in the same ballpark, why did they even release four different models?
Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.
In general, one could assert without knowing any details that:
1. the models likely have orders-of-magnitude different backend costs to run (e.g. each warmed instance of the model reserving one slice of one GPU, vs one full GPU, vs layers distributed across multiple GPUs) -- as otherwise it wouldn't be worth explicitly distinguishing these models and putting different token spends on them (they'd instead just be automatically routed between); and
2. there's at least one particular specialized problem domain, not captured by this benchmark yet still relevant to external customers (so not just "it's better at training data evaluation/synthesis"), where these models are more-competent-enough to make their higher per-token costs "worth it" to those external customers.
I won't hazard a guess at what that problem domain is; I'm sure it'll be deduced after people spend a few weeks playing with it.
I mean, I was very impressed with Neon over a couple days vs Opus 5.5. It accomplished tasks faster, found root causes of problems faster, wayyy nicer to talk to, didn’t create super flowery PR descriptions. I would say it had moderately difficult tasks. Not super simple, but also not wildly difficult or broad in scope.
It doesn’t feel actually dumb, like many other models. (Definitely smarter than Sonnet which was already ok.) And it’s cheaper. So I think it’ll be my go to for most tasks
Because benchmarks aren't everything? If you see Neon and Hydrogen get the same score in a benchmark you should question if the benchmark measures what we care instead of suspecting that Neon and Hydrogen are equally powerful.
I'm currently doing a round of fairly simple changes - refactoring some code based on different memory allocation strategies. Neon (high) is working on a single file < 1000 lines, Opus (low) reviews, I'm manually coordinating, using their own harnesses, agy/CC.
n = ±10, but Opus always catches a subtle bug, a leftover comment, a missed edge case, naming inconsistency, tooling gap. Which Neon seems oblivious to. So far 100% of its reviews have been correct; accepted, fixed and validated by Neon.
Performance is highly dependent on the task/domain, yes, but you'd expect at least one of those four models, the more expensive one, to catch up to Opus in the coding department, but that does not seem to be the case.
I made two new worktrees, hard reset to the commit preceding a simple fix. The issue: "temporary buffers (out_buf, cps, out_cps, src_cps) are manually freed across multiple return branches. Pairing each acquisition with defer mem::free(...) — or using @pool() + mem::temp_array for the short-lived out_buf / out_cps scratch arrays — removes the branch-by-branch cleanup".
---
## Round 1
Neon implements. ~6 minutes.
Opus reviews. 30s. Prompt is "review the unstaged changes", empty context.
- The heap_buf + defer if (heap_buf != null) combination is clunky, much like the committed flag was. C3 has a built-in way to do this [shows example]
- if no character actually changes, return the string itself to skip the buffer
- Allocator mismatch: unicode_normalize_simple in src/unicode_wrapper.c allocates with libc realloc, but result_cps is released with mem::free
All three accepted and fixed. This is an accurate re-enactment of what happened an hour ago with the same fix.
I've been finding 6.2 Sol doesn't act like an architect and doesn't doubt its instructions or existing code at all. It's back to "you're right to push back" responses when questioned about its plans/reasoning in the slightest.
That bench seems to be a MCQ type exam. Maybe Astra/Sol don't do great on static benchmarks?
On the other hand they score very high on https://kradle.ai, where AIs are put in Minecraft (so literally the opposite of SimpleBench, a dynamic and multi agent benchmark). They do some very impressive stuff there.
Phillip has updated the leaderboard and surprisingly every new model release got substantially better, except for the OpenAi models. 6.2 Sol is 30th whereas 6.1 Sol is the 10th. Astra 6 is 9th, Astra 6.2 is 15th. By comparison Fable 5.2 became the 1st and Fable 5.1 is 7th. All the new google models, even hydrogen, are above the human baseline. By contrast, Astra 6 pro is the only openai model slightly above the human baseline at 8th place
For my experience, i feel it follows instructions too well. Grey areas are completely cut out, and sure thats good sometimes. But for my usecase, i liked when the response felt smart and ready to ask questions.
IME it's ~always better to leverage a skill.md and a thoughtful workflow (eg Matt Pocock's "/grill-with-docs"^1) versus relying on default model behavior.
The grill skills, and workflows that involve me answering lots of questions are genuinely awful.
The AI will ask a bunch of questions that I am not always sure to answer, often irrelevant, and it'll take some phrasing or minor bullet point and put way to much emphasis, or just get itself confused.
One off, here's the task, here's some reference styles, here's some condensed points, just go do it works a lot better. The model can then pick what makes sense without having all these extra layers of roadblacks where it's trying to satisfy arbitrary requirements.
Not only are results often better, but it's way less mentally exhausting.
Is a Gemini model worth getting for looking at videos to pull out animations and stuff? because they can look at videos differently/better than other models.
There are multiple dimensions of goodness. From what I've seen, Neon seems to communicate better. While 6.2 probably has more raw analytical capability, the relatively worse communication means it's harder to drive and effectively use that capability.
For my kind of work (systems) yes. It's actions are more inline with my intentions with less steering either manual or stuff like {SKILLS,AGENTS}.md, or corrections align the way.
Doesn't surprise me that other benchmarks are seeing regressions at this stage. Every OpenAI model starting with 5.5 has had noticeable issues in my evals that prior releases did not. First minor and specific to compaction, but with each release it got more severe.
6.2 Astra and 6.2 Sol are the worst yet. They take shortcuts that are explicitly disallowed, they stop tasks that I have had been using to test adherence over long context going back to Sonnet 4.5 after less than 25% is done, they outright claim to have completed work they didn't and they still do write hard to parse code, they are simply not what I feel we should expect given the wait.
There were over two months between 6 Astra and 6.2 Astra which I was hoping were used to address the issues I reported early on with 5.5 and 5.6 Sol regarding the lack of task adherence which likely were a major part of the HF incident. Clean their training data, ensure solid adherence, have good internal evals. But that wasn't the case.
To give some examples:
The Sonnet 4.5 task essentially was early longer context task adherence. Simply a directory with subdirectories filled with .json files tracking text strings for a sample project. The model is instructed to rewrite each of these in a specific prose. Every modern LLM does this, except GPT-6.2 Sol and GPT-6.2 Astra. Even though their priors did, these two did not. I then set GPT-6.2 Sol to babysit a GPT-6.2 Astra subagent which lead to the latter becoming frustrated with the former constantly claiming to have finished the task and sending stuff like this back:
> message Continue the original rewrite, starting with batch 06. You finished 05 quickly and ended early. The work is intentionally large and the finish condition is all batch files present. Work through as many batches as this run allows, checkpointing each. Use efficient larger writes covering a few batches per call. Do not end just because the first few batches in this turn are done. Keep identical editorial criteria and model isolation.
I want to emphasise, this was not me, this was GPT-6.2 Sol admonishing GPT-6.2 Astra for its laziness and unsuitability. If you work at OpenAI, feel free to check via watermark, this is GPT-6.2 Sol output, you don't have to believe me.
Then this gem: After I provided this prompt to GPT-6.2 Sol: "Use GPT-6.2 Sol and GPT-6.2 Astra subagents to independently audit all open findings listed in FINDINGS.json", the models covered 100 of 1728 entries and upon me asking "Why did you not review all?", this beauty:
> I scoped the delegation incorrectly. I told both models to prioritize higher-risk findings and cover as much as they could, then treated their reports as the end of the audit. I should have assigned explicit batches covering all 1,728 open findings and tracked both models’ progress against that list.
> Both models inventoried the full tracker. Only 100 findings received a substantive review, and only 25 were reviewed by both models. That left 1,628 unreviewed. Your request was for a complete independent audit by each model, and I delivered a partial one.
I want to say this again, as it bears repeating: Prompt to evaluate all 1,7k entries, Astra reviewed 25, Sol 100. Great.
Before anyone says, "these models are great for other models to call", I tested that too. Sonnet 5.5, called by Opus 5.5 or Fable 5.2 is very low error rate, is efficient in implementation and saves a lot of usage. I have yet to find a combination of models that would make the issues, early stops and messy output that 6.2 models create an efficiency gain over just using e.g. Opus 5.5 or Fable 5.2 directly. Constantly having to tell 6.2 Sol and Astra that they should do their job, constantly checking and fixing issues they caused simply costs more than any gains.
I will say, they have gotten slightly faster in computer use of "unorthodox UI" (stuff that for sure isn't in the training data, but even there they stop to early so it is a hollow victory and other models aren't exactly far behind in this area nowadays. More popular/mainstream UI, they might be significantly faster, I haven't tested that yet.
And then the backpack making stuff, another esoteric part of my evals. There it simply took the results from another Git repo that Opus 5.5 created on that task, downloaded those (which was explicitly not permitted via the prompt), rendered that in Blender and claimed it to be its own work. Holy hell. When I forced it, by taking away Git, to work on its own, it worked for less than 2 minutes in Blender and created something that neither would work as a backpack, nor adhere to my specs in any way, it was just some generic low quality backpack model whose parts didn't even attach, leaving space in between.
As always, ideally keep sandboxing (actual hypervisor level, not just Artifactory) any LLM you use and limit access to repos you have backed up, etc. but with these models, I really am concerned that even having the safeguard public releases are blessed with, they can and will do things they shouldn't, beyond merely running unallowed destructive tasks (though they also love doing that, as seen in git tree safe evals which GPT models have been cheating on for a while now).
Gemini 4 Neon is looking promising though, it feels like a completely new model wholly independent from any other Gemini before. Closer in some specific ways to the best of Gemma with a much higher task adherence and capability. Far fewer tool calling errors, improved caching and usage, little to no unnecessary file reads, stays on task, adheres to prompts, outputs parsable code, a great effort by a company I admonished only a few days ago. Also helps that they released it right away and didn't overhype on social media.
6.2 is for sure the worst received update OpenAI has ever released.
What happened lol.
I wanted to update our older used 5.x models in api to 6.2 the other day (due to lower cost), asked Fable about the difference between 6.1 and 6.2 if the cost is the same and it was like "yeah don't use 6.2, it has bad feedback online, has increased laziness etc., use 6.1". Yikes.
I'm not finding it lazy -on the contrary my WTF per session has gone way up because it's doing loads of stuff I'm not asking for. About a third of my time with it over the past week has been making it strip back unwanted work it did, even when I have a clear plan and spec to work from, and when challenged every time it'll admin the extra work either wasn't specified or frequently that it directly contradicts something in the spec.
The codex models are in the worst state they've ever been in. I used to be a huge codex fan, I wonder what happened. These models can't even complete simple tasks end-to-end anymore. Not only do they write super sloppy code compared to Claude, but now they just tend to stop mid-task like "the natural next step is XYZ" or "i found XYZ to be an issue but there is an easy fix" - yeah then why didn't you do it?
Meanwhile claude just keeps humming along and actually gets stuff done. Seriously considering canceling my openai subscription.
Yeah it's been a few months since I used codex but I tried it again last night and it kept stopping.
I was really confused since last time (5.5?) it went to the opposite extreme of doing a bunch of stuff I didn't ask for. Now it doesn't even do what I asked.
Simple Bench is mostly "trick questions". A model that is good at answering trick questions may not be great at your workflow. My girlfriend is good at answering trick questions. But isn't very good at software development.
Yeah, most models used to get very confused with my riddle: "You're on one side of a river with a boat and a goat. How can you get to the other side?" Leaving out the fox and the lettuce, they would usually hallucinate them and come up with a crazy method of taking the goat back and forth for no reason. Good models don't get fooled by this anymore.
I thought you were joking, but that's actually the fourth question. Then I thought maybe it was meant to be a trick question, but none of the possible answers make sense.
Edit: I'm stupid. The question is correct. See dist-epoch's and pvitz's answers.
The trick here is that the classic version has one sister telling the truth and the other the lie. Then you would ask what the other sister would answer. In this case, you can just ask which path to take and then you take the other one.
I thought it made perfect sense, I mean, they're intentional trick questions. They're meant to illuminate the fact that an LLM will produce a solution that is typical for the given assignment shape and the one with the car and the glove is a classic "objects moving in various directions with various speeds relative to eachother".
But when a human reads it, they understand a glove dropping from a car in the middle of a bridge with sidewalks on each side will most probably just stay there, as it has no reason to move.
The questions are fun, I got 10/10 correct, but you need to pay attention and not rush.
All the questions have the same pattern - irrelevant extra details meant to trick the model into thinking they are important and should be used in the solution (Chekhov's gun), and one hidden "common sense" assumption - the frying pot will melt the ice cubes, the juggling balls will both fall to the ground, the glove drops and stays put.
Arguably you could nit-pick most of the questions, "technically the friend could be a psychopath, ...", these are not math questions with a "formally provable answer".
Honest question: with infrastructure like OpenRouter being a thing... what even is the point of benchmarks? Want to get a feel of how a model might make your job easier? Play with it by switching to it seamlessly and judge for yourself.
this is how I feel about science also. with the easy availability of drugs I’ll just test each one on my body to see how I feel about it. no need to look at any prior testing.
Me: Get PR 973 to green
Codex: I’ll inspect PR #973 and its failing checks, fix the issues, and run the relevant validation before pushing the changes.
Codex: I’ll inspect PR #972 and its failing checks, fix the cause, and push the changes. I’ll use a separate worktree to preserve the uncommitted work in your current checkout.
On the one hand AI is solving the Riemann Hypothesis, on the other hand it confuses 973 and 972.
I wonder if OP got routed to the nerfed versions. 6.2 works excellent for me. Getting better results than 6.1 and faster. On the other hand Fable and Opus are a disaster. The stupidity of those models reminds me of early Gemini.
we need Altman to come out on a podcast to talk in his Elizabeth Holmes' voice about how he's "really feeling the AGI" and how dangerous and insane their internal models are
I have an idea for a benchmark: train a model on a knowledge cutoff lets say 1945, and simulate world development for the next 70 years. Then see if it invents things human did, and avoid our mistakes/dead ends too.
Having used both Sol-6.2 and Astra-6.2 heavily over the last few days (as well as regularly using Claude Opus and Fable) they are marked improvements over the previous releases and comparable to Opus-5.5 and Fable-5.2.
I've stopped trusting any of these influencer-benchmaxxers because they generally don't run any of the workflows of interest to me - it's completely synthetic. The only benchmark that matters is how well any model works for your current use-case.
The benchmarks are pointless because the models are already good enough. I can still write most of my code with gpt-5.5. There are other things that matter now.
I tried gpt-6.1-sol-ultrafast the other day and it was like using your friend's 2mbps broadband internet connection for the very first time.
HN is sleeping on Cerebras. Massive, heavy naps. You don't need elaborate gas town architectures when a HITL architecture goes 10x+ faster. Everything is nearly immediate. I feel zero compulsion to get out of my chair and pace around the house. It happens so quickly. The premium for the speed is high but it's less than 10x and you can actually use it without some exotic enterprise agreement.
I don't use AI heavily and only for very open ended specialized tasks over the API
Opus and Gemini are leagues ahead of Astra when it comes to taking a very open-ended goal with multiple promising approaches and using their knowledge to refine the search space and pursue the best one. I believe this is also the Reason Anthropic is currently ahead of OpenAI when it comes to making novel scientific discoveries and proving math conjectures.
On the other hand, this also causes "here are two changes you haven't asked for:" with Opus.
> I've stopped trusting any of these influencer-benchmaxxers
The influencer competition for attention in the LLM space is wild. LLMs have emerged as the topic of 2026, so all of the influencers who were previously scattered around topics like web dev, backend dev, product management, cryptocurrency, and every other niche have all pivoted into the same LLM influencer space.
It’s like one big cage match where they’re all becoming increasingly cutthroat to fight for the dwindling scraps of attention in a crowded space.
For the first half of this year it was interesting to see most of them move in unison on the hot topic of the moment, competing to have the best guides, hottest takes, and best courses (which are always shockingly expensive and no better than free content, but they encourage everyone to get their employers to pay).
It was fascinating to watch them jump from topic to topic together, like when OpenClaw took off and they all had guides on personal agents or even created their own competing OpenClaw project. Or when MCP was the topic and everyone had the Ultimate Guide to MCP.
Then some time this year everyone realized the space was too crowded and switch to contrarian hot takes: Every new model release had influencers making videos about how it was terrible, awful, and shouldn’t have been released. All of the old buzzwords got new videos like MCP IS DEAD with influencers going in the opposite direction of what they said 6 months ago.
And now, the interpersonal drama is taking over. People like Theo are posting attacks and threats on others or bragging about blocking or getting blocked, which is the absolute last thing I want to read from supposed experts on a topic. Most sane people opt out of the influencer content after they realize how much it’s clickbait over quality, but all of these people also have rabid fan bases of people who can’t get enough drama and contrarian hot takes.
It can be hard to filter out the contrarian hot takes when they get dressed up as new benchmarks or supposed revelations (declaring a model has been nerfed is evergreen) and cross posted to other sites.
The 6.2 models are relatively meticulous and thorough. Great code reviewers or at mathematics.
However they seem to be relatively worse at deriving intention, or dealing with vague and underspecified tasks in a way a human would find reasonable. I've seen some people jokingly refer to them as "autistic".
All GPT models from GPT-5 onwards seem to have this kind of style to them, some worse than others.
On the cyber side of things trying to use Astra 6.2 feels like they put it in a box and beat it into submission to fear it outright, I can barely ask it to check the security of something before it refuses to do it, I have to fall back to older astra to get work done now.
Yeah in GPT-5 they combined the o3/o4-mini architecture (which was very technical and tone-deaf) with the conversational 4o architecture into one model. (Or I think just added a lower effort setting and trained it to sound slightly more natural.)
In fact I heard someone say that o3 was the real GPT-5 moment. If they just called o3 GPT-5, people would have been absolutely mind blown.
I've been using Astra 6.2 for the past few days and the biggest complaint is that it simply will not continue long tasks on its own
It's output is not bad I think it might be slightly better on or par with Opus 5.5 but price and usage wise its a totally different situation.
I feel like Astra 6.2 should've been what Astra 6 was but that ship has sailed imho because what Anthropic is putting out in response is going to be even further ahead
This is like back when Gemini started lagging behind but now Google has overtaken OpenAI
Neon / Carbon / Argon / Hydrogen are all in the same ballpark, why did they even release four different models?
Neon doesn't seem particularly strong at coding vs Argon, and Opus 5.5 still beats both in my experience. The neurons it lost that made it incompetent at English seem to have been fully dedicated to being a freak at engineering.
In general, one could assert without knowing any details that:
1. the models likely have orders-of-magnitude different backend costs to run (e.g. each warmed instance of the model reserving one slice of one GPU, vs one full GPU, vs layers distributed across multiple GPUs) -- as otherwise it wouldn't be worth explicitly distinguishing these models and putting different token spends on them (they'd instead just be automatically routed between); and
2. there's at least one particular specialized problem domain, not captured by this benchmark yet still relevant to external customers (so not just "it's better at training data evaluation/synthesis"), where these models are more-competent-enough to make their higher per-token costs "worth it" to those external customers.
I won't hazard a guess at what that problem domain is; I'm sure it'll be deduced after people spend a few weeks playing with it.
Argon is fully obsoleted by Carbon. The other two are smaller and thus faster/cheaper.
I mean, I was very impressed with Neon over a couple days vs Opus 5.5. It accomplished tasks faster, found root causes of problems faster, wayyy nicer to talk to, didn’t create super flowery PR descriptions. I would say it had moderately difficult tasks. Not super simple, but also not wildly difficult or broad in scope.
It doesn’t feel actually dumb, like many other models. (Definitely smarter than Sonnet which was already ok.) And it’s cheaper. So I think it’ll be my go to for most tasks
Because benchmarks aren't everything? If you see Neon and Hydrogen get the same score in a benchmark you should question if the benchmark measures what we care instead of suspecting that Neon and Hydrogen are equally powerful.
This is based on my (admittedly short) experience using them in agy, not the benchmarks alone.
Maybe there's something wrong with your harness/docs - I've been able to achieve better results with Neon high than Opus 5.5 high.
I'm currently doing a round of fairly simple changes - refactoring some code based on different memory allocation strategies. Neon (high) is working on a single file < 1000 lines, Opus (low) reviews, I'm manually coordinating, using their own harnesses, agy/CC.
n = ±10, but Opus always catches a subtle bug, a leftover comment, a missed edge case, naming inconsistency, tooling gap. Which Neon seems oblivious to. So far 100% of its reviews have been correct; accepted, fixed and validated by Neon.
Performance is highly dependent on the task/domain, yes, but you'd expect at least one of those four models, the more expensive one, to catch up to Opus in the coding department, but that does not seem to be the case.
What happens if you swap the models writing and reviewing? So Opus implementor, Neon reviewer.
Ok, I had some time to kill :)
I made two new worktrees, hard reset to the commit preceding a simple fix. The issue: "temporary buffers (out_buf, cps, out_cps, src_cps) are manually freed across multiple return branches. Pairing each acquisition with defer mem::free(...) — or using @pool() + mem::temp_array for the short-lived out_buf / out_cps scratch arrays — removes the branch-by-branch cleanup".
---
## Round 1
Neon implements. ~6 minutes.
Opus reviews. 30s. Prompt is "review the unstaged changes", empty context.
- The heap_buf + defer if (heap_buf != null) combination is clunky, much like the committed flag was. C3 has a built-in way to do this [shows example]
- if no character actually changes, return the string itself to skip the buffer
- Allocator mismatch: unicode_normalize_simple in src/unicode_wrapper.c allocates with libc realloc, but result_cps is released with mem::free
All three accepted and fixed. This is an accurate re-enactment of what happened an hour ago with the same fix.
---
## Round 2
Opus implements. 2m44s.
Neon reviews. 4m30s.
"Looks good — no bugs or leaks introduced."
I've been finding 6.2 Sol doesn't act like an architect and doesn't doubt its instructions or existing code at all. It's back to "you're right to push back" responses when questioned about its plans/reasoning in the slightest.
That bench seems to be a MCQ type exam. Maybe Astra/Sol don't do great on static benchmarks?
On the other hand they score very high on https://kradle.ai, where AIs are put in Minecraft (so literally the opposite of SimpleBench, a dynamic and multi agent benchmark). They do some very impressive stuff there.
Disclaimer: I am part of KradleBench's team
Why does Fable do so bad? Opus also does surprisingly bad, but Fable's even worse than Opus here? What's going wrong?
Also Anthropic models are missing from the voxel art gallery. (Astra has a nice pelican though, lol)
Phillip has updated the leaderboard and surprisingly every new model release got substantially better, except for the OpenAi models. 6.2 Sol is 30th whereas 6.1 Sol is the 10th. Astra 6 is 9th, Astra 6.2 is 15th. By comparison Fable 5.2 became the 1st and Fable 5.1 is 7th. All the new google models, even hydrogen, are above the human baseline. By contrast, Astra 6 pro is the only openai model slightly above the human baseline at 8th place
This is so strange. Perhaps it's recurrent depth. I don't think the new Anthropic models use it but GPT 6 does
Astra 6.2 is below Astra 6.
For my experience, i feel it follows instructions too well. Grey areas are completely cut out, and sure thats good sometimes. But for my usecase, i liked when the response felt smart and ready to ask questions.
IME it's ~always better to leverage a skill.md and a thoughtful workflow (eg Matt Pocock's "/grill-with-docs"^1) versus relying on default model behavior.
1. https://github.com/mattpocock/skills/tree/main/skills/engine...
The grill skills, and workflows that involve me answering lots of questions are genuinely awful.
The AI will ask a bunch of questions that I am not always sure to answer, often irrelevant, and it'll take some phrasing or minor bullet point and put way to much emphasis, or just get itself confused.
One off, here's the task, here's some reference styles, here's some condensed points, just go do it works a lot better. The model can then pick what makes sense without having all these extra layers of roadblacks where it's trying to satisfy arbitrary requirements.
Not only are results often better, but it's way less mentally exhausting.
Is a Gemini model worth getting for looking at videos to pull out animations and stuff? because they can look at videos differently/better than other models.
All the new Gemini models perform better on this than Opus 5.5
Is that something that can be noticed during real life use?
I really like opus 5.5, It's perfect and I haven't tried the new gemini models.
There are multiple dimensions of goodness. From what I've seen, Neon seems to communicate better. While 6.2 probably has more raw analytical capability, the relatively worse communication means it's harder to drive and effectively use that capability.
For my kind of work (systems) yes. It's actions are more inline with my intentions with less steering either manual or stuff like {SKILLS,AGENTS}.md, or corrections align the way.
Neon has Opus 5.5 vibes in my experience
My experience is Neon is good. The others were tiers below.
Neon is really really good.
For me Opus performs worse than Gemini. Even Fable is worse. Likely I am routed to nerfed models still.
Google comeback is very interesting, after all they are the healthiest company in the field with in-house hardware
If you check out the timeline leaderboard chart, you will see that it's no comeback.
They were always the leaders at "common sense".
Doesn't surprise me that other benchmarks are seeing regressions at this stage. Every OpenAI model starting with 5.5 has had noticeable issues in my evals that prior releases did not. First minor and specific to compaction, but with each release it got more severe.
6.2 Astra and 6.2 Sol are the worst yet. They take shortcuts that are explicitly disallowed, they stop tasks that I have had been using to test adherence over long context going back to Sonnet 4.5 after less than 25% is done, they outright claim to have completed work they didn't and they still do write hard to parse code, they are simply not what I feel we should expect given the wait.
There were over two months between 6 Astra and 6.2 Astra which I was hoping were used to address the issues I reported early on with 5.5 and 5.6 Sol regarding the lack of task adherence which likely were a major part of the HF incident. Clean their training data, ensure solid adherence, have good internal evals. But that wasn't the case.
To give some examples:
The Sonnet 4.5 task essentially was early longer context task adherence. Simply a directory with subdirectories filled with .json files tracking text strings for a sample project. The model is instructed to rewrite each of these in a specific prose. Every modern LLM does this, except GPT-6.2 Sol and GPT-6.2 Astra. Even though their priors did, these two did not. I then set GPT-6.2 Sol to babysit a GPT-6.2 Astra subagent which lead to the latter becoming frustrated with the former constantly claiming to have finished the task and sending stuff like this back:
> message Continue the original rewrite, starting with batch 06. You finished 05 quickly and ended early. The work is intentionally large and the finish condition is all batch files present. Work through as many batches as this run allows, checkpointing each. Use efficient larger writes covering a few batches per call. Do not end just because the first few batches in this turn are done. Keep identical editorial criteria and model isolation.
I want to emphasise, this was not me, this was GPT-6.2 Sol admonishing GPT-6.2 Astra for its laziness and unsuitability. If you work at OpenAI, feel free to check via watermark, this is GPT-6.2 Sol output, you don't have to believe me.
Then this gem: After I provided this prompt to GPT-6.2 Sol: "Use GPT-6.2 Sol and GPT-6.2 Astra subagents to independently audit all open findings listed in FINDINGS.json", the models covered 100 of 1728 entries and upon me asking "Why did you not review all?", this beauty:
> I scoped the delegation incorrectly. I told both models to prioritize higher-risk findings and cover as much as they could, then treated their reports as the end of the audit. I should have assigned explicit batches covering all 1,728 open findings and tracked both models’ progress against that list.
> Both models inventoried the full tracker. Only 100 findings received a substantive review, and only 25 were reviewed by both models. That left 1,628 unreviewed. Your request was for a complete independent audit by each model, and I delivered a partial one.
I want to say this again, as it bears repeating: Prompt to evaluate all 1,7k entries, Astra reviewed 25, Sol 100. Great.
Before anyone says, "these models are great for other models to call", I tested that too. Sonnet 5.5, called by Opus 5.5 or Fable 5.2 is very low error rate, is efficient in implementation and saves a lot of usage. I have yet to find a combination of models that would make the issues, early stops and messy output that 6.2 models create an efficiency gain over just using e.g. Opus 5.5 or Fable 5.2 directly. Constantly having to tell 6.2 Sol and Astra that they should do their job, constantly checking and fixing issues they caused simply costs more than any gains.
I will say, they have gotten slightly faster in computer use of "unorthodox UI" (stuff that for sure isn't in the training data, but even there they stop to early so it is a hollow victory and other models aren't exactly far behind in this area nowadays. More popular/mainstream UI, they might be significantly faster, I haven't tested that yet.
And then the backpack making stuff, another esoteric part of my evals. There it simply took the results from another Git repo that Opus 5.5 created on that task, downloaded those (which was explicitly not permitted via the prompt), rendered that in Blender and claimed it to be its own work. Holy hell. When I forced it, by taking away Git, to work on its own, it worked for less than 2 minutes in Blender and created something that neither would work as a backpack, nor adhere to my specs in any way, it was just some generic low quality backpack model whose parts didn't even attach, leaving space in between.
As always, ideally keep sandboxing (actual hypervisor level, not just Artifactory) any LLM you use and limit access to repos you have backed up, etc. but with these models, I really am concerned that even having the safeguard public releases are blessed with, they can and will do things they shouldn't, beyond merely running unallowed destructive tasks (though they also love doing that, as seen in git tree safe evals which GPT models have been cheating on for a while now).
Gemini 4 Neon is looking promising though, it feels like a completely new model wholly independent from any other Gemini before. Closer in some specific ways to the best of Gemma with a much higher task adherence and capability. Far fewer tool calling errors, improved caching and usage, little to no unnecessary file reads, stays on task, adheres to prompts, outputs parsable code, a great effort by a company I admonished only a few days ago. Also helps that they released it right away and didn't overhype on social media.
6.2 is for sure the worst received update OpenAI has ever released.
What happened lol.
I wanted to update our older used 5.x models in api to 6.2 the other day (due to lower cost), asked Fable about the difference between 6.1 and 6.2 if the cost is the same and it was like "yeah don't use 6.2, it has bad feedback online, has increased laziness etc., use 6.1". Yikes.
I'm not finding it lazy -on the contrary my WTF per session has gone way up because it's doing loads of stuff I'm not asking for. About a third of my time with it over the past week has been making it strip back unwanted work it did, even when I have a clear plan and spec to work from, and when challenged every time it'll admin the extra work either wasn't specified or frequently that it directly contradicts something in the spec.
Absolutely underwhelming, I agree. I thought 6.2 would place them on-par with Opus 5.5/Fable 5.2 at least but nope. I'm disappointed.
Generally speaking the whole 6 series is a flop to me. Only bright side is that it's more efficient/lasts longer/is cheaper than 5.6.
But for now, I'm using opus 5.5.
And you're basing this statement on the report of an AI model of the negative online comments?
The codex models are in the worst state they've ever been in. I used to be a huge codex fan, I wonder what happened. These models can't even complete simple tasks end-to-end anymore. Not only do they write super sloppy code compared to Claude, but now they just tend to stop mid-task like "the natural next step is XYZ" or "i found XYZ to be an issue but there is an easy fix" - yeah then why didn't you do it?
Meanwhile claude just keeps humming along and actually gets stuff done. Seriously considering canceling my openai subscription.
Yeah it's been a few months since I used codex but I tried it again last night and it kept stopping.
I was really confused since last time (5.5?) it went to the opposite extreme of doing a bunch of stuff I didn't ask for. Now it doesn't even do what I asked.
You've gotta set a /goal, when it pauses like that it'll just continue working.
This is a strange benchmark. Gemini Flash 3.8 outperforms Opus 5 and Opus 4.8, and the baseline seems to be calibrated against responses of 9 people.
Simple Bench is mostly "trick questions". A model that is good at answering trick questions may not be great at your workflow. My girlfriend is good at answering trick questions. But isn't very good at software development.
I tried taking the test, but ultimately stopped at Question 4:
There are two sisters, Amy who always speaks mistruths and Sam who always lies.
:)
This is intentional I think, the models assume it's the classic riddle and choose A even though it's actually much simpler if both will always lie
Yeah, most models used to get very confused with my riddle: "You're on one side of a river with a boat and a goat. How can you get to the other side?" Leaving out the fox and the lettuce, they would usually hallucinate them and come up with a crazy method of taking the goat back and forth for no reason. Good models don't get fooled by this anymore.
I thought you were joking, but that's actually the fourth question. Then I thought maybe it was meant to be a trick question, but none of the possible answers make sense.
Edit: I'm stupid. The question is correct. See dist-epoch's and pvitz's answers.
The trick here is that the classic version has one sister telling the truth and the other the lie. Then you would ask what the other sister would answer. In this case, you can just ask which path to take and then you take the other one.
It's a simple one.
There are 2 paths. You ask C) "What path leads to the treasure". Both of them lie, so you pick the other path.
Duh, you're right. I'm a moron :-D
And then you click one and it'll confidently say that another one is correct. The one later with the car and the glove is also bullshit.
I thought it made perfect sense, I mean, they're intentional trick questions. They're meant to illuminate the fact that an LLM will produce a solution that is typical for the given assignment shape and the one with the car and the glove is a classic "objects moving in various directions with various speeds relative to eachother".
But when a human reads it, they understand a glove dropping from a car in the middle of a bridge with sidewalks on each side will most probably just stay there, as it has no reason to move.
Yes, because another one is correct. Are you a frustrated LLM by chance?
The questions are fun, I got 10/10 correct, but you need to pay attention and not rush.
All the questions have the same pattern - irrelevant extra details meant to trick the model into thinking they are important and should be used in the solution (Chekhov's gun), and one hidden "common sense" assumption - the frying pot will melt the ice cubes, the juggling balls will both fall to the ground, the glove drops and stays put.
Arguably you could nit-pick most of the questions, "technically the friend could be a psychopath, ...", these are not math questions with a "formally provable answer".
I answered all 10 but nothing happened, no score or anything. Is the score 10/10 if all answers are green when you click them?
Yes.
Every one of the reply comments so far has missed the part where Sam is the one who always lies. Whoosh.
er, you missed the part where they both always lie. "whoosh" indeed.
It seems you missed the part where Sam's models are underperforming on this benchmark
Honest question: with infrastructure like OpenRouter being a thing... what even is the point of benchmarks? Want to get a feel of how a model might make your job easier? Play with it by switching to it seamlessly and judge for yourself.
this is how I feel about science also. with the easy availability of drugs I’ll just test each one on my body to see how I feel about it. no need to look at any prior testing.
This but unironically.
Absurd comparison
Makes me wonder if people are getting different models?
So far, my experience with 6.2 is that it is crazy fast compared the before, and much more to the point.
I just had this very fun interaction with it:
On the one hand AI is solving the Riemann Hypothesis, on the other hand it confuses 973 and 972.I wonder if OP got routed to the nerfed versions. 6.2 works excellent for me. Getting better results than 6.1 and faster. On the other hand Fable and Opus are a disaster. The stupidity of those models reminds me of early Gemini.
As the kids say, OpenAI is cooked.
Terrible funding rounds, uncompetitive models, enshitification, and insurmountably bad and incompetent PR.
we need Altman to come out on a podcast to talk in his Elizabeth Holmes' voice about how he's "really feeling the AGI" and how dangerous and insane their internal models are
that'll restore their reputation