I'll use this opportunity while it's still early to share my experience with Argon.
I got it via the "Pro" plan, exclusively for personal task usage. So no agentic programming.
For starters, the limits are dreadfully low. Five messages on medium is already enough to trigger a 4 hour cooldown. Secondly, the best part is the integration with Google search and Maps. Makes it seamless for trip planning. Lastly it's the only "SOTA" model in a long time that has actively hallucinated details. I don't know if it's because the Gemini harness is not asking it to verify information, but I've had some real Deepseek V3-like lies in responses.
Either way, I find it difficult to get invested in Gemini because nobody will have access to Carbon for a few months anyways.
I similarly had Argon respond with a vast hallucination. It surprised me because it wasn't a little wrong on the details but a fully false verbose response that completely fabricated fabricated everything. I'd recently had Claude Code help me create transcripts for some audio notes from my mother in law and decided to see how some other models would handle the same task sans harness. Argon made up a long conversation from a police sting operation with under cover operatives seemingly just because it read the file names and identified that they would have come from a recording device.
Honestly the fiction writing was pretty impressive!
Suddenly very curious what the llm roleplay people think about this one lol. Fortunately I've already learned to stifle my curiosity on that specific subject.
It goes way back, I work in AI/Ml (taken over by transformers/LLMs) space since early days of this "AI Summer". We get some nice grad school students interning from top universities. I met one during a trip to the office, from top notch school, introduced me to Triton programming when it first came out. It was early days of LLaMA just coming, when The Bloke was regularly dropping finetunes, I was joking asking him what the hacker stack was for "AI waifus" that the cool kids were using. Lo and behold he drops the name of the SillyTavern project. Which later was really educational in setting up local inference servers and in other ways, may have had to, ahem, delete some browser history.
But my experience is that the agent matters a bit more than the model. It is fun on ChatGPT because of its recent (?) bias to always be creating images for conversations around fiction.
what are some of the vagaries of how that use case depends on the agent aka harness? i’ve only been focused on sort of how to make models reliable for my math / cs / engineering and arts and crafts work loads, but RP is an important category for some folks. like are the fails of the various models different from more technical yet creative work loads?
eg im working on rigorizing a proof that a fun algorithm of mine is not only the first provably efficient algorithm for a particular problem, but its also the fastest in practice. ive found that for the sort of stuff i need to assume the model will get confused in the out of training distribution stuff from my personal library of ideas, so they will at best be a confused secretary intern mashup
Yes! I was having Argon summarize some project updates and it... just made up entire paragraphs that had no basis in reality. It felt like I was back in GPT3 days.
Will have to give Carbon a try, but if it is anything like what I saw before it was entirely unusable.
If you uploaded the audio via the web site or app and got terrible results I'd bet Argon wasn't the model doing the transcribing. The only way to know for sure is to try again with the API.
Yeah... Given that everyone else seems to make models available either immediately or within days of announcement, idk why Google feels it's competitive or acceptable to do these 'available at some point' releases.
Capabilities are also improving so fast that this fomo doesn't really keep people excited. If I can't run it, I'll sorta just click off. And given how they handled the Argon release, I've not even bothered look at this Carbon one. Fully expected it to not be usable yet.
The other labs do the same process, but they don't announce the model until it is about to be released. For some reason Google is announcing at the start of th process.
My guess is Google knows it is (way) behind the and needs to keep drumming up interest. Although at some point it becomes counterproductive (raising "yet another not available model; is it vaporware" vibes).
Creating a false sense of scarcity is essential to get people talking about it. I get almost endless inquiries from peers now that I have access to Mythos.
I'm not sure about this. I myself am curious about this model, but not for my own use. I've already written it off as too expensive. The money won't be flowing their way regardless.
> argon was announced to coincide with performance review deadline. carbon was announced to coincide with quarterly earnings release.
Same as it ever was. Google is really completing their IBM-ification. We need to release these models now because year end reviews are about to hit and I need a win.
It's late in life to learn this, but let me be the one to let you in on a little secret: people often say only some of the things they know, they often selectively tell you things while not revealing everything they know.
You are assuming that Carbon is ready for release, deployed to their datacenters, that all scaling and other issues have been worked out, and they are confident in all of the above. Not to mention additional internal constraints like alignment testing or other safety procedures their own policies and guardrails require.
If they are smart, which they clearly have some smart people if they are making the best frontier model, they would account for the above needs in their timeline, and announce the model when they can prove they have it but also before they are ready to release it. This is so they can be at the top of the frontier model lists, boost their brand for both Gemini and Google, get the bump in stock price and investor confidence, steer adoption decisions for people making decisions for what platform they might adopt, etc.
Eh, maybe I'm too much of an AI safety person, but I'd rather the labs be forced to announce when they are working on large SOTA models like they are a clinical trial. That way if it's never announced they have to say something like "training didn't work out" or "it took over the server farm and we had to burn the hard disks".
What's the betting we see another "dramatic" Anthropic release either shortly before the IPO, or at same time they release their Q3 results which were conveniently omitted from their S-1 filing.
hmm, I haven't tried Carbon yet. But I have been using Argon on and off via agent-shell - https://github.com/xenodium/agent-shell and unofficial ACP client.
I am also using it via Google pro tier and did not immediately run into quota issues. But as someone else said in previous discussion of this - Google Pro tier already includes lot of stuff (including Youtube premium etc), the AI plans are just a nice addon.
yeah thats basically the only reason i have the google AI plan
YT Premium, increased drive/photos data, & home premium (though i may have that already tbh)
Only using flash & argon for toy projects since its included though Argon availablity is very small, similar to the included claude usage avaialble in antigravity
> I find it difficult to get invested in Gemini because nobody will have access to Carbon for a few months anyways.
I feel like you're being overly dramatic here. Argon was 13 - 15 days.
Argon was announced September 30, made available on October 13 to Ultra everywhere, and Pro in the Gemini App. Pro plans got Argon on October 15 in AGY.
This is the first time they've slow-rolled a model. Who knows how long they'll wait on Carbon.
The Ultra-lite plans ($99/month) get some pretty good quota where you can actually use Argon some more. Maybe Google shouldn't be grouping together plans, but it should also be noted that $20/month also gets you 5TB of storage on Drive. If you paid for Dropbox, you get 3TB for $18/month (without any AI model usage).
If we go back only 45 days, Opus 5.5 was not even out yet. So for more than a third of the current SOTA model's existence, Google's response to it was under its own embargo. When it was finally out, paying users get 3-4 prompts a day, which means Argon will be experienced by a tiny minority of people, another unforced error on Google's side.
By the time Carbon comes out, I fully expect Anthropic, ChatGPT and/or a Chinese lab to have its answer ready to ship in less time than it took Google to approve a release.
Since you're a googler, can you trigger some internal process to improve the connection between google services?
Gemini cannot create "my maps". I like creating a map of stuff for travel and now I ask it to create an importable file (which it doesn't always get right) but it should honestly do that itself.
And I don't mean that from within the Gemini app, I'd like a shareable list of places, which you can do via a very outdated maps UI
This would be true if Gemini didn't already have some surface level access with Google maps. They just stopped short of supporting the feature I cared about
You also get YouTube premium and home premium on the Google ultra plan. I didn’t realize how nice it was not to see ads on YouTube, and the home summary of who walked next to my home each day is bizarre. The grouping doesn’t make sense at all, since I would have bought it anyways, but it’s a nice bonus.
It's annoying that there is no way to just pay the marginal difference between Premium Lite and Premium for AI Pro subscribers. I have a bunch of interacting subsidies from Google Fi, Pixel phone purchases, and Google One AI Pro, but it's hard to keep them straight.
I have 160G of screen captures from Tumblr from my then angsty teenage kid. I think my spouse takes a few Gig of photos for each outing we go on. Several people dress for fancy thing via a process involving many clothes changes and even more selfies.
It’s hard to get interested in Google products because they throw them out all the time. If it’s not a step change in functionality why even go near their baggage.
That's the problem for me. Google can have the world's best model; it doesn't matter to me, because I effectively can't use it.
Both Opus 5.5 and Sol 6.1 are extremely good and provide a lot of inference on their $200 plans, and then there are the Chinese models - DeepSeek 4.1 Flash, MiMo, and StepFun all offer a lot of value per cost.
At the moment, I don't see a niche where it makes sense to use Google's models, regardless of how good they are.
Google has a $99 plan that I’m using right now and I found flash to be reasonable for dev, but I spent my now past career at Google working around Gemini Flash limitations (when it was much worse than it is now) and leveraging subagents in antigravity. I want to dip my toes into anthropic or openAI models, but getting setup seems to require either another plan or expensive tokens, and I’ve found DeepSeek to be a reasonable back up/second model that I’m not eager to just go in and see what I can do over there.
In my experience, (using 3.8 flash) the Google models do a fairly good job at summary notes and simple documentation (do this, do that). It doesn't seem to be as bad as Claude for producing headache-inducing text.
Good to know. I figured I'd start with Gemini because if I used Claude, I'd probably get addicted to the results. Then I would have to commission a velvet painting of me dressed in Dickensian beggar clothes with big eyes and a quivering lower lip, holding my hands out, asking for tokens.
> For starters, the limits are dreadfully low. Five messages on medium is already enough to trigger a 4 hour cooldown.
Is that chat messages that involve tool calls (web search, file reads, etc) or actually just messages?
When I'm on the $20/month OpenAI plan, I can get about 3-5 turns out of Astra before the 5 hour limit is up.
When Fable was on the $20/month Anthropic plan, I remember getting about the same.
3-4 turns out of the $20/month plan would seem about par with other providers for their best model, but really Carbon should occupy that slot, not Argon.
If Argon's limits are that low, it wouldn't make sense to use that plan for anything when models of similar intelligence are available with much higher limits on other $20 plans.
In my experience with antigravity-cli for coding Argon gets the task finished much faster than 3.8 Flash and with a higher quality output. The quotas on Pro are about 2 hours of usage per week for Argon. Its a really nice model to use. Only thing that is annoying is that it still struggles with tool use on the gemini website.
Same. I subscribed to Gemini pro and requested a refund literally the same day. Asked it to put a technical image on a presentation (gave it the image) and it kept generating a different image of a peaceful landscape... Yeah, very pleasant but totally unrelated.
Agreed on the limits, they're quite frustrating. I've still been preferring Argon to Opus 5.5 though, because even though Argon gets details wrong, I find that it's much more creative and willing to explore ideas. In my experience Opus tends to be rather condescending and won't make room for less orthodox ideas to develop. For what I'm doing, I'd much rather have good ideas that I implement mostly by hand rather than regressing to the mean and have the LLM write the code.
> because nobody will have access to Carbon for a few months anyways
From Google saying that Argon would be 'available soon' (as they did just now for Carbon), it took them 13 days to release it. So... unclear that Carbon won't see the same trajectory.
Note that Argon was rolled out for IDEs for people on paid AI Pro on Oct 15/Oct 16 (a couple days after Ultra users), but of course, usage limits are extremely low
It's kind of an odd release. Yes, it's technically the top intelligence model by like one point above Claude according to Artificial Analysis, but it's like barely cheaper?
It looks like the best sort of bang for your buck is still GPT Sol and GPT Astra. Claude and Gemini have just been going for that number one intelligence spot by sacrificing a lot of efficiency.
It does sort of show their hands a bit that their larger goals might be outside of just consumer sales but also seeking hyper intelligence that could bring value in new ways of discovery (if it ends up being a legit outlet for such things consistently)
Gemini models always suffered from this. They were almost always best in terms of intelligence, but one of the worst to use in practice, because they have a lot of failure modes on simple tasks that simply give factually incorrect answers, or completely break and output gibberish.
Yeah that "Pro" usage is a joke! I wish google paid more attention to what the people want! it's a shame I loved Gemini 2.5 and 3 when they first came out they were super amazing!
>Lastly it's the only "SOTA" model in a long time that has actively hallucinated details. I don't know if it's because the Gemini harness is not asking it to verify information, but I've had some real Deepseek V3-like lies in responses.
I don't know what it is about Google models, but they are the worst at hallucinations. I don't understand how they can so consistently be this bad.
> Lastly it's the only "SOTA" model in a long time that has actively hallucinated details.
Yesterday I used Google AI-Search (so Gemini Flash 3.5?) for opinions on a traffic accident and it continously constructed law-interpretations and then referenced the wrong paragraphs. It talked about rules on road-crossings and cited paragraphs pertaining to alcohol intoxication as sources.
I have experienced all of this too. I also find Argon still seems to have the sycophancy problem that some of the other labs have stamped out to a large degree.
same. I have Pro plan so I was pretty excited when Argon was released. But it was such a disappointment.
the problem with gemini is the whole app itself, not the model.
when you use other chat app, like chatgpt or deepseek chat, the app will use web search for everything to verify. and if it can not find anything useful, or the current fact is disputed, it will try to call web search tool again and again until the consensus.
meanwhile gemini will refuse to do web search if possible, and only do it a single time if demanded. worse, it uses the same result from google search, so the top ranking answers are almost all SEO garbage.
since then I just stopped asking gemini again. and I don't think the newer model will change the situation.
It is expensive as hell. That said, on my weird personal benchmark (https://botbowlbench.com/) it reliably beats Opus 5.5, which surprised me a fair bit.
I am having a very positive experience with Argon: since it was launched I accessed it through Ultragravity and connected via a Google Cloud account. For one month I have been trying to move the needle in SHA256 cryptoanalysis with Claude to not avail and Argon moved "two needles".
Since then I tried it for experimenting on diverse fields. For example, this week there was a discussion here in HN about the non-continuation of Python funding on the JIT, so I "played" again to removing the GIL and moving to the JIT in Python, it did it. I played with many other things, even the Riemann Hypothesis (it just removed their rational parameter rounding only moves 0.875 towards 0.87495702, nothing significant though).
Many other relatively great things were done this week in parallel. I have some budget for research.
It's terrible. I tried it via API in OpenCode. It's so slow, and the lack of reasoning output makes it feel even slower. And I also noticed the lies. I gave it a high-res image of a map to classify labelled locations, and it started going on about how smart it was for the map creator to use squares as the symbol for one of the categories of labelled places, and gave me a list of all the places with squares along with their category, but the map has no squares. It's circles, triangles, and stars.
Trip planning is the most often used use-case for agents/bots I see in marketing. I'm lucky to travel once a year, but if I did have a trip, I wouldn't feel comfortable letting an LLM take the wheel. Too many variables for wrong flight or room, closed locations in itinerary, etc.
Now, critical server infrastructure? Sure, have at it Claude, make no mistakes!
I would literally never let an AI plan my trip. I do have very specific interests though and it's nice to have an overview of suggestions of places to go.
Plus, it's great at researching the history of places you're visiting, so you can find cultural experiences a bit easier.
I don't travel often, but I do live in Europe where it's easy to visit a new city or country in the weekend. This makes this functionality a quite common use for me.
I mean this specifically for general tasks, like personal research. Small stuff. I was asking for espresso machine recommendations and it literally came up with a non-existent machine that supposedly met all my requirements. I've never seen that with ChatGPT, Claude or Kimi. It's quite surprising how it did that _after_ doing a Google search.
Gemini has been like that forever. Even when given the full manual of my bread baking machine, it still gave me the wrong program number; when I was lost at the airport, it gave me completely false directions, etc. Neither ChatGPT nor Claude has let me down like that in recent years.
The only thing Gemini has over those is integration with my Google account, which is genuinely useful. But I reckon Google could release open weights for their whole product line without affecting market share much.
It has already become irrelevant to anyone paying attention. Raw scores at published benchmarks mean next to nothing in modern use. They are either gamed to hell or unsolved to the point where differences can be explained by randomness. Speed, cost per token, tokens per task and hallucination levels are the useful axes you can compare along these days if you're not doing scientific frontier level research.
They probably prefer being used by the deep-pocketed organizations that have access to Gemini 4 Carbon even if it means losing the entire rest of the market.
Also, their tooling still is complete shit. I'll not be forced to use Antigravity thing even if it is available and clearly better than Opus or Astra. Their subscription is useless. API can be used with anything you got but that's gonna cost.
You should be running your own benchmarks often given the nature of these products. Releasing these models whenever with whatever corresponding benchmarks provides you no assurance that things will stay the same.
Having a great time with an ice cream brand at a store is no assurance against hollowing out the brand tomorrow.
This is the only use case I've found it superior to ChatGPT or Claude for - helping me understand higher level math. It's slightly easier to understand than the other models
Having an AI do a quick check of research studies is about as effective as just trusting a Vogue magazine article. You are just fooling yourself into thinking there is something useful there, there is not.
Well OP is asking about a specific paper and you are asking for it to be disproven, different requests and the LLM is just doing what you want.
Classic "you're holding it wrong" (maybe enhanced by exercise/nutrition science being what it is with a study and expert representing every side of an argument)
I'd personally probably do it like
> Someone told me about a recent paper which showed that sleep deprivation had surprisingly little impact on muscle hypertrophy, and that mechanical stimulus seemed to be the dominant factor.
> Can you comprehensively fact-check this, and let me know what the general scientific consensus on the topic is?
Even then you can't be certain of course. Even reading the cited studies you can't be certain unless you know enough to interpret them. But you'll probably be a bit more informed than you would be if you were just going off news article titles you see on social media like the average person.
Not OP, but I asked Gemini 3.8 Flash this question (via self-hosted LibreChat) and got an excellent answer:
I'm trying to figure out how heat dissipation works. Specifically, I have a block of aluminum with extruded fins (as a heat sink) and I'm trying to understand how many 30-watt LEDs I can place on it. I have a fan as well, and I'd like to understand how fast I need to run it at different power levels (the LEDs are dimmable). I can gather measurements empirically (which I'll do), but I also want to understand the basic mathematical basis I should use for estimating these values.
How can you determine if it's explanation is good if you don't already know the answer? You are not qualified to know if it's answer is correct, by hypothesis.
Just talk to 3.8 flash about whatever (caveats: ancient knowledge cutoff of jan 2025, small parameter model so if niche '90s korean animated series is your jam, it's gonna be rough). The model is a real joy to talk to, I even will feed it SOTA model output to clean up and clarify for me.
Its the first model ever that I have been worried about a successor losing the "feel" that it has.
Gemini is the best at convincing you that you understand scientific concepts. Unless you have applied that knowledge in other contexts and been given feedback that your understanding is correct you have no idea whether it is good at actually explaining them.
That's generally true for all LLMs, but as with any tool - you can actually use it in a way that produces actual, verifiable results. Obviously it requires a lot of work that 99% of people won't put into what is essentially seen as a 'magic black box', but I do agree with GP that Gemini is quite useful in actually breaking things down in an understandable way, compared to most other models.
My experience with it is mostly using it against an actual, academic source, where I can easily tell if it starts making things up. I wouldn't rely on it for anything that I can't confirm myself.
I had a research question in my area of expertise. Tried it with ChatGPT, Claude, and Gemini. I think all three could have solved the question, but ChatGPT was an unreadable mess with too many sections and headings, Claude got stuck thinking and never produced an output. Gemini gave succinct explanation that I think was similar to ChatGPT conceptually but that I was able to follow and verify. The answer was actually fairly trivial to understand; I just hadn't gone at the problem from that perspective.
I know it is common for people to try AI in domains they aren't experts in, but it is also common for people to use AI in domains they are experts in.
Gemini team has a long history of releasing great benchmark charts and disappearing into the ether. There has never been a moment where I have preferred them to Claude/Codex. When I get access I eval them on my home assistant set, a small coder benchmark, etc. and they are too expensive, and too low quality.
The benefit for Google is their chat interface has privileged access to their data (maps, etc.) but their model itself is low quality and, if I'm being honest, has the promotion-oriented measurable-output bias that FAANGs tend to have. The later Llamas had the same problem.
About to say the same... even with the "Gemini AI Plus" plan that should be renamed to "Gemini Flash" because you are only allowed to use "the mangled model" using it for something as basic as taking notes and saving to google keep or even google docs, heck yeah a simple "add this to my calendar" completly fails in gemini... Gemini sucks for everything even its own google apps... Claude and Codex even DeepSeek and Kimi are milestones better than anything Google has launched.
It is unfortunate that having such an amazing engineers that are capable to solve any leetcode exercise in 5 minutes on the spot with a white board, they aren't able to even deliver something useful on the AI space.
As of their latest update (AA's), I find their index hard to trust, and a bit shady. 60% of the benchmarks contributing to the AA index are now private, nobody can replicate them. And they explicitly say that they offer benchmark consulting to labs. This is a serious conflict of interest.
The latest update knocked down only openweights models from the leaderboard.
Carbon scoring below Opus 5.5 on coding highlights how magnificent Opus 5.5 is as a coding agent. With just a $20/month sub, it performs a ton of work at a very high quality.
The way benchmarks work is very very different from how you work with an agent.
The benchmarks often have ambiguous instructions, and there is no interaction with the agent. So a docker image is started, the agent is called in non-interactive mode with a prompt saying to read the instructions and finish the task. The instructions are often somewhat ambiguous and the agent has no way to ask for clarification. If it picks the wrong option out of many seemingly equally likely options that test will be mark as failed.
The benchmarks of a model that hasn't been released yet are definitely suspicious.
I also think Google should stop "releasing" models that aren't actually available. For their own sake, because by the time Gemini 4 Carbon becomes available, nobody will notice. There won't be a new press release or much excitement around it.
There's an issue with this index where many models are listed in the model chooser, but missing from the index. E.g. Qwen 3.8 Flash Next is listed but missing, as are older Claude (4.6, 4.5, etc) or OpenAI models. This makes it a bit tricky to see the improvements over time.
They removed Qwen Flash Next and GLM Flash from their benchmarks, as of two days ago. They're not publishing any figures at all for them (they show as "N/A").
edit to add: It's very disturbing, because these open models (GLM particularly) were among the best of all LLM's on their benchmarks. On the intelligence/cost frontier, specifically. Archive.org remembers that GLM Flash was in their green quadrant just a few days ago, next to Haiku:
well if there was any doubt about why openweight models were ranking lower on every iteration of the benchmarks... this should remove all doubt. to just flat out remove is very overt
You know what. When Gemini 3.1 Pro Preview came out it was better than Opus at the time. It was my main model for 3 weeks.
Until it got lobotomised. Then it lost most of it's capability.
Similar situation happened to Anthropic's Fable. It was an amazing model until Opus 5.5 came out.
So now, I'm not even looking at a "new" model until it is a month or two old. By then we will know if they will destroy ability to make it cheaper to host.
I remember when Opus 5.5 came out, we had a discussion about whether it was nerfed a week later [1]. It's not possible to do so with Gemini 4 models since none has been made available the day it is announced.
Without us being able to access Carbon when it is announced, there's no way to know the Carbon we get a few weeks later is the same thing as what has been benchmarked here. I think benchmarks should verify what is benchmarked is exactly the same as what's been released.
I tried Argon for a large historical HTR transcription project recently, since I heard it was really good at this. It was really good, but like all top US models I've used it was
* Definitively very good at domain specific knowledge. It can read sloppy 19th century danish gothic handwriting with faded ink just fine. Not perfectly, but probably better than I could have done with a day of struggle for every page.
* Expensive.
* Bad at actually doing what I said. I had told it in the prompt to be careful not to miss text, but I hadn't explicitly told it that some images may have text in double-column layout. So on those images - over half of my 300-page document - it produced an excellent transcription of the first column, and then considered itself done. Want to try again with a more careful prompt? That'll be another $20.
For ocr, you have to practice, practice, practice with your prompts on small sample sizes before one-shotting a book. I create a json schema for the output that describes the structure in detail (using ai from my prose descriptions of structure of course). I have specific tough pages I use on openrouter to test every new model I am interested in. Gemini has and does absolutely dominate for this. But flash can be better because it doesn't overthink things.
I did do test images, but apparently not enough. It's not 100% of images it drops the second column on either. On many it gets both, on some it only gets the second column.
I also found that it ignores other instructions sporadically ("Don't use letter variants or ligatures in the output, use the normal Danish/Norwegian alphabet when at all possible", still a few Unicode tall s'es in the output.)
It's good, but it's so frustrating since "worse" models manage to follow such instructions much better.
> They are like almost like malicious compliance geniuses lol.
They're like a monkey's paw or evil genie I think.
You'll get exactly what you ask for, but lots of weirdness you didn't expect.
To be fair to the model makers, deciding what to do with an under-specified prompt is hard, particularly in an RL environment where you can't just ask the user.
For stuff I want to run for a while, I'm defaulting to telling it to ask me the questions upfront to reduce the incidence of these kinds of problems.
It kind of doesn’t make sense. Modern google is typically kind of bad at building products and very good at getting them in customer hands. This is the opposite of that.
It does make sense, when you consider that Google had its earnings report yesterday, and missed its targets. $GOOG is already down 13%. It's an announcement purely for their investors.
Gemini 2.5 flash and 3 flash were highly underrated models that performed way better than the peers and we used the hell out of them.
Since then none of the Google models have competed well with the others at their price points. Long context performance drop, instruction following issues, unnecessarily complex and sloppish outputs.
Argon is no different, it feels just an iteration better than 3.8 flash. I don't trust the AA ranking yet.
I agree. Flash 3.8 is not good and actually quite expensive, given its tendency to overthink everything. We've been transitioning lots of workloads that were formerly on Flash to Haiku 5.5. Great model.
Even with gemini 2.5 or so i remember the models used to just simply hang in production and randomly used to collate tokes and give in the output stream vs giving one by one which was weird.
Also the guardrails were implemented in a very bad manner. By default the models used to raise a default guardrail error on basic queries.
I want to talk about my experience with Argon, noting that I used it via OpenRouter on a Raspberry Pi, spending around $15 in credit.
My experience is that Google is aiming to create a generalist model – but a truly generalist one!
In fact, I find it particularly accurate in general terms and even in long conversations, maintaining excellent consistency and a wealth of information in response to complex questions.
I’ve tested it both as a coding agent – though, precisely because of what I mentioned above about its ‘generalism’, I don’t think it measures up to Fable 5.1 or Astra – and I’ve really appreciated its qualities when supervising my agents’ activities. It was able to keep track of them, suggest alternatives, and monitor the entire lengthy process in parallel without ever straying too far from the request or losing the thread – as often happens with Claude and OpenAI (you only need to test it in Claude Code itself, in both Ultracode and the agent team mode – albeit experimental – to realise how, once the task gets underway, both of them lose their way).
Just so everyone is aware of the "coincidences" that happen with this index:
- Two days ago AA announced their v5 Index which added a few new benchmarks and tweaked a few things.
- Today the newest model announced from a big lab tops the charts.
I'm not saying that AA is definitely taking money from the big labs and changing the benchmark to make the labs' new models look better, but it kind of looks that way. This pattern of behavior has happened before, where the index gets tweaked right around when a new model comes out, and the new model is on top. AA does not, AFAIK, provide a way to see how the new models would have scored under the older index versions. You could probably calculate it yourself, but I'm not going to do that much legwork just for an HN comment.
It's interesting because they publish the index methodology, so you can't really accuse them of not being transparent, but I don't really believe that the choices to update the index were completely independent of the model they happened to be evaluating at the time.
Related: The DXOMark camera scoring method also had similar behavior, where they would release new testing methodology right before flagship smartphones were released. Again, they are technically being transparent about the process, but it kind of smells like the "independent" testers may not be as independent as the average user would hope.
I had a look at the changes and the biggest impact seems to be from adding Terminal-Bench-Science 0.1 and having it contribute 15% to the overall score.
I have no doubt that these are very relevant task for people in those respective fields, but I suspect they have next to no relevance to some 99.9% of model users. Hence why especially smaller labs with smaller models didn't really train for such use cases and you can see it.
Personally, it's not something that matters at all to me, as I don't work in cheminformatics or bioinformatics.
This is also arguably more of a test of a model's knowledge base than reasoning ability.
But most importantly, there is also the question to which extent the aggregate score then remains a meaningful result when you mix up too many too disparate benchmarks.
The coding agents subindex, in contrast, looks at least superficially plausible [1].
True, but coding is where something like 80%-90% of their revenue comes from; few other use cases are as token-hungry, and none of those are nearly as widespread. Many users will of course use the models just as chatbots, but they don't generate much in terms of revenue. And that is ultimately what determines where labs focus their resources.
Yeah, it is very suspicious. Each time they update their ranking algorithm, the closed models from US labs jump up and any previously high ranking open model jumps down. It happened with MiMo, Kimi, GLM. All were right there amongst the very top, and each time that happened, AA followed quickly with an index update that shunted them dramatically lower.
Check out vals.ai for an alternative. Not sure how independent they really are either, but it's at least good for a cross reference.
This could be another point informing your prior as to how much benchmaxxing and distilling of closed models is being performed by the open weights publishers as well.
I don't understand this conspiracy/suspicion vibe around AI benchmarks -- the best benchmarks get maxed quickly in today's world, especially if the datasets are open. We're in a situation where the AA's of the world show 0-1 point difference between models, so by definition they are NOT good discriminators. These tests NEED improvement.
Do they work with frontier labs? Probably. They should work with them at least a little so that we can get an independent read on the model quality. Do they inject new tests that specifically favor one particular lab? Clearly no, as different labs have moved up over time. Do they inject new tests that help to provide some separation on scoring? God, I hope so.
Nota bene, another change they announced was dramatically increasing the amount of held back test data; this would be a simple explanation for seeing open weights models, all of which are known to distill from frontier labs and benchmarks, drop in the rankings.
We’re literally talking about the same thing, but ascribing different motivations.
I do think the test a new model and realize their index is broken- the new model isn’t on top, so they change the weights to make sure it’s on top.
The difference is that you seem to think they are doing so out of some innocent feeling that the new model is actually better, while I think they are doing so because they were paid to do so. I think my interpretation is more likely, but that’s just me.
And how do they realize their current index is broken? They either have some recency bias, or indeed favor the big American labs (more likely, since a new Chinese model topping the charts triggers a shakeup).
It's odd because I wonder if anybody actually cares about such scores. It's not like millions of end users look at the score and use it to make a decision what to buy. Subscriptions can easiliy be canceled and people anyway use the models they "feel" work best for them. Hardly anybody I know uses an even remotely scientific method to evaluate model performance. It's mostly vibes with a little bit of (mostly flawed) A/B testing sprinkled in.
I think part of it is to justify capex. "Yeah, we spent x billions on this, but look, we have the best models, we're beating OpenAI and Anthropic, we're winning, it's all worth it!"
Only because it aggregates all the other benchmarks. It still feels off. It felt off before the last two re-rankings, and it still feels off. Probably because the benchmarks its based on aren’t measuring the right things.
For specific benchmarks, I’ve downloaded and used basically every model out there under 35b parameters - especially the tiny models: ling-3.0-tiny, spark-x2.6-4b, nanbeige, etc. etc. some of those tiny models are ranked VERY high on benchmarks like deepswe or terminalbench but are obviously incapable when you try to use them for swe or terminal based tasks.
If that’s benchmaxxing or a bad benchmark, idk, but it’s made me not trust the benchmarks much. If it benchmarks poorly, it’s probably bad. If it benchmarks well, it might be good.
My paranoid impression was more that the updating and addition of tests — obviously essential in this game — was delayed to provide advertisement for Kimi K3.1. The same happened for K3 — and this is why it was famous — but its position had collapsed by the time of 3.1
Since 3 Pro we already know that Google's model are very good in benchmarks, but when you start programming with them... not really good and each model has its quirks that make me continue the project with something else, once I fixed gemini's bugs.
The time between public announcement -> public release seems to be a tacit admission that Google is still a bit behind the frontier labs. Google is justifying the delay by pointing to cyber risk and a closed-access period reserved for analysis of safeguards. I'm dubious of this justification. But if it is true, I'm pretty sure the other labs are doing similar sorts of things in private. I think most of us prefer it that way—if Anthropic public announced Fable 5.5 two weeks ago with no availability that would add to my annoyance. Maybe they learned their lesson with Mythos, or maybe it's just that Google also does not have an IPO looming. Who knows?
It's performative stuff to try and appear like they care about alignment and safety.
"Hey we are launching this thing but we've got to let the safety folks see it first".
So they look like they care about alignment etc.
To be honest there is a lot of rage on here about it being announced before being able to access it. I don't understand why people are getting so upset about this - like no one gets into a rage because they see a movie trailer before it is released?
This is just a hunch since I have not used Argon very much, and I have not used it for coding (deepseek 4.1 flash 1017 with dsh, and qwen 3.8 max are so good for coding tasks that local models have difficulty with).
My hunch is that Google will end up being best for non coding tasks. Google can concede coding to other vendors who pay Google for infra.
Artificial Analysis is not trustworthy anymore. Every time an open-weight model, especially Chinese, looks to be nearing the top, AA updates the index to (ostensibly) specifically push closed models higher. Just recently with v5, OpenAI models were pushed up.
> I think what's happening is Chines models genuinely try to benchmax on it and hence they suffer when a new version comes out.
I read somewhere on Substack that a lot of this was driven by the inclusion of more knowledge work benchmarks, where the US labs are stronger, versus coding which is where most Chinese models appear to put their energies for now.
Or it could be that the benchmaxxed models are getting pushed down by the introduction of unsaturated benchmarks. Muse Wonder also went down 52 -> 43 after the update.
It's not that hard to create your own uncontaminated eval for the problems you care about, and compare.
Here's mine for coding in Rust, and I definitely see the same pattern of "most chinese models do much worse than on public evals that have been around long enough to train on" that AA v5 surfaces: https://slopcop.com/power-ranking
People always love to think the AI race is a duopoly (OpenAI vs. Anthropic), but from what I have experienced, Gemini was always holding on its own, and I find myself using it more than Claude.
True, I use LLMs as glorified search engines to unearth material to read myself, but Gemini is much more useful if you decide to learn yourself. It also offers interesting expansions to your questions.
While I don't offload my work to LLMs (e.g.: Write this project, I'll be watching TV), I find them useful for closing my knowledge gap by collating information, and Google's Gemini is good at that.
I have Gemini because of my Google One membership. I already was on their 2TB plan and it gave me Pro level, IIRC, and it was generous. Now I'm on Ultra mostly because of Google Drive storage I get, but Gemini also gets a nice boost, as well.
I'm on $20 Claude plan, and it's also ample for my needs, for now.
The $20 plan gives you ~120 minutes of high thinking time from Argon weekly. Thinking time isn't an exact metric, context size matters a lot, but I have been using it on the $20 and still have 25% left with ~85 minutes of thinking time on high for a project I have been working on.
I think this is because of capacity issues. Higher paying per token users probably get preferential capacity allocations to those on monthly plans. The allowance for what you get in the monthly plan increases as the models age and the likely capacity problems improve.
They have an insane amount of resources but are so bloated, have a ton of bureaucracy, and have lost a lot of their top talent. Looks like they have figured things out now though.
They have top of the line research, I mean, they invented the thing, and they are still doing stuff that goes beyond making a bigger LLM, as it seems to be the only thing the others are doing, research-wise.
They also have hardware capabilities no one else has, they invented the TPUs, and have lots of them.
They also have the brand recognition, ecosystem, user base and lots of money.
So why are they lagging behind on the actual products, and by lagging behind? I mean they are always in the position where they are just behind the frontier, but still in the game.
It seems almost deliberate. Maybe they are playing the long game, waiting for the bubble to burst. Maybe they don't want to compete against the more profitable parts of the company. Or maybe their researchers are not interested in making an actual product unless being forced to.
Not everything is writing code and search. My uncle works in mortgages and the (very large) firm he works for is entirely Gemini. The way he talks about it is with wonderment in his eyes while he goes into detail on how it went through multiple documents and found unaccounted for cases. I think Google is doing fine in these kinds of industries.
I wouldn't even bet that Google is better than OpenAI and Anthropic for these categories, but they seem to be well used outside of our direct space.
Antigravity harness seems much worse than Codex. Argon seems very knowledgeable but very sloppy. If I force it to self review it will catch its own errors and laziness.
At this point frontier models are good enough to be a pair programmer. But only OAI and Anthropic offer good limits. Maybe they're burning money but if the Google limits are the future of usage style accounts then we all have to get used to API
I haven’t tried Gemini for coding yet. I did consider buying their subscription, but based on what I’ve seen from other forums, Gemini has never been a frontier in coding, even with good “benchmaxing”.
Google is killing its own product. Gemini Argon is much better at writing and to some extend coding, but my teammates keep using deepseek because it is available and don't have to check vertex ai, ai studio, gemini, antigravity and all those shit...
why do i not see as much traction and stirring up on the internet for gemini models as much as claude or gpt even though on bench's they are very much THE frontier. is the reason very limited access early or or just not enough publicity or hype as the other labs do?
I mean, that's the main reason. How would you expect to see the kind of hype that Claude or GPT can generate when nobody can use it? The hype of those models are primarily generated from people using them to build impressive things and share those results. Can't do that with an unreleased model, right?
Beyond that, it seems a lot of people have learned that these agent leaderboards really don't tell the whole story. They're a good litmus test, per se, and there's plenty of hype being generated around these scores as-is. But anybody using AI in practice knows that benchmarks can only say so much, and one model scoring better than another doesn't necessarily imply that that model is actually better in the ways those people care about. You can see that in this thread as people are pointing out that even these benchmarks seem to suggest that Opus 5.5 is still leading in coding (which is what a lot of people, especially on HN, ultimately care about).
>just not enough publicity or hype as the other labs do?
Well, it's effectively both. Google is certainly not that pressed to generate as much hype as these other companies. They can afford to take their time. It's also worth noting that Google is definitely targeting different consumers, investors, etc., so their strategy will definitely be different.
For what it's worth, I've been using Argon since it's been available in Cursor and I really like it (at least at its current price). I think Google really did close the gap a meaningful amount (maybe not all the way, but enough to prove they're still in the game). Carbon being significantly better while being the same cost is pretty exciting to me. But until I can use it, I don't really care about it.
I use it purely for finding stuff in text documents and OCR, at which it excels, but I've found that for any kind of analysis it's some bewildering mixture of smug, defensive, paranoid, cheerful, and wrong about everything.
I feel like the headline here is "the Artificial Analysis index is becoming an increasingly poor measure of capability, having previously been quite good".
Google has to get it's shit together. All the stats and charts doesn't mean squat because when I try it now on a provider all I get is trash :(. Makes me question these metrics entirely and really makes me question the authority that google used to perceivably have.
"We're taking a deliberate approach to rollout and are actively engaged in the U.S. government’s voluntary process for pre-release model testing. Starting this week, we will make Carbon available to a subset of cyber defenders and will gradually phase access to additional trusted partners through our Fairwind Program. We’re continuing to gather feedback and iterate on guardrails before making Carbon available to developers, businesses, and consumers via our product surfaces as soon as possible."
By the time this model is available for public use I'd imagine it won't be #1 anymore, which is pretty sad. Even when Google gets back on the ball they seem to find a way to fuck it up.
These benchmarks and rankings seem more noise than signal these days. They're uniformly divorced from sincerely-attested real world experience. And the real world experience isn't driving any consensus either.
So here's a meta-benchmark: at what point does the act of choosing an LLM product for a reasoning task become less reliable (in the sense of "predictive of future success") than hiring a human being for that same task? Have we already cross that threshold? I think we might have.
And once we do... what happens then? Will the truly elite models be hidden in a sea of mediocrities simply because the goofball "managers" doing the hiring are too dumb to recognize their genius?
I don’t care and in my opinion AA does a disservice to their users by allowing such blatant manipulation. What would happen if every AI company were to include their unreleased models like this? Put the unreleased models on a separate page and leave the headline benchmark rankings for models that you can actually use.
It's unobtanium so nobody should care about it. Any frontier AI company can nowadays create models with ridiculous resource and token usage that will get extremely high scores, but they're of no use when you can't use them.
I take this opportunity to give feedback on Argon. The limitations are a problem. I have a Pro subscription and Argon already stops working after 3-4 prompts with long delays afterwards. It's virtually useless right now even when you pay monthly for it. It is better for some tasks than 3.1 Pro, though. Not much in my experience but it can sometimes fix errors that 3.1 has problems with. I never use Flash models since they often give abysmal results.
A secret model I trained last afternoon is the actual #1 on benchmarks.
No, you can't use it. You can't even see it, and I won't provide you any information. But you must trust me that I am the genius who made the best model in the whole world.
Spare me the celebration. Artificial Analysis should not be granting any company the favor of a #1 ranking for a model that isn't released to the public.
Google's model release is pure PR puffery. It's no coincidence that it was released on the day of its earnings call.
I'll use this opportunity while it's still early to share my experience with Argon.
I got it via the "Pro" plan, exclusively for personal task usage. So no agentic programming.
For starters, the limits are dreadfully low. Five messages on medium is already enough to trigger a 4 hour cooldown. Secondly, the best part is the integration with Google search and Maps. Makes it seamless for trip planning. Lastly it's the only "SOTA" model in a long time that has actively hallucinated details. I don't know if it's because the Gemini harness is not asking it to verify information, but I've had some real Deepseek V3-like lies in responses.
Either way, I find it difficult to get invested in Gemini because nobody will have access to Carbon for a few months anyways.
I similarly had Argon respond with a vast hallucination. It surprised me because it wasn't a little wrong on the details but a fully false verbose response that completely fabricated fabricated everything. I'd recently had Claude Code help me create transcripts for some audio notes from my mother in law and decided to see how some other models would handle the same task sans harness. Argon made up a long conversation from a police sting operation with under cover operatives seemingly just because it read the file names and identified that they would have come from a recording device.
Honestly the fiction writing was pretty impressive!
Suddenly very curious what the llm roleplay people think about this one lol. Fortunately I've already learned to stifle my curiosity on that specific subject.
It seems a strangely underdiscussed topic on HN.
It goes way back, I work in AI/Ml (taken over by transformers/LLMs) space since early days of this "AI Summer". We get some nice grad school students interning from top universities. I met one during a trip to the office, from top notch school, introduced me to Triton programming when it first came out. It was early days of LLaMA just coming, when The Bloke was regularly dropping finetunes, I was joking asking him what the hacker stack was for "AI waifus" that the cool kids were using. Lo and behold he drops the name of the SillyTavern project. Which later was really educational in setting up local inference servers and in other ways, may have had to, ahem, delete some browser history.
I will try here: https://myownlore.com :)
But my experience is that the agent matters a bit more than the model. It is fun on ChatGPT because of its recent (?) bias to always be creating images for conversations around fiction.
what are some of the vagaries of how that use case depends on the agent aka harness? i’ve only been focused on sort of how to make models reliable for my math / cs / engineering and arts and crafts work loads, but RP is an important category for some folks. like are the fails of the various models different from more technical yet creative work loads?
eg im working on rigorizing a proof that a fun algorithm of mine is not only the first provably efficient algorithm for a particular problem, but its also the fastest in practice. ive found that for the sort of stuff i need to assume the model will get confused in the out of training distribution stuff from my personal library of ideas, so they will at best be a confused secretary intern mashup
Yes! I was having Argon summarize some project updates and it... just made up entire paragraphs that had no basis in reality. It felt like I was back in GPT3 days.
Will have to give Carbon a try, but if it is anything like what I saw before it was entirely unusable.
If you uploaded the audio via the web site or app and got terrible results I'd bet Argon wasn't the model doing the transcribing. The only way to know for sure is to try again with the API.
Very wise of you to keep MIL receipts. You never know when you might need them...
Yeah... Given that everyone else seems to make models available either immediately or within days of announcement, idk why Google feels it's competitive or acceptable to do these 'available at some point' releases.
Capabilities are also improving so fast that this fomo doesn't really keep people excited. If I can't run it, I'll sorta just click off. And given how they handled the Argon release, I've not even bothered look at this Carbon one. Fully expected it to not be usable yet.
The other labs do the same process, but they don't announce the model until it is about to be released. For some reason Google is announcing at the start of th process.
Didn't Anthropic also announce Mythos at the start of the process, five months before public release?
People who are already using it (both internally and/or select partners) can speak more freely about it if it's announced.
This is especially true around earnings, not necessarily for some sinister reason, other than it's material information that can influence the market.
My guess is Google knows it is (way) behind the and needs to keep drumming up interest. Although at some point it becomes counterproductive (raising "yet another not available model; is it vaporware" vibes).
Deepmind has over a decade of DNA of not releasing - it's a significant cultural difference.
Creating a false sense of scarcity is essential to get people talking about it. I get almost endless inquiries from peers now that I have access to Mythos.
I'm not sure about this. I myself am curious about this model, but not for my own use. I've already written it off as too expensive. The money won't be flowing their way regardless.
argon was announced to coincide with performance review deadline. carbon was announced to coincide with quarterly earnings release.
they don't really care how end-users will receive these models, they're just announcing for internal targets.
> argon was announced to coincide with performance review deadline. carbon was announced to coincide with quarterly earnings release.
Same as it ever was. Google is really completing their IBM-ification. We need to release these models now because year end reviews are about to hit and I need a win.
I don't think you are their target audience, unless you are in their promotions committee or a large holder of their stock.
The target is to make a celebratory announcement.
It's late in life to learn this, but let me be the one to let you in on a little secret: people often say only some of the things they know, they often selectively tell you things while not revealing everything they know.
You are assuming that Carbon is ready for release, deployed to their datacenters, that all scaling and other issues have been worked out, and they are confident in all of the above. Not to mention additional internal constraints like alignment testing or other safety procedures their own policies and guardrails require.
If they are smart, which they clearly have some smart people if they are making the best frontier model, they would account for the above needs in their timeline, and announce the model when they can prove they have it but also before they are ready to release it. This is so they can be at the top of the frontier model lists, boost their brand for both Gemini and Google, get the bump in stock price and investor confidence, steer adoption decisions for people making decisions for what platform they might adopt, etc.
Eh, maybe I'm too much of an AI safety person, but I'd rather the labs be forced to announce when they are working on large SOTA models like they are a clinical trial. That way if it's never announced they have to say something like "training didn't work out" or "it took over the server farm and we had to burn the hard disks".
goog released it just before the earnings call. anthropic and openai don't have these issues.
What's the betting we see another "dramatic" Anthropic release either shortly before the IPO, or at same time they release their Q3 results which were conveniently omitted from their S-1 filing.
hmm, I haven't tried Carbon yet. But I have been using Argon on and off via agent-shell - https://github.com/xenodium/agent-shell and unofficial ACP client.
In my limited usage, it has fixed issues which has not been correctly fixed by sol-6.1 (medium) - https://github.com/auto-gnufied/markdown-modern/commit/a189a...
I am also using it via Google pro tier and did not immediately run into quota issues. But as someone else said in previous discussion of this - Google Pro tier already includes lot of stuff (including Youtube premium etc), the AI plans are just a nice addon.
yeah thats basically the only reason i have the google AI plan YT Premium, increased drive/photos data, & home premium (though i may have that already tbh) Only using flash & argon for toy projects since its included though Argon availablity is very small, similar to the included claude usage avaialble in antigravity
I might consider Google AI Pro if it had a family addition that included YT Premium Family, but alas it does not
Interesting idea .. "Google Prime" including access to all their best stuff.
> I find it difficult to get invested in Gemini because nobody will have access to Carbon for a few months anyways.
I feel like you're being overly dramatic here. Argon was 13 - 15 days.
Argon was announced September 30, made available on October 13 to Ultra everywhere, and Pro in the Gemini App. Pro plans got Argon on October 15 in AGY.
This is the first time they've slow-rolled a model. Who knows how long they'll wait on Carbon.
The Ultra-lite plans ($99/month) get some pretty good quota where you can actually use Argon some more. Maybe Google shouldn't be grouping together plans, but it should also be noted that $20/month also gets you 5TB of storage on Drive. If you paid for Dropbox, you get 3TB for $18/month (without any AI model usage).
(googler, opinions are my own)
If we go back only 45 days, Opus 5.5 was not even out yet. So for more than a third of the current SOTA model's existence, Google's response to it was under its own embargo. When it was finally out, paying users get 3-4 prompts a day, which means Argon will be experienced by a tiny minority of people, another unforced error on Google's side.
By the time Carbon comes out, I fully expect Anthropic, ChatGPT and/or a Chinese lab to have its answer ready to ship in less time than it took Google to approve a release.
Since you're a googler, can you trigger some internal process to improve the connection between google services?
Gemini cannot create "my maps". I like creating a map of stuff for travel and now I ask it to create an importable file (which it doesn't always get right) but it should honestly do that itself.
And I don't mean that from within the Gemini app, I'd like a shareable list of places, which you can do via a very outdated maps UI
>can you trigger some internal process to improve the connection between google services?
You might as well be asking an Adobe employee to bring up improving integration with Canva.
This would be true if Gemini didn't already have some surface level access with Google maps. They just stopped short of supporting the feature I cared about
What do you mean?
Doesn't every Googler have weekly meetings with Tim Google?
You also get YouTube premium and home premium on the Google ultra plan. I didn’t realize how nice it was not to see ads on YouTube, and the home summary of who walked next to my home each day is bizarre. The grouping doesn’t make sense at all, since I would have bought it anyways, but it’s a nice bonus.
It's annoying that there is no way to just pay the marginal difference between Premium Lite and Premium for AI Pro subscribers. I have a bunch of interacting subsidies from Google Fi, Pixel phone purchases, and Google One AI Pro, but it's hard to keep them straight.
Ya, it was kind of a mess for me also, I had to get support to put my accounts in order after I subscribed.
problem is, i don't care about the extra storage at all.
my whole family of 5 people is using in total 160 GB. personally I use about 15 GB
I have 160G of screen captures from Tumblr from my then angsty teenage kid. I think my spouse takes a few Gig of photos for each outing we go on. Several people dress for fancy thing via a process involving many clothes changes and even more selfies.
It’s hard to get interested in Google products because they throw them out all the time. If it’s not a step change in functionality why even go near their baggage.
> For starters, the limits are dreadfully low.
That's the problem for me. Google can have the world's best model; it doesn't matter to me, because I effectively can't use it.
Both Opus 5.5 and Sol 6.1 are extremely good and provide a lot of inference on their $200 plans, and then there are the Chinese models - DeepSeek 4.1 Flash, MiMo, and StepFun all offer a lot of value per cost.
At the moment, I don't see a niche where it makes sense to use Google's models, regardless of how good they are.
Google has a $99 plan that I’m using right now and I found flash to be reasonable for dev, but I spent my now past career at Google working around Gemini Flash limitations (when it was much worse than it is now) and leveraging subagents in antigravity. I want to dip my toes into anthropic or openAI models, but getting setup seems to require either another plan or expensive tokens, and I’ve found DeepSeek to be a reasonable back up/second model that I’m not eager to just go in and see what I can do over there.
In my experience, (using 3.8 flash) the Google models do a fairly good job at summary notes and simple documentation (do this, do that). It doesn't seem to be as bad as Claude for producing headache-inducing text.
3.8 flash is pretty good. it's not expensive, works great for reasoning tasks and it's fast.
for coding .. it works ok but i prefer the openai or anthropic models, much better results.
Good to know. I figured I'd start with Gemini because if I used Claude, I'd probably get addicted to the results. Then I would have to commission a velvet painting of me dressed in Dickensian beggar clothes with big eyes and a quivering lower lip, holding my hands out, asking for tokens.
> For starters, the limits are dreadfully low. Five messages on medium is already enough to trigger a 4 hour cooldown.
Is that chat messages that involve tool calls (web search, file reads, etc) or actually just messages?
When I'm on the $20/month OpenAI plan, I can get about 3-5 turns out of Astra before the 5 hour limit is up.
When Fable was on the $20/month Anthropic plan, I remember getting about the same.
3-4 turns out of the $20/month plan would seem about par with other providers for their best model, but really Carbon should occupy that slot, not Argon.
If Argon's limits are that low, it wouldn't make sense to use that plan for anything when models of similar intelligence are available with much higher limits on other $20 plans.
In my experience with antigravity-cli for coding Argon gets the task finished much faster than 3.8 Flash and with a higher quality output. The quotas on Pro are about 2 hours of usage per week for Argon. Its a really nice model to use. Only thing that is annoying is that it still struggles with tool use on the gemini website.
Same. I subscribed to Gemini pro and requested a refund literally the same day. Asked it to put a technical image on a presentation (gave it the image) and it kept generating a different image of a peaceful landscape... Yeah, very pleasant but totally unrelated.
Agreed on the limits, they're quite frustrating. I've still been preferring Argon to Opus 5.5 though, because even though Argon gets details wrong, I find that it's much more creative and willing to explore ideas. In my experience Opus tends to be rather condescending and won't make room for less orthodox ideas to develop. For what I'm doing, I'd much rather have good ideas that I implement mostly by hand rather than regressing to the mean and have the LLM write the code.
> because nobody will have access to Carbon for a few months anyways
From Google saying that Argon would be 'available soon' (as they did just now for Carbon), it took them 13 days to release it. So... unclear that Carbon won't see the same trajectory.
In 13 days' time there'll likely be better Claude and OpenAI models available.
Note that Argon was rolled out for IDEs for people on paid AI Pro on Oct 15/Oct 16 (a couple days after Ultra users), but of course, usage limits are extremely low
It's kind of an odd release. Yes, it's technically the top intelligence model by like one point above Claude according to Artificial Analysis, but it's like barely cheaper?
It looks like the best sort of bang for your buck is still GPT Sol and GPT Astra. Claude and Gemini have just been going for that number one intelligence spot by sacrificing a lot of efficiency.
It does sort of show their hands a bit that their larger goals might be outside of just consumer sales but also seeking hyper intelligence that could bring value in new ways of discovery (if it ends up being a legit outlet for such things consistently)
> has actively hallucinated details
Gemini models always suffered from this. They were almost always best in terms of intelligence, but one of the worst to use in practice, because they have a lot of failure modes on simple tasks that simply give factually incorrect answers, or completely break and output gibberish.
Yeah that "Pro" usage is a joke! I wish google paid more attention to what the people want! it's a shame I loved Gemini 2.5 and 3 when they first came out they were super amazing!
>Lastly it's the only "SOTA" model in a long time that has actively hallucinated details. I don't know if it's because the Gemini harness is not asking it to verify information, but I've had some real Deepseek V3-like lies in responses.
I don't know what it is about Google models, but they are the worst at hallucinations. I don't understand how they can so consistently be this bad.
> Lastly it's the only "SOTA" model in a long time that has actively hallucinated details.
Yesterday I used Google AI-Search (so Gemini Flash 3.5?) for opinions on a traffic accident and it continously constructed law-interpretations and then referenced the wrong paragraphs. It talked about rules on road-crossings and cited paragraphs pertaining to alcohol intoxication as sources.
I have it on $200 Ultra and it feels like Opus 5.5 (I do not think about limits) rather than gpt-6* models where I constantly hit a wall.
I had the same experience. 4 messages on medium triggers the cooldown. The utility isn't really relevant if you're on the Pro plan.
I have experienced all of this too. I also find Argon still seems to have the sycophancy problem that some of the other labs have stamped out to a large degree.
Yeah, I thought it was just me - it hallucinates more than Flash, Flash-lite and Pro.
I’ve had good experience coding with it. In a bit of an ironical twist I now find Gemini a much better coder and Claude a much better search engine.
same. I have Pro plan so I was pretty excited when Argon was released. But it was such a disappointment.
the problem with gemini is the whole app itself, not the model.
when you use other chat app, like chatgpt or deepseek chat, the app will use web search for everything to verify. and if it can not find anything useful, or the current fact is disputed, it will try to call web search tool again and again until the consensus.
meanwhile gemini will refuse to do web search if possible, and only do it a single time if demanded. worse, it uses the same result from google search, so the top ranking answers are almost all SEO garbage.
since then I just stopped asking gemini again. and I don't think the newer model will change the situation.
It is expensive as hell. That said, on my weird personal benchmark (https://botbowlbench.com/) it reliably beats Opus 5.5, which surprised me a fair bit.
Arena.ai has reviewed it using many tasks against SOTA models.
https://www.youtube.com/watch?v=NO2I3ueVQrk&lc=UgxRXtipA3QbA...
tldr; it's not as good as opus 5.5
I am having a very positive experience with Argon: since it was launched I accessed it through Ultragravity and connected via a Google Cloud account. For one month I have been trying to move the needle in SHA256 cryptoanalysis with Claude to not avail and Argon moved "two needles".
Since then I tried it for experimenting on diverse fields. For example, this week there was a discussion here in HN about the non-continuation of Python funding on the JIT, so I "played" again to removing the GIL and moving to the JIT in Python, it did it. I played with many other things, even the Riemann Hypothesis (it just removed their rational parameter rounding only moves 0.875 towards 0.87495702, nothing significant though).
Many other relatively great things were done this week in parallel. I have some budget for research.
It's terrible. I tried it via API in OpenCode. It's so slow, and the lack of reasoning output makes it feel even slower. And I also noticed the lies. I gave it a high-res image of a map to classify labelled locations, and it started going on about how smart it was for the map creator to use squares as the symbol for one of the categories of labelled places, and gave me a list of all the places with squares along with their category, but the map has no squares. It's circles, triangles, and stars.
> Makes it seamless for trip planning.
Trip planning is the most often used use-case for agents/bots I see in marketing. I'm lucky to travel once a year, but if I did have a trip, I wouldn't feel comfortable letting an LLM take the wheel. Too many variables for wrong flight or room, closed locations in itinerary, etc.
Now, critical server infrastructure? Sure, have at it Claude, make no mistakes!
I would literally never let an AI plan my trip. I do have very specific interests though and it's nice to have an overview of suggestions of places to go.
Plus, it's great at researching the history of places you're visiting, so you can find cultural experiences a bit easier.
I don't travel often, but I do live in Europe where it's easy to visit a new city or country in the weekend. This makes this functionality a quite common use for me.
The best [whatever] is the one that is the most available.
> Lastly it's the only "SOTA" model in a long time that has actively hallucinated details
Could you expand on this? I’ve never seen a frontier model that doesn’t hallucinate details fairly often, what do you mean by “actively” here?
I mean this specifically for general tasks, like personal research. Small stuff. I was asking for espresso machine recommendations and it literally came up with a non-existent machine that supposedly met all my requirements. I've never seen that with ChatGPT, Claude or Kimi. It's quite surprising how it did that _after_ doing a Google search.
Gemini has been like that forever. Even when given the full manual of my bread baking machine, it still gave me the wrong program number; when I was lost at the airport, it gave me completely false directions, etc. Neither ChatGPT nor Claude has let me down like that in recent years.
The only thing Gemini has over those is integration with my Google account, which is genuinely useful. But I reckon Google could release open weights for their whole product line without affecting market share much.
Weird, using AGY I get tons of argon and its on a weekly limit.
Doesn't mean jack without access to it.
I used to defend Google's models months ago, but switched to Luna for less intelligent models.
But with this bs of not making it even public, pushing off release of newer models late, I lost faith in Google.
yeah, IMO AA should be excluding unreleased models from their benches or risk becoming an irrelevant marketing platform that no one uses anymore.
It has already become irrelevant to anyone paying attention. Raw scores at published benchmarks mean next to nothing in modern use. They are either gamed to hell or unsolved to the point where differences can be explained by randomness. Speed, cost per token, tokens per task and hallucination levels are the useful axes you can compare along these days if you're not doing scientific frontier level research.
They probably prefer being used by the deep-pocketed organizations that have access to Gemini 4 Carbon even if it means losing the entire rest of the market.
They've only recently emerged as something people reference and now they're already dying.
Curious what was the source of faith you previously had?
Yeah. It is vaporware at this stage.
Also, their tooling still is complete shit. I'll not be forced to use Antigravity thing even if it is available and clearly better than Opus or Astra. Their subscription is useless. API can be used with anything you got but that's gonna cost.
Not to mention blocking API usage with subs.
I never use any google models for any agent work any more, in case they block me.
Started migrating gdrive, and gmail to other providers at this point after horror stories in HN, that they lost access to their gmails, etc.
How the fuck can something that was announced less than a day ago be "vaporware".
I have a car that is better than Elon's, it actually does 2 backflips, my friend saw it happen. Will be released soon.
Prove me wrong.
If you're talking about track record, Argon had the same sort of announcement and was released.
It's announced with trust me bro benches, no you can't verify it yourself, this might not even be the model you get.
It's been provided to AA under clandestine circumstances, no you can't verify it yourself, this might not even be the model you get.
People on HN for some reason "wow this thing that I can't use is so great, really excited about all these ads I just looked at"
You should be running your own benchmarks often given the nature of these products. Releasing these models whenever with whatever corresponding benchmarks provides you no assurance that things will stay the same.
Having a great time with an ice cream brand at a store is no assurance against hollowing out the brand tomorrow.
True facts, I look forward to you sharing the link where I can use this new model with my own benches.
Gemini, for whatever reason, has consistently been the best for explaining scientific concepts for me.
Especially with the recent addition of graphic visualization widgets
It manages to strike the perfect balance between depth and style of prose that other Frontier models just can't quite seem to match
This is the only use case I've found it superior to ChatGPT or Claude for - helping me understand higher level math. It's slightly easier to understand than the other models
GPT has gone downhill in terms of prose readability.
On Antrhopic side Claude 5.5 improved over 5, but honestly in this dimension the bar was extremely low.
https://www.youtube.com/watch?v=NO2I3ueVQrk&lc=UgxRXtipA3QbA...
Comparison with other models.
Do you have some prompt examples I can run. Thanks
I'll publish a few of my convos
1. Sleep debts effect on muscle size and strength gain
https://share.gemini.google/Gvdb5Q02INrl
2. Fish oil and extra cellular vesicles
https://share.gemini.google/fem9EBCe4mfy
3. Pharmacology question about human-equivalent dose
https://share.gemini.google/RWNYJ2eiNjcU
Yea, but you are just accepting AI slop.
Here is your first example, "corrected": https://share.gemini.google/DCSLT4PPap5q
Having an AI do a quick check of research studies is about as effective as just trusting a Vogue magazine article. You are just fooling yourself into thinking there is something useful there, there is not.
Well OP is asking about a specific paper and you are asking for it to be disproven, different requests and the LLM is just doing what you want.
Classic "you're holding it wrong" (maybe enhanced by exercise/nutrition science being what it is with a study and expert representing every side of an argument)
I'd personally probably do it like
> Someone told me about a recent paper which showed that sleep deprivation had surprisingly little impact on muscle hypertrophy, and that mechanical stimulus seemed to be the dominant factor.
> Can you comprehensively fact-check this, and let me know what the general scientific consensus on the topic is?
Even then you can't be certain of course. Even reading the cited studies you can't be certain unless you know enough to interpret them. But you'll probably be a bit more informed than you would be if you were just going off news article titles you see on social media like the average person.
Not OP, but I asked Gemini 3.8 Flash this question (via self-hosted LibreChat) and got an excellent answer:
How can you determine if it's explanation is good if you don't already know the answer? You are not qualified to know if it's answer is correct, by hypothesis.
Just talk to 3.8 flash about whatever (caveats: ancient knowledge cutoff of jan 2025, small parameter model so if niche '90s korean animated series is your jam, it's gonna be rough). The model is a real joy to talk to, I even will feed it SOTA model output to clean up and clarify for me.
Its the first model ever that I have been worried about a successor losing the "feel" that it has.
Gemini is the best at convincing you that you understand scientific concepts. Unless you have applied that knowledge in other contexts and been given feedback that your understanding is correct you have no idea whether it is good at actually explaining them.
That's generally true for all LLMs, but as with any tool - you can actually use it in a way that produces actual, verifiable results. Obviously it requires a lot of work that 99% of people won't put into what is essentially seen as a 'magic black box', but I do agree with GP that Gemini is quite useful in actually breaking things down in an understandable way, compared to most other models.
My experience with it is mostly using it against an actual, academic source, where I can easily tell if it starts making things up. I wouldn't rely on it for anything that I can't confirm myself.
I had a research question in my area of expertise. Tried it with ChatGPT, Claude, and Gemini. I think all three could have solved the question, but ChatGPT was an unreadable mess with too many sections and headings, Claude got stuck thinking and never produced an output. Gemini gave succinct explanation that I think was similar to ChatGPT conceptually but that I was able to follow and verify. The answer was actually fairly trivial to understand; I just hadn't gone at the problem from that perspective.
I know it is common for people to try AI in domains they aren't experts in, but it is also common for people to use AI in domains they are experts in.
Gemini team has a long history of releasing great benchmark charts and disappearing into the ether. There has never been a moment where I have preferred them to Claude/Codex. When I get access I eval them on my home assistant set, a small coder benchmark, etc. and they are too expensive, and too low quality.
The benefit for Google is their chat interface has privileged access to their data (maps, etc.) but their model itself is low quality and, if I'm being honest, has the promotion-oriented measurable-output bias that FAANGs tend to have. The later Llamas had the same problem.
About to say the same... even with the "Gemini AI Plus" plan that should be renamed to "Gemini Flash" because you are only allowed to use "the mangled model" using it for something as basic as taking notes and saving to google keep or even google docs, heck yeah a simple "add this to my calendar" completly fails in gemini... Gemini sucks for everything even its own google apps... Claude and Codex even DeepSeek and Kimi are milestones better than anything Google has launched.
It is unfortunate that having such an amazing engineers that are capable to solve any leetcode exercise in 5 minutes on the spot with a white board, they aren't able to even deliver something useful on the AI space.
As of their latest update (AA's), I find their index hard to trust, and a bit shady. 60% of the benchmarks contributing to the AA index are now private, nobody can replicate them. And they explicitly say that they offer benchmark consulting to labs. This is a serious conflict of interest.
The latest update knocked down only openweights models from the leaderboard.
I agree that they are a bit shady, but private benchmarks are the only way to prevent benchmaxxing
Can’t blame them. Silicon Valley is expensive.
Carbon scoring below Opus 5.5 on coding highlights how magnificent Opus 5.5 is as a coding agent. With just a $20/month sub, it performs a ton of work at a very high quality.
The way benchmarks work is very very different from how you work with an agent.
The benchmarks often have ambiguous instructions, and there is no interaction with the agent. So a docker image is started, the agent is called in non-interactive mode with a prompt saying to read the instructions and finish the task. The instructions are often somewhat ambiguous and the agent has no way to ask for clarification. If it picks the wrong option out of many seemingly equally likely options that test will be mark as failed.
That's still the point though.
I get that you're saying the model might get a higher score because of luck, but at the same time you want the model to be "lucky".
I wish Argon were as lucky as Opus.
The benchmarks of a model that hasn't been released yet are definitely suspicious.
I also think Google should stop "releasing" models that aren't actually available. For their own sake, because by the time Gemini 4 Carbon becomes available, nobody will notice. There won't be a new press release or much excitement around it.
There's an issue with this index where many models are listed in the model chooser, but missing from the index. E.g. Qwen 3.8 Flash Next is listed but missing, as are older Claude (4.6, 4.5, etc) or OpenAI models. This makes it a bit tricky to see the improvements over time.
They removed Qwen Flash Next and GLM Flash from their benchmarks, as of two days ago. They're not publishing any figures at all for them (they show as "N/A").
https://artificialanalysis.ai/models/qwen3-8-flash-next
https://artificialanalysis.ai/models/glm-5-3-flash/
edit to add: It's very disturbing, because these open models (GLM particularly) were among the best of all LLM's on their benchmarks. On the intelligence/cost frontier, specifically. Archive.org remembers that GLM Flash was in their green quadrant just a few days ago, next to Haiku:
https://web.archive.org/web/20261014031624/https://artificia...
It's completely gone now. AAI (as far as I've seen) haven't publicly commented about why they did this.
well if there was any doubt about why openweight models were ranking lower on every iteration of the benchmarks... this should remove all doubt. to just flat out remove is very overt
They both appear today on the chart archived, the one you say they don’t appear on today.
You know what. When Gemini 3.1 Pro Preview came out it was better than Opus at the time. It was my main model for 3 weeks.
Until it got lobotomised. Then it lost most of it's capability.
Similar situation happened to Anthropic's Fable. It was an amazing model until Opus 5.5 came out.
So now, I'm not even looking at a "new" model until it is a month or two old. By then we will know if they will destroy ability to make it cheaper to host.
I remember when Opus 5.5 came out, we had a discussion about whether it was nerfed a week later [1]. It's not possible to do so with Gemini 4 models since none has been made available the day it is announced.
Without us being able to access Carbon when it is announced, there's no way to know the Carbon we get a few weeks later is the same thing as what has been benchmarked here. I think benchmarks should verify what is benchmarked is exactly the same as what's been released.
I tried Argon for a large historical HTR transcription project recently, since I heard it was really good at this. It was really good, but like all top US models I've used it was
* Definitively very good at domain specific knowledge. It can read sloppy 19th century danish gothic handwriting with faded ink just fine. Not perfectly, but probably better than I could have done with a day of struggle for every page.
* Expensive.
* Bad at actually doing what I said. I had told it in the prompt to be careful not to miss text, but I hadn't explicitly told it that some images may have text in double-column layout. So on those images - over half of my 300-page document - it produced an excellent transcription of the first column, and then considered itself done. Want to try again with a more careful prompt? That'll be another $20.
For ocr, you have to practice, practice, practice with your prompts on small sample sizes before one-shotting a book. I create a json schema for the output that describes the structure in detail (using ai from my prose descriptions of structure of course). I have specific tough pages I use on openrouter to test every new model I am interested in. Gemini has and does absolutely dominate for this. But flash can be better because it doesn't overthink things.
I did do test images, but apparently not enough. It's not 100% of images it drops the second column on either. On many it gets both, on some it only gets the second column.
I also found that it ignores other instructions sporadically ("Don't use letter variants or ligatures in the output, use the normal Danish/Norwegian alphabet when at all possible", still a few Unicode tall s'es in the output.)
It's good, but it's so frustrating since "worse" models manage to follow such instructions much better.
> That'll be another $20.
That seems arguably pretty cheap compared to the human alternatives.
I do get super annoyed at how specific you've got to be with these tools. They are like almost like malicious compliance geniuses lol.
> They are like almost like malicious compliance geniuses lol.
They're like a monkey's paw or evil genie I think.
You'll get exactly what you ask for, but lots of weirdness you didn't expect.
To be fair to the model makers, deciding what to do with an under-specified prompt is hard, particularly in an RL environment where you can't just ask the user.
For stuff I want to run for a while, I'm defaulting to telling it to ask me the questions upfront to reduce the incidence of these kinds of problems.
Good for them, having a frontier model that nobody can actually use.
It kind of doesn’t make sense. Modern google is typically kind of bad at building products and very good at getting them in customer hands. This is the opposite of that.
It does make sense, when you consider that Google had its earnings report yesterday, and missed its targets. $GOOG is already down 13%. It's an announcement purely for their investors.
Gemini 2.5 flash and 3 flash were highly underrated models that performed way better than the peers and we used the hell out of them.
Since then none of the Google models have competed well with the others at their price points. Long context performance drop, instruction following issues, unnecessarily complex and sloppish outputs.
Argon is no different, it feels just an iteration better than 3.8 flash. I don't trust the AA ranking yet.
I agree. Flash 3.8 is not good and actually quite expensive, given its tendency to overthink everything. We've been transitioning lots of workloads that were formerly on Flash to Haiku 5.5. Great model.
Gemini 2.5 flash was amazing via the API for a lot of use cases. Great blend of fast, accurate, and cheap.
Even with gemini 2.5 or so i remember the models used to just simply hang in production and randomly used to collate tokes and give in the output stream vs giving one by one which was weird.
Also the guardrails were implemented in a very bad manner. By default the models used to raise a default guardrail error on basic queries.
250tk/s is really the sweet spot for chat IMO, results from Flash were fast and usually good quality.
I want to talk about my experience with Argon, noting that I used it via OpenRouter on a Raspberry Pi, spending around $15 in credit.
My experience is that Google is aiming to create a generalist model – but a truly generalist one! In fact, I find it particularly accurate in general terms and even in long conversations, maintaining excellent consistency and a wealth of information in response to complex questions.
I’ve tested it both as a coding agent – though, precisely because of what I mentioned above about its ‘generalism’, I don’t think it measures up to Fable 5.1 or Astra – and I’ve really appreciated its qualities when supervising my agents’ activities. It was able to keep track of them, suggest alternatives, and monitor the entire lengthy process in parallel without ever straying too far from the request or losing the thread – as often happens with Claude and OpenAI (you only need to test it in Claude Code itself, in both Ultracode and the agent team mode – albeit experimental – to realise how, once the task gets underway, both of them lose their way).
Just so everyone is aware of the "coincidences" that happen with this index:
- Two days ago AA announced their v5 Index which added a few new benchmarks and tweaked a few things.
- Today the newest model announced from a big lab tops the charts.
I'm not saying that AA is definitely taking money from the big labs and changing the benchmark to make the labs' new models look better, but it kind of looks that way. This pattern of behavior has happened before, where the index gets tweaked right around when a new model comes out, and the new model is on top. AA does not, AFAIK, provide a way to see how the new models would have scored under the older index versions. You could probably calculate it yourself, but I'm not going to do that much legwork just for an HN comment.
It's interesting because they publish the index methodology, so you can't really accuse them of not being transparent, but I don't really believe that the choices to update the index were completely independent of the model they happened to be evaluating at the time.
Related: The DXOMark camera scoring method also had similar behavior, where they would release new testing methodology right before flagship smartphones were released. Again, they are technically being transparent about the process, but it kind of smells like the "independent" testers may not be as independent as the average user would hope.
I had a look at the changes and the biggest impact seems to be from adding Terminal-Bench-Science 0.1 and having it contribute 15% to the overall score.
Example problems are:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
I have no doubt that these are very relevant task for people in those respective fields, but I suspect they have next to no relevance to some 99.9% of model users. Hence why especially smaller labs with smaller models didn't really train for such use cases and you can see it.
Personally, it's not something that matters at all to me, as I don't work in cheminformatics or bioinformatics.
This is also arguably more of a test of a model's knowledge base than reasoning ability.
But most importantly, there is also the question to which extent the aggregate score then remains a meaningful result when you mix up too many too disparate benchmarks.
The coding agents subindex, in contrast, looks at least superficially plausible [1].
[1] https://artificialanalysis.ai/agents/coding-agents#coding-ag...
Ironically coding has no relevance to something like 95% of users (I forget the exact OAI statistic), but it is the mainstay LLM benchmark.
True, but coding is where something like 80%-90% of their revenue comes from; few other use cases are as token-hungry, and none of those are nearly as widespread. Many users will of course use the models just as chatbots, but they don't generate much in terms of revenue. And that is ultimately what determines where labs focus their resources.
Yeah, it is very suspicious. Each time they update their ranking algorithm, the closed models from US labs jump up and any previously high ranking open model jumps down. It happened with MiMo, Kimi, GLM. All were right there amongst the very top, and each time that happened, AA followed quickly with an index update that shunted them dramatically lower.
Check out vals.ai for an alternative. Not sure how independent they really are either, but it's at least good for a cross reference.
I also look at the ECI from Epoch.ai as well, but they haven’t evaluated Carbon yet. Probably because the model isn’t widely available yet.
This could be another point informing your prior as to how much benchmaxxing and distilling of closed models is being performed by the open weights publishers as well.
I don't understand this conspiracy/suspicion vibe around AI benchmarks -- the best benchmarks get maxed quickly in today's world, especially if the datasets are open. We're in a situation where the AA's of the world show 0-1 point difference between models, so by definition they are NOT good discriminators. These tests NEED improvement.
Do they work with frontier labs? Probably. They should work with them at least a little so that we can get an independent read on the model quality. Do they inject new tests that specifically favor one particular lab? Clearly no, as different labs have moved up over time. Do they inject new tests that help to provide some separation on scoring? God, I hope so.
Nota bene, another change they announced was dramatically increasing the amount of held back test data; this would be a simple explanation for seeing open weights models, all of which are known to distill from frontier labs and benchmarks, drop in the rankings.
Have you considered the other direction? They test a new model, and realize their current index is broken, so they improve it?
At the previous iterations they were accused of favoring OpenAI. Now they are accused of favoring Google.
We’re literally talking about the same thing, but ascribing different motivations.
I do think the test a new model and realize their index is broken- the new model isn’t on top, so they change the weights to make sure it’s on top.
The difference is that you seem to think they are doing so out of some innocent feeling that the new model is actually better, while I think they are doing so because they were paid to do so. I think my interpretation is more likely, but that’s just me.
So explain why the Chinese are not paying to put their model on top, surely that would drive a shit ton of business to them.
Or are you implying that the Chinese can't afford the amounts paid by OpenAI/Google.
Can’t afford isn’t the only option. Unwilling to pay the bribe, or not invited to the party are also possible reasons.
And how do they realize their current index is broken? They either have some recency bias, or indeed favor the big American labs (more likely, since a new Chinese model topping the charts triggers a shakeup).
It's odd because I wonder if anybody actually cares about such scores. It's not like millions of end users look at the score and use it to make a decision what to buy. Subscriptions can easiliy be canceled and people anyway use the models they "feel" work best for them. Hardly anybody I know uses an even remotely scientific method to evaluate model performance. It's mostly vibes with a little bit of (mostly flawed) A/B testing sprinkled in.
So ... why bother to cheat?
> So ... why bother to cheat?
I think part of it is to justify capex. "Yeah, we spent x billions on this, but look, we have the best models, we're beating OpenAI and Anthropic, we're winning, it's all worth it!"
From all of the terrible benchmarks of AIs, the AA one is the least terrible.
Only because it aggregates all the other benchmarks. It still feels off. It felt off before the last two re-rankings, and it still feels off. Probably because the benchmarks its based on aren’t measuring the right things.
Not questioning that it's off/biased/..., but what are you basing your "feel" on? Anything empirical or really just a feel?
Entirely vibes based for the AA rankings.
For specific benchmarks, I’ve downloaded and used basically every model out there under 35b parameters - especially the tiny models: ling-3.0-tiny, spark-x2.6-4b, nanbeige, etc. etc. some of those tiny models are ranked VERY high on benchmarks like deepswe or terminalbench but are obviously incapable when you try to use them for swe or terminal based tasks.
If that’s benchmaxxing or a bad benchmark, idk, but it’s made me not trust the benchmarks much. If it benchmarks poorly, it’s probably bad. If it benchmarks well, it might be good.
My paranoid impression was more that the updating and addition of tests — obviously essential in this game — was delayed to provide advertisement for Kimi K3.1. The same happened for K3 — and this is why it was famous — but its position had collapsed by the time of 3.1
Since 3 Pro we already know that Google's model are very good in benchmarks, but when you start programming with them... not really good and each model has its quirks that make me continue the project with something else, once I fixed gemini's bugs.
I was having immense fun wit 2.5 Pro. Going as far as copying the whole repo into context with repomix.
The scores look similar to how it feels to use Argon: great at world knowledge, not so good at coding compared to Opus.
It's hard to understand the phased rollout of a model that isn't particularly sota, unlike Mythos a few months ago.
they are too afraid of botched launches
They can release even Jambon and Jargon, why should I care when I cannot test it?
Creating the hype
The time between public announcement -> public release seems to be a tacit admission that Google is still a bit behind the frontier labs. Google is justifying the delay by pointing to cyber risk and a closed-access period reserved for analysis of safeguards. I'm dubious of this justification. But if it is true, I'm pretty sure the other labs are doing similar sorts of things in private. I think most of us prefer it that way—if Anthropic public announced Fable 5.5 two weeks ago with no availability that would add to my annoyance. Maybe they learned their lesson with Mythos, or maybe it's just that Google also does not have an IPO looming. Who knows?
It's performative stuff to try and appear like they care about alignment and safety.
"Hey we are launching this thing but we've got to let the safety folks see it first".
So they look like they care about alignment etc.
To be honest there is a lot of rage on here about it being announced before being able to access it. I don't understand why people are getting so upset about this - like no one gets into a rage because they see a movie trailer before it is released?
I guess it's also because the model is available to at least one group, and this releases embargo on information about it.
This is just a hunch since I have not used Argon very much, and I have not used it for coding (deepseek 4.1 flash 1017 with dsh, and qwen 3.8 max are so good for coding tasks that local models have difficulty with).
My hunch is that Google will end up being best for non coding tasks. Google can concede coding to other vendors who pay Google for infra.
Then you'll miss auto improvements on code
Is this a pro 3.1 situation that by the time ot reached the users it produced no more work than the flash variant or competitor for absurd pricing?
Artificial Analysis is not trustworthy anymore. Every time an open-weight model, especially Chinese, looks to be nearing the top, AA updates the index to (ostensibly) specifically push closed models higher. Just recently with v5, OpenAI models were pushed up.
I don't think that's the right read. Deepseek Flash 4.1 1017 did not get affected much.
I think what's happening is Chines models genuinely try to benchmax on it and hence they suffer when a new version comes out.
> I think what's happening is Chines models genuinely try to benchmax on it and hence they suffer when a new version comes out.
I read somewhere on Substack that a lot of this was driven by the inclusion of more knowledge work benchmarks, where the US labs are stronger, versus coding which is where most Chinese models appear to put their energies for now.
Or it could be that the benchmaxxed models are getting pushed down by the introduction of unsaturated benchmarks. Muse Wonder also went down 52 -> 43 after the update.
It's not that hard to create your own uncontaminated eval for the problems you care about, and compare.
Here's mine for coding in Rust, and I definitely see the same pattern of "most chinese models do much worse than on public evals that have been around long enough to train on" that AA v5 surfaces: https://slopcop.com/power-ranking
Is hallucinations and deep thinking tangential optimization steps ?
Don't bet against Google
People always love to think the AI race is a duopoly (OpenAI vs. Anthropic), but from what I have experienced, Gemini was always holding on its own, and I find myself using it more than Claude.
True, I use LLMs as glorified search engines to unearth material to read myself, but Gemini is much more useful if you decide to learn yourself. It also offers interesting expansions to your questions.
While I don't offload my work to LLMs (e.g.: Write this project, I'll be watching TV), I find them useful for closing my knowledge gap by collating information, and Google's Gemini is good at that.
Gemini is just too expensive for what you describe. Their 20$ plan only gives you access to the free tier model
I have Gemini because of my Google One membership. I already was on their 2TB plan and it gave me Pro level, IIRC, and it was generous. Now I'm on Ultra mostly because of Google Drive storage I get, but Gemini also gets a nice boost, as well.
I'm on $20 Claude plan, and it's also ample for my needs, for now.
The $20 plan gives you ~120 minutes of high thinking time from Argon weekly. Thinking time isn't an exact metric, context size matters a lot, but I have been using it on the $20 and still have 25% left with ~85 minutes of thinking time on high for a project I have been working on.
I think this is because of capacity issues. Higher paying per token users probably get preferential capacity allocations to those on monthly plans. The allowance for what you get in the monthly plan increases as the models age and the likely capacity problems improve.
Thanks David Guetta!
They have an insane amount of resources but are so bloated, have a ton of bureaucracy, and have lost a lot of their top talent. Looks like they have figured things out now though.
A lot of people do though by looking at the current stock price.
Never bet against their contempt for paying customers.
The situation with Google is a bit unexpected.
They have top of the line research, I mean, they invented the thing, and they are still doing stuff that goes beyond making a bigger LLM, as it seems to be the only thing the others are doing, research-wise.
They also have hardware capabilities no one else has, they invented the TPUs, and have lots of them.
They also have the brand recognition, ecosystem, user base and lots of money.
So why are they lagging behind on the actual products, and by lagging behind? I mean they are always in the position where they are just behind the frontier, but still in the game.
It seems almost deliberate. Maybe they are playing the long game, waiting for the bubble to burst. Maybe they don't want to compete against the more profitable parts of the company. Or maybe their researchers are not interested in making an actual product unless being forced to.
Really? Been 3 years since ChatGPT and Google still going from misery to another in their AI products.
Even if these numbers are correct, they don’t have any useful product for coding or search. Not sure what you’re referring to
Not everything is writing code and search. My uncle works in mortgages and the (very large) firm he works for is entirely Gemini. The way he talks about it is with wonderment in his eyes while he goes into detail on how it went through multiple documents and found unaccounted for cases. I think Google is doing fine in these kinds of industries.
I wouldn't even bet that Google is better than OpenAI and Anthropic for these categories, but they seem to be well used outside of our direct space.
It worked for Microsoft with cloud. Existing industry penetration goes a long way.
Antigravity harness seems much worse than Codex. Argon seems very knowledgeable but very sloppy. If I force it to self review it will catch its own errors and laziness.
Argon ran out so fast on the $20 Pro plan that it's practically unusable. I can't imagine how restrictive the limits must be for Carbon.
At this point frontier models are good enough to be a pair programmer. But only OAI and Anthropic offer good limits. Maybe they're burning money but if the Google limits are the future of usage style accounts then we all have to get used to API
I haven’t tried Gemini for coding yet. I did consider buying their subscription, but based on what I’ve seen from other forums, Gemini has never been a frontier in coding, even with good “benchmaxing”.
Let’s see, maybe they’ll surprise this time.
I have a fairly narrow use case: I'm developing apps in Dart/Flutter. Gemini works well for that.
Boggles my mind that they still offer 3.1. Absolutely 0 reason to use it when 3.8 Flash is better in every respect.
Nice coverup for the Q3 results. Wall St. don't give a F*
Don't care, it's not released so in reality it's undefined, not #1
Google is killing its own product. Gemini Argon is much better at writing and to some extend coding, but my teammates keep using deepseek because it is available and don't have to check vertex ai, ai studio, gemini, antigravity and all those shit...
why do i not see as much traction and stirring up on the internet for gemini models as much as claude or gpt even though on bench's they are very much THE frontier. is the reason very limited access early or or just not enough publicity or hype as the other labs do?
>is the reason very limited access early
I mean, that's the main reason. How would you expect to see the kind of hype that Claude or GPT can generate when nobody can use it? The hype of those models are primarily generated from people using them to build impressive things and share those results. Can't do that with an unreleased model, right?
Beyond that, it seems a lot of people have learned that these agent leaderboards really don't tell the whole story. They're a good litmus test, per se, and there's plenty of hype being generated around these scores as-is. But anybody using AI in practice knows that benchmarks can only say so much, and one model scoring better than another doesn't necessarily imply that that model is actually better in the ways those people care about. You can see that in this thread as people are pointing out that even these benchmarks seem to suggest that Opus 5.5 is still leading in coding (which is what a lot of people, especially on HN, ultimately care about).
>just not enough publicity or hype as the other labs do?
Well, it's effectively both. Google is certainly not that pressed to generate as much hype as these other companies. They can afford to take their time. It's also worth noting that Google is definitely targeting different consumers, investors, etc., so their strategy will definitely be different.
For what it's worth, I've been using Argon since it's been available in Cursor and I really like it (at least at its current price). I think Google really did close the gap a meaningful amount (maybe not all the way, but enough to prove they're still in the game). Carbon being significantly better while being the same cost is pretty exciting to me. But until I can use it, I don't really care about it.
I use it purely for finding stuff in text documents and OCR, at which it excels, but I've found that for any kind of analysis it's some bewildering mixture of smug, defensive, paranoid, cheerful, and wrong about everything.
I feel like the headline here is "the Artificial Analysis index is becoming an increasingly poor measure of capability, having previously been quite good".
Even Haiku is better than Grok
It's interesting it's not ahead of Claude on the coding benchmark right?
Google has to get it's shit together. All the stats and charts doesn't mean squat because when I try it now on a provider all I get is trash :(. Makes me question these metrics entirely and really makes me question the authority that google used to perceivably have.
I have pro account and previous model "Argon" still not available on my account.
I have it on both web and agy, but you know when you wait for something and in the end it turns out to be not that good...
Mine showed up after clearing cache for some reason.
is it on web interface or antigravity ?
gemini.google.com (Pro plan).
AGY never even update often, and only has gemini 3.8 flash (for Google model).
How can one access Carbon? I only see Argon via gemini.google.com
Is this API-only or US-only for now?
It’s available only to those who read the announcement.
My ignorance laid bare
https://blog.google/innovation-and-ai/models-and-research/ge...
"We're taking a deliberate approach to rollout and are actively engaged in the U.S. government’s voluntary process for pre-release model testing. Starting this week, we will make Carbon available to a subset of cyber defenders and will gradually phase access to additional trusted partners through our Fairwind Program. We’re continuing to gather feedback and iterate on guardrails before making Carbon available to developers, businesses, and consumers via our product surfaces as soon as possible."
By the time this model is available for public use I'd imagine it won't be #1 anymore, which is pretty sad. Even when Google gets back on the ball they seem to find a way to fuck it up.
These benchmarks and rankings seem more noise than signal these days. They're uniformly divorced from sincerely-attested real world experience. And the real world experience isn't driving any consensus either.
So here's a meta-benchmark: at what point does the act of choosing an LLM product for a reasoning task become less reliable (in the sense of "predictive of future success") than hiring a human being for that same task? Have we already cross that threshold? I think we might have.
And once we do... what happens then? Will the truly elite models be hidden in a sea of mediocrities simply because the goofball "managers" doing the hiring are too dumb to recognize their genius?
I don’t care and in my opinion AA does a disservice to their users by allowing such blatant manipulation. What would happen if every AI company were to include their unreleased models like this? Put the unreleased models on a separate page and leave the headline benchmark rankings for models that you can actually use.
AA (like most other VC-funded benchmark sites) is pay-to-play.
Want a private benchmark and see where you'd rank? $$$.
Want to show up day 0? $$$$.
Want to show up day -0? $$$$$.
What the AA charts say is irrelevant to me.
I couldn't care less, honestly
well, $GOOG is down 13%...
slick
It's unobtanium so nobody should care about it. Any frontier AI company can nowadays create models with ridiculous resource and token usage that will get extremely high scores, but they're of no use when you can't use them.
I take this opportunity to give feedback on Argon. The limitations are a problem. I have a Pro subscription and Argon already stops working after 3-4 prompts with long delays afterwards. It's virtually useless right now even when you pay monthly for it. It is better for some tasks than 3.1 Pro, though. Not much in my experience but it can sometimes fix errors that 3.1 has problems with. I never use Flash models since they often give abysmal results.
So I'm stuck with using Gemini Pro 3.1 for now.
A secret model I trained last afternoon is the actual #1 on benchmarks.
No, you can't use it. You can't even see it, and I won't provide you any information. But you must trust me that I am the genius who made the best model in the whole world.
owaoo
Spare me the celebration. Artificial Analysis should not be granting any company the favor of a #1 ranking for a model that isn't released to the public.
Google's model release is pure PR puffery. It's no coincidence that it was released on the day of its earnings call.
And i guess people dont care by google releases anymore
Argon did not leave a mark or change anything
Why would anyone care about carbon as well
Sunder quoted their tweet in his release announcement. Like that was the main thing he wanted to say.
Of course he would, he paid a lot for the privilege.