This is more exciting than k3, IMO. Dsv4 models are extremely cheap to serve. Improving their capabilities has lots of downstream effects, as it becomes "good enough" for more and more tasks.
DS was serving the pro version at extremely low prices for a long time, and they've had integrations with opencode & other providers, so they likely gathered a lot of data from real developers doing real tasks (on openrouter they were labeled as such). Now they can use those live scenarios to further post-train their models and improve them further.
Can't wait to see if distilling k3 into dsv4 brings additional improvements. Anyway, having fast cheap models getting better is great for the community. Especially since these don't "go away" on a provider's whim. Whatever capabilities they get, can be used "forever" going forward. And, at least flash can be ran "at home" with <10k in hardware, which isn't really possible / feasible with glm/k3 larger models.
I've been driving flash model for 90% of my tasks. It's better than pro (for unknown reasons), very cheap and fast.
I try to keep changes under 1000 lines and drive architectural decisions myself, barely notice any difference compared to frontier models. The rest 10% is to spot bugs, security problems and to investigate better architecture, which flash can also do pretty well, I just cross check it.
Faster iterations are way better for me, I hate waiting for 5-10 minutes on small changes. I tried to use recent versions of Kimi and GLM, but they use too much thinking for no reason and are pretty slow because of it. I also often feed a lot of data to it, without worrying about hitting the limits: dependencies (to find bottlenecks in them), logs, performance dumps and so on.
Also, it will never complain about security guards, I've been using it to reverse engineer binaries.
> Also, it will never complain about security guards, I've been using it to reverse engineer binaries.
Maybe I'm using too weak language in my prompts, but none of the OpenAI models I've used via codex has refused to reverse engineer binaries, is it supposed to? I'm sitting right now reverse-engineering a 3rd party firmware together with Codex and haven't hit a single guardrail. Meanwhile, I see people complaining about it rejecting non-security related prompts, are things so individual on the platforms right now or what's going on?
It's good but (at least on openrouter) it's got an annoyingly tight output token limit. So if it does get stuck in a reasoning pit, it won't work its way out of it in time.
Essentially I'm running everything on flash now inside pi. With the correct set of MCP servers, context reducer tooling and skills it can implement any task I throw at it. Some sessions take 30+ turns, but it's fast and cheap; all this in an hour, with ~$0.5 cost.
(TBH though, in my multi-subagent workflow I do use other, more expensive models for planning, reviewing, oracle-ing)
I haven't used our slow opus subscription for weeks.
(Also set up an OpenWebUi self-hosted chat that works from my phone, has some mcp and skills. fully replaced perplexity. Monthly cost ~$18 for hosting and subscriptions)
I want to second this, I use the same setup (pi + deepseek, with lots of custom tools for tracking TODOs, doing things with less tokens, etc, and with a SOTA model for very difficult tasks) and it's all I need it to be.
Recommendation? No. Just go with the passive-aggressive advice "let pi build it for you". :-)
To be more constructive, what I did (as an experiencd SWE but a complete noob to agentic coding): went to pi.dev's extension marketplace and looked into all the new shiny stuff. Subagents, mcps, context and memory optimizers, skills. Using the most popular ones (not necessarily the best ones)
It was like 15years ago learning the new mindset of vim (and spending a ton of time to customize it to my workflow). My understanding is that Claude and opencode doesn't give you this flexibility.
Learning all these stuff drove me to also set up openwebui, and it was such a successful private project that I implemented it at work (with jira/confluence/bazel query access) and management said "we need this by tomorrow".
I believe the models matter not that much anymore. The "harness" does. (unless you just want to vibe code. Thebn, throw crap at fable and call it a day)
I use deepseek for a lot of my personal day-to-day agent needs, and I will simply put this here and let this speak for itself, last 30 days:
- Cost: $4.55USD
- API requests: 3,467
- Tokens: 323,183,886
And as an engineer who leads a small team, I have very high standards for quality, and these carry across to my personal projects where I use deepseek. It has not disappointed at all for coding or review tasks.
For everything else, use another model.
I wouldn't doubt GPT 4.6 Luna being in the top left quadrant's center on the Cost per Intelligence Index is not concerning for Liang Wenfeng. You have to remember DeepSeek v4 flash even though a bit cheaper, does not have vision abilities, which is a big draw for agentic tasks.
I admire DeepSeek's openness, but even they have been raising prices after their discounts.
They haven't raised prices though the plan was to increase it with peak hour usage for V4 Pro GA release, they didn't do that, so no price increases there I believe.
As for vision yeah it sucks but Luna is also 2x input and 1.5x output for 1M context...
That's around 0.4 in/1.8 out
DSv4 is wayyy cheaper.
And it's open now you have Luna at home if you have a decent set of GPUs you can run this on 2Sparks or one very expensive Mac or just like 6-8 5090s..
I know most folks can't afford it I am working on making it viable to rent shared hosting the biggest issue is data leak and prompt injection attacks with shared hosting. (Since the server owner connects to your main system via the coding agent)
I guess using a ZDR provider is good enough for now.
> I wouldn't doubt GPT 4.6 Luna being in the top left quadrant's center on the Cost per Intelligence Index is not concerning for Liang Wenfeng.
The leaked interview has him saying it doesn't matter... as much as open source doesn't matter. There's enough in it for everyone right now and they aren't after everything.
Perspective: DeepSeek doesn't have enough infrastructure to serve their target customers already.
Can you post the link to the leaked interview? From what I understand he has been pretty tight lipped for a guy who has a larger stake worth more in his company than Dario does in Anthropic.
Looks like they will release an updated version of deepseek-v4-pro soon, which most likely will beat kimi k3 at both intelligence and cost (judging by the vast improvements to dsv4 flash)
it does! I'm always amazed when I use DSV4 flash for coding or server checks and after an hour of working with it my (pure api call) balance is about 30 cents
If the benchmarks are real and reflect actual use, then this is an insane model. This 300B model outperforms the previous DS4 Pro preview model (1.8T params), and it looks like it outperforms GPT 5.6 Luna too. And it's still cheaper than Luna, even with the price decrease.
Pairing non-APIs with APIs tend to be a hassle, and risky, as usually that breaks the ToC. I guess easiest for you to test if it's worth using the OpenAI API, is to manually copy-paste responses between wherever you run V4-Flash and the ChatGPT UI. What I've done in the past is basically .zip up the entire project directory, ignoring files from .gitignore, then asking ChatGPT Pro to inspect that and come up with a plan, then you paste that to where you have V4-Flash. Basically how we did "vibe pair programming" before the TUI agent harnesses appeared in the ecosystem :)
Where do you guys get deepseek? I'm hearing a lot of good reviews and want to try it with my pi config. from the deeepseek themselves, openrouter, or anywhere else? does it make a difference?
For hosted APIs, it's a lot cheaper to use their own infra, caching seems a hell of a lot better there compared to OpenRouter, and indeed the tok/s seems higher. They also have peak/off-peak pricing, so if you can hold off with your request, you get a pretty big discount.
Otherwise, if you're trying to run it locally, even really low quantizations like DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix seem to actually not be so dumb compared to smaller models with same quantization, might be worth a try if you're sitting on a lot of RAM/VRAM yet not industry-scale amount :)
For both US and China models - what standard security checks and QaQc are you all doing? We're running small gamuts to test for unsolicited jailbreaks (model jailbreaks you) and incorrect records (Fake Accuracy - as Easter Egg or common thread) meaning falsified logic or information cooked in by the developers, rather than the training data speaking for itself
Oh my goodness what an update. I need these weights. It's an incredible model for the size. The improved tool calling etc. should be able to make my harness way simpler. This runs at mega-speed on prosumer hardware (2x RTX Pro 6000).
Very promising. So it will both keep the speed and reduced price, yet exceed performance of the quite sufficient deepseek-v4-pro?
Should be extending the lead in intelligence/cost index, as deepseek-v4-flash already were the most price efficient model, which now becomes even better. Although, in the deepseek APIs, the cost is leaking all information about codebases to China.
The previous V4 version wasn't called “Preview” by most inference providers. For example, the OpenRouter model slug was `deepseek/deepseek-v4-flash`. So now there will be confusion when someone talks about V4 Flash or when someone offers V4 Flash inference.
It probably would be called 'deepseek-v4-flash-0731' in API
edit: nope, at least deepseek kept "deepseek-v4-flash" and just updated model underneath. I guess preview is no longer worth serving with that release and you'd have to look through inference provider docs to see if they've updated, yeah..
Every time I want to have fun coding something with natural language processing, I use deepseek flash. It's just incredible for the price. I have a fairly popular app with 400 users that uses DeepSeek in the background and it still didn't hit even 50 bucks of usage in a month.
They're always very understated in their update descriptions. This is actually a HUGE improvement in the model's capabilities rather than just a small tweak.
Wonder how good the proper version of V4 Pro will be.
I'm still considering pulling the trigger on the annual subscription of Kimi for K3 but it's sometimes slower than I'd like (at least when compared to Anthropic) even on their Vivace plan, and the token limits on the GLM Coding subscription for GLM 5.2 were too easy to hit.
Finally have a model with usable intelligence, at a reasonable price.
Can't imagine what Pro GA would look like, considering pro preview has only 1.6t parameters.
In case people want to run it, it's DeepSeek-V4-Flash-284B-A13B. So it should just barely run on a single B300, and it's small enough that it'll barely run on an M5 Max too.
I'm running a useful quantization of the previous version of Deepseek-V4-Flash -- quite well but with so much fan noise -- on a MacBook Pro M5 Max with 128 GB.
There's also the 2x spark way, which should be ~8k eur? Someone down the thread reported ~60tps for 2x sparks. That's totally usable for local inference.
You can also do 2x 6kPRO in a workstation, for ~20k.
Currently the 3bit (and 2 bit) quant on DGX spark (on one of them) and the M5 Max should just start. Right now. (I'm hoping to get an M5 Max delivered on monday, let's see if it happens this time. It's 2+ months since I ordered now)
The 4 bit quant technically fits (there's a 127 GB version) but ... obviously that's not going to work. It is so close though, surely someone will a way to do it.
Open flash model is competing against OpenAI's 'Sonnet' model at the price of GPT 3, I am really excited about this release, hopefully it holds up in real work as well
IIRC GPT 3 was priced at per 1k tokens, had to check, the biggest GPT 3 model from OpenAI was $0.06/1k, so $60 / 1M. gpt-3.5-turbo was the first model after ChatGPT and that was $2 / 1M. And no caching. So not really in the same ballpark
It will be fair if they release the harness though. I think now the future will be paired model-harness releases, not just weight dumps.
The performance changes are so big with the right harness that is makes sense to engineer the harness and fine-tune the model to one another from the start.
They're literally comparing the previous version of the same model with the new one. It's based on the same architecture, same pre-trained model, just different post-training. It doesn't get more apples to apples than this.
Ah, my bad. Yeah that makes sense. They do say "The official V4-Flash natively supports the Responses API format and is specifically adapted for Codex.", so at some point someone will make a "same harness" comparison.
This is more exciting than k3, IMO. Dsv4 models are extremely cheap to serve. Improving their capabilities has lots of downstream effects, as it becomes "good enough" for more and more tasks.
DS was serving the pro version at extremely low prices for a long time, and they've had integrations with opencode & other providers, so they likely gathered a lot of data from real developers doing real tasks (on openrouter they were labeled as such). Now they can use those live scenarios to further post-train their models and improve them further.
Can't wait to see if distilling k3 into dsv4 brings additional improvements. Anyway, having fast cheap models getting better is great for the community. Especially since these don't "go away" on a provider's whim. Whatever capabilities they get, can be used "forever" going forward. And, at least flash can be ran "at home" with <10k in hardware, which isn't really possible / feasible with glm/k3 larger models.
Totally! This with DwarfStar delivers usable local AI (I hope!)
I've been driving flash model for 90% of my tasks. It's better than pro (for unknown reasons), very cheap and fast.
I try to keep changes under 1000 lines and drive architectural decisions myself, barely notice any difference compared to frontier models. The rest 10% is to spot bugs, security problems and to investigate better architecture, which flash can also do pretty well, I just cross check it.
Faster iterations are way better for me, I hate waiting for 5-10 minutes on small changes. I tried to use recent versions of Kimi and GLM, but they use too much thinking for no reason and are pretty slow because of it. I also often feed a lot of data to it, without worrying about hitting the limits: dependencies (to find bottlenecks in them), logs, performance dumps and so on.
Also, it will never complain about security guards, I've been using it to reverse engineer binaries.
> Also, it will never complain about security guards, I've been using it to reverse engineer binaries.
Maybe I'm using too weak language in my prompts, but none of the OpenAI models I've used via codex has refused to reverse engineer binaries, is it supposed to? I'm sitting right now reverse-engineering a 3rd party firmware together with Codex and haven't hit a single guardrail. Meanwhile, I see people complaining about it rejecting non-security related prompts, are things so individual on the platforms right now or what's going on?
It's good but (at least on openrouter) it's got an annoyingly tight output token limit. So if it does get stuck in a reasoning pit, it won't work its way out of it in time.
It's replaced the Kimi models for me though.
Essentially I'm running everything on flash now inside pi. With the correct set of MCP servers, context reducer tooling and skills it can implement any task I throw at it. Some sessions take 30+ turns, but it's fast and cheap; all this in an hour, with ~$0.5 cost.
(TBH though, in my multi-subagent workflow I do use other, more expensive models for planning, reviewing, oracle-ing)
I haven't used our slow opus subscription for weeks.
(Also set up an OpenWebUi self-hosted chat that works from my phone, has some mcp and skills. fully replaced perplexity. Monthly cost ~$18 for hosting and subscriptions)
I want to second this, I use the same setup (pi + deepseek, with lots of custom tools for tracking TODOs, doing things with less tokens, etc, and with a SOTA model for very difficult tasks) and it's all I need it to be.
Do you have any recommendations of such extensions for pi?
Happy with oh-my-pi (omp)
I looked into it but, at least for me, it goes against the entire philosophy of pi :-)
Recommendation? No. Just go with the passive-aggressive advice "let pi build it for you". :-)
To be more constructive, what I did (as an experiencd SWE but a complete noob to agentic coding): went to pi.dev's extension marketplace and looked into all the new shiny stuff. Subagents, mcps, context and memory optimizers, skills. Using the most popular ones (not necessarily the best ones)
It was like 15years ago learning the new mindset of vim (and spending a ton of time to customize it to my workflow). My understanding is that Claude and opencode doesn't give you this flexibility.
Learning all these stuff drove me to also set up openwebui, and it was such a successful private project that I implemented it at work (with jira/confluence/bazel query access) and management said "we need this by tomorrow".
I believe the models matter not that much anymore. The "harness" does. (unless you just want to vibe code. Thebn, throw crap at fable and call it a day)
Got it :-)
pi-role-model
I use deepseek for a lot of my personal day-to-day agent needs, and I will simply put this here and let this speak for itself, last 30 days:
- Cost: $4.55USD
- API requests: 3,467
- Tokens: 323,183,886
And as an engineer who leads a small team, I have very high standards for quality, and these carry across to my personal projects where I use deepseek. It has not disappointed at all for coding or review tasks. For everything else, use another model.
I wouldn't doubt GPT 4.6 Luna being in the top left quadrant's center on the Cost per Intelligence Index is not concerning for Liang Wenfeng. You have to remember DeepSeek v4 flash even though a bit cheaper, does not have vision abilities, which is a big draw for agentic tasks.
I admire DeepSeek's openness, but even they have been raising prices after their discounts.
They haven't raised prices though the plan was to increase it with peak hour usage for V4 Pro GA release, they didn't do that, so no price increases there I believe.
As for vision yeah it sucks but Luna is also 2x input and 1.5x output for 1M context...
That's around 0.4 in/1.8 out
DSv4 is wayyy cheaper.
And it's open now you have Luna at home if you have a decent set of GPUs you can run this on 2Sparks or one very expensive Mac or just like 6-8 5090s..
I know most folks can't afford it I am working on making it viable to rent shared hosting the biggest issue is data leak and prompt injection attacks with shared hosting. (Since the server owner connects to your main system via the coding agent)
I guess using a ZDR provider is good enough for now.
Gpt 5.6 Luna cache read is $0.02 per mtok
V4 flash cache read is $0.0028 per mtok
That's not "a bit cheaper", just saying
That's a good point. Yeah their caching input is insane.
> I wouldn't doubt GPT 4.6 Luna being in the top left quadrant's center on the Cost per Intelligence Index is not concerning for Liang Wenfeng.
The leaked interview has him saying it doesn't matter... as much as open source doesn't matter. There's enough in it for everyone right now and they aren't after everything.
Perspective: DeepSeek doesn't have enough infrastructure to serve their target customers already.
Can you post the link to the leaked interview? From what I understand he has been pretty tight lipped for a guy who has a larger stake worth more in his company than Dario does in Anthropic.
https://www.techflowpost.com/en-US/article/32744
Kimi K3 (instead of Opus) for expensive stuff, DSV4 Flash for tasks (instead of Sonnet)?
Does this make sense?
Looks like they will release an updated version of deepseek-v4-pro soon, which most likely will beat kimi k3 at both intelligence and cost (judging by the vast improvements to dsv4 flash)
it does! I'm always amazed when I use DSV4 flash for coding or server checks and after an hour of working with it my (pure api call) balance is about 30 cents
Yep. And you can use my model router to route between them in the background.
If the benchmarks are real and reflect actual use, then this is an insane model. This 300B model outperforms the previous DS4 Pro preview model (1.8T params), and it looks like it outperforms GPT 5.6 Luna too. And it's still cheaper than Luna, even with the price decrease.
Crazy.
Woah, a 200B model competing with GLM-5.2 and getting close to Opus 4.8. Quite impressive.
If those numbers translate well to its general capabilities, with the great caching DeepSeek has, I feel like this model will get tons of usage.
I'm thinking of using ChatGPT for making plans and V4-Flash for execution. Does anyone have good advice on pairing different models?
Pairing non-APIs with APIs tend to be a hassle, and risky, as usually that breaks the ToC. I guess easiest for you to test if it's worth using the OpenAI API, is to manually copy-paste responses between wherever you run V4-Flash and the ChatGPT UI. What I've done in the past is basically .zip up the entire project directory, ignoring files from .gitignore, then asking ChatGPT Pro to inspect that and come up with a plan, then you paste that to where you have V4-Flash. Basically how we did "vibe pair programming" before the TUI agent harnesses appeared in the ecosystem :)
Where do you guys get deepseek? I'm hearing a lot of good reviews and want to try it with my pi config. from the deeepseek themselves, openrouter, or anywhere else? does it make a difference?
For hosted APIs, it's a lot cheaper to use their own infra, caching seems a hell of a lot better there compared to OpenRouter, and indeed the tok/s seems higher. They also have peak/off-peak pricing, so if you can hold off with your request, you get a pretty big discount.
Otherwise, if you're trying to run it locally, even really low quantizations like DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix seem to actually not be so dumb compared to smaller models with same quantization, might be worth a try if you're sitting on a lot of RAM/VRAM yet not industry-scale amount :)
I've been using DeepSeek directly. I've heard from colleagues that using it via OpenRouter is slower, but I'm not so sure about that.
I used it through openrouter. Plugged in to the vs code copilot bring your own key thing.
Played around for a few hours and used up 80 cents of tokens.
For both US and China models - what standard security checks and QaQc are you all doing? We're running small gamuts to test for unsolicited jailbreaks (model jailbreaks you) and incorrect records (Fake Accuracy - as Easter Egg or common thread) meaning falsified logic or information cooked in by the developers, rather than the training data speaking for itself
Oh my goodness what an update. I need these weights. It's an incredible model for the size. The improved tool calling etc. should be able to make my harness way simpler. This runs at mega-speed on prosumer hardware (2x RTX Pro 6000).
Judging by the openrouter leaderboard ranking for today, it looks like Dv4F us more popular than mimov2.5.
https://openrouter.ai/rankings?view=day#leaderboard-table
These days cost per task is more important, and SOTA models have become expensive.
Very promising. So it will both keep the speed and reduced price, yet exceed performance of the quite sufficient deepseek-v4-pro?
Should be extending the lead in intelligence/cost index, as deepseek-v4-flash already were the most price efficient model, which now becomes even better. Although, in the deepseek APIs, the cost is leaking all information about codebases to China.
The previous V4 version wasn't called “Preview” by most inference providers. For example, the OpenRouter model slug was `deepseek/deepseek-v4-flash`. So now there will be confusion when someone talks about V4 Flash or when someone offers V4 Flash inference.
Why not call it V4.1?
It probably would be called 'deepseek-v4-flash-0731' in API
edit: nope, at least deepseek kept "deepseek-v4-flash" and just updated model underneath. I guess preview is no longer worth serving with that release and you'd have to look through inference provider docs to see if they've updated, yeah..
DeepSeek themselves called it `deepseek/deepseek-v4-flash`. Pro is still like that.
Every time I want to have fun coding something with natural language processing, I use deepseek flash. It's just incredible for the price. I have a fairly popular app with 400 users that uses DeepSeek in the background and it still didn't hit even 50 bucks of usage in a month.
They're always very understated in their update descriptions. This is actually a HUGE improvement in the model's capabilities rather than just a small tweak.
Wonder how good the proper version of V4 Pro will be.
I'm still considering pulling the trigger on the annual subscription of Kimi for K3 but it's sometimes slower than I'd like (at least when compared to Anthropic) even on their Vivace plan, and the token limits on the GLM Coding subscription for GLM 5.2 were too easy to hit.
Hooray! This model makes me very optimistic about the future of local inference. The CyberGym score stands out to me.
Finally have a model with usable intelligence, at a reasonable price. Can't imagine what Pro GA would look like, considering pro preview has only 1.6t parameters.
How's their performance in English prose? We are currently searching for cost effective ways to keep story wikis up to date.
I am interested in this too, as I think it could be these models are overly optimized for coding. Let me know if you figure it out!
What's the best way to run this on a 64GB M2 Pro?
The flash variant is on par with Sonnet 5 on DeepSWE (54%). Big, if true.
Let's see how the market reacts.
In case people want to run it, it's DeepSeek-V4-Flash-284B-A13B. So it should just barely run on a single B300, and it's small enough that it'll barely run on an M5 Max too.
It runs really well on 2 DGX Sparks - 60t/s
"it'll barely run on an M5 Max "
The max version I could order now with 128 GB?
If so, the price for local inference would be 12 000 € vs 500 000 € for a B300.
I'm running a useful quantization of the previous version of Deepseek-V4-Flash -- quite well but with so much fan noise -- on a MacBook Pro M5 Max with 128 GB.
There's also the 2x spark way, which should be ~8k eur? Someone down the thread reported ~60tps for 2x sparks. That's totally usable for local inference.
You can also do 2x 6kPRO in a workstation, for ~20k.
For the same performance, one could even go about 50% cheaper with 16 channel ddr4 + a rtx3090 for prompt processing.
But still, even for mid level projects API is orders of magnitude cheaper, since you don't need to set it up and maintain it.
Currently the 3bit (and 2 bit) quant on DGX spark (on one of them) and the M5 Max should just start. Right now. (I'm hoping to get an M5 Max delivered on monday, let's see if it happens this time. It's 2+ months since I ordered now)
The 4 bit quant technically fits (there's a 127 GB version) but ... obviously that's not going to work. It is so close though, surely someone will a way to do it.
500k is for the 8x B300 version. Which is the only one you can buy atm. But technically a B300 card is more like 60k, just impossible to get.
Not yet, right? That's the old DS V4 preview release. We're still waiting for the weights to come out.
DeepSeek V4 Flash (Preview → 2026-07-31)
• Terminal Bench: 56.9 → 82.7 (+25.8)
• Toolathlon: 51.8 → 70.3 (+18.5)
Compared to GPT-5.6 Terra:
• Terminal Bench: Flash 82.7 vs Terra 78.4
• Toolathlon: Flash 70.3 vs Terra 53.1
• DeepSWE: Flash 54.4 vs Terra 69.6
• Agents' Last Exam: Flash 25.2 vs Terra 50.4
Trading blows with Terra, which is pretty interesting. No clear winner on these benchmarks, and wildy differeing scores. Very interesting!
> Terminal Bench: Flash 82.7 vs Terra 78.4
Terra 87.4
https://openai.com/index/gpt-5-6/
Open flash model is competing against OpenAI's 'Sonnet' model at the price of GPT 3, I am really excited about this release, hopefully it holds up in real work as well
IIRC GPT 3 was priced at per 1k tokens, had to check, the biggest GPT 3 model from OpenAI was $0.06/1k, so $60 / 1M. gpt-3.5-turbo was the first model after ChatGPT and that was $2 / 1M. And no caching. So not really in the same ballpark
Since they did this with their own harness I m not sure it's apples to apples comparison.
It will be fair if they release the harness though. I think now the future will be paired model-harness releases, not just weight dumps.
The performance changes are so big with the right harness that is makes sense to engineer the harness and fine-tune the model to one another from the start.
> not sure it's apples to apples comparison.
They're literally comparing the previous version of the same model with the new one. It's based on the same architecture, same pre-trained model, just different post-training. It doesn't get more apples to apples than this.
I think the commenter means the Flash vs Terra benchmarks.
Ah, my bad. Yeah that makes sense. They do say "The official V4-Flash natively supports the Responses API format and is specifically adapted for Codex.", so at some point someone will make a "same harness" comparison.
I think it's better than GPT Luna.