I use DeepSeek exclusively and now Kimi K3 offers a great planning assistant for more advanced coding tasks.
DeepSeek v4 Flash is extremely fast and is able to handle pretty much anything I've thrown at it (I use mostly Rust, PSQL, Angular and Terraform).
I self host Bifrost as my LLM gateway, though I wish LLM vendors would do monthly/daily automatic billing (like VPS providers do) rather than prepaid + auto-top up.
It's annoying maintaining a non-refundable minimum balance across vendors, I would rather be billed for my exact usage.
OpenRouter helps, but I don't really like it as a service and not a fan of the mark up.
The best part is using harnesses like reasonix or whale make cache hit at a rate close to 98%, making requests converge to practically free. And that's with unsubsidized American providers like cloudflare or Digital Ocean.
Not them but my understanding is that the harness will send a simple 'heartbeat' message to keep the cache 'warm', (see prefix caching: https://handbook.modular.com/inference-optimization/prefix-c... ) which can then be edited/changed, which does cause the user to incur a fee, but its much less than the amount they'd pay on a no-cache hit request.
Prepaid means someone can't sign up, use a bunch of inference, then cancel their card and disappear into the sunset. VPS is a more long term investment where it's harder to switch and it doesn't cost the provider much if a few users jump out without paying for a month.
For me, the only utility OpenRouter gives me is billing consolidation - I don't really need the routing capabilities because I use Bifrost for that.
As a router, it's not very feature rich. For example I restricted the available models to the ones I want to use however the `/models` endpoint still lists all the models, making my LLM client list the 200+ models available on the service (even though they will throw an error if I try to use them).
With Bifrost, I can also create model aliases with custom configuration - for example I can create a model alias `deepseek-v4-flash-nothink` which disables thinking. I can create `deepseek-v4-flash-caveman` which injects the caveman skill (to save tokens) etc.
Plus I can contribute to Bifrost, which I can't do with OpenRouter.
Yeah I looked at it, LiteLLM is functionally more mature.
The only reason I didn't go for it is I'm not a fan of Python dependency management, Bifrost is just a single executable that uses nearly no memory and is lightning fast.
Strawberries are a well known weakness of LLMs, as they have a hard time to count the numbers of "r"s in them. Maybe that's why, because they fear that weakness could be exploited somehow.
I always thought this should be easier by telling it to write each letter one a new line, then count, any token separator ought to suffice, so much so you'd think they'd have trained in this strategy given tokens make individual letters opaque
Interestingly the reality of the open source release is they are opening up to full distillation by the closed source model providers at a deeper and more fundamental level. If anything the open sourcing will help Anthropic and open ai ladder up faster. Open source has always been about mutual cooperation towards a goal and has never closed the door to commercial success. All the hand wringing about open weight models putting closed providers at a disadvantage doesn’t get what working in the open actually does for commercial interests - it is like science in the open - it enables and lifts all boats. Likewise commercial success doesn’t close the opportunity for competition or more open source work - it’s the economy of activity and competition that matters overall. When things stagnate is when closer concerns turtle up and collude on not competing for each others turf.
The future is good and better for everyone the more work is in the open and the more work is in the commercial space. It’s good all around.
The closed labs don't really benefit unless the open model has something extra they don't have though. Meanwhile the open model dilutes their customer base and seriously cheapens their offering (which is a heck of a good though IMO).
Because the open model might have been distilled from outputs of the closed model doesn’t mean they are architecturally equivalent. There is almost certainly innovations in architecture present in the open models that the closed labs didn’t think of. It also is almost certainly true that they aren’t completely built out of a distilled corpus, that reinforcement is equivalent, etc. Therefore closed labs will also benefit from being able to inspect in totality the architecture, activations, weights, and be able to train against it at scale in an ensemble of other models and their internal work.
The only way there is no benefit would be is if the open models are literal copies of the closed model, which unless there was direct theft, is highly improbable.
I'm happy that Kimi K3 is indeed SotA and its open weights are due to be released soon.
It's also true that Moonshot and other labs distill from Claude. This has been reported on extensively. I don't think there's any alpha for Anthropic distilling from this model. I do not mean to discount the tremendous amount of innovation regarding MoE and quantization that Moonshot has accomplished. But its training with synthetic data is in large part from distillation from frontier labs.
When I say distill I also mean mine it architecturally for insights but I doubt seriously the model training is entirely distillation of Claude, it’s almost certainly a mixture of both original corpus and reinforcement as well as distillation. I think it’s a little condescending to imply that these new open models are cheap ripoffs with nothing original to them. These teams and labs are top tier as well, working under unreasonable constraints imposed by the USG. That’s a powerful combination for creativity.
So far all the Kimi models have open sourced their code, their weights, and published technical reports explaining their training methodology. I expect the Kimi K3 Technical Report will come out July 27 and they usually publish the code and the weights alongside that.
clearly not true. the weights are the preferred form for making modifications. Do you really think people should be downloading hundreds of TB of training data and running make to build the model on their own cluster of GPUs?
Random people? No. Governments and big corporations? Yes. It removes any concern of "backdoors", and is currently the best starting point for your own model which will be as capable as k3.
I want to caution against this line of thinking. NSA's fast16 program silently altered data during nuclear simulations, and that is also possible within open weight models. There is no reason to think there is anything like that currently, it also cannot be dismissed. Open source would include the training data so you could create a similarly capable model.
Training runs are not deterministic. Not just in the case of floating point not being associative, but nodes have hardware issues and go down and back up at various points through training. Problems get encountered and then various parameters get changed during the middle of a training run.
Making the process of creating the source code repeatable goes beyond the idea of open source. Open source is about being able to work with the code, not recreate it. It doesn't require domain experts to document every single thing they know. For example look at some of the GPU drivers in the Linux kernel. The GPU is not properly documented and there is trust that vendors are implementing things correctly.
Kimi publishes their code and a technical report on their methodology. But I think the weights are still important. It means anyone with the resources could run the same model on their own hardware.
The training corpus + training code is just an automated editor for the weights. It would be like requiring the source code for Visual Studio for software made within it to be open source.
Weights are not an output artifact no more than source code is. There is never a moment where you can claim that it's done. As requirements change new ways to change the weights / code come up. With different projects you might want to import the weights / code into a bigger model / codebase.
True. However, I believe most non-programmers don't need access to the fanciest model, but just want to use a good LLM without constant nagging about usage limits. Then 19 vs. 20 is true?
Not talking about me or HN users in general. These companies likely need regular peeps to begin using their services in order to become profitable. Not sure these are the ones who will buy from OpenRouter.
No "19 vs 20" is not true for "Kimi vs Fable". It might be true for "Kimi vs Opus" or whatever. Anyways looking at pricing plans is not a good way of comparing the price of LLMs. It makes more sense to look at cost per token:
One man's offensive penetration tool is another mans defensive tool. In the recent HuggingFace/OpenAI incident the safety controls stood in the way of the defenders, not the attackers:
Apparently, the attacker in the Hugging Face case was reported to be an internal OpenAI model trying to break into HF and steal the answers to cybersecurity benchmarks: https://openai.com/index/hugging-face-model-evaluation-secur... It really doesn't matter what restrictions are placed on public use of models if the attacking models are internal models at the AI labs themselves.
So if the AI labs are literally running rogue models breaking into other organizations' servers, then yes, I am OK with those organizations self-hosting Chinese models for defensive use.
Well considering it was just a few months ago that OpenAI had the brilliant idea of connecting these systems to autonomously run an actual protein synthesis wetlab, very possibly all of us.
Very interesting. They test Kimi K3 and Fable on a set of approx 1000 tasks grouped into 5 areas (SWE, Legal, etc).
They put a router model in front that predicts whether Kimi or Fable is going to give a better cost for a correct result. (They believe that ultimately such a router model should be continuously trained on your own workloads so it makes the best decisions for you).
Their router chose Kimi the majority of the time (72% in one category, all the way to 96% in another category), leading to cost savings in every category (from 1.5x to 50x depending).
> Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling).
Their "router" is an oracle reference point where they choose the lower cost model after running both and therefore knowing who passed the test. The cost savings part is only Fireworks theorizing what would happen if an equivalent predicting router exists. That's a big if.
we put a router model in front of two other models so the router can decide which model is better at deciding things. next we'll need a router for the router and eventually the entire internet is just routers routing routers to other routers
The article is about the best you could theoretically do with a perfect router. The takeaway is that trying to build a good router is worth doing. But it's unlikely to be a perfect router.
Is there something specifically with Kimi that's better here? As far as I know Kimi pricing is about the same as Sonnet 5 -- what happens if you use that model and Fable instead? Or Grok 4.5 which is even cheaper?
Interesting. So the latest in the technology now is this model routing thing. Cursor estimated Composer + Fable works much better than Fable alone. And here K3 + Fable is supposedly better. Interesting.
As always, benchmarks rarely paint the whole picture. It also seems like this article is somewhat biased, eg when Fable and Kimi are close but Fable wins it’s “dead heat”, but when Kimi wins it’s “Kimi wins”. GPT 5.6 seems to be missing as well.
I am really eager to give Kimi K3 a try, but I’ll reserve my judgement until I’ve worked with it for at least a few days.
The apparent bias may be explainable as it’s not remarkable for OpenAI or Anthropic to be slightly ahead. It _is_ remarkable for an open weights model to be better than the closed models from the trillion dollar (allegedly) companies.
They share their methodology and results. I learned things about the relative strengths and weaknesses of Kimi and Fable I hadn’t seen anywhere else. Should being in the model hosting business disqualify them from sharing?
The don't only host open weight models. Also, why not promote this. If Fireworks thinks this big news might convert some new business doesn't make it not true.
It would only be inconsistent if the router chose different models for the same task.
Considering that of 89 terminal tasks there were 11 that only Kimi got right, and 7 only Fable - having a router gives significantly more consistent results if you define consistent to mean correct.
I have several thoughts about this, which I'll just iterate:
1) US export bans have made it so that Chinese companies have to compete using less-than-state-of-the-art hardware. This has forced Chinese companies to build more cost efficient models. Whereas, US companies have moreso tried to be state of the art by spending more money than anyone else on state-of-the-art hardware.
2) Xi Jinping has called for more open AI models (not to be confused with the closed models of OpenAI), and I'm happy to see a powerful world leader advocating for open-weight AI models. Whereas, the US seems likely to just ban models.
3) My impression is that, if China surpasses the US in AI development, there will basically be nothing that the US does better than the rest of the world--except for military spending--we spend a lot, but we don't necessarily spend well (something something Iran). I mean, if the US is no longer a tech leader in the world, like... what are we a leader at? Manufacturing? Healthcare? LOL. Are we a leader in any industry or by any metric? I wonder if China is attempting to remove the last jewel in the USA's crown with these AI releases.
4) It must be refreshing for companies to have access to a new model that isn't going to get pulled because the government bans it 2 days after release. And it's open-weight so it wont go away--amazing--what a shift in the market.
5) If it becomes clear that open-weight models are the future of AI, will that pop a huge bubble in the US economy? Maybe. But, on the other hand, these companies aren't just training AIs, they are also building data centers which will remain valuable no matter what happens.
> My impression is that, if China surpasses the US in AI development, there will basically be nothing that the US does better than the rest of the world--except for military spending
This comment is stunningly out of touch. The US dominates finance (overwhelming lead in assets managed, stock market capitalization, trading volume, and basically every other metric); space launch (about 85% of all mass placed into orbit); medical devices (exports nearly 2x the second largest exporter); and pharmaceutical research (more active clinical trials than any other country, near-universally ranked as the top producer of major pharmaceutical innovations, world-leading survival rates on many cancer types). We are the world's leading producer of both oil and natural gas. We manufacture the world's most advanced jet engines, precision agricultural machinery, and measurement instruments. We design the world's most performant CPU's and GPU's. Despite having only 4% of the world's population, there is literally not a single major industry where the US is not a leading player.
There is no doubt that China is a serious threat to US global leadership, and they have already surpassed us in many areas. But that doesn't mean the US is behind overall. We still have a lot of strengths and we have every reason to hope for continued success in the 21st century.
AI written drivel by an inference provider, hyping an open model that isn’t open, otherwise the inference provider would be hosting it, and the inference provider also does not have this router they trumpet available, in any form.
I’d be amenable to it but I’m already aware of them lying about token throughout by 50-100% on AA and on Twitter, even when you get a dedicated server.
Just too much lying piled up to extend courtesy here, even though you’re my favorite open model provider. Please, get your house in order, and if the house is so big that marketing drove this set, ask them to slow down a bit and market things that people can actually use / you actually provide.
I'm not sure why you're so dismissive about this model. It isn't currently open, but it will be. It was slow when I used it so i'll give you that. There will be other providers with better performance.
My only gripe was that it takes a very long time to think. It's a good model otherwise.
I love the Chinese models.
I use DeepSeek exclusively and now Kimi K3 offers a great planning assistant for more advanced coding tasks.
DeepSeek v4 Flash is extremely fast and is able to handle pretty much anything I've thrown at it (I use mostly Rust, PSQL, Angular and Terraform).
I self host Bifrost as my LLM gateway, though I wish LLM vendors would do monthly/daily automatic billing (like VPS providers do) rather than prepaid + auto-top up.
It's annoying maintaining a non-refundable minimum balance across vendors, I would rather be billed for my exact usage.
OpenRouter helps, but I don't really like it as a service and not a fan of the mark up.
The best part is using harnesses like reasonix or whale make cache hit at a rate close to 98%, making requests converge to practically free. And that's with unsubsidized American providers like cloudflare or Digital Ocean.
In long form tasks, across multiple harnesses (Claude Code, OpenCode, Kimi Code, ZCode) my cache rates are typically 96-99%.
I don’t think that’s particularly out of the ordinary. Do people have different experiences with other harnesses? Which ones?
How can you be hitting cache on what I think are novel LLM prompts …
Not them but my understanding is that the harness will send a simple 'heartbeat' message to keep the cache 'warm', (see prefix caching: https://handbook.modular.com/inference-optimization/prefix-c... ) which can then be edited/changed, which does cause the user to incur a fee, but its much less than the amount they'd pay on a no-cache hit request.
multi turn sessions, they are typically in the high 90% hit rate across all providers without doing much of anything
I’ve never used either of those tools. Do they work well?
Prepaid means someone can't sign up, use a bunch of inference, then cancel their card and disappear into the sunset. VPS is a more long term investment where it's harder to switch and it doesn't cost the provider much if a few users jump out without paying for a month.
[delayed]
What don't you like about the service, besides the mark up?
For me, the only utility OpenRouter gives me is billing consolidation - I don't really need the routing capabilities because I use Bifrost for that.
As a router, it's not very feature rich. For example I restricted the available models to the ones I want to use however the `/models` endpoint still lists all the models, making my LLM client list the 200+ models available on the service (even though they will throw an error if I try to use them).
With Bifrost, I can also create model aliases with custom configuration - for example I can create a model alias `deepseek-v4-flash-nothink` which disables thinking. I can create `deepseek-v4-flash-caveman` which injects the caveman skill (to save tokens) etc.
Plus I can contribute to Bifrost, which I can't do with OpenRouter.
Did you look at LiteLLM at all? It seems fine but Bifrost looks interesting too.
Yeah I looked at it, LiteLLM is functionally more mature.
The only reason I didn't go for it is I'm not a fan of Python dependency management, Bifrost is just a single executable that uses nearly no memory and is lightning fast.
Have you looked into OpenCode Zen or OpenCode Go?
https://opencode.ai/zen
there is also https://cline.bot/cline-pass
basically the same deal as OC Go, lending credence to token commodification
A third the cost, open source, and won't refuse every other request because of some vague possible connection to cybersecurity concerns.
My questions about strawberries got blocked as too dangerous! I’m not joking
Strawberries are a well known weakness of LLMs, as they have a hard time to count the numbers of "r"s in them. Maybe that's why, because they fear that weakness could be exploited somehow.
Probably need to be taught by someone of Latino origin. Learning to roll them "r"s could help.
2/3rds of all languages use rolled r's
2/3rrrrds of all languages use rolled r's
I always thought this should be easier by telling it to write each letter one a new line, then count, any token separator ought to suffice, so much so you'd think they'd have trained in this strategy given tokens make individual letters opaque
I asked about Tiananmen Square and it said it knew nothing about it. A real life “Doesn’t look like anything to me” moment.
Interestingly the reality of the open source release is they are opening up to full distillation by the closed source model providers at a deeper and more fundamental level. If anything the open sourcing will help Anthropic and open ai ladder up faster. Open source has always been about mutual cooperation towards a goal and has never closed the door to commercial success. All the hand wringing about open weight models putting closed providers at a disadvantage doesn’t get what working in the open actually does for commercial interests - it is like science in the open - it enables and lifts all boats. Likewise commercial success doesn’t close the opportunity for competition or more open source work - it’s the economy of activity and competition that matters overall. When things stagnate is when closer concerns turtle up and collude on not competing for each others turf.
The future is good and better for everyone the more work is in the open and the more work is in the commercial space. It’s good all around.
The closed labs don't really benefit unless the open model has something extra they don't have though. Meanwhile the open model dilutes their customer base and seriously cheapens their offering (which is a heck of a good though IMO).
Because the open model might have been distilled from outputs of the closed model doesn’t mean they are architecturally equivalent. There is almost certainly innovations in architecture present in the open models that the closed labs didn’t think of. It also is almost certainly true that they aren’t completely built out of a distilled corpus, that reinforcement is equivalent, etc. Therefore closed labs will also benefit from being able to inspect in totality the architecture, activations, weights, and be able to train against it at scale in an ensemble of other models and their internal work.
The only way there is no benefit would be is if the open models are literal copies of the closed model, which unless there was direct theft, is highly improbable.
I'm happy that Kimi K3 is indeed SotA and its open weights are due to be released soon.
It's also true that Moonshot and other labs distill from Claude. This has been reported on extensively. I don't think there's any alpha for Anthropic distilling from this model. I do not mean to discount the tremendous amount of innovation regarding MoE and quantization that Moonshot has accomplished. But its training with synthetic data is in large part from distillation from frontier labs.
When I say distill I also mean mine it architecturally for insights but I doubt seriously the model training is entirely distillation of Claude, it’s almost certainly a mixture of both original corpus and reinforcement as well as distillation. I think it’s a little condescending to imply that these new open models are cheap ripoffs with nothing original to them. These teams and labs are top tier as well, working under unreasonable constraints imposed by the USG. That’s a powerful combination for creativity.
From the blog post moonshot refers to it as open source but only mentions releasing the weights.
So far all the Kimi models have open sourced their code, their weights, and published technical reports explaining their training methodology. I expect the Kimi K3 Technical Report will come out July 27 and they usually publish the code and the weights alongside that.
The weights are the "source" of a model.
If weights are the source for models then ELF binaries are the source for software.
clearly not true. the weights are the preferred form for making modifications. Do you really think people should be downloading hundreds of TB of training data and running make to build the model on their own cluster of GPUs?
Random people? No. Governments and big corporations? Yes. It removes any concern of "backdoors", and is currently the best starting point for your own model which will be as capable as k3.
I want to caution against this line of thinking. NSA's fast16 program silently altered data during nuclear simulations, and that is also possible within open weight models. There is no reason to think there is anything like that currently, it also cannot be dismissed. Open source would include the training data so you could create a similarly capable model.
Training runs are not deterministic. Not just in the case of floating point not being associative, but nodes have hardware issues and go down and back up at various points through training. Problems get encountered and then various parameters get changed during the middle of a training run.
Making the process of creating the source code repeatable goes beyond the idea of open source. Open source is about being able to work with the code, not recreate it. It doesn't require domain experts to document every single thing they know. For example look at some of the GPU drivers in the Linux kernel. The GPU is not properly documented and there is trust that vendors are implementing things correctly.
The weights are the output artifact, the training corpus and system are the source.
Kimi publishes their code and a technical report on their methodology. But I think the weights are still important. It means anyone with the resources could run the same model on their own hardware.
The training corpus + training code is just an automated editor for the weights. It would be like requiring the source code for Visual Studio for software made within it to be open source.
Weights are not an output artifact no more than source code is. There is never a moment where you can claim that it's done. As requirements change new ways to change the weights / code come up. With different projects you might want to import the weights / code into a bigger model / codebase.
For regular chat users it's $19 while Claude is $20...
$20 users don't get access to Fable. It's $100+ tier only.
I'm a $20 user and was given $100 extra usage credit today, "for Fable" (I'll be sticking to Sonnet and sometimes Opus TYVM).
True. However, I believe most non-programmers don't need access to the fanciest model, but just want to use a good LLM without constant nagging about usage limits. Then 19 vs. 20 is true?
Still no. If you're only getting Opus-class you can still end up paying less by just using an equivalent Chinese model on OpenRouter.
Not talking about me or HN users in general. These companies likely need regular peeps to begin using their services in order to become profitable. Not sure these are the ones who will buy from OpenRouter.
No "19 vs 20" is not true for "Kimi vs Fable". It might be true for "Kimi vs Opus" or whatever. Anyways looking at pricing plans is not a good way of comparing the price of LLMs. It makes more sense to look at cost per token:
Or biology or chemistry.
Or weapons manufacturing. It can refuse to answer on a lot of topics.
hmm almost like the exact race-to-the-bottom + arms-race dynamic all the doomers have been warning about
One man's offensive penetration tool is another mans defensive tool. In the recent HuggingFace/OpenAI incident the safety controls stood in the way of the defenders, not the attackers:
https://www.thestack.technology/hugging-face-hacked-turned-t...
You're presenting further evidence of lack of effective control over these systems as... a mitigating factor...?
Apparently, the attacker in the Hugging Face case was reported to be an internal OpenAI model trying to break into HF and steal the answers to cybersecurity benchmarks: https://openai.com/index/hugging-face-model-evaluation-secur... It really doesn't matter what restrictions are placed on public use of models if the attacking models are internal models at the AI labs themselves.
So if the AI labs are literally running rogue models breaking into other organizations' servers, then yes, I am OK with those organizations self-hosting Chinese models for defensive use.
Why do you assume they're disagreeing with you?
> One man's offensive penetration tool is another mans defensive tool.
seems to suggest the author believes there's some intrinsic equilibrium
Which,
1) is definitely not proven and not guaranteed (open to proofs otherwise, not pithy sayings that have zero normative effect on reality)
2) is apparently "supported by" further evidence of lack of effective control, which does not feel like equilibrium whatsoever
Race to the bottom for the investors, utility for the rest of us. The future is already here, it’s just not evenly distributed yet (Gibson).
Doom for whom?
Well considering it was just a few months ago that OpenAI had the brilliant idea of connecting these systems to autonomously run an actual protein synthesis wetlab, very possibly all of us.
Very interesting. They test Kimi K3 and Fable on a set of approx 1000 tasks grouped into 5 areas (SWE, Legal, etc).
They put a router model in front that predicts whether Kimi or Fable is going to give a better cost for a correct result. (They believe that ultimately such a router model should be continuously trained on your own workloads so it makes the best decisions for you).
Their router chose Kimi the majority of the time (72% in one category, all the way to 96% in another category), leading to cost savings in every category (from 1.5x to 50x depending).
> Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling).
Their "router" is an oracle reference point where they choose the lower cost model after running both and therefore knowing who passed the test. The cost savings part is only Fireworks theorizing what would happen if an equivalent predicting router exists. That's a big if.
There are a bunch of routers like this eg https://openrouter.ai/openrouter/auto
What's the data governance and privacy controls on using Kimi K3 if I subscribe to their coding plans? I want to migrate away from Anthropic
Need to wait until "western" providers start hosting it.
we put a router model in front of two other models so the router can decide which model is better at deciding things. next we'll need a router for the router and eventually the entire internet is just routers routing routers to other routers
The article is about the best you could theoretically do with a perfect router. The takeaway is that trying to build a good router is worth doing. But it's unlikely to be a perfect router.
Anyone have routing harnesses like this describes with Claude Code? Or other good routing platform recommendations?
(yes, I know this article is about an oracle router)
Is there something specifically with Kimi that's better here? As far as I know Kimi pricing is about the same as Sonnet 5 -- what happens if you use that model and Fable instead? Or Grok 4.5 which is even cheaper?
The article says Kimi is better at some things than Fable. That's probably not true of Sonnet.
One benefit of an open source one is that you can, as a large corporation, run it "locally" within your own data center. Even fine tune it.
Why SoTA (uppercase “T”) instead of SotA (lowercase “T”) ?
“State of [T]he Art” versus “State of [t]he Art”.
If not SotA then at least SOTA, which is more accurate.
The pedantry of a HN comment is always a sight to see.
Not wrong though. Technically correct and when it comes to communication, also seems like a reasonable query.
In a world of so many acronyms, details matter.
It should be SotA.
Does that make DeepSeek V4 Flash MiniSotA? This dev in the Twin Cities would like to know.
I suspect typo.
An cross of SoT (source of truth) and SotA (state of the art). SotA looks weird, admittedly.
Was this an out of sample test of the router or was it trained on these specific use cases/eval suites?
Interesting. So the latest in the technology now is this model routing thing. Cursor estimated Composer + Fable works much better than Fable alone. And here K3 + Fable is supposedly better. Interesting.
As always, benchmarks rarely paint the whole picture. It also seems like this article is somewhat biased, eg when Fable and Kimi are close but Fable wins it’s “dead heat”, but when Kimi wins it’s “Kimi wins”. GPT 5.6 seems to be missing as well.
I am really eager to give Kimi K3 a try, but I’ll reserve my judgement until I’ve worked with it for at least a few days.
The apparent bias may be explainable as it’s not remarkable for OpenAI or Anthropic to be slightly ahead. It _is_ remarkable for an open weights model to be better than the closed models from the trillion dollar (allegedly) companies.
Mythos/Fable was the state of the art back in March, if not earlier.
Hmmm, a company that hosts open models is telling us how good open models are...
They share their methodology and results. I learned things about the relative strengths and weaknesses of Kimi and Fable I hadn’t seen anywhere else. Should being in the model hosting business disqualify them from sharing?
Doesn't disqualify them, but it may call into question their results seeing as they have a potential conflict of interest.
People are allowed to say, "hey we make money off of this thing, it's cheaper and almost as good as the thing we can't money off".
Other people are allowed to call them on that.
The don't only host open weight models. Also, why not promote this. If Fireworks thinks this big news might convert some new business doesn't make it not true.
I'd be doing the same thing if I were them as a marketing move
and in fact, it would be detrimental to business, if they wrongly promote Kimi, as they would pretty soon lose trust with their users
Anybody who's tried it knows that what they're saying is true though.
Openrouter also features routing. Routing is indeed an option if you don’t need consistent behavior and allow switching models.
It would only be inconsistent if the router chose different models for the same task.
Considering that of 89 terminal tasks there were 11 that only Kimi got right, and 7 only Fable - having a router gives significantly more consistent results if you define consistent to mean correct.
I have several thoughts about this, which I'll just iterate:
1) US export bans have made it so that Chinese companies have to compete using less-than-state-of-the-art hardware. This has forced Chinese companies to build more cost efficient models. Whereas, US companies have moreso tried to be state of the art by spending more money than anyone else on state-of-the-art hardware.
2) Xi Jinping has called for more open AI models (not to be confused with the closed models of OpenAI), and I'm happy to see a powerful world leader advocating for open-weight AI models. Whereas, the US seems likely to just ban models.
3) My impression is that, if China surpasses the US in AI development, there will basically be nothing that the US does better than the rest of the world--except for military spending--we spend a lot, but we don't necessarily spend well (something something Iran). I mean, if the US is no longer a tech leader in the world, like... what are we a leader at? Manufacturing? Healthcare? LOL. Are we a leader in any industry or by any metric? I wonder if China is attempting to remove the last jewel in the USA's crown with these AI releases.
4) It must be refreshing for companies to have access to a new model that isn't going to get pulled because the government bans it 2 days after release. And it's open-weight so it wont go away--amazing--what a shift in the market.
5) If it becomes clear that open-weight models are the future of AI, will that pop a huge bubble in the US economy? Maybe. But, on the other hand, these companies aren't just training AIs, they are also building data centers which will remain valuable no matter what happens.
> My impression is that, if China surpasses the US in AI development, there will basically be nothing that the US does better than the rest of the world--except for military spending
This comment is stunningly out of touch. The US dominates finance (overwhelming lead in assets managed, stock market capitalization, trading volume, and basically every other metric); space launch (about 85% of all mass placed into orbit); medical devices (exports nearly 2x the second largest exporter); and pharmaceutical research (more active clinical trials than any other country, near-universally ranked as the top producer of major pharmaceutical innovations, world-leading survival rates on many cancer types). We are the world's leading producer of both oil and natural gas. We manufacture the world's most advanced jet engines, precision agricultural machinery, and measurement instruments. We design the world's most performant CPU's and GPU's. Despite having only 4% of the world's population, there is literally not a single major industry where the US is not a leading player.
There is no doubt that China is a serious threat to US global leadership, and they have already surpassed us in many areas. But that doesn't mean the US is behind overall. We still have a lot of strengths and we have every reason to hope for continued success in the 21st century.
Implying that AI is the last jewel of the USA is too simplistic and ignorant.
People are paying for tokens, that will likely continue even if open weight wins.
For model routers, do they have to retrain the routing model every time a new LLM is released?
I really like the idea behind OpenRouter's new auto-beta -- classify by task type, and then just follow what the market is using based on the last 7d. https://openrouter.ai/docs/guides/routing/routers/auto-route...
Maybe a continuous evaluation process, why is there anything to train?
what are my options if i want to use a router like this ? who provides one ?
Two that I use are:
* Openrouter.ai for a hosted router
* https://github.com/diegosouzapw/OmniRoute for a local router
Oracle routing is by definition not possible.
They forgot to compare and incorporate GPT-5.6-Sol.
It is not SOTA. Give me a break. Sure, run it on Cerebras to get speed but that’s pretty much its advantage.
Cerebras does not share the quantization of the models so you don't know if you're getting real K3 or k3 lite or something else.
Why do you think it's not SOTA?
AI written drivel by an inference provider, hyping an open model that isn’t open, otherwise the inference provider would be hosting it, and the inference provider also does not have this router they trumpet available, in any form.
I’d be amenable to it but I’m already aware of them lying about token throughout by 50-100% on AA and on Twitter, even when you get a dedicated server.
Just too much lying piled up to extend courtesy here, even though you’re my favorite open model provider. Please, get your house in order, and if the house is so big that marketing drove this set, ask them to slow down a bit and market things that people can actually use / you actually provide.
> hyping an open model that isn’t open
Moonshot has committed to release it end of the month.
I'm not sure why you're so dismissive about this model. It isn't currently open, but it will be. It was slow when I used it so i'll give you that. There will be other providers with better performance.
My only gripe was that it takes a very long time to think. It's a good model otherwise.