I don't know enough about the specific models they're comparing against to say this definitively, but it looks to me like they're comparing their pre-trained models with others' post-trained models.
The metric upon which their 10x claim is based (bits-per-byte) is exactly the metric which is optimized during pre-training. Post-trained models are fine-tuned to optimize other metrics, which is known to be detrimental to performance on bits-per-byte evaluations. So bits-per-byte evaluations will always make a pre-trained model look favorable in comparison to a comparable model which has also undergone post-training.
Can someone confirm whether the models they are comparing against (DeepSeek V4, Kimi K2, and Nemotron 3 Ultra) have been post-trained?
Cost will keep dropping, but the frontier will keep getting pushed. The whole "give me today's model 10x cheaper and I'm good" line is a fallacy. It isn't true now and it never will be for the top 1% of tasks, which will create the most economic gains.
this is basically the only thing pre-training teams work on in labs. compute efficiency is the metric, the assumption that scaling = intelligence is considered a given.
If you're like me, a SWE who is curious about ML/LLM training but unfamiliar with the terms, I got an agent to explain to me how to read the charts.
Basically, you can think of a LLM as a function which generates a probability distribution of words. If the next word in a series is "they", and one model predicts that word 40% of the time, and another model predicts that word 1% of the time, the latter model is worse as it is more surprised by the true distribution.
You can convert these probabilities into "bits":
surprise in bits = −log₂(probability of the actual token)
Probability of actual token,Surprise
1,0 bits
1/2,1 bit
1/8,3 bits
1/1024,10 bits
This is then normalised by text length:
Bits per byte = total next-token surprise in bits / number of bytes in the evaluated text
So the lower you go on the charts, the less surprises in the LLMs distribution (a better model).
Super awesome. Wish they would release the paper about what they did to achieve this. I remember nous released the token superposition paper which improved pretraining FLOPs some, but not 50x: https://nousresearch.com/token-superposition. Wondering if they also found some cool tokenization strategiesa
I don't know enough about the specific models they're comparing against to say this definitively, but it looks to me like they're comparing their pre-trained models with others' post-trained models.
The metric upon which their 10x claim is based (bits-per-byte) is exactly the metric which is optimized during pre-training. Post-trained models are fine-tuned to optimize other metrics, which is known to be detrimental to performance on bits-per-byte evaluations. So bits-per-byte evaluations will always make a pre-trained model look favorable in comparison to a comparable model which has also undergone post-training.
Can someone confirm whether the models they are comparing against (DeepSeek V4, Kimi K2, and Nemotron 3 Ultra) have been post-trained?
they say they're looking at base models, so I think it's fairly compared as written.
A lot of people are betting their money on infinite growth forever of AI performance, compute usage, user base, subscription price.
I think cost will decrease forever.
Cost will keep dropping, but the frontier will keep getting pushed. The whole "give me today's model 10x cheaper and I'm good" line is a fallacy. It isn't true now and it never will be for the top 1% of tasks, which will create the most economic gains.
this is basically the only thing pre-training teams work on in labs. compute efficiency is the metric, the assumption that scaling = intelligence is considered a given.
> We match DeepSeek V4 Pro Base using ~50x fewer FLOPs – that’s around half of GPT3’s pretraining compute, or ~$0.5M on GB200.
If this holds up that's a really big deal.
Huge if true. As it were.
If you're like me, a SWE who is curious about ML/LLM training but unfamiliar with the terms, I got an agent to explain to me how to read the charts.
Basically, you can think of a LLM as a function which generates a probability distribution of words. If the next word in a series is "they", and one model predicts that word 40% of the time, and another model predicts that word 1% of the time, the latter model is worse as it is more surprised by the true distribution.
You can convert these probabilities into "bits":
surprise in bits = −log₂(probability of the actual token)
This is then normalised by text length:Bits per byte = total next-token surprise in bits / number of bytes in the evaluated text
So the lower you go on the charts, the less surprises in the LLMs distribution (a better model).
For more info: https://smalldocs.org/s/DfvdGuFsiR3LlzXw1H5J0K#k=AJ8V1AQECYj...
not a good enough submarine
Super awesome. Wish they would release the paper about what they did to achieve this. I remember nous released the token superposition paper which improved pretraining FLOPs some, but not 50x: https://nousresearch.com/token-superposition. Wondering if they also found some cool tokenization strategiesa
Imagine the sheer amount of power you could save by releasing the paper.
But thanks to the Jevon Paradox, the global power consumption would probably increase.
https://en.wikipedia.org/wiki/Jevons_paradox
Solar is now not only the cheapest energy it's cheaper to intitially deploy than non-renewables.
Doesn't mean we should waste energy; it does mean that we have crossed a threshold beyond which energy concerns change shape.
Cool story. If it's true the company will be bought by open AI/Anthropic and Chinese labs will discover the trick and open source it by next quarter.