> Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokens, a tokens per parameter of 1:500,000.
I don’t think that’s a fair comparison since (as far as I know) humans aren’t born with randomly initialised neural weights that are setup in a very generic structure, but have at the very least extremely strong inductive biases. I think it’s fair to count the hundreds of millions of years of evolution between the first animals with a primitive neural system and humans today, and the amount of “tokens” that were “trained on” is likely enormous. I think it’s better to consider the 200M tokens as a sort of “fine tuning” to a sort of genetic “pre-training”.
It's nice to throw around claims like that, or like "microtubules connect the brain the quantum realm" (Penrose). Fortunately that's not how science works.
> Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokens, a tokens per parameter of 1:500,000.
I don’t think that’s a fair comparison since (as far as I know) humans aren’t born with randomly initialised neural weights that are setup in a very generic structure, but have at the very least extremely strong inductive biases. I think it’s fair to count the hundreds of millions of years of evolution between the first animals with a primitive neural system and humans today, and the amount of “tokens” that were “trained on” is likely enormous. I think it’s better to consider the 200M tokens as a sort of “fine tuning” to a sort of genetic “pre-training”.
That 200M figure needs much more justification as well, humans recieve huge amounts of data through their senses, particularly vision and touch.
If those tokens were to be passed on they'd have to be encoded somewhere, like DNA. And in DNA, there's no room. So ... where are they?
The data is massively compressed and pruned. Where are the input tokens in an LLM?
DNA is probably the seed for the random initialization.
It's nice to throw around claims like that, or like "microtubules connect the brain the quantum realm" (Penrose). Fortunately that's not how science works.
Science starts with a hypothesis. If DNA has everything encoded, then it stands to reason that it influences the initial state of the brain.
Paper: https://arxiv.org/abs/2610.37842