Is the model "thinking of the last word for the next line already" or is "the word that came last in the second line was already showing high p from considering only the first line"? Shouldn't we care about such a distinction?
If “aim” is already highly probable after the first line, the example is much less interesting.
A more cool question is whether the model is carrying a latent representation of the destination that isn’t yet reflected in the immediate token probabilities?
Is the model "thinking of the last word for the next line already" or is "the word that came last in the second line was already showing high p from considering only the first line"? Shouldn't we care about such a distinction?
If “aim” is already highly probable after the first line, the example is much less interesting.
A more cool question is whether the model is carrying a latent representation of the destination that isn’t yet reflected in the immediate token probabilities?
I don’t know. Anthropic is investigating: https://www.anthropic.com/research/tracing-thoughts-language...