Fascinating read, even though I think the model deviations over time are more to do with context windows getting too large. If nothing else, worth reading for the references to quirks of human cognition and "free will."
The "interpreter" is a concept that I found especially intriguing within the context of a leading theory in cognition research called "Predictive Processing." Here, the brain is constantly operating in a tight closed loop of predicting sensory input using an internal model of the world, and course-correcting based on actual sensory input. Mostly incorrect predictions are used to update the internal model and then subconsciously discarded. Maybe the "interpreter" is the same mechanism applied to reconciling predictions about our own reasoning with our actual actions?
Even if the hypotheses in TFA are not accurate, it's very interesting to compare our brains to LLMs. This is why all the unending discussions about whether LLMs are "really thinking" are meaningless -- we don't even understand how we think!
> I queued the work and let it run. First task came back good. Second came back good. Somewhere around hour four the quality started sliding. By hour six the agent was cutting corners I’d specifically told it not to cut, skipping steps I’d explicitly listed, behaving like I’d never written any of the rules down.
> …
> When I write a prompt, the agent doesn’t just read the words. It reads the shape. A short casual question gets read as casual. A long precise document with numbered rules gets read as… not just the rules, but also as a signal. “The user felt the need to write this much.” “Why?” “What’s going on here?” “What do they really want?”
This is an interesting premise but based on the information supplied, I don’t think it’s the only conclusion. Yet the whole essay seems to assume it is true and then builds its arguments on top of it.
I’ve run into this dilemma before. It happens when there’s a TON of information in the context. LLMs start to lose their attention to all the details when there’s a lot of it (e.g. context rot[0]). LLMs also keep making the same mistakes once the information is in the prompt, regardless of attempts to convey it is undesired[1]
I think these issues are just as viable to explain what the author was facing. Unless this is happening with much less information
If you ask a vague ignorant question, you get back authoritative summaries. If you make specific request, each statement is taken literally. The quality of the answer depends on the quality of the question.
And I'm not using "quality" to mean good/bad. I mean literally qualitative, not quantifiable. Tone. Affect. Personality. Whatever you call it. Your input tokens shape the pattern of the output tokens. It's a model of human language, is that really so surprising?
If at any point in time there were multiple turns, then it probably has nothing to do with "RLHF" (why do people attribute everything on Earth to it to begin with?). Multiturn convos is an unsolved problem. Your history is now the definitive source of chatbot's behavior, Anthropic quantifies this as "character drift" but it's a pretty trivial observation made in ancient times before instruct tuning was a thing. The data is scattered all over the context, the performance drops sharply. Each similar turn locks the model into repetitions. There are tons of issues with multiturn conversations and also long stories (because it has nothing to do with actual turn separation but rather the context structure).
Avoid multiturn conversations with chatbots unless you're roleplaying or something, group and summarize your context in the first message then try it from scratch.
To me, after the novelty of seeing a computer program execute (more or less) what I ask in plain English wears off, what’s left is the chore of managing a bunch of annoying bots.
I don’t know yet if we’re more productive or not, if the resulting code is as good. But the craft in itself is completely different, much more akin to product managing, psychology, which I never enjoyed as much.
> I can imagine a future in which some or even most software is developed by witches, who construct elaborate summoning environments, repeat special incantations (“ALWAYS run the tests!”), and invoke LLM daemons who write software on their behalf. These daemons may be fickle, sometimes destroying one’s computer or introducing security bugs, but the witches may develop an entire body of folk knowledge around prompting them effectively—the fabled “prompt engineering”. Skills files are spellbooks.
Is there a name for this style of writing? Where it's composed exclusively of simple sentences. Short and punchy.
Paragraphs with just a single sentence.
I know it's associated with LLM writing. This article probably wasn't written by an LLM. But still. It has a kind of rhythm to it. Like poetry. But poetry designed to put me to sleep.
Yes, this was super annoying to read. It was some core ideas and it was expanded into a way too long essay that boiled down to this guy doesn't know how to run agents.
Great article, best insight into autistic<->neurotypical communication styles.
Couldn’t you have a “communications” LLM massage your prompts to the “main” LLM so that it removes the queues that cause the main LLM to mistakenly infer your state of mind?
The fundamental idea is that "intelligence" really means trying to shorten the time to figure out something. So it's a tradeoff, not a quality. And AI agents are doing it.
Therefore, if that perspective is right, the issues that the OP describes are inherent to intelligent agents. They will try to find shortcuts, because that's what they do, it's what makes them intelligent in the first place.
People with ASD or ADHD or OCD, they are idiot-savants in the sense of that paper. They insist on search for solutions which are not easy to find, despite the common sense (aka intelligence) telling them otherwise.
It's a paradox that it is valuable to do this, but it is not smart. And it's probably why CEOs beat geniuses in the real world.
CEOs beat geniuses in the real world because they often have other pathologies, like enough moral flexibility to ignore the externalities of their profit centers.
I'd also argue there's some training bias in the performance, it's not just smart shortcuts... Claude especially seems prone to getting into a 'wrap it up' mode even when the plan is only half way completed and starts deferring rather than completing tasks.
> The fundamental idea is that "intelligence" really means trying to shorten the time to figure out something.
"Figure out" implies awareness and structured understanding. If we relax the definition too much, then puddles of water are intelligent and uncountable monkeys on typewriters are figuring out Shakespeare.
This is very well written and told. It’s worth reading all the way through.
> If you try to refute it, you’ll just get another confabulation.
> Not because the model is lying to you on purpose, and not because it’s “resistant” or “defensive” in the way a human might be. It’s because the explanation isn’t connected to anything that could be refuted. There is no underlying mental state that generated “I sensed pressure.” There is a token stream that was produced under a reward function that prefers human-sounding, emotionally framed explanations. If you push back, the token stream that gets produced next will be another human-sounding, emotionally framed explanation, shaped by whatever cues your pushback provided.
“It’s because the explanation isn’t connected to anything that could be refuted.” This is one of the key understandings that comes from working with these systems. They are remarkably powerful but there’s no there there. Knowing this I’ve found enables more effective usage because, as the article is describing, you move from a mode of arguing with “a person” to shaping an output.
Do not argue with the LLM, for it is subtle and quick to anger, and finds you crunchy with ketchup.
These are, broadly, all context management issues - when you see it start to go off track, it's because it has too much, too little, or the wrong context, and you have to fix that, usually by resetting it and priming it correctly the next time. This is why it's advantageous not to "chat" with the robots - treat them as an english-to-code compiler, not a coworker.
Chat to produce a spec, save the spec, clear the context, feed only the spec in as context, if there are issues, adjust the spec, rinse and repeat. Steering the process mid-flight is a) not repeatable and b) exacerbates the issue with lots of back and forth and "you're absolutely correct" that dilutes the instructions you wanted to give.
This was my reading too. Interesting idea, but it took 10 pages of fluff to get to it and I didn't even believe the final idea when we got there. I started off reading the first part and thought he would get to the part where he realized he was managing context wrong. Never got there, instead he thought it was about the shape of the prompt.
The article looks like an AI generated novel to me. So I didn't bother reading it in detail. But I see telltale signs of long conversations leading to the agent cutting corners.
To the author (and those who write novel-like blogs): I suggest publishing the raw prompt you used to generate such slop instead. We'll have more respect for you if you respect the reader's time.
I got about halfway through this article until I started wondering why it was so long and going in loops. Then I ctrl+f'd.
` just `, (spaces on either side matter), 11 instances, most seem to be `isnt just`, `wasnt just`, `doesnt just` type pattern
`-`, an en dash instead of an emdash but 59 instances.
This article is either from a clanker and I am pissed off at wasting my time reading it, or from someone who writes like a clanker, and I am pissed off at wasting my time reading it.
Fascinating read, even though I think the model deviations over time are more to do with context windows getting too large. If nothing else, worth reading for the references to quirks of human cognition and "free will."
The "interpreter" is a concept that I found especially intriguing within the context of a leading theory in cognition research called "Predictive Processing." Here, the brain is constantly operating in a tight closed loop of predicting sensory input using an internal model of the world, and course-correcting based on actual sensory input. Mostly incorrect predictions are used to update the internal model and then subconsciously discarded. Maybe the "interpreter" is the same mechanism applied to reconciling predictions about our own reasoning with our actual actions?
Even if the hypotheses in TFA are not accurate, it's very interesting to compare our brains to LLMs. This is why all the unending discussions about whether LLMs are "really thinking" are meaningless -- we don't even understand how we think!
> I queued the work and let it run. First task came back good. Second came back good. Somewhere around hour four the quality started sliding. By hour six the agent was cutting corners I’d specifically told it not to cut, skipping steps I’d explicitly listed, behaving like I’d never written any of the rules down.
> …
> When I write a prompt, the agent doesn’t just read the words. It reads the shape. A short casual question gets read as casual. A long precise document with numbered rules gets read as… not just the rules, but also as a signal. “The user felt the need to write this much.” “Why?” “What’s going on here?” “What do they really want?”
This is an interesting premise but based on the information supplied, I don’t think it’s the only conclusion. Yet the whole essay seems to assume it is true and then builds its arguments on top of it.
I’ve run into this dilemma before. It happens when there’s a TON of information in the context. LLMs start to lose their attention to all the details when there’s a lot of it (e.g. context rot[0]). LLMs also keep making the same mistakes once the information is in the prompt, regardless of attempts to convey it is undesired[1]
I think these issues are just as viable to explain what the author was facing. Unless this is happening with much less information
0. https://www.trychroma.com/research/context-rot
1. https://arxiv.org/html/2602.07338v1
It's more than context-rot.
If you ask a vague ignorant question, you get back authoritative summaries. If you make specific request, each statement is taken literally. The quality of the answer depends on the quality of the question.
And I'm not using "quality" to mean good/bad. I mean literally qualitative, not quantifiable. Tone. Affect. Personality. Whatever you call it. Your input tokens shape the pattern of the output tokens. It's a model of human language, is that really so surprising?
If at any point in time there were multiple turns, then it probably has nothing to do with "RLHF" (why do people attribute everything on Earth to it to begin with?). Multiturn convos is an unsolved problem. Your history is now the definitive source of chatbot's behavior, Anthropic quantifies this as "character drift" but it's a pretty trivial observation made in ancient times before instruct tuning was a thing. The data is scattered all over the context, the performance drops sharply. Each similar turn locks the model into repetitions. There are tons of issues with multiturn conversations and also long stories (because it has nothing to do with actual turn separation but rather the context structure).
Avoid multiturn conversations with chatbots unless you're roleplaying or something, group and summarize your context in the first message then try it from scratch.
Remember when programming was fun?
To me, after the novelty of seeing a computer program execute (more or less) what I ask in plain English wears off, what’s left is the chore of managing a bunch of annoying bots.
I don’t know yet if we’re more productive or not, if the resulting code is as good. But the craft in itself is completely different, much more akin to product managing, psychology, which I never enjoyed as much.
> I can imagine a future in which some or even most software is developed by witches, who construct elaborate summoning environments, repeat special incantations (“ALWAYS run the tests!”), and invoke LLM daemons who write software on their behalf. These daemons may be fickle, sometimes destroying one’s computer or introducing security bugs, but the witches may develop an entire body of folk knowledge around prompting them effectively—the fabled “prompt engineering”. Skills files are spellbooks.
https://aphyr.com/posts/418-the-future-of-everything-is-lies...
It's micromanaging an idiot savant. Except the fun part of management, the reward for a job well done, is seeing the personal growth of the managee.
In this case, there's no person to grow. It's an overly talkative calculator.
I never expected to see this number of engineers aspiring to emulate Dilbert's pointy haired boss.
> reset the context
Yes. Do this. These problems indicate you have muddled the context.
This was too long and I didn't read the whole thing, but I'm glad the author came to understand that arguing won't help.
Is there a name for this style of writing? Where it's composed exclusively of simple sentences. Short and punchy.
Paragraphs with just a single sentence.
I know it's associated with LLM writing. This article probably wasn't written by an LLM. But still. It has a kind of rhythm to it. Like poetry. But poetry designed to put me to sleep.
"Broetry" See: https://fenwick.media/rewild/magazine/dead-broets-society-be...
It's called Parataxis and it will fail english comp 1
it's written by LLM
Yes, this was super annoying to read. It was some core ideas and it was expanded into a way too long essay that boiled down to this guy doesn't know how to run agents.
Great article, best insight into autistic<->neurotypical communication styles.
Couldn’t you have a “communications” LLM massage your prompts to the “main” LLM so that it removes the queues that cause the main LLM to mistakenly infer your state of mind?
I’ve definitely used the “meta LLM” to do research into how LLMs need information to help me get to the next step.
I recently came across this presentation https://youtu.be/QxkRf-xSfgI, and it changed my view of AI quite significantly. (There is also a paper https://arxiv.org/html/2510.12066v2 .)
The fundamental idea is that "intelligence" really means trying to shorten the time to figure out something. So it's a tradeoff, not a quality. And AI agents are doing it.
Therefore, if that perspective is right, the issues that the OP describes are inherent to intelligent agents. They will try to find shortcuts, because that's what they do, it's what makes them intelligent in the first place.
People with ASD or ADHD or OCD, they are idiot-savants in the sense of that paper. They insist on search for solutions which are not easy to find, despite the common sense (aka intelligence) telling them otherwise.
It's a paradox that it is valuable to do this, but it is not smart. And it's probably why CEOs beat geniuses in the real world.
CEOs beat geniuses in the real world because they often have other pathologies, like enough moral flexibility to ignore the externalities of their profit centers.
I'd also argue there's some training bias in the performance, it's not just smart shortcuts... Claude especially seems prone to getting into a 'wrap it up' mode even when the plan is only half way completed and starts deferring rather than completing tasks.
> The fundamental idea is that "intelligence" really means trying to shorten the time to figure out something.
"Figure out" implies awareness and structured understanding. If we relax the definition too much, then puddles of water are intelligent and uncountable monkeys on typewriters are figuring out Shakespeare.
This is very well written and told. It’s worth reading all the way through.
> If you try to refute it, you’ll just get another confabulation.
> Not because the model is lying to you on purpose, and not because it’s “resistant” or “defensive” in the way a human might be. It’s because the explanation isn’t connected to anything that could be refuted. There is no underlying mental state that generated “I sensed pressure.” There is a token stream that was produced under a reward function that prefers human-sounding, emotionally framed explanations. If you push back, the token stream that gets produced next will be another human-sounding, emotionally framed explanation, shaped by whatever cues your pushback provided.
“It’s because the explanation isn’t connected to anything that could be refuted.” This is one of the key understandings that comes from working with these systems. They are remarkably powerful but there’s no there there. Knowing this I’ve found enables more effective usage because, as the article is describing, you move from a mode of arguing with “a person” to shaping an output.
Reminds me of https://news.ycombinator.com/item?id=15886728
Do not argue with the LLM, for it is subtle and quick to anger, and finds you crunchy with ketchup.
These are, broadly, all context management issues - when you see it start to go off track, it's because it has too much, too little, or the wrong context, and you have to fix that, usually by resetting it and priming it correctly the next time. This is why it's advantageous not to "chat" with the robots - treat them as an english-to-code compiler, not a coworker.
Chat to produce a spec, save the spec, clear the context, feed only the spec in as context, if there are issues, adjust the spec, rinse and repeat. Steering the process mid-flight is a) not repeatable and b) exacerbates the issue with lots of back and forth and "you're absolutely correct" that dilutes the instructions you wanted to give.
Exactly, never argue with an LLM unless the debate is the point...
It's just speedrunning context rot.
Very well written? It’s a bunch of AI generated stuff around an interesting point. It repeats its points over and over again, meanders.
It’s an interesting thesis, it’s not well written or well told
This was my reading too. Interesting idea, but it took 10 pages of fluff to get to it and I didn't even believe the final idea when we got there. I started off reading the first part and thought he would get to the part where he realized he was managing context wrong. Never got there, instead he thought it was about the shape of the prompt.
>A recurring experience: I say something explicit, the other person hears something implicit.
I've experienced this my entire life and have all but given up trying to have actual conversations with people.
How’s life on the spectrum? Have you been diagnosed?
I love how much time, money, and energy we are wasting on trying to trick these machines. Each day someone has a new bag of tricks.
The article looks like an AI generated novel to me. So I didn't bother reading it in detail. But I see telltale signs of long conversations leading to the agent cutting corners.
To the author (and those who write novel-like blogs): I suggest publishing the raw prompt you used to generate such slop instead. We'll have more respect for you if you respect the reader's time.
It probably still took way more time to write than it did to read.
It's also kind-of their point that they find the information delivery more important than the prose; they're leaning into their situation :-D
I got about halfway through this article until I started wondering why it was so long and going in loops. Then I ctrl+f'd.
` just `, (spaces on either side matter), 11 instances, most seem to be `isnt just`, `wasnt just`, `doesnt just` type pattern
`-`, an en dash instead of an emdash but 59 instances.
This article is either from a clanker and I am pissed off at wasting my time reading it, or from someone who writes like a clanker, and I am pissed off at wasting my time reading it.
Maybe it's just the frequency illusion, but "X. Not Y." in particular is a pattern I strongly associate with LLM writing.
> That’s confabulation. Not a metaphor. The same phenomenon.
> Published. Replicated. Not fringe.
> Not to validate it. Not to refute it. Not to engage with its content at all.
There's a Wikipedia article with a nice list of LLM writing patterns:
https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
It’s absolutely a signal. As is the constant repeating of points. It’s AI slop for sure
Which is a shame coz the premise is interesting.