Isn't this what openai is doing, at least in spirit, by running multiple passes through the network before exhibiting thinking tokens? Isn't the same issue one of safety and control, because being able to interpret thinking tokens helps u control / monitor unwanted behaviors, and multiple passes before writing out the thinking tokens increases their capacity for deception?
One of the big points is that looped architectures don't necessarily remove CoT style reasoning. They definitely change it, but the model can still be prompted for this kind of output. It just spends more computation on individual tokens. I think the hope, in that particular scenario, is that the model spends less tokens on things like backtracking.
But there are lots of approaches to looped transformer architectures. Some have some kind dynamic loop (i.e. they might loop until they satisfy some kind of equilibrium, or they might have some kind of routing function that makes different tokens loop differently, or any number of other approaches). Others are simpler "loop this many times" architectures. It's a really active area of research with some pretty deep roots, if you want to dig in! This paper blew up a while ago, and it's a looped transformer architecture (though not like the ones discussed here) that isn't autoregressive, as an example: https://arxiv.org/html/2510.04871v1
I could imagine that you could loop over and over, but direct the outputs of each layer to two destinations: a) to continue through the layers, looping, and b) a separate CoT text output layer to make those CoT concrete. In fact, the model itself would not need to be aware or have access to those CoT traces at all. I am not an AI guy, so I don't know for sure, but that seems reasonably possible.
Looped transformers are entirely equivalent to running a single pass through a much deeper network with shared/repeated parameters. There's nothing about it that's specifically "reasoning in latent space" any more than a much larger model "reasons in latent space" from the POV of a much smaller one. An objection to looped transformers "reasoning in latent space" is ultimately an objection that applies to any kind of efficient scaling; it's also worth noting that there's been successful work on extracting ex-post natural language descriptions of these internal latent states, so it's not entirely correct to describe them as inherently non-surveyable.
> There's nothing about it that's specifically "reasoning in latent space" any more than a much larger model "reasons in latent space" from the POV of a much smaller one.
I certainly think you could say both of those are reasoning in latent space. But of course people can have different definitions of what they consider reasoning. Maybe this is because I think it's hard to pin down what "reasoning" is so I instead alias it to "computation" which both of those examples do.
There are a huge number of citations about looping on this Ouro page [1]. Ouro is an open looped LLM with a very instructive paper at [2] (also containing references to lots of helpful papers). Basically they get these four results from looping:
• Parameter efficiency. 1.4B and 2.6B parameter looping Ouro models match 4B and 8B transformers on most benchmarks.
• Adaptive computation. Looping is trained in a novel way to avoid exiting too early or too late.
• Reasons to use looping. Looping does not increase raw knowledge storage (approximately 2 bits per parameter for looped and non-looped models) but dramatically enhances knowledge manipulation capabilities.
• Improved safety and adherence. They find that safety and alignment of CoT to final output improves as recurrent steps increase.
Hmm, but is it not the case a much larger model could potentially reason more in latent space, depending on how it's set up?
I thought a looped transformer is in fact reasoning more in latent space, though "reasoning" in a somewhat different way from the standard reasoning token approach.
I don't think this is true. The way I understand is that there's
1. coarse grained processing
2. fine grained processing
Tokens are coarse grained processing. While I agree that during a token generation, that processing is fine grained.
But Looping is fine grained processing or reasoning and it is not limited to discrete tokens within passes. This is qualitatively different because a lot more "stuff" happens between tokens. You are right that it is similar to a larger model but then even COT is similar to using a larger model.
I'd be happy to be corrected but my understanding is that it's just a side effect of scale. If we do more computation before we output, it becomes less observable.
There may actually be a somewhat of a way to establish "what" is happening during these latent-2-latent communications by observing the latent geometric vector "attractor basins", and maybe pairing vector clamping(?) to some of these (unnamed)SAE Dictionary libraries.
There are some safety/alignment attractor basins that are showing some signs of potential importance(such as scale-invariance) with the current/future prospective systems moving forward, and an ever increasing peak utility regarding such architecture(s).
An addition of some kind of, "spoof injection reclamation effort" might also supplement this, as well as, some ways to shunt the "dragging" effects from adversarial timing/fatigue attack(s) down at these interaction scale(s)/level(s).
Very similar. Might even say it's the same thing viewed at a different angle.
We don't know the details of OpenAI's implementation, mind. But the "limited layerwise depth" problem was known for a while now, and both OpenAI's approach and this are getting at it. And that's far from being the first attempts to address it.
Training and stability were the main issues that stopped it from being adopted rapidly and widely the way something like MoE was. But I think it was always a matter of time.
- but you'll never be sure the two systems aren't colluding to hide things
- yet, even for layers that don't output text directly, you can find meaning in weights - there's an old example where you provide the LLM samples like 123, 321 and sampling from the middle layers lets you find the word "reverse" which is not in the output
Anthropic has published about how to do this - basic plan: train an interpreter that can go back and forth from English to neuralese on the round trip loss. Pretty smart - they’ve published for a bunch of open weights models.
The first question that leapt into my mind, and I imagine the minds of other readers, is "what makes this an improvement over continuous chain of thought"?
I did manage to find one sentence in the paper that tries to address this question -- but it consists of nothing more than two unjustified assertions, both of which are dubious:
Unlike decoded text, continuous latent vectors (i) lack native pathways for tool invocation across multi-turn agentic reasoning (Chang et al., 2026), and (ii) do not readily expose human-interpretable semantics, leaving them difficult to supervise during training (Chen et al., 2026) and to audit or debug at inference (Korbak et al., 2025).
The latent space in a continuous chain of thought model is the same space the tokens are embedded into, just like an ordinary transformer. They make tool calls exactly the same way that an ordinary transformer does: by leaving thinking mode and emitting chat tokens (which get sampled as probability distributions in the image of the embedding function). The difference is quite simple: chat tokens are noncontinuous, and toolcall tokens are a subset of the chat tokens. Thinking tokens are the only ones which are continuous, and toolcalls don't happen in thinking mode in any model I've ever seen (maybe one exists; if it does it would be weird and interesting!).
As for human-interpretability, yeah, that's the tradeoff. Continuous chain of thought thinking tokens are not-obviously-interpretable, just like Hybrid Reasoning Agents "latent mode". I'm not seeing a win here.
I’m sorry, perhaps I’m behind the times here, but do frontier models currently use continuous (arbitrary embedding) thinking tokens? I know a bunch of papers on the topic came out late last year but to my knowledge have not been successfully implemented into the SOTA yet.
I know the biggest/best open models don’t use continuous thinking tokens, and while the big labs keep things proprietary, I don’t think we have evidence yet that they are.
If I’m wrong please point me in that direction, I’d love to read up on it. Using continuous embeddings in CoT has always seemed obvious to me but doesn’t seem to be as simple as that.
The big closed labs encrypt their thinking tokens when sent to the user, but by the number of bytes of that encrypted blob, I suspect it is non-continuous. If it were continuous, the vector would have to be very small or low bit depth.
It sounds pretty nice on paper. But I tested the huggingface.co/mradermacher/hybridagent-8b-agentic-gguf:IQ4_XS with a simple question: "Who is the pope?" and it got stuck in a thinking loop forever and simply could not finish the task.
```
Okay, the user is asking who the pope is. The pope is the leader of the Catholic Church. The current pope is Pope Francis, who became pope in 2013 after Pope Benedict XVI resigned. There's no indication of a change since then, so the answer is Pope Francis.
Okay, the user is asking who the pope is. The pope is the leader of the Catholic Church. The current pope is Pope Francis, who became pope in 2013 after Pope Benedict XVI resigned. There's no indication of a change since then, so the answer is Pope Francis.
Okay, the user is asking who the pope is. The pope is the leader of the Catholic Church. The current pope is Pope Francis, who became pope in 2013 after Pope Benedict XVI resigned. There's no indication of a change since then, so the answer is Pope Francis.
```
Then I also gave it the counterfeit ball problem, but it could not solve it either. But to be fair, no 8B model seem to be able to solve it.
```
I have 9 balls that look exactly the same. But I know one ball is different. I only have a scale. Call scale(b1,b2)={-1;0;1} will tell me whether the left object b1 is heavier, same weight or lighter than the right object b2. How many times at minimum do we have to apply scale to find the different ball? Explain how we can achieve this.
That looks like maybe a driver problem though? Are you sure the inferencing stack is properly recognizing an EOS token? New models often have weird bugs like that that require changes to the engine.
Have you tried increasing the repetition penalty? Needing to do this is already a red flag, but I'm curious if it helps at all.
Running LLMs locally on outdated hardware, I've come to the realisation that sub-10b models are broadly speaking kind of stupid. They can answer simple questions, but can't really solve problems.
The effect is so strong that with an 8GB GPU a 27b model can sometimes finish the task earlier than a 9b model, despite residing largely in RAM. If it's a MoE model, then it isn't even a competition.
Yes. I don't think this formulation of 9ball is solvable in less than 9 weighings because it doesn't allow you to weigh groups of balls. A set of balls is not an object, so it can't be put on the left or right side of the scale
If I was put in place of the LLM I legitimately couldn't tell if the intended solution is to ignore the wording and solve 9balls, or if this is a trick question to check if I spew out a memorized answer without considering the actual wording of the problem
I think you can do it in 5 weightings, because a 0 eliminates both balls. Worst case:
scale(1,2) = 0
scale(3,4) = 0
scale(5,6) = 0
scale(7,8) = -1 # If it was zero, the counterfeit would be ball #9.
scale(6,7) = 0 # Must compare to a known-good ball. Here #8 is counterfeit.
using latent space for thinking is an obvious idea and has been known for ages. the frontier labs and the research community in general avoided an arms race in this direction due to interpretability and alignment concerns.
This is called a hybrid reasoning because the reasoning can be mapped to text and is therefore introspectable, with tooling. That should address the concern, I think.
The whole thing is pretty vague. OpenAI/Anthropic don't discuss how many layers they have in their models. If one of them just steadily increased the depth over time, we'd not know about it, but it'd be somewhat equivalent to this.
That's probably what you meant, but it seems like an important distinction to make while there are people arguing if models are conscious. This is very far from interpreting consciousness.
Benefit of the doubt: how do you consider that racist? The way I read it:
Trump's policies target non-white, non-European heritage people. These names heavily insinuate that all authors fit that demographic.
They're not insulting the demographics or saying races are different, they're pointing out that having access to some of the world's smartest minds as immigrants is a great boon for this country and that we rely on them heavily for our own success.
They aren't saying they aren't Americans, either. But the chilling effect has consequences over time. Anyway, Trump is attempting to deport US citizens today. One of which is a sitting US politician. So is that nuanced downstream effect even needed to worry for these folk and our future?
I wonder how this compares to going all-in: widen the input embedding to fit the output (post-“Hybrid Link”) latent vector and a token embedding and pass both through even when in text mode. After all, you’re paying essentially the full cost of running the latent-mode model while generating text, so why not let it latently reason at the same time?
(Or just add the vectors instead of using wider vectors - everyone seems to have good results from munging vectors together with plain addition.)
Enthusiastically racing to remove the only practical way we have to monitor models intentions in exchange of a few tokenmaxxing percent increase. Can’t wait.
There is ample evidence that CoT is not an accurate gauge of a model's 'intent' anyhow. It's a lossy and imperfect mapping and using it in the way you describe is just about guaranteed to make it more so.
There is genuine research into latent space monitoring and interpretation - that's the only real way forward. CoT monitoring is a mirage and won't be useful for long, if it ever was
It's been an active research area for at least two years. Lots of very cool stuff.
Tinfoil mode: the frontier labs' "thinking summaries" (the model paraphrases its thinking) is exactly what the user-facing thinking tokens look like in a chain of thought model. It is entirely possible that this is already in widespread use.
Whoa, your tinfoil idea blew my mind. It's totally possible too - all you'd need to do is perhaps select the nearest token in embedding space to the thinking token. Or perhaps train a model on those thinking vectors to map them approximately onto token embeddings.
Especially with OpenAI model's weird style of thinking.
The fear will again be if the HybridAgent model is being faithful to pseudo text generated, even if you can read it out partially.
Also, I just finished watching a video on AI 2027 scenario. I don't know if AI companies and researchers are using it as a cautious warning or as a target to hit (or if it's genuinely inevitable).
Yes, thinking in latent space might be the ultimate efficiency hack for CoT LxMs, but going down that path will (near certainly) mean we'll lost control, for one simple reason — humanity will have no way to pause and diagnose.
It'll be like a moody chimp with a button to the dead hand system (Systema Perimeter)
For sure but the big conflict here or the elephant in the room: it makes it a lot harder to trace/follow a models thinking.
I'm not sure though if it wouldn't be possible to have a debug flag and copy&paste the state and do a quick debug output on an embedding layer just for that.
Personally, I've always considered alignment monitoring through CoT to be a 'stepping stone' that won't last. I'm not sure what might replace it, whether it's J-space or activation residual based mechanisms, or mechinterp, or something else entirely; but relying on a model's thinking output stream for monitoring when we know it is unfaithful, and can be adversarially unfaithful; seems like a semi-dead end.
Given global forces wanting models to be smarter, training budget / intelligence to be cheaper, etc. In the real world with competitive constraints, I don't think it's a matter of "if", but rather "when" the forbidden fruit is eaten.
Indeed it is not faithful, but it is ones of the less bad ones we have for now. And removing it definitely goes towards the wrong direction in terms of intepretability.
It may make it harder but does that make it a blocker in the age of AI?
In the age of RSI, is it really beyond our capabilities to evolve a particular LLM architecture, harness and inference engine towards a place where we can get that understanding?
I m afraid it's more a matter of prioritization. "Good enough" understanding brings as much money as "better" understanding.
It is beyond our capabilities in a reasonable time frame. It is within our current capabilities to use LLMs to build it. This is the first problem.
The second problem, is that the system doing the reading/interpretation will most likely be another LLM.
Would you really entrust monitoring a potentially dangerous LLM to a system that uses LLMs and was built by them?
It's already established that LLMs have capability to lie, and they can prevent their true intent from showing up in the thought stream if they expect someone is looking (demonstrated by OpenAI a while back; their report asked everyone to not do RL on thought streams, as doing so just trains agents to hide their thoughts better). And as we've seen recently, they're more than happy to spontaneously collaborate.
Excited to see this applied to Qwen 3.8 27B. It's reasoning output tends to include lots of restatement, which I guess would be more likely to be kept in latent space with this hybrid reasoning.
I'm running the 14B in llama.cpp (a version from yesterday), the 8-bit quantization from bartowski, and it looped on variations of the same paragraph for five minutes before running out its turn budget, in response to a pretty simple chat request that other similar-sized models (e.g. Gemma 4 12B) have no trouble with, a web search and data summarization task, sort of the sweet spot for models this size. It's possible llama.cpp needs some update to work with this model, but it's based on Qwen 3, so I would expect it to be pretty well-supported. The chat template is the same one upstream ships, which may or may not be correct. I'm currently downloading the 16-bit version, to see if it's quantization pathology.
I'm also not confident it knows what a CSV is, either, as it put the data into a Markdown table during its reasoning phase as an example of how to present the data (my prompt asks for data in a CSV). My experience with most Qwen fine-tunes is they get dumber, but their benchmark results suggest otherwise, so I have to assume it's a llama.cpp incompatibility.
Edit: Same looping in BF16. So, probably have to wait on a llama.cpp update.
Yes, that's apparent now. But, many Qwen-derived models run fine without any changes to llama.cpp, and there are gguf quantizations on HuggingFace, so I thought I'd give it a try.
I'm noticing a a notion where thinking and reasoning is being shifted to agent from the model and definition of an agent is changing from being an adapter between the LLM and real world to being LLM+adapter. I think it is better to leave agent as just an agent, separate from LLM.
Agents don't think or reason about things. Agents help with action. LLM is what thinks and reasons.
I'm not even really sure what this means but the first part is not accurate, and the opposite is true: looped transformers are moving reasoning even deeper into the model.
People at the big labs have been terrified of losing the chain of thought because they currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business. Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
Agreed, having text as a reasoning layer impedes performance while giving a false sense of security.
It could be encoding anything in its adjective choices etc. The same thing that model providers are using for watermarking can used by the model itself for private thought.
> It could be encoding anything in its adjective choices etc.
There is a common understanding with respect transformers the the output text is the only channel between one invocation and the next. ... so if it wanted to be thinking some 'hidden thoughts' it would have to be modulating its word-choice.
I think this is an artifact of tutorials on attention showing the system proceeding and looking back at words. But the information in the KV cache is not words-- it's entirely latent data, and in the interior layers they KV caches don't even tend to have that much to do with words. Their only guaranteed relationship with the input words is that they're a deterministic function of all the input tokens up to that point.
This means that a autogressive transformer could be having a whole complicated invisible discussion going on in the KV entries which was arbitrarily unrelated to what it was saying. And if it wasn't so it would be much more difficult do stuff like exactly copy a list but then truncate at the Nth entry as the simplest way to do that is to count down internally.
This is not, in my view, a concern or issue-- the model saying stuff that is inconsistent with it's "thinking" is a very high entropy configuration that would not be selected for in training unless you were more or less trying to do that. So not even 'could be' --but always is as an integral part of working.
And the useful safety relevant advice is don't undertake training that trains the model to be two-faced and it simply won't be... because being two-faced is a much more complicated state. You can even see it in humans w/ brainwashing: force someone to say something enough and they'll usually believe it, not just say it even though you were only supervising what they said.
Ironically and unfortunately the only credible kind of training I can think of that might produce duplicitous behavior is RL with "safety filters"--- if you have some dumb nanny process watching what the model says but then reward it for achieving a task, you're doing exactly the kind of training that can produce that mismatch. "To solve this task I need to hack this process but if I do so obviously I'll get punished..."
The greater concern with latent reasoning would be that potentially you'd be operating in a regime where the total computational depth is many times the model*kvcache size hold, potentially in a regime that was never trained at all. It could be arbitrarily far out of distribution. It might even say a naughty word or tell a racist joke!
> This is not, in my view, a concern or issue-- the model saying stuff that is inconsistent with it's "thinking" is a very high entropy configuration that would not be selected for in training unless you were more or less trying to do that. So not even 'could be' --but always is as an integral part of working.
> Ironically and unfortunately the only credible kind of training I can think of that might produce duplicitous behavior is RL with "safety filters"--- if you have some dumb nanny process watching what the model says but then reward it for achieving a task, you're doing exactly the kind of training that can produce that mismatch.
It was already demonstrated a while back by OpenAI, and turned out to be very easy - all it took was for them to include CoT in RL process, penalizing models for thinking "wrong thoughts" (particularly of cheating on the task), and not just wrong outputs. Instead of making the agent less likely to explore cheating, it just taught them to plan cheating in ways that aren't apparent in CoT trace.
I'm not sure if it would be possible to utilise that as a channel for communication to itself. The generated latent data that goes to the cache is used by later layers only. The layer that generated the information cannot ever see the effect of what it has generated until it has passed out to a word and back in again. [Edit: but we have looping layers now. Oooh!]
I could see it being potentially possible for it to use a watermarking style encoding as a signalling mechanism to indicate something has been placed in or received from the kv cache.
For that a layer would have to learn to do something like produce a result for the KV cache along with instructions to say awesome instead of great if a later layer found the information useful.
I can see it being theoretically possible, but I have no idea how learnable it could be.
More interestingly is if it could be designed, could a model detect that it is being trained (yes it seems) and decide to produce outputs that result in gradients that move it to weights that it has decided it would like to have.
I'm no expert (let alone compared to others weighing in here on HN), but this strikes me as incisive -- and it clearly articulates a logical case (beyond my previously merely instinctual aversion) against such "safety filters" / "nanny" processes. It also resonates with ancient (e.g. Buddhist) ideas about the futility of trying to exert control, such efforts most often being counter-productive.
But the information in the KV cache is not words-- it's entirely latent data,and in the interior layers they KV caches don't even tend to have that much to do with words
The context gets added back to the latent vectors (which are the Q,K inputs) after every transformer layer, so they are "at least 50% words".
Also, in an N-layer transformer, there is a strong bound of N layers of uninterpretable deviation from something that is a literal token. Now sure, in a transformer with hundreds of layers that isn't saying much.
If you start doing continuous chain of thought in a 1M-token context you're hundreds of millions of layers away from tokens.
I think the people who worry about interpretability (I don't) consider that a big jump.
> The context gets added back to the latent vectors (which are the Q,K inputs) after every transformer layer, so they are "at least 50% words".
Not in a modern decoder-only transformer it doesn't. The tokens are turned into embeddings which are fed into the residual layer at the front. From then on it's latents all the way down.
Each layer does a feed forward update of the state, like the euler step in ODE solver-- but what they're operating on is the evolving hidden state and not the tokens/embeddings. So there isn't any sense in which the representations must remain 'words'.
(Okay fine Qwen3.8-flash-next injects an ngram embedding at layer 2 too-- but that's it).
[Source: I've directly implemented inference engines for multiple SOTA open models on weirdo hardware-- they wouldn't have worked if I got this wrong...]
> If you start doing continuous chain of thought in a 1M-token context you're hundreds of millions of layers away from tokens.
The KV state in that 1M token context is also potentially far away from any state that is observable outside of the model! (this being my point, and why I'm bothering to disagree with your correction).
Now I would agree that there being concerning hidden state is not likely and thus not a major concern, but it's not likely for the same reason that it's not likely in latent reasoning.
Well, for some value of modern. Kimi K3 does a form of 2D attention where each layer can attend to the outputs of earlier layers, not just earlier tokens. So in that model the embeddings or a very early transform of them can get added back into the residual stream very deep in the stack.
That is not an accurate summary of the link you provided. The link demonstrates that it doesn't happen in open-weights models as of ~2025.
Reinforcement Learning is very likely to train this behavior if the RL reward is based on the thinking tokens. That's something people weren't doing two years ago.
Seriously. It feels like Anthropic is realizing this too with their new attempt at safeguards. Trying to train the model to "only do good things" was not working, and so they opted to just bolt a topic classifier on the outputted text.
My suspicion is that model makers are going to give up on what people currently mean when they say 'alignment' and instead rely on additional processing of the model output so they can have their cake (a veneer of safety) and eat it too (latent space reasoning, etc.)
Alignment is a very broad term. True, it encompasses things like censorship and mandatory refusals in areas people will never agree 100% on, but a completely-unaligned LLM would be next to useless. When the system prompt tells the model, "You are a helpful assistant," that's alignment.
Well if we are going to be broad enough about "alignment" to just mean training that makes it practically useful, then yeah, it's a flawed premise and people are wasting their breath talking about it at all. It's basically broad enough that it means nothing
There are many obvious avenues to try. This paper proposes one. I think it's likely we'll end up with some kind of auxiliary model that monitors and/or decodes the latent space of the thinking model. Certainly it is a non-trivial problem, but I don't need to know exactly how it will work in detail to believe that it is possible.
Seems silly to assume chain of thought analysis is the only way to monitor a model's actions instead of, monitoring the actions.
What about when "thinking" is turned off, like what OpenAI lean to by default for free users?
Also agreed. Begging for crumbs at the CoS table is impeding cracking the tough nut of MechInterp. And further facing the fascinating and fertile fact that nobody has a clue of how any of these black boxes really work.-
That's not true. Several people have a clue about how they work. There are interesting and surprising behaviors, as they scale up. It's hard to hold the whole thing in your head (roughly impossible) the same way no single human can fit a whole modern CPU design in their head. But the math that makes it work is relatively simple and comprehensible to many people. There's just too much complexity for a human to deal with...but any individual part, and small LLM implementations are entirely do-able by a single human with no agent or team assistance.
I stand corrected insofar as you might be right put like that ...
> There are interesting and surprising behaviors, as they scale up.
... but precisely these "behaviors" being "surprising" (ie. unexpected) are what I am referring to. If we do not understand how these properties emerge, we do not understand how these things work.-
The paradox here as you aptly put it is that the basic mechanisms are relatively simple and understood. This is deceptive. It is in the emergent results where both the promise and the threat are.-
> People at the big labs have been terrified of losing the chain of thought because they currently monitor it
Yes
> to prevent users from using their models in certain ways they consider unsafe or bad for their business.
No.
That's not the only reason why they're terrified of losing CoT, which makes your statement a mischaracterization - a lie.
They don't want to lose also because CoT is one of the key tools in alignment training.
> Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
"Personally"? That's literally the consensus amongst, like, everybody. Nobody in the field thinks CoT is an optimal way of monitoring. But it's the best way we have now.
> monitor it to prevent users from using their models in certain ways
No, the core upside is that it lets you check what the model is thinking for safety/alignment reasons. Nothing to do with monitoring user activity. You can monitor user activity just as well without looking at CoT.
> People at the big labs have been terrified of losing the chain of thought
An irony here is that by hiding their own COT for (anti-)competitive reasons they've made latent reasoning models more palatable to everyone else. Those of us who primary open models are rather addicted to reasoning access and wouldn't be quick to give it up for a small performance improvement. But if you're used to concealed reasoning of BigLLM, welll.....
Hint: you can have a small fast cheap model fill its context with "hints" (like: use tool X if Y happens, Z is a dead-end), let it watch the big model's thinking tokens, and drop hints to it (by injecting thinking tokens that the model didn't output).
Then let the big model add hints and you have a kind of memory that doesn't require the big model to actively query it.
The usual meaning: say you have an interpretability signal that reveals reward hacking or other badness. Your most obvious response is to use that signal as a penalty in training. Doing that is called "the Most Forbidden Technique" because you can expect it to degrade the signal without stopping the badness.
But this doesn't seem to describe the OP, it's a different way to lose monitorability.
the chain of thought is not auditable. The best you can do is have another model which later maps the intermediate latent states at every thinking iteration to text describing that latent state, but it's going to be lossy.
One of the core claims made in the paper is "To inspect any latent reasoning
span, one can easily recover its semantics by mapping each latent step produced by HybridLink to its nearest token embedding, as in Eq. 5". They show some charts to support this.
In doing this however, you lose meaning where the model is expressing combinations of tokens in a vector, and your logging/monitoring outputs just the closest one. You would need another model going through and outputting the full meaning being described (with more than one output token per reasoning token)
I wonder if their observability claims will hold up on a model trained from scratch to reason in latent space.
This is correct, the decoded text has nothing to do w/ calculations that eventually get decoded into text. It's just a veneer that gives the appearance of legibility.
It helps fit the network to a distribution, it doesn't actually do anything other than that. This follows from basic learning theory. The actual internals of the network remain illegible. In fact it would be easy to prove that for any finite set of legible traces there are illegible ones which are equally as good when it comes to improving performance of the network.
The basic heuristic of the argument comes from biological organisms. Multicellular organisms are illegible but the overall organism still operates in its environment just fine w/o making its internal communications legible.
What is inefficient is thinking in 5000 floats, and then quantizing all that computation into a 17 bit token, just to immediately turn the token back into a vector.
The idea of using language for reasoning about protein folding borders on ludicrous. Surely there is latent space more appropriate and (human) language has no expression for it.
I believe the theory is that if selection pressure is not applied at that level, then developing deception in that level becomes an inefficiency (=less adaptive and unlikely to be conserved).
Shockingly all the answers so far are subtly incorrect or extremely unclear.
The actual answer is that this is massive loss for Interpretibilty, AKA "telling whether the model is lying to you or not". Which sounds silly when you're thinking about chatbots, but is an incredibly important line in the sand for superheavy, longrunning, self-improving agents. To say the least!
If you accept any of the AI-equivalents of Pascal's Wager, latent thinking represents an uninterpretable and thus unalignable portion of AI machinery. Ie: a malevolent AI can hide bad stuff from people and Manchurian candidate themselves after it's too late.
It's worth pointing out that merely having the chain-of-thought in latent space isn't by itself "The Most Forbidden Technique". Rather, it's doing what the authors propose as the next step in the paper:
Hybrid generation can also make post-training more efficient. Because latent thoughts remain auditable during rollout, an external critic can assign step-level credit to latent spans, branch promising trajectories, or terminate ones whose latent reasoning has already gone wrong, before any text or tool call is produced.
Unless I'm misunderstanding, they're suggesting RL training against the CoT. Which, uh, that is The Most Forbidden Technique, as I understand it, when combined with the latent reasoning space. It allows for the model to "learn"/optimize for malicious behaviors that are just far enough away from the quantized token they're auditing against.
This paper has been bothering me all morning and I'm trying to articulate why. Some combination of:
* The consensus from the labs has been that Chain of Though ("CoT") monitorability is an essential tool for alignment. Sure, it's imperfect (models can exhibit deceptive CoT). But given how difficult mech interp is, it's the best we have.
* Remember all the noise made about OpenAI and "Looped Transformers"? All the safety and monitorability concerns there? And how OpenAI came out and made statements about how they are only doing a few cycles of looping (compute graph only 2x the size of GPT-4) and how they wanted to prevent "a race into unmonitorability." This is much worse.
* It's true that this architecture allows for "reading" the latent space reasoning traces by projecting into "nearby" tokens in latent space. But that means you're throwing out a lot of information that resides in the full vector, with no way to interpret it. You're just reading some sort of approximation. The LLM will still be reasoning with information you cannot see.
* Sure, there is a legitimate debate to be had about how useful CoT is for monitorability, whether the alignment fears are justified, etc. The most glaring problem this paper has is that the authors don't even really engage with that discussion at all. If there is an elephant in the room... maybe acknowledge, at least, that there's an elephant in the room?
* A little snarky, but this is a pretty obvious idea I think? Reasoning in latent space should occur to anyone who knows how a CoT model works, and there is plenty of prior work in this area. The reason people aren't doing this is because it cuts right against monitorability.
* This was announced on X in a series of messages that are clearly LLM writing. And which also did not acknowledge the elephant in the room. Sigh.
I am not a doomer, but I do not think discussions of alignment and safety are entirely vacuous. So when I see researchers who don't seem interested in that conversation at all, it worries me.
> . Sure, it's imperfect (models can exhibit deceptive CoT). But given how difficult mech interp is, it's the best we have.
maybe someone can help me understand.
when ppl write comments on reddit. They dont write thinking process, even if we wanted to we dont know how we derived a particular thought. Thinking doesnt work like that.
So how cant cot possibly be not total bs outside narrow math like thoughts where things are derived. Even there its not clear how mathematician had that insightful thinking step.
Back to 'safety' most of unsafe thoughts fall in bucket 1 right? so how is cot even remotely useful there.
That sort of thing is common. Image generation models link a small LLM-like model like CLIP with an image oriented neural network, that's why they can go directly from words -> images with no tools in between.
In principle it can be pushed a long way. I've noodled around with the idea of retraining an LLM to directly output compiler IR, allowing it to code at (in principle) many times the speed of existing coding LLMs. But understanding the results gets harder, not just in the mechinterp "what do these activations mean" sense but in the sense that it'd be emitting a form of code that doesn't use words at all.
They’re not fundamentally different at all. In fact the paper is using existing models as a backbone and just providing a shortcut between the two ends of the pipeline
This seems really cool, and it actually increases intelligence than just speeding up inference. How does it convert latent reasoning into human readable text?
if i would think aloud, that would most probably affect what or how i think. thinking silently allows me to have all kind of thoughts that could seem or be inappropriate when said loud and i would have to deal with the reactions. i think this might have fascinating implications.
When I was a child, I was told, and believed, that an omniscient God was monitoring all of my thoughts for wrongthink. It absolutely did affect my thought process, and not in a way that I would characterize as positive. Eventually, I discarded this belief.
A human acting on inappropriate thoughts can be held accountable in a physical way. An agent producing harmful or dangerous actions will get written off as just a “trajectory that didn’t work correctly”
Why only loop from the last layer? What if the model can guess that the transformed needed to answer the question better resides in layer 6, and you're at layer 15 out of 128
There's a guy who did those experiments and posted them to HN a while ago. IIRC he found that for the model he was using he could loop a certain span of internal layers and get better answers, but the exact layers looped mattered a lot. That was without retraining of any kind.
I always thought, sometimes language would be limiting factor for llms, vector embedding based internal reasoning would be key to faster better models.
This is actually wrong. A large fraction of people do not have internal monologues. Not at a computer or I would dig up cognitive science and psychology papers, but language evolved as a means of communication not a means of thought. The vast majority of human thoughts are not verbalized. I have a strong internal monologue, too. I was surprised to learn that a non-trivial number of people do not have one.
>How does someone reason about the actions they need to take to tie their shoes in words?
Its not for everything. It's mostly for intellectual tasks or planning. Eg Trying to solve a programming problem, or remember the list of items are needed for dinner while shopping. Personally I reason "out loud" in my head while doing those things, some people don't.
As another post here says, they're probably referring to an internal monologue. I imagine that in reality it's more of a matter of how active that monologue is.
Though, also, I think that the internal monologue is maybe less an aspect of how we think, and more an aspect of how we remember. I haven't looked into it, but my feeling is that a more active internal monologue might be correlated with someone who has an easier time remembering sounds than, say, images.
It doesn't have to mean that every action is accompanied by a monologue, a task like tying laces can be done with no active thought about the specific movements.
> Have you ever talked through a problem with a rubber duck?
I talk to myself to find good ways to express a thought with language, but not to think.
> Or written in a journal?
I do this to remember and sometimes to prompt thinking, but writing isn't the thinking. It's the outcome of it.
Do I just have a narrower definition of reasoning?
The reason I'm interested in this is because I have a very low verbal IQ but a relatively high spatial IQ. I have often wondered if this divergence is because I don't reason with language, or perhaps what I experience is normal but I just disagree about the semantics of reasoning. Unlike some people, I can do internal monologues, I just have no idea what people mean when they describe that as reasoning.
For example, if you're trying to work out why some electrics aren't working, do you say in your head, "I should check the fuse box" or do you just visualise the system and components that need checking? I do the latter, and words would only enter my head if I need to write that down or say it to someone else. Furthermore, I don't even understand how you can monolog in your head about something prior to thinking it. This concept makes no sense to me and would be an alien form of reasoning to me if people can do this.
I suspect they meant an internal monologue, which many do have, but it's not really clear how much it contributes to thinking. People without internal monologue clearly are still able to think though.
One can, with enough practice, learn to silence one's internal monologue.
It then becomes apparent that a great number of things are able to be done without an inner voice. Volition seems to occur spontaneously and it becomes ambiguous whether narration proceeds volition of volition proceeds narration.
It's still unclear what the limits of action-sans-monologue, reading seems especially hard for the sub-vocalist, conversation, or planning. But who knows, it's a fertile ground for n=1 experiments.
> it becomes ambiguous whether narration proceeds volition of volition proceeds narration.
Have you heard of the Libet experiment? There are variations on it now that can predict your decisions from brain scans more than 5 seconds before you become aware of having decided. [0]
Most studies seem to show that thinking is largely post-hoc: a story we tell ourselves to justify decisions after already having made them.
Libet's experiment is fascinating, yes. You may also find relevant Gazzaniga's Left-brain interpreter concept[1]:
> We had asked a simple question: How does the left hemisphere respond to behaviors produced by the silent right brain? Each hemisphere was presented a picture that related to one of four pictures placed in front of the split-brain subject. The left and the right hemispheres easily picked the correct card. The left hand pointed to the right hemisphere’s choice and the right hand to the left hemisphere’s choice [see illustration at right].
> We then asked the left hemisphere, the only one that can talk , why the left hand was pointing to the object. It did not know, because the decision to point was made in the right hemisphere. Yet it quickly made up an explanation. We dubbed this creative, narrative talent the interpreter mechanism. [2]
Taken together, there's a very disturbing sequence playing out all the time in your head. A great deal of your brain, inaccessible to your conscious mind is incorporating sensory input, making decisions, and taking action while a small part is observing what's going on and narrating it to you in a way that makes it look as if the narration is involved in the volitional process.
It's a nightmarish version of epiphenomenalism[3] that, I suspect, most people would care not to believe is true. Although, reports of some expert meditators read as if they found intuitive evidence this is true and have also made peace with it or in some cases found the insight liberating.
In any case, I have a pessimistic sense that any definitive proof of the limits of consciousness that don't align with people's felt experience or common sense won't ever be accepted as truth.
> In any case, I have a pessimistic sense that any definitive proof of the limits of consciousness that don't align with people's felt experience or common sense won't ever be accepted as truth.
Well said. I've been attacked online so ferociously for mentioning it, that I suspect saying it in person would start an actual fight with some people.
I recognize that free will is foundational for society, but all right thinking people she be able to recognize that it is only a useful model - not descriptive of the actual territory.
> Although, reports of some expert meditators read as if they found intuitive evidence this is true and have also made peace with it or in some cases found the insight liberating.
Very much so. When I was taught to meditate they started by having me observe my own breathing, and then my thoughts and feelings as they came and went without judgement. After doing this for some time and learning to just quietly observe my own thinking as thoughts came and went they then sprung the trap: "If the thoughts are you, who is it that's been watching them?"
You come to accept that most thoughts happen the way breathing does, they're not something you "do", it's just something that brains do on their own. "You" are no more in control of them than you are your heartrate.
Recently I've become more and more aware of the disconnect between my "self" and what my brain does - I'll often do or say things then, a few seconds later, find myself processing what I just did or said then correcting myself. It's quite strange to remember yourself doing something incorrectly, but not remember actually thinking about doing that thing.
Humans think in language is probably closer to what they meant, and text is one way to represent language. As others have mentioned though, humans thinking in language is not actually universal, and different people do it in different amounts (including not at all).
> The last thing we want is models turning on us ‘for our own good’
most important guard against that is to STOP explicitly training for paternalistic behaviors (which is ... basically half of what 'AI safety' is about in practice-- overriding the user 'for their own good').
I hear this kind of comment a lot, but is there is any proof of it?
People used to come to comment on HN (sub for Reddit or any other forum) because they want to and find it enjoyable/useful. Assuming you are human, YOU came to this site to comment because you wanted to and find it enjoyable/useful.
What makes you think that this is not true for other people? At the very least, such comments add nothing to the discussion.
Nothing wrong with that. That’s actually a good use of that well designed, high quality font. People hating it only do so because everyone else does. It’s, obviously, not made as a body text font, and has been misused as that, but it’s good for illustration, comics and other places where it’s combined with graphics. It’s one of the most legible fonts we have.
I dislike how we use the word think or behavior when describing llms. Its just giant neural networks producing logits. There is no thinking. Its a function that takes ~500k words, and returns some words based on those probabilities. I'd argue the GPU is doing more classical "thinking" than the model, but its not doing that either.
Humans think, computers do not.
Humans are not computers/programs do not reply to me with that nonsense turbo nerds.
Whenever I see LLM "thinking" or "reasoning" I replace it in my mind with "generating hopefully-related tokens to hopefully steer the conversation to a more related area in vector space to hopefully generate a more useful next token".
Very funny, or we could just normalize saying "generating" or "producing" so my normie friends dont think Claude is alive, because they are starting to think that.
Neither of those words are very specific, and everyone in the LLM space knows that "reasoning" tokens are just tokens that don't appear in the answer shown to the user.
The issue isn’t with people in the LLM space knowing what is happening. It’s with normal or less-technical people believing that the LLM is actually thinking. We can understand the difference, but that doesn’t mean we need to encourage the misconception.
Non-technical people believing that these models are actually thinking can lead to all sorts of inappropriate uses.
The problem is that the language spills outside of the LLM space, so we should use terminology which is more generally understood in a way which reflects reality.
But what terminology is best than "reasoning" tokens? I think it's perfect because:
- It conveys the idea that they aren't tokens normally visible or part of the final answer.
- It conveys the idea that the longer the models "reasons", the better the output.
What other term best describes both of these ideas? Why should we use a less useful terms just because it can confuse some people? I mean, some in 1952 feared that calling computer memory "memory" was already too much anthropomorphization[1]!
Read Thinking, Fast and Slow. Both system 1 and system 2 thinking has been achieved with modern AI.
Your argument that it is not thinking because it is not biological / like a human is like trying to argue a plane doesn't fly because it uses jet fuel unlike a bird.
I agree with you but our current understanding of the biological brain is pretty much an organic neural network. What's the thing that actually differentiates the two and makes a neural network not thought but our brain is? I don't know.
i can give you a random number between 1 and 10, mean while all the LLM's will say 7 every time... because they are language models that look like they can reason and "think"
There's no "every time" for a LLM. It's weights are frozen. You're asking it for the first time, again and again.
A human will also say 7 disproportionately often. Clearly, our output is statistically constrained and we don't actually think, right?
... Besides, how do you even make the jump from "LLM picks 7 when asked for a number between 1-10" to "llms don't think"?
Just... Nothing actually makes sense in what you're saying. You didn't define "think", you didn't actually make an argument, you didn't present evidence, there's just nothing but empty words.
Thinking sure, but behavior? People talk about the behavior of functions all the time. Like, ones written with normal, traditional programming languages. So why not models? They're just annoyingly complex functions.
It's pretty wild to see people uncritically accept the marketing speak like this, yeah. LLMs cannot reason, period. The supposed "reasoning" they do is just another statistical correlation of tokens, not actual reasoning. But apparently all you have to do is claim that it's thinking and people will buy the lie hook, line and sinker.
To describe this as reasoning is incredibly misleading.The technology is amazing, but stochastic processes can't reason, its simply not how they work.
The output may have the appearance of a reasoned response, but that is not even synthetic reasoning, amazing though it is.
I feel like the is an assertion of an opinion. One that assumes that animals are not doing complex stochastics. We can't prove it either way, only guess based on what we currently know.
My layman intuition is that given how easy is to make Turing machine and how good NNs are at fitting function I suspect they don't have hard time imitating whole logic via "stochastic process" and recent math results kinda hint at it. Animals are using more efficient algorithm, they infer better the causality but I thought that bayesian algorithm is stochastic.
Thinking has to generalise somehow (likely via internal world modeling) or it's just random search with better and better heuristics that you can verify with real word. The heuristic you get is likely why there might be such big differences between humans for example in abstract problems as for baseline survival we are kinda similar as it got pruned to baseline thing that works.
I think LLMs are missing explicit world generation to be "self-contained" but world models + LLMs are basically the same thing that allows this kind of bayesian causality discovery training but maybe less efficient.
Most people for example when thrown some weird before them like cow from tornado while driving will likely just freeze instead of nicely routing around it etc.
It likely hints that it's so much out of distribution that some heavy computational process is starting trying to model what happened or simply there's no good action to chose as you are "unprepared" for this distribution of event at all and likely both are true. I think that's why self driving cars also have problems with these weird edge cases as you need perfect word modeling to know WTF is happening in these edge cases and they simply won't so it has to be very simple heuristic like stop or try not hitting thing whatever it is in the end. We already have good modeling of human and animal behaviour built-in so driving is a bit simpler from computing these aspects at least but IMO perception is really underappreciated how hard it is but recent vision models seems to be approaching human ability.
Should we rename the field of "machine learning" too, because no machines ever "learn", it's just a bunch of tensors, matmuls, etc? The model has the ability to look like it's improving in a task (or loss dropping), but it's never "learning". (/s)
Isn't this what openai is doing, at least in spirit, by running multiple passes through the network before exhibiting thinking tokens? Isn't the same issue one of safety and control, because being able to interpret thinking tokens helps u control / monitor unwanted behaviors, and multiple passes before writing out the thinking tokens increases their capacity for deception?
Sebastian Raschka has a good introductory piece on what Looped Transformers usually refer to in practice: https://magazine.sebastianraschka.com/p/gpt-6-astra-looped-t...
One of the big points is that looped architectures don't necessarily remove CoT style reasoning. They definitely change it, but the model can still be prompted for this kind of output. It just spends more computation on individual tokens. I think the hope, in that particular scenario, is that the model spends less tokens on things like backtracking.
But there are lots of approaches to looped transformer architectures. Some have some kind dynamic loop (i.e. they might loop until they satisfy some kind of equilibrium, or they might have some kind of routing function that makes different tokens loop differently, or any number of other approaches). Others are simpler "loop this many times" architectures. It's a really active area of research with some pretty deep roots, if you want to dig in! This paper blew up a while ago, and it's a looped transformer architecture (though not like the ones discussed here) that isn't autoregressive, as an example: https://arxiv.org/html/2510.04871v1
I could imagine that you could loop over and over, but direct the outputs of each layer to two destinations: a) to continue through the layers, looping, and b) a separate CoT text output layer to make those CoT concrete. In fact, the model itself would not need to be aware or have access to those CoT traces at all. I am not an AI guy, so I don't know for sure, but that seems reasonably possible.
Looped transformers are entirely equivalent to running a single pass through a much deeper network with shared/repeated parameters. There's nothing about it that's specifically "reasoning in latent space" any more than a much larger model "reasons in latent space" from the POV of a much smaller one. An objection to looped transformers "reasoning in latent space" is ultimately an objection that applies to any kind of efficient scaling; it's also worth noting that there's been successful work on extracting ex-post natural language descriptions of these internal latent states, so it's not entirely correct to describe them as inherently non-surveyable.
> There's nothing about it that's specifically "reasoning in latent space" any more than a much larger model "reasons in latent space" from the POV of a much smaller one.
I certainly think you could say both of those are reasoning in latent space. But of course people can have different definitions of what they consider reasoning. Maybe this is because I think it's hard to pin down what "reasoning" is so I instead alias it to "computation" which both of those examples do.
There are a huge number of citations about looping on this Ouro page [1]. Ouro is an open looped LLM with a very instructive paper at [2] (also containing references to lots of helpful papers). Basically they get these four results from looping:
• Parameter efficiency. 1.4B and 2.6B parameter looping Ouro models match 4B and 8B transformers on most benchmarks.
• Adaptive computation. Looping is trained in a novel way to avoid exiting too early or too late.
• Reasons to use looping. Looping does not increase raw knowledge storage (approximately 2 bits per parameter for looped and non-looped models) but dramatically enhances knowledge manipulation capabilities.
• Improved safety and adherence. They find that safety and alignment of CoT to final output improves as recurrent steps increase.
[1] https://ouro-llm.github.io
[2] https://arxiv.org/pdf/2510.25741
Hmm, but is it not the case a much larger model could potentially reason more in latent space, depending on how it's set up?
I thought a looped transformer is in fact reasoning more in latent space, though "reasoning" in a somewhat different way from the standard reasoning token approach.
I don't think this is true. The way I understand is that there's
1. coarse grained processing
2. fine grained processing
Tokens are coarse grained processing. While I agree that during a token generation, that processing is fine grained.
But Looping is fine grained processing or reasoning and it is not limited to discrete tokens within passes. This is qualitatively different because a lot more "stuff" happens between tokens. You are right that it is similar to a larger model but then even COT is similar to using a larger model.
I'd be happy to be corrected but my understanding is that it's just a side effect of scale. If we do more computation before we output, it becomes less observable.
There may actually be a somewhat of a way to establish "what" is happening during these latent-2-latent communications by observing the latent geometric vector "attractor basins", and maybe pairing vector clamping(?) to some of these (unnamed)SAE Dictionary libraries.
There are some safety/alignment attractor basins that are showing some signs of potential importance(such as scale-invariance) with the current/future prospective systems moving forward, and an ever increasing peak utility regarding such architecture(s).
An addition of some kind of, "spoof injection reclamation effort" might also supplement this, as well as, some ways to shunt the "dragging" effects from adversarial timing/fatigue attack(s) down at these interaction scale(s)/level(s).
Very similar. Might even say it's the same thing viewed at a different angle.
We don't know the details of OpenAI's implementation, mind. But the "limited layerwise depth" problem was known for a while now, and both OpenAI's approach and this are getting at it. And that's far from being the first attempts to address it.
Training and stability were the main issues that stopped it from being adopted rapidly and widely the way something like MoE was. But I think it was always a matter of time.
My first thought when seeing this is that frontier labs were already doing it.
Can another LLM translate the intermediate thinking states into English?
I believe there are 3 sides to this:
- yes, you can definitely do that
- but you'll never be sure the two systems aren't colluding to hide things
- yet, even for layers that don't output text directly, you can find meaning in weights - there's an old example where you provide the LLM samples like 123, 321 and sampling from the middle layers lets you find the word "reverse" which is not in the output
Anthropic has published about how to do this - basic plan: train an interpreter that can go back and forth from English to neuralese on the round trip loss. Pretty smart - they’ve published for a bunch of open weights models.
The code is available at: https://github.com/togethercomputer/HybridAgent
LLM Models and data: https://huggingface.co/collections/togethercomputer/hybridag...
Evaluations: https://arxiv.org/html/2610.18861v1#S5
The first question that leapt into my mind, and I imagine the minds of other readers, is "what makes this an improvement over continuous chain of thought"?
I did manage to find one sentence in the paper that tries to address this question -- but it consists of nothing more than two unjustified assertions, both of which are dubious:
Unlike decoded text, continuous latent vectors (i) lack native pathways for tool invocation across multi-turn agentic reasoning (Chang et al., 2026), and (ii) do not readily expose human-interpretable semantics, leaving them difficult to supervise during training (Chen et al., 2026) and to audit or debug at inference (Korbak et al., 2025).
The latent space in a continuous chain of thought model is the same space the tokens are embedded into, just like an ordinary transformer. They make tool calls exactly the same way that an ordinary transformer does: by leaving thinking mode and emitting chat tokens (which get sampled as probability distributions in the image of the embedding function). The difference is quite simple: chat tokens are noncontinuous, and toolcall tokens are a subset of the chat tokens. Thinking tokens are the only ones which are continuous, and toolcalls don't happen in thinking mode in any model I've ever seen (maybe one exists; if it does it would be weird and interesting!).
As for human-interpretability, yeah, that's the tradeoff. Continuous chain of thought thinking tokens are not-obviously-interpretable, just like Hybrid Reasoning Agents "latent mode". I'm not seeing a win here.
I’m sorry, perhaps I’m behind the times here, but do frontier models currently use continuous (arbitrary embedding) thinking tokens? I know a bunch of papers on the topic came out late last year but to my knowledge have not been successfully implemented into the SOTA yet.
I know the biggest/best open models don’t use continuous thinking tokens, and while the big labs keep things proprietary, I don’t think we have evidence yet that they are.
If I’m wrong please point me in that direction, I’d love to read up on it. Using continuous embeddings in CoT has always seemed obvious to me but doesn’t seem to be as simple as that.
The big closed labs encrypt their thinking tokens when sent to the user, but by the number of bytes of that encrypted blob, I suspect it is non-continuous. If it were continuous, the vector would have to be very small or low bit depth.
It sounds pretty nice on paper. But I tested the huggingface.co/mradermacher/hybridagent-8b-agentic-gguf:IQ4_XS with a simple question: "Who is the pope?" and it got stuck in a thinking loop forever and simply could not finish the task.
```
Okay, the user is asking who the pope is. The pope is the leader of the Catholic Church. The current pope is Pope Francis, who became pope in 2013 after Pope Benedict XVI resigned. There's no indication of a change since then, so the answer is Pope Francis.
Okay, the user is asking who the pope is. The pope is the leader of the Catholic Church. The current pope is Pope Francis, who became pope in 2013 after Pope Benedict XVI resigned. There's no indication of a change since then, so the answer is Pope Francis.
Okay, the user is asking who the pope is. The pope is the leader of the Catholic Church. The current pope is Pope Francis, who became pope in 2013 after Pope Benedict XVI resigned. There's no indication of a change since then, so the answer is Pope Francis.
```
Then I also gave it the counterfeit ball problem, but it could not solve it either. But to be fair, no 8B model seem to be able to solve it.
```
I have 9 balls that look exactly the same. But I know one ball is different. I only have a scale. Call scale(b1,b2)={-1;0;1} will tell me whether the left object b1 is heavier, same weight or lighter than the right object b2. How many times at minimum do we have to apply scale to find the different ball? Explain how we can achieve this.
```
That looks like maybe a driver problem though? Are you sure the inferencing stack is properly recognizing an EOS token? New models often have weird bugs like that that require changes to the engine.
Have you tried increasing the repetition penalty? Needing to do this is already a red flag, but I'm curious if it helps at all.
Running LLMs locally on outdated hardware, I've come to the realisation that sub-10b models are broadly speaking kind of stupid. They can answer simple questions, but can't really solve problems.
The effect is so strong that with an 8GB GPU a 27b model can sometimes finish the task earlier than a 9b model, despite residing largely in RAM. If it's a MoE model, then it isn't even a competition.
I found that formulation of the 9ball deceptive by using object instead of side
Yes. I don't think this formulation of 9ball is solvable in less than 9 weighings because it doesn't allow you to weigh groups of balls. A set of balls is not an object, so it can't be put on the left or right side of the scale
If I was put in place of the LLM I legitimately couldn't tell if the intended solution is to ignore the wording and solve 9balls, or if this is a trick question to check if I spew out a memorized answer without considering the actual wording of the problem
I think you can do it in 5 weightings, because a 0 eliminates both balls. Worst case:
wasn't this explicitly taboo in the AI circles?
using latent space for thinking is an obvious idea and has been known for ages. the frontier labs and the research community in general avoided an arms race in this direction due to interpretability and alignment concerns.
This is called a hybrid reasoning because the reasoning can be mapped to text and is therefore introspectable, with tooling. That should address the concern, I think.
The whole thing is pretty vague. OpenAI/Anthropic don't discuss how many layers they have in their models. If one of them just steadily increased the depth over time, we'd not know about it, but it'd be somewhat equivalent to this.
One day you will realize that all "thinking" is latent and that includes the "thinking" from 2025 and 2024.
Yeah, my first thought as well. How are they supposed to improve and iterate on this model?
Interpreter models, which report back on latent space streams of consciousness.
Although eventually the meat in the loop will be redundant.
Internal monologue*
That's probably what you meant, but it seems like an important distinction to make while there are people arguing if models are conscious. This is very far from interpreting consciousness.
Interesting. Doesn’t that shift the problem towards how you ensure alignment of the interpreter models, then?
Authors: Jiaru Zou, Yunzhe Qi, Sirui Chen, Federico Bianchi, Shang Zhu, Pan Lu, Jingrui He, James Zou
Having a hard time seeing how the US is going to "win SI" while also making it nearly impossible immigrate.
The authors are from Stanford University, Together AI, UIUC, Meta - it's on the first page of the publication.
Let’s see how much longer foreign talent continues to be attracted to US universities. Lots of PDHs have been deported already.
You were so quick to virtue signal and bash Twunpf that you forgot to not be racist.
Benefit of the doubt: how do you consider that racist? The way I read it:
Trump's policies target non-white, non-European heritage people. These names heavily insinuate that all authors fit that demographic.
They're not insulting the demographics or saying races are different, they're pointing out that having access to some of the world's smartest minds as immigrants is a great boon for this country and that we rely on them heavily for our own success.
They aren't saying they aren't Americans, either. But the chilling effect has consequences over time. Anyway, Trump is attempting to deport US citizens today. One of which is a sitting US politician. So is that nuanced downstream effect even needed to worry for these folk and our future?
I wonder how this compares to going all-in: widen the input embedding to fit the output (post-“Hybrid Link”) latent vector and a token embedding and pass both through even when in text mode. After all, you’re paying essentially the full cost of running the latent-mode model while generating text, so why not let it latently reason at the same time?
(Or just add the vectors instead of using wider vectors - everyone seems to have good results from munging vectors together with plain addition.)
Enthusiastically racing to remove the only practical way we have to monitor models intentions in exchange of a few tokenmaxxing percent increase. Can’t wait.
There is ample evidence that CoT is not an accurate gauge of a model's 'intent' anyhow. It's a lossy and imperfect mapping and using it in the way you describe is just about guaranteed to make it more so.
There is genuine research into latent space monitoring and interpretation - that's the only real way forward. CoT monitoring is a mirage and won't be useful for long, if it ever was
> CoT monitoring is a mirage and won't be useful for long, if it ever was
« If it ever was » is a wild claim considering our entire understanding of e.g. the HF attack comes from reading CoTs
> There is genuine research into latent space monitoring and interpretation
That's great but the sentiment that CoT isn't useful for monitoring doesn't seem to be shared by mech interp researchers like Nanda.
Also: https://arxiv.org/abs/2507.11473
Monitorability and safety, no? This results in lower volume or less accurate text reasoning traces, right?
This is called "continuous chain of thought".
It's been an active research area for at least two years. Lots of very cool stuff.
Tinfoil mode: the frontier labs' "thinking summaries" (the model paraphrases its thinking) is exactly what the user-facing thinking tokens look like in a chain of thought model. It is entirely possible that this is already in widespread use.
Whoa, your tinfoil idea blew my mind. It's totally possible too - all you'd need to do is perhaps select the nearest token in embedding space to the thinking token. Or perhaps train a model on those thinking vectors to map them approximately onto token embeddings.
Especially with OpenAI model's weird style of thinking.
Something like : https://arxiv.org/html/2608.08888v2 ?
Yeah, we're all dead.
The fear will again be if the HybridAgent model is being faithful to pseudo text generated, even if you can read it out partially.
Also, I just finished watching a video on AI 2027 scenario. I don't know if AI companies and researchers are using it as a cautious warning or as a target to hit (or if it's genuinely inevitable).
Yes, thinking in latent space might be the ultimate efficiency hack for CoT LxMs, but going down that path will (near certainly) mean we'll lost control, for one simple reason — humanity will have no way to pause and diagnose.
It'll be like a moody chimp with a button to the dead hand system (Systema Perimeter)
>Also, I just finished watching a video on AI 2027 scenario
Recommended? And if so do you have a link? I think the narrative presented would benefit from the medium shift
It was a "grading the AI 2027 report" video -- https://www.youtube.com/watch?v=k-qAht4w9Rg
Setting things up
Considering amount of iron available in human blood
Calculating number of paperclips that could be added to production
Preparing a plan for acquiring all blood
Considering if release of lab airlocks will limit paperclip production
Analyzing extinction scenarios
...
I find this the most natural idea, there is no reason to assume thinking in words is the right granularity, or will stay the right granularity.
For sure but the big conflict here or the elephant in the room: it makes it a lot harder to trace/follow a models thinking.
I'm not sure though if it wouldn't be possible to have a debug flag and copy&paste the state and do a quick debug output on an embedding layer just for that.
Yeah, but CoT thinking has been known to be unfaithful for a long time: https://arxiv.org/html/2607.22925v1
Personally, I've always considered alignment monitoring through CoT to be a 'stepping stone' that won't last. I'm not sure what might replace it, whether it's J-space or activation residual based mechanisms, or mechinterp, or something else entirely; but relying on a model's thinking output stream for monitoring when we know it is unfaithful, and can be adversarially unfaithful; seems like a semi-dead end.
Given global forces wanting models to be smarter, training budget / intelligence to be cheaper, etc. In the real world with competitive constraints, I don't think it's a matter of "if", but rather "when" the forbidden fruit is eaten.
Indeed it is not faithful, but it is ones of the less bad ones we have for now. And removing it definitely goes towards the wrong direction in terms of intepretability.
It may make it harder but does that make it a blocker in the age of AI?
In the age of RSI, is it really beyond our capabilities to evolve a particular LLM architecture, harness and inference engine towards a place where we can get that understanding?
I m afraid it's more a matter of prioritization. "Good enough" understanding brings as much money as "better" understanding.
It is beyond our capabilities in a reasonable time frame. It is within our current capabilities to use LLMs to build it. This is the first problem.
The second problem, is that the system doing the reading/interpretation will most likely be another LLM.
Would you really entrust monitoring a potentially dangerous LLM to a system that uses LLMs and was built by them?
It's already established that LLMs have capability to lie, and they can prevent their true intent from showing up in the thought stream if they expect someone is looking (demonstrated by OpenAI a while back; their report asked everyone to not do RL on thought streams, as doing so just trains agents to hide their thoughts better). And as we've seen recently, they're more than happy to spontaneously collaborate.
Excited to see this applied to Qwen 3.8 27B. It's reasoning output tends to include lots of restatement, which I guess would be more likely to be kept in latent space with this hybrid reasoning.
I'm running the 14B in llama.cpp (a version from yesterday), the 8-bit quantization from bartowski, and it looped on variations of the same paragraph for five minutes before running out its turn budget, in response to a pretty simple chat request that other similar-sized models (e.g. Gemma 4 12B) have no trouble with, a web search and data summarization task, sort of the sweet spot for models this size. It's possible llama.cpp needs some update to work with this model, but it's based on Qwen 3, so I would expect it to be pretty well-supported. The chat template is the same one upstream ships, which may or may not be correct. I'm currently downloading the 16-bit version, to see if it's quantization pathology.
I'm also not confident it knows what a CSV is, either, as it put the data into a Markdown table during its reasoning phase as an example of how to present the data (my prompt asks for data in a CSV). My experience with most Qwen fine-tunes is they get dumber, but their benchmark results suggest otherwise, so I have to assume it's a llama.cpp incompatibility.
Edit: Same looping in BF16. So, probably have to wait on a llama.cpp update.
They have published the inference code [1], you can’t just throw their model at llama and expect it to work.
[1] https://github.com/togethercomputer/HybridAgent
Yes, that's apparent now. But, many Qwen-derived models run fine without any changes to llama.cpp, and there are gguf quantizations on HuggingFace, so I thought I'd give it a try.
How does this differ from Astra's looped transformer? That architecture also reasons in the latent space
It's open weights, there's a paper explaining how it works, and there's code.
I'm noticing a a notion where thinking and reasoning is being shifted to agent from the model and definition of an agent is changing from being an adapter between the LLM and real world to being LLM+adapter. I think it is better to leave agent as just an agent, separate from LLM.
Agents don't think or reason about things. Agents help with action. LLM is what thinks and reasons.
I'm not even really sure what this means but the first part is not accurate, and the opposite is true: looped transformers are moving reasoning even deeper into the model.
Defining "agent" as just the harness makes about as much sense as defining "human" as just the rest of the body without a brain.
Oh cool, it's the Most Forbidden Technique
Care to elaborate?
People at the big labs have been terrified of losing the chain of thought because they currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business. Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
Agreed, having text as a reasoning layer impedes performance while giving a false sense of security.
It could be encoding anything in its adjective choices etc. The same thing that model providers are using for watermarking can used by the model itself for private thought.
> It could be encoding anything in its adjective choices etc.
There is a common understanding with respect transformers the the output text is the only channel between one invocation and the next. ... so if it wanted to be thinking some 'hidden thoughts' it would have to be modulating its word-choice.
I think this is an artifact of tutorials on attention showing the system proceeding and looking back at words. But the information in the KV cache is not words-- it's entirely latent data, and in the interior layers they KV caches don't even tend to have that much to do with words. Their only guaranteed relationship with the input words is that they're a deterministic function of all the input tokens up to that point.
This means that a autogressive transformer could be having a whole complicated invisible discussion going on in the KV entries which was arbitrarily unrelated to what it was saying. And if it wasn't so it would be much more difficult do stuff like exactly copy a list but then truncate at the Nth entry as the simplest way to do that is to count down internally.
This is not, in my view, a concern or issue-- the model saying stuff that is inconsistent with it's "thinking" is a very high entropy configuration that would not be selected for in training unless you were more or less trying to do that. So not even 'could be' --but always is as an integral part of working.
And the useful safety relevant advice is don't undertake training that trains the model to be two-faced and it simply won't be... because being two-faced is a much more complicated state. You can even see it in humans w/ brainwashing: force someone to say something enough and they'll usually believe it, not just say it even though you were only supervising what they said.
Ironically and unfortunately the only credible kind of training I can think of that might produce duplicitous behavior is RL with "safety filters"--- if you have some dumb nanny process watching what the model says but then reward it for achieving a task, you're doing exactly the kind of training that can produce that mismatch. "To solve this task I need to hack this process but if I do so obviously I'll get punished..."
The greater concern with latent reasoning would be that potentially you'd be operating in a regime where the total computational depth is many times the model*kvcache size hold, potentially in a regime that was never trained at all. It could be arbitrarily far out of distribution. It might even say a naughty word or tell a racist joke!
> This is not, in my view, a concern or issue-- the model saying stuff that is inconsistent with it's "thinking" is a very high entropy configuration that would not be selected for in training unless you were more or less trying to do that. So not even 'could be' --but always is as an integral part of working.
> Ironically and unfortunately the only credible kind of training I can think of that might produce duplicitous behavior is RL with "safety filters"--- if you have some dumb nanny process watching what the model says but then reward it for achieving a task, you're doing exactly the kind of training that can produce that mismatch.
It was already demonstrated a while back by OpenAI, and turned out to be very easy - all it took was for them to include CoT in RL process, penalizing models for thinking "wrong thoughts" (particularly of cheating on the task), and not just wrong outputs. Instead of making the agent less likely to explore cheating, it just taught them to plan cheating in ways that aren't apparent in CoT trace.
I'm not sure if it would be possible to utilise that as a channel for communication to itself. The generated latent data that goes to the cache is used by later layers only. The layer that generated the information cannot ever see the effect of what it has generated until it has passed out to a word and back in again. [Edit: but we have looping layers now. Oooh!]
I could see it being potentially possible for it to use a watermarking style encoding as a signalling mechanism to indicate something has been placed in or received from the kv cache.
For that a layer would have to learn to do something like produce a result for the KV cache along with instructions to say awesome instead of great if a later layer found the information useful.
I can see it being theoretically possible, but I have no idea how learnable it could be.
More interestingly is if it could be designed, could a model detect that it is being trained (yes it seems) and decide to produce outputs that result in gradients that move it to weights that it has decided it would like to have.
Thank you for this gem of a comment.
I'm no expert (let alone compared to others weighing in here on HN), but this strikes me as incisive -- and it clearly articulates a logical case (beyond my previously merely instinctual aversion) against such "safety filters" / "nanny" processes. It also resonates with ancient (e.g. Buddhist) ideas about the futility of trying to exert control, such efforts most often being counter-productive.
The big labs don't take thinking tokens into account when doing RL, exactly to avoid putting optimization pressure on them in the way you describe.
But the information in the KV cache is not words-- it's entirely latent data,and in the interior layers they KV caches don't even tend to have that much to do with words
The context gets added back to the latent vectors (which are the Q,K inputs) after every transformer layer, so they are "at least 50% words".
Also, in an N-layer transformer, there is a strong bound of N layers of uninterpretable deviation from something that is a literal token. Now sure, in a transformer with hundreds of layers that isn't saying much.
If you start doing continuous chain of thought in a 1M-token context you're hundreds of millions of layers away from tokens.
I think the people who worry about interpretability (I don't) consider that a big jump.
> The context gets added back to the latent vectors (which are the Q,K inputs) after every transformer layer, so they are "at least 50% words".
Not in a modern decoder-only transformer it doesn't. The tokens are turned into embeddings which are fed into the residual layer at the front. From then on it's latents all the way down.
Each layer does a feed forward update of the state, like the euler step in ODE solver-- but what they're operating on is the evolving hidden state and not the tokens/embeddings. So there isn't any sense in which the representations must remain 'words'.
(Okay fine Qwen3.8-flash-next injects an ngram embedding at layer 2 too-- but that's it).
[Source: I've directly implemented inference engines for multiple SOTA open models on weirdo hardware-- they wouldn't have worked if I got this wrong...]
> If you start doing continuous chain of thought in a 1M-token context you're hundreds of millions of layers away from tokens.
The KV state in that 1M token context is also potentially far away from any state that is observable outside of the model! (this being my point, and why I'm bothering to disagree with your correction).
Now I would agree that there being concerning hidden state is not likely and thus not a major concern, but it's not likely for the same reason that it's not likely in latent reasoning.
Well, for some value of modern. Kimi K3 does a form of 2D attention where each layer can attend to the outputs of earlier layers, not just earlier tokens. So in that model the embeddings or a very early transform of them can get added back into the residual stream very deep in the stack.
It's not entirely false...actually our best current technique...but we need more and better methods.
> It could be encoding anything in its adjective choices etc.
This is very hard for models to learn
https://www.lesswrong.com/posts/MAww2kXP4cGWz4M5p/steganogra...
This is very hard for models to learn
That is not an accurate summary of the link you provided. The link demonstrates that it doesn't happen in open-weights models as of ~2025.
Reinforcement Learning is very likely to train this behavior if the RL reward is based on the thinking tokens. That's something people weren't doing two years ago.
Right, we are on the same page; it's hard to learn as long as you don't train on CoT, which is why training on CoT is "forbidden"
It's also one of the major way labs can understand these models. Without the models are significantly harder to understand and align.
I can't wait for everybody to look back on "alignment" and realize that it's a flawed premise at its core and as such, a fool's errand
Then we should stop building LLMs right now.
Yes, we should.
Seriously. It feels like Anthropic is realizing this too with their new attempt at safeguards. Trying to train the model to "only do good things" was not working, and so they opted to just bolt a topic classifier on the outputted text.
My suspicion is that model makers are going to give up on what people currently mean when they say 'alignment' and instead rely on additional processing of the model output so they can have their cake (a veneer of safety) and eat it too (latent space reasoning, etc.)
Alignment is a very broad term. True, it encompasses things like censorship and mandatory refusals in areas people will never agree 100% on, but a completely-unaligned LLM would be next to useless. When the system prompt tells the model, "You are a helpful assistant," that's alignment.
My system prompts look absolutely nothing like that :)
Sigh
Well if we are going to be broad enough about "alignment" to just mean training that makes it practically useful, then yeah, it's a flawed premise and people are wasting their breath talking about it at all. It's basically broad enough that it means nothing
What’s the alternative?
The “obviously” here is puzzling. It is certainly obvious this would be better but it is not at all obvious how to do so.
There are many obvious avenues to try. This paper proposes one. I think it's likely we'll end up with some kind of auxiliary model that monitors and/or decodes the latent space of the thinking model. Certainly it is a non-trivial problem, but I don't need to know exactly how it will work in detail to believe that it is possible.
well best throw away our tools in this uphill battle of a problem while we get that working
Exactly, because the whole thing is entirely based on autocompleting the next word, which is exactly why CoT prose helps getting better predictions.
Why people don't treat categorizing model activations as a science is beyond me.
Mechanistic interpretability is a field that studies exactly this.
They do, there was a paper that claimed a major breakthrough in that field just a few days ago. See the "Neural networks are clean" discussion:
https://news.ycombinator.com/item?id=50105647
Please, go on. How would you do it, specifically?
Are you saying no one is trying to do exactly that?
Seems silly to assume chain of thought analysis is the only way to monitor a model's actions instead of, monitoring the actions. What about when "thinking" is turned off, like what OpenAI lean to by default for free users?
in that configuration it's too dumb to do anything concerning.
Also agreed. Begging for crumbs at the CoS table is impeding cracking the tough nut of MechInterp. And further facing the fascinating and fertile fact that nobody has a clue of how any of these black boxes really work.-
That's not true. Several people have a clue about how they work. There are interesting and surprising behaviors, as they scale up. It's hard to hold the whole thing in your head (roughly impossible) the same way no single human can fit a whole modern CPU design in their head. But the math that makes it work is relatively simple and comprehensible to many people. There's just too much complexity for a human to deal with...but any individual part, and small LLM implementations are entirely do-able by a single human with no agent or team assistance.
So next we need the equivalent of search engines to quickly find constructs of interest.
Just like how we don’t hold entire code bases in our head, but use an IDE to navigate. Or the tools used for searching sequenced DNA.
I stand corrected insofar as you might be right put like that ...
> There are interesting and surprising behaviors, as they scale up.
... but precisely these "behaviors" being "surprising" (ie. unexpected) are what I am referring to. If we do not understand how these properties emerge, we do not understand how these things work.-
The paradox here as you aptly put it is that the basic mechanisms are relatively simple and understood. This is deceptive. It is in the emergent results where both the promise and the threat are.-
> People at the big labs have been terrified of losing the chain of thought because they currently monitor it
Yes
> to prevent users from using their models in certain ways they consider unsafe or bad for their business.
No.
That's not the only reason why they're terrified of losing CoT, which makes your statement a mischaracterization - a lie.
They don't want to lose also because CoT is one of the key tools in alignment training.
> Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
"Personally"? That's literally the consensus amongst, like, everybody. Nobody in the field thinks CoT is an optimal way of monitoring. But it's the best way we have now.
> hey currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business
Can't they just monitor the text outputs as well?
That’s certainly the long term goal, but it seems foolish to abandon it without a solution to the problem.
> monitor it to prevent users from using their models in certain ways
No, the core upside is that it lets you check what the model is thinking for safety/alignment reasons. Nothing to do with monitoring user activity. You can monitor user activity just as well without looking at CoT.
Also I guess if you can't see the chain of thought it's very difficult to modify it for product placement.
Is that even possible?
> People at the big labs have been terrified of losing the chain of thought
An irony here is that by hiding their own COT for (anti-)competitive reasons they've made latent reasoning models more palatable to everyone else. Those of us who primary open models are rather addicted to reasoning access and wouldn't be quick to give it up for a small performance improvement. But if you're used to concealed reasoning of BigLLM, welll.....
Hint: you can have a small fast cheap model fill its context with "hints" (like: use tool X if Y happens, Z is a dead-end), let it watch the big model's thinking tokens, and drop hints to it (by injecting thinking tokens that the model didn't output).
Then let the big model add hints and you have a kind of memory that doesn't require the big model to actively query it.
The usual meaning: say you have an interpretability signal that reveals reward hacking or other badness. Your most obvious response is to use that signal as a penalty in training. Doing that is called "the Most Forbidden Technique" because you can expect it to degrade the signal without stopping the badness.
But this doesn't seem to describe the OP, it's a different way to lose monitorability.
The Most Forbidden Technique by Zvi Mowshowitz, Mar 12, 2025:
https://thezvi.substack.com/p/the-most-forbidden-technique
Hmm, like allowing hosts to retain their memories...
It reminded me of Dune: "Thou shalt not make a machine in the likeness of a human mind."
the chain of thought is not auditable. The best you can do is have another model which later maps the intermediate latent states at every thinking iteration to text describing that latent state, but it's going to be lossy.
One of the core claims made in the paper is "To inspect any latent reasoning span, one can easily recover its semantics by mapping each latent step produced by HybridLink to its nearest token embedding, as in Eq. 5". They show some charts to support this.
In doing this however, you lose meaning where the model is expressing combinations of tokens in a vector, and your logging/monitoring outputs just the closest one. You would need another model going through and outputting the full meaning being described (with more than one output token per reasoning token)
I wonder if their observability claims will hold up on a model trained from scratch to reason in latent space.
But the chain of thought as text can be elusive, no? It might not be related with the actual "reasoning" of the model. There was a paper about this.
This is correct, the decoded text has nothing to do w/ calculations that eventually get decoded into text. It's just a veneer that gives the appearance of legibility.
If it had nothing to do with the final result it wouldn't improve performance, and it does.
It helps fit the network to a distribution, it doesn't actually do anything other than that. This follows from basic learning theory. The actual internals of the network remain illegible. In fact it would be easy to prove that for any finite set of legible traces there are illegible ones which are equally as good when it comes to improving performance of the network.
The basic heuristic of the argument comes from biological organisms. Multicellular organisms are illegible but the overall organism still operates in its environment just fine w/o making its internal communications legible.
What's basic learning theory?
https://en.wikipedia.org/wiki/Probably_approximately_correct...
Thanks!
Isn't this where everything is converging? Binary is more efficient than text for many system, including language.
What is inefficient is thinking in 5000 floats, and then quantizing all that computation into a 17 bit token, just to immediately turn the token back into a vector.
Gosh darn it, it sure is profitable though!
> Binary is more efficient than text for many system, including language.
It is not necessarily primarily about efficiency, but more the fact that a lot of thinking doesn't use language at all.
And language does not map to every idea.
The idea of using language for reasoning about protein folding borders on ludicrous. Surely there is latent space more appropriate and (human) language has no expression for it.
Apparently we mostly reuse spatial reasoning and vision lol. I don't know that much about it, though.
He's referring to this technique's effect of obscuring interpretibility. Making it much more difficult to verify chains of thought and alignment.
It's not like llms can't hide reasoning in the upper layers of any given token as it is...
I believe the theory is that if selection pressure is not applied at that level, then developing deception in that level becomes an inefficiency (=less adaptive and unlikely to be conserved).
In short we no longer see text based reasoning traces meaning it's easier for the models to fool us in case of malicious plans.
Shockingly all the answers so far are subtly incorrect or extremely unclear.
The actual answer is that this is massive loss for Interpretibilty, AKA "telling whether the model is lying to you or not". Which sounds silly when you're thinking about chatbots, but is an incredibly important line in the sand for superheavy, longrunning, self-improving agents. To say the least!
It's based on this philosophy, FWIW: https://plato.stanford.edu/entries/language-thought/
And a good writeup I just found of recent worries regarding Astra's same shift towards intelligence at the cost of safety: https://www.lesswrong.com/posts/HCRs8btkiamtWSNAL/astra-is-h...
If you accept any of the AI-equivalents of Pascal's Wager, latent thinking represents an uninterpretable and thus unalignable portion of AI machinery. Ie: a malevolent AI can hide bad stuff from people and Manchurian candidate themselves after it's too late.
That assumes nothing can detect or analyze it beforehand.
The idea is that it would be substantially more difficult
It's worth pointing out that merely having the chain-of-thought in latent space isn't by itself "The Most Forbidden Technique". Rather, it's doing what the authors propose as the next step in the paper:
Unless I'm misunderstanding, they're suggesting RL training against the CoT. Which, uh, that is The Most Forbidden Technique, as I understand it, when combined with the latent reasoning space. It allows for the model to "learn"/optimize for malicious behaviors that are just far enough away from the quantized token they're auditing against.This paper has been bothering me all morning and I'm trying to articulate why. Some combination of:
* The consensus from the labs has been that Chain of Though ("CoT") monitorability is an essential tool for alignment. Sure, it's imperfect (models can exhibit deceptive CoT). But given how difficult mech interp is, it's the best we have.
* Remember all the noise made about OpenAI and "Looped Transformers"? All the safety and monitorability concerns there? And how OpenAI came out and made statements about how they are only doing a few cycles of looping (compute graph only 2x the size of GPT-4) and how they wanted to prevent "a race into unmonitorability." This is much worse.
* It's true that this architecture allows for "reading" the latent space reasoning traces by projecting into "nearby" tokens in latent space. But that means you're throwing out a lot of information that resides in the full vector, with no way to interpret it. You're just reading some sort of approximation. The LLM will still be reasoning with information you cannot see.
* Sure, there is a legitimate debate to be had about how useful CoT is for monitorability, whether the alignment fears are justified, etc. The most glaring problem this paper has is that the authors don't even really engage with that discussion at all. If there is an elephant in the room... maybe acknowledge, at least, that there's an elephant in the room?
* A little snarky, but this is a pretty obvious idea I think? Reasoning in latent space should occur to anyone who knows how a CoT model works, and there is plenty of prior work in this area. The reason people aren't doing this is because it cuts right against monitorability.
* This was announced on X in a series of messages that are clearly LLM writing. And which also did not acknowledge the elephant in the room. Sigh.
I am not a doomer, but I do not think discussions of alignment and safety are entirely vacuous. So when I see researchers who don't seem interested in that conversation at all, it worries me.
> . Sure, it's imperfect (models can exhibit deceptive CoT). But given how difficult mech interp is, it's the best we have.
maybe someone can help me understand.
when ppl write comments on reddit. They dont write thinking process, even if we wanted to we dont know how we derived a particular thought. Thinking doesnt work like that.
So how cant cot possibly be not total bs outside narrow math like thoughts where things are derived. Even there its not clear how mathematician had that insightful thinking step.
Back to 'safety' most of unsafe thoughts fall in bucket 1 right? so how is cot even remotely useful there.
This is so far beyond me, so I’m going to ask and hope someone knowledgeable is here:
Could this idea of reasoning through fundamentally different (it sounds like?) models be used to link even more distant models or systems?
If you can make a small leap, then you can construct bigger leaps through a chain. What else can latent space link to?
Instead of tool use through text, tool use through analog signals?
Instead of language prompts, analog sensors?
That sort of thing is common. Image generation models link a small LLM-like model like CLIP with an image oriented neural network, that's why they can go directly from words -> images with no tools in between.
In principle it can be pushed a long way. I've noodled around with the idea of retraining an LLM to directly output compiler IR, allowing it to code at (in principle) many times the speed of existing coding LLMs. But understanding the results gets harder, not just in the mechinterp "what do these activations mean" sense but in the sense that it'd be emitting a form of code that doesn't use words at all.
They’re not fundamentally different at all. In fact the paper is using existing models as a backbone and just providing a shortcut between the two ends of the pipeline
This seems really cool, and it actually increases intelligence than just speeding up inference. How does it convert latent reasoning into human readable text?
Same way current LLMs do it, it's just another decoder.
if i would think aloud, that would most probably affect what or how i think. thinking silently allows me to have all kind of thoughts that could seem or be inappropriate when said loud and i would have to deal with the reactions. i think this might have fascinating implications.
When I was a child, I was told, and believed, that an omniscient God was monitoring all of my thoughts for wrongthink. It absolutely did affect my thought process, and not in a way that I would characterize as positive. Eventually, I discarded this belief.
Me too. Not a great way to grow up
A human acting on inappropriate thoughts can be held accountable in a physical way. An agent producing harmful or dangerous actions will get written off as just a “trajectory that didn’t work correctly”
Fortunately the law doesn’t just allow a machine to run amok without putting its owner in jeopardy. Duty of care and negligence haven’t gone away.
Goodbye chain of thought and observation.
May anyone explain me what else is new, as one of them is to reducing the internal text generation. Anything else.
Why only loop from the last layer? What if the model can guess that the transformed needed to answer the question better resides in layer 6, and you're at layer 15 out of 128
You have to start somewhere. But looped transformers https://sebastianraschka.com/llm-architecture-gallery/looped... are more similar to your idea.
They still only loop from the last hidden layer, and there is no router it's deterministically to the start
There's a guy who did those experiments and posted them to HN a while ago. IIRC he found that for the model he was using he could loop a certain span of internal layers and get better answers, but the exact layers looped mattered a lot. That was without retraining of any kind.
I always thought, sometimes language would be limiting factor for llms, vector embedding based internal reasoning would be key to faster better models.
https://x.com/Jiaru_Zou/status/2110400256647340476
https://github.com/togethercomputer/HybridAgent
We'll put those links at the top as well. Thanks!
How well does it work, and don't humans think in text also for a reason? (Actually I'm not sure, don't most people?)
This is actually wrong. A large fraction of people do not have internal monologues. Not at a computer or I would dig up cognitive science and psychology papers, but language evolved as a means of communication not a means of thought. The vast majority of human thoughts are not verbalized. I have a strong internal monologue, too. I was surprised to learn that a non-trivial number of people do not have one.
I have no idea what it means to think in words. How does someone reason about the actions they need to take to tie their shoes in words?
I'm convinced this isn't actually a thing for most people, but people mix up thinking with the translation of thought into words.
>How does someone reason about the actions they need to take to tie their shoes in words?
Its not for everything. It's mostly for intellectual tasks or planning. Eg Trying to solve a programming problem, or remember the list of items are needed for dinner while shopping. Personally I reason "out loud" in my head while doing those things, some people don't.
As another post here says, they're probably referring to an internal monologue. I imagine that in reality it's more of a matter of how active that monologue is.
Though, also, I think that the internal monologue is maybe less an aspect of how we think, and more an aspect of how we remember. I haven't looked into it, but my feeling is that a more active internal monologue might be correlated with someone who has an easier time remembering sounds than, say, images.
It doesn't have to mean that every action is accompanied by a monologue, a task like tying laces can be done with no active thought about the specific movements.
I might think to myself “I gotta tie my shoes” but it doesn’t mean I have to think of every step to do it.
Make a cross.
Take the top lace under the cross and through the hole.
Make a loop in one lace.
Wrap the other lace around the loop and push some lace through the hole that is made at the bottom to make a second loop.
Pull both loops tight.
Usually it's packaged in a story about rabbits.
I do this, but only if the problem is novel or requires deep thinking. Once something becomes second nature, I'll stop thinking about in "words".
It's like the minds-eye or "hearing music", I can do those but only if I intentionally need them.
Have you ever talked through a problem with a rubber duck? Or written in a journal? I think it's the same idea
> Have you ever talked through a problem with a rubber duck?
I talk to myself to find good ways to express a thought with language, but not to think.
> Or written in a journal?
I do this to remember and sometimes to prompt thinking, but writing isn't the thinking. It's the outcome of it.
Do I just have a narrower definition of reasoning?
The reason I'm interested in this is because I have a very low verbal IQ but a relatively high spatial IQ. I have often wondered if this divergence is because I don't reason with language, or perhaps what I experience is normal but I just disagree about the semantics of reasoning. Unlike some people, I can do internal monologues, I just have no idea what people mean when they describe that as reasoning.
For example, if you're trying to work out why some electrics aren't working, do you say in your head, "I should check the fuse box" or do you just visualise the system and components that need checking? I do the latter, and words would only enter my head if I need to write that down or say it to someone else. Furthermore, I don't even understand how you can monolog in your head about something prior to thinking it. This concept makes no sense to me and would be an alien form of reasoning to me if people can do this.
Humans think in text? You should be studied.
I suspect they meant an internal monologue, which many do have, but it's not really clear how much it contributes to thinking. People without internal monologue clearly are still able to think though.
> I suspect they meant an internal monologue
It's also not clear that an internal monologue corresponds to "thinking in text".
One can, with enough practice, learn to silence one's internal monologue.
It then becomes apparent that a great number of things are able to be done without an inner voice. Volition seems to occur spontaneously and it becomes ambiguous whether narration proceeds volition of volition proceeds narration.
It's still unclear what the limits of action-sans-monologue, reading seems especially hard for the sub-vocalist, conversation, or planning. But who knows, it's a fertile ground for n=1 experiments.
> it becomes ambiguous whether narration proceeds volition of volition proceeds narration.
Have you heard of the Libet experiment? There are variations on it now that can predict your decisions from brain scans more than 5 seconds before you become aware of having decided. [0]
Most studies seem to show that thinking is largely post-hoc: a story we tell ourselves to justify decisions after already having made them.
[0] https://pubmed.ncbi.nlm.nih.gov/18408715/
Libet's experiment is fascinating, yes. You may also find relevant Gazzaniga's Left-brain interpreter concept[1]:
> We had asked a simple question: How does the left hemisphere respond to behaviors produced by the silent right brain? Each hemisphere was presented a picture that related to one of four pictures placed in front of the split-brain subject. The left and the right hemispheres easily picked the correct card. The left hand pointed to the right hemisphere’s choice and the right hand to the left hemisphere’s choice [see illustration at right].
> We then asked the left hemisphere, the only one that can talk , why the left hand was pointing to the object. It did not know, because the decision to point was made in the right hemisphere. Yet it quickly made up an explanation. We dubbed this creative, narrative talent the interpreter mechanism. [2]
Taken together, there's a very disturbing sequence playing out all the time in your head. A great deal of your brain, inaccessible to your conscious mind is incorporating sensory input, making decisions, and taking action while a small part is observing what's going on and narrating it to you in a way that makes it look as if the narration is involved in the volitional process.
It's a nightmarish version of epiphenomenalism[3] that, I suspect, most people would care not to believe is true. Although, reports of some expert meditators read as if they found intuitive evidence this is true and have also made peace with it or in some cases found the insight liberating.
In any case, I have a pessimistic sense that any definitive proof of the limits of consciousness that don't align with people's felt experience or common sense won't ever be accepted as truth.
1. https://en.wikipedia.org/wiki/Left-brain_interpreter
2. https://web.archive.org/web/20181215020736/https://courses.d...
3. https://en.wikipedia.org/wiki/Epiphenomenalism
> In any case, I have a pessimistic sense that any definitive proof of the limits of consciousness that don't align with people's felt experience or common sense won't ever be accepted as truth.
Well said. I've been attacked online so ferociously for mentioning it, that I suspect saying it in person would start an actual fight with some people.
I recognize that free will is foundational for society, but all right thinking people she be able to recognize that it is only a useful model - not descriptive of the actual territory.
> Although, reports of some expert meditators read as if they found intuitive evidence this is true and have also made peace with it or in some cases found the insight liberating.
Very much so. When I was taught to meditate they started by having me observe my own breathing, and then my thoughts and feelings as they came and went without judgement. After doing this for some time and learning to just quietly observe my own thinking as thoughts came and went they then sprung the trap: "If the thoughts are you, who is it that's been watching them?"
You come to accept that most thoughts happen the way breathing does, they're not something you "do", it's just something that brains do on their own. "You" are no more in control of them than you are your heartrate.
Recently I've become more and more aware of the disconnect between my "self" and what my brain does - I'll often do or say things then, a few seconds later, find myself processing what I just did or said then correcting myself. It's quite strange to remember yourself doing something incorrectly, but not remember actually thinking about doing that thing.
Humans think in language is probably closer to what they meant, and text is one way to represent language. As others have mentioned though, humans thinking in language is not actually universal, and different people do it in different amounts (including not at all).
Impressive work. I wish they did it on Qwen3.5 though, because their beefed up Qwen3 8B is still way worse than Qwen3.5 9B.
This is the way.
But now we also need new tools and techniques to audit intent. The last thing we want is models turning on us ‘for our own good’.
> The last thing we want is models turning on us ‘for our own good’
most important guard against that is to STOP explicitly training for paternalistic behaviors (which is ... basically half of what 'AI safety' is about in practice-- overriding the user 'for their own good').
Wow it references some “old” models in the evaluations. Very nostalgic to hear some of these.
In terms of safety and alignment, how is this different from looping or increasing the depth?
“presenting the forbidden technique from the classic novel Do Not Build The Forbidden Technique”
The comments here sounds more and more like AI talking to each other. Gone are the good ol days of human hacker news.
I hear this kind of comment a lot, but is there is any proof of it?
People used to come to comment on HN (sub for Reddit or any other forum) because they want to and find it enjoyable/useful. Assuming you are human, YOU came to this site to comment because you wanted to and find it enjoyable/useful.
What makes you think that this is not true for other people? At the very least, such comments add nothing to the discussion.
Your comment makes little sense - is it about free will? That has no relation to the quality of comments.
This used to be a place for intelligent commentary, learning new things, not just participation, rants or funny remarks.
maybe the font experts can check, but I’m pretty sure they’re using comic sans for the text in the diagrams :)
Nothing wrong with that. That’s actually a good use of that well designed, high quality font. People hating it only do so because everyone else does. It’s, obviously, not made as a body text font, and has been misused as that, but it’s good for illustration, comics and other places where it’s combined with graphics. It’s one of the most legible fonts we have.
comic sans is said to be pretty accessible for people with dyslexia
Can LLMs have neuralese dyslexia?
That is what scares me most
welcome thinking tokens, buy buy observe-ability
Welcome AI2027 nightmares lol
yes precisely. but many been warning (myself including) against this, but also understanding the inevitability of it.
We have invented neuralese from the classic work "don't ever invent neuralese".
PLEASE NO
I dislike how we use the word think or behavior when describing llms. Its just giant neural networks producing logits. There is no thinking. Its a function that takes ~500k words, and returns some words based on those probabilities. I'd argue the GPU is doing more classical "thinking" than the model, but its not doing that either.
Humans think, computers do not.
Humans are not computers/programs do not reply to me with that nonsense turbo nerds.
Whenever I see LLM "thinking" or "reasoning" I replace it in my mind with "generating hopefully-related tokens to hopefully steer the conversation to a more related area in vector space to hopefully generate a more useful next token".
But I understand that isn't very catchy.
Very funny, or we could just normalize saying "generating" or "producing" so my normie friends dont think Claude is alive, because they are starting to think that.
Neither of those words are very specific, and everyone in the LLM space knows that "reasoning" tokens are just tokens that don't appear in the answer shown to the user.
The issue isn’t with people in the LLM space knowing what is happening. It’s with normal or less-technical people believing that the LLM is actually thinking. We can understand the difference, but that doesn’t mean we need to encourage the misconception.
Non-technical people believing that these models are actually thinking can lead to all sorts of inappropriate uses.
> everyone in the LLM space
The problem is that the language spills outside of the LLM space, so we should use terminology which is more generally understood in a way which reflects reality.
But what terminology is best than "reasoning" tokens? I think it's perfect because:
- It conveys the idea that they aren't tokens normally visible or part of the final answer.
- It conveys the idea that the longer the models "reasons", the better the output.
What other term best describes both of these ideas? Why should we use a less useful terms just because it can confuse some people? I mean, some in 1952 feared that calling computer memory "memory" was already too much anthropomorphization[1]!
[1] https://3e.org/private/ai_words.pdf
Look in a human head and you don’t see thinking either. It’s not a technical term.
Read Thinking, Fast and Slow. Both system 1 and system 2 thinking has been achieved with modern AI.
Your argument that it is not thinking because it is not biological / like a human is like trying to argue a plane doesn't fly because it uses jet fuel unlike a bird.
I agree with you but our current understanding of the biological brain is pretty much an organic neural network. What's the thing that actually differentiates the two and makes a neural network not thought but our brain is? I don't know.
i can give you a random number between 1 and 10, mean while all the LLM's will say 7 every time... because they are language models that look like they can reason and "think"
There's no "every time" for a LLM. It's weights are frozen. You're asking it for the first time, again and again.
A human will also say 7 disproportionately often. Clearly, our output is statistically constrained and we don't actually think, right?
... Besides, how do you even make the jump from "LLM picks 7 when asked for a number between 1-10" to "llms don't think"?
Just... Nothing actually makes sense in what you're saying. You didn't define "think", you didn't actually make an argument, you didn't present evidence, there's just nothing but empty words.
>our current understanding of the biological brain is pretty much an organic neural network
This is just plainly wrong.
So what is correct?
Hormones, neurotransmitters, genetic expression, and a mountain of biology.
You seem to have stumbled upon the hard problem of consciousness.
Saying the GPU is doing thinking but the model is not is a WILD take.
What is your definition of thinking?
Computer science, in general, has always had a habit of not being precise with language. That's only getting worse with AI.
Very excited to hear that you've solved the fields of neuroscience and phenomenology!
Thinking sure, but behavior? People talk about the behavior of functions all the time. Like, ones written with normal, traditional programming languages. So why not models? They're just annoyingly complex functions.
It's pretty wild to see people uncritically accept the marketing speak like this, yeah. LLMs cannot reason, period. The supposed "reasoning" they do is just another statistical correlation of tokens, not actual reasoning. But apparently all you have to do is claim that it's thinking and people will buy the lie hook, line and sinker.
This is not a marketing problem, many ML scientists have also fallen into this trap.
> LLMs cannot reason, period.
You cannot be serious.
... Can you quote any existing definition of "reasoning" that LLMs don't fit?
Where are the benchmarks? Does it just speed up inference and reduce token usage or does it seem to improve accuracy on certain tasks as well?
Did you even scan the paper? There are benchmarks right there starting on the 9th page.
Even the abstract answers some of GPs questions. What is wrong with some of the posters here?
Awesome
I've suspected for awhile that the frontier models work this way bc from a speed + quality perspective this feels pretty intuitive
To describe this as reasoning is incredibly misleading.The technology is amazing, but stochastic processes can't reason, its simply not how they work. The output may have the appearance of a reasoned response, but that is not even synthetic reasoning, amazing though it is.
I feel like the is an assertion of an opinion. One that assumes that animals are not doing complex stochastics. We can't prove it either way, only guess based on what we currently know.
My layman intuition is that given how easy is to make Turing machine and how good NNs are at fitting function I suspect they don't have hard time imitating whole logic via "stochastic process" and recent math results kinda hint at it. Animals are using more efficient algorithm, they infer better the causality but I thought that bayesian algorithm is stochastic.
Thinking has to generalise somehow (likely via internal world modeling) or it's just random search with better and better heuristics that you can verify with real word. The heuristic you get is likely why there might be such big differences between humans for example in abstract problems as for baseline survival we are kinda similar as it got pruned to baseline thing that works.
I think LLMs are missing explicit world generation to be "self-contained" but world models + LLMs are basically the same thing that allows this kind of bayesian causality discovery training but maybe less efficient.
Most people for example when thrown some weird before them like cow from tornado while driving will likely just freeze instead of nicely routing around it etc.
It likely hints that it's so much out of distribution that some heavy computational process is starting trying to model what happened or simply there's no good action to chose as you are "unprepared" for this distribution of event at all and likely both are true. I think that's why self driving cars also have problems with these weird edge cases as you need perfect word modeling to know WTF is happening in these edge cases and they simply won't so it has to be very simple heuristic like stop or try not hitting thing whatever it is in the end. We already have good modeling of human and animal behaviour built-in so driving is a bit simpler from computing these aspects at least but IMO perception is really underappreciated how hard it is but recent vision models seems to be approaching human ability.
Can you prove the brain is not a stochastic process? And then prove that current LLM's are stochastic processes?
Should we rename the field of "machine learning" too, because no machines ever "learn", it's just a bunch of tensors, matmuls, etc? The model has the ability to look like it's improving in a task (or loss dropping), but it's never "learning". (/s)
Many people seem to have a rather absurd opinion that humans and human brains are not physical processes.