Cool study, we definitely need more principled studies on the role of harnesses. I'd also say that there aren't too many benchmarks where the more complicated harnesses consistently outperform extremely simple agents. But I'm also biased, because I wrote https://github.com/swe-agent/mini-swe-agent/ , which is probably the most minimal agent out there (it started as just 100 lines, all included), and it's used in a lot of benchmarks like DeepSWE, terminalbench, programbench (seems like it's still top of the ranking for TB3, but wasn't evaluated with the best models on TB4).
I'd really love to see more studies about effectiveness of AI in general. As in, what works best and how to use it and such.
Because I feel that the technology and space is - so - hyped and fast moving that a lot of cultish feeling rituals seem to pop up, none of which are backed by evidence. Anthropic openly recommend giving the agents.md file an architectural overview of the code, and the one time this was studied they found the opposite - that the agents.md file is best for concrete commands about how to build stuff and such, and - not - huge overviews. This was, and still is, the official recommendation from Anthropic as far as I can tell.
And then there are the benchmarks, how feel vague and not concrete, and everyone kind of knows they're not the best cuz you can't just assign these tools one fixed number ( for multiple reasons ), but everyone still looks at them and compares them.
People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all.
It all feels a bit weird to me, and I've been on the lookout for exactly these kinds of studies more lately, because I think having this research, even if not done on the exact newest models or not the exact, newest thing, are still - vastly - superior to the alternative.
Haven't gone through full PDF as its very detailed, few things have resonated with me so far.
Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc.
You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model).
Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.
As far as I can tell, the paper says "bash capable", without ever describing what that means. How would one know whether a given model is "bash capable" or not?
I would have to imagine, that Luna would very much fall into the camp of "bash capable". At which point- it seems to me that adding any tools beyond just Bash requires some rigorous testing and verification that value is being added.
This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research.
The focus of the study was the different harness approaches and how they scale across model sizes. The fact that they used any particular set of models is irrelevant.
I'm not totally convinced that models are fungible, the claudes/gpts/Gemini all have pretty individual feels when you're working with them. I wouldn't be surprised if the approaches don't scale or even work the same in a poly model setup
Agree. Harnesses are effective because they interact with the underlying model effectively. If the latest models were fundamentally different, excluding them would be a miss. But I don’t think they are, at least not in ways that would affect these observations.
I think the confounding issue is that by now, millions of sessions of Claude Code and Codex are now in the training set for these models. So they have been trained to work the way these harnesses are configured, and at least in the case of Claude Code the harness itself is greatly stripped down because the model has absorbed it.
I'm going to cherry pick one example where newer models are noticeably improving at least in my experience.
What is a noticeable improvement with something that struggles to read a message longer than 200 characters without missing information in the middle, may be a 0.000000001% improvement with a model that... almost never misses info in the first place.
I have to say, I am starting to hate this line of reasoning. Yes, LLMs move extremely fast and a lot of improvements are done in a short amount of time.
And there might be a point to these arguments, vaguely. However:
There never seems to be - any - kind of counter example or reasoning behind the rationale. You have an in depth and empirical study, done by researchers who, frankly, now their shit (most of the time)
And on the other hand a random internet comment saying "nope" because...the models aren't the latest.
If the latest models really would make a difference, you should at least provide some kind of evidence towards that. As it stands though, every time these comments come up this is missing.
There seems to just be a vaguely defined understanding that "everything changes all the time, and nothing you ever research is transferable to state-of-the-art models"
Which brings me to my second point about these kinds of arguments:
LLM models often - aren't - fundamentally different. Yes, they are vastly more capable. And yes, there are emergent properties. But at their core, they function very much similarly. And for quite a while now, there have not been any of these drastic changes we saw when LLMs first become "good enough" for agentic coding.
I am tired of dismissing empirical evidence and studies every. single. time for reasons without evidence and seemingly a vague sense of "no, but my model is different"
I tried to get my lawyer-mom switched to Linux a bit ago, and she loved it in generally, but none of the Office competitors had good enough compat to work.
The only other thing keeping her on Windows is Adobe PDF pro, since it can do OCR where, when you edit it, it reflow the text in a font that matches the scanned in one to look like the original. (I also got weird "This feels like it enables fraud vibes" from this, but, no, turns out it's a totally common workflow for lawyers to need to do this ... I hate it.)
Cool to see 1/2 of the problems keeping her on Windows solved.
> Planning improves success at additional cost for weaker models but mainly reduces cost, with small decreases in success rate, for stronger models.
> Predefined tools raise success rates for models with weak bash control, whereas bash-only yields higher success at lower cost for bash-capable models, most clearly on shell-centric task types.
> context management extends execution trajectories without substantially altering agent behavior and is most beneficial under tight context budgets
> planning sustains the trajectories of models that abandon tasks too early and trims repeated verification in models that verify too long
> structured tools support models with limited shell proficiency, while bash-only enables capable models to combine multiple code modifications in a single tool call
Seems fairly intuitive to me, based on feeling. But also fairly kind of obvious; bash-only tooling has higher success for bash-capable models, compared to using predefined tools for models that aren't good at bash? Yeah... They all seem a bit "duh" to me. The final piece of the conclusion is agreeable regardless of how they arrived at it though:
> Harness design is thus a conditional systems problem in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default.
I think lots of people treat the harness/model/prompts combo as interchangeable, but in my experience the quality and efficiently depends heavily on the combo of the harness/model, and using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses. Most likely because they use their own traces when training future model iterations.
bash scripts, famously the last word in software engineering. all these castles of sand we've built atop the beautiful, perfect, timeless Bourne Again SHell. all for naught. fools!
Testing the "obvious", "duh" things is incredibly valuable science. It provides a more solid foundation on which to build because it reduces the assumption space.
> using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses
not really, there was a recent benchmark with claude and codex and it showed no difference in ability with a harness like pi agent compared to their native harnesses, pi was in fact cheaper per task.
It might be 'duh' but it means we need a formal list of what each model is good at, and to pick or change harnesses to closer fit the model. Like an llm recipe book. Not just for remote models, but also local ones where how you run the model is critical too.
Todo/task-tracking tools (TaskCreate/Get/Update/List, TodoWrite) are no longer available on Opus 4.8, Sonnet 5, Fable 5, Mythos 5, and newer models; set CLAUDE_CODE_ENABLE_TODO_TOOLS=1 to bring them back"
Anthropic appears to agree frontier models don't need in-session planning tools.
Cool study, we definitely need more principled studies on the role of harnesses. I'd also say that there aren't too many benchmarks where the more complicated harnesses consistently outperform extremely simple agents. But I'm also biased, because I wrote https://github.com/swe-agent/mini-swe-agent/ , which is probably the most minimal agent out there (it started as just 100 lines, all included), and it's used in a lot of benchmarks like DeepSWE, terminalbench, programbench (seems like it's still top of the ranking for TB3, but wasn't evaluated with the best models on TB4).
I'd really love to see more studies about effectiveness of AI in general. As in, what works best and how to use it and such.
Because I feel that the technology and space is - so - hyped and fast moving that a lot of cultish feeling rituals seem to pop up, none of which are backed by evidence. Anthropic openly recommend giving the agents.md file an architectural overview of the code, and the one time this was studied they found the opposite - that the agents.md file is best for concrete commands about how to build stuff and such, and - not - huge overviews. This was, and still is, the official recommendation from Anthropic as far as I can tell.
And then there are the benchmarks, how feel vague and not concrete, and everyone kind of knows they're not the best cuz you can't just assign these tools one fixed number ( for multiple reasons ), but everyone still looks at them and compares them.
People share skills and superpowers and plugins and mcps and very, very few of them have and kind of proof they do much at all.
It all feels a bit weird to me, and I've been on the lookout for exactly these kinds of studies more lately, because I think having this research, even if not done on the exact newest models or not the exact, newest thing, are still - vastly - superior to the alternative.
Haven't gone through full PDF as its very detailed, few things have resonated with me so far.
Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc.
You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model).
Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.
As far as I can tell, the paper says "bash capable", without ever describing what that means. How would one know whether a given model is "bash capable" or not?
I would have to imagine, that Luna would very much fall into the camp of "bash capable". At which point- it seems to me that adding any tools beyond just Bash requires some rigorous testing and verification that value is being added.
This is done on Nemotron models + mistral, so its not very relevant to the current frontier of cheap chinese models + big models from Claude/GPT. Big miss not having qwen or deepseek in this research.
The focus of the study was the different harness approaches and how they scale across model sizes. The fact that they used any particular set of models is irrelevant.
I'm not totally convinced that models are fungible, the claudes/gpts/Gemini all have pretty individual feels when you're working with them. I wouldn't be surprised if the approaches don't scale or even work the same in a poly model setup
Agree. Harnesses are effective because they interact with the underlying model effectively. If the latest models were fundamentally different, excluding them would be a miss. But I don’t think they are, at least not in ways that would affect these observations.
I think the confounding issue is that by now, millions of sessions of Claude Code and Codex are now in the training set for these models. So they have been trained to work the way these harnesses are configured, and at least in the case of Claude Code the harness itself is greatly stripped down because the model has absorbed it.
Nope. Sorry. Not how this works.
How does it work then?
I'm going to cherry pick one example where newer models are noticeably improving at least in my experience.
What is a noticeable improvement with something that struggles to read a message longer than 200 characters without missing information in the middle, may be a 0.000000001% improvement with a model that... almost never misses info in the first place.
I have to say, I am starting to hate this line of reasoning. Yes, LLMs move extremely fast and a lot of improvements are done in a short amount of time.
And there might be a point to these arguments, vaguely. However:
There never seems to be - any - kind of counter example or reasoning behind the rationale. You have an in depth and empirical study, done by researchers who, frankly, now their shit (most of the time)
And on the other hand a random internet comment saying "nope" because...the models aren't the latest.
If the latest models really would make a difference, you should at least provide some kind of evidence towards that. As it stands though, every time these comments come up this is missing.
There seems to just be a vaguely defined understanding that "everything changes all the time, and nothing you ever research is transferable to state-of-the-art models"
Which brings me to my second point about these kinds of arguments:
LLM models often - aren't - fundamentally different. Yes, they are vastly more capable. And yes, there are emergent properties. But at their core, they function very much similarly. And for quite a while now, there have not been any of these drastic changes we saw when LLMs first become "good enough" for agentic coding.
I am tired of dismissing empirical evidence and studies every. single. time for reasons without evidence and seemingly a vague sense of "no, but my model is different"
Awesome work!
I tried to get my lawyer-mom switched to Linux a bit ago, and she loved it in generally, but none of the Office competitors had good enough compat to work.
The only other thing keeping her on Windows is Adobe PDF pro, since it can do OCR where, when you edit it, it reflow the text in a font that matches the scanned in one to look like the original. (I also got weird "This feels like it enables fraud vibes" from this, but, no, turns out it's a totally common workflow for lawyers to need to do this ... I hate it.)
Cool to see 1/2 of the problems keeping her on Windows solved.
The conclusions:
> Planning improves success at additional cost for weaker models but mainly reduces cost, with small decreases in success rate, for stronger models.
> Predefined tools raise success rates for models with weak bash control, whereas bash-only yields higher success at lower cost for bash-capable models, most clearly on shell-centric task types.
> context management extends execution trajectories without substantially altering agent behavior and is most beneficial under tight context budgets
> planning sustains the trajectories of models that abandon tasks too early and trims repeated verification in models that verify too long
> structured tools support models with limited shell proficiency, while bash-only enables capable models to combine multiple code modifications in a single tool call
Seems fairly intuitive to me, based on feeling. But also fairly kind of obvious; bash-only tooling has higher success for bash-capable models, compared to using predefined tools for models that aren't good at bash? Yeah... They all seem a bit "duh" to me. The final piece of the conclusion is agreeable regardless of how they arrived at it though:
> Harness design is thus a conditional systems problem in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default.
I think lots of people treat the harness/model/prompts combo as interchangeable, but in my experience the quality and efficiently depends heavily on the combo of the harness/model, and using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses. Most likely because they use their own traces when training future model iterations.
> bash-only tooling has higher success for bash-capable models, compared to using predefined tools for models that aren't good at bash?
No. The conclusion is that:
bash-capable models + bash-only tools > bash-capable models + predefined tools
In other words, MCP was just a bunch of bullshit that maybe helped a little bit until the models got good at bash, and now it's basically useless.
bash scripts, famously the last word in software engineering. all these castles of sand we've built atop the beautiful, perfect, timeless Bourne Again SHell. all for naught. fools!
"Everybody knows foul air causes sickness."
"Duh, of course Mars has canals."
Testing the "obvious", "duh" things is incredibly valuable science. It provides a more solid foundation on which to build because it reduces the assumption space.
> using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses
not really, there was a recent benchmark with claude and codex and it showed no difference in ability with a harness like pi agent compared to their native harnesses, pi was in fact cheaper per task.
It might be 'duh' but it means we need a formal list of what each model is good at, and to pick or change harnesses to closer fit the model. Like an llm recipe book. Not just for remote models, but also local ones where how you run the model is critical too.
Todo/task-tracking tools (TaskCreate/Get/Update/List, TodoWrite) are no longer available on Opus 4.8, Sonnet 5, Fable 5, Mythos 5, and newer models; set CLAUDE_CODE_ENABLE_TODO_TOOLS=1 to bring them back"
Anthropic appears to agree frontier models don't need in-session planning tools.
https://github.com/anthropics/claude-code/issues/80487