This seems a strange thing to test given that Claude Code is optimized for Anthropic models.
A test of how different models perform in more of a model-agnostic harness like OpenCode would be much more interesting, perhaps paired with a control experiment of how those same models performed on the same task when using their respective native harnesses.
Yeah, it's an interesting test because he gave Kimi credit for running inside the harness he liked, which I presume is Claude Code. But he also said it was noted for its harness sensitivity, and then said it didn't do as well as Opus.
A lot of folks don't seem to understand that it's the harness and its behavior paired with a particular model that produces the result, and you have to focus on both aspects. He barely acknowledges the importance of the harness in his write-up, beyond the two notes above.
There's a very interesting benchmark comparison below of Opus 4.7 run under three different harnesses : OpenCode, Cursor and Claude Code where it's not very close at all and Opus's native harness, Claude Code, performs worst of all three.
The pass@1 scores are 50/45/40 for OpenCode/Cursor/Claude Code respectively.
This seems a strange thing to test given that Claude Code is optimized for Anthropic models.
A test of how different models perform in more of a model-agnostic harness like OpenCode would be much more interesting, perhaps paired with a control experiment of how those same models performed on the same task when using their respective native harnesses.
Yeah, it's an interesting test because he gave Kimi credit for running inside the harness he liked, which I presume is Claude Code. But he also said it was noted for its harness sensitivity, and then said it didn't do as well as Opus.
A lot of folks don't seem to understand that it's the harness and its behavior paired with a particular model that produces the result, and you have to focus on both aspects. He barely acknowledges the importance of the harness in his write-up, beyond the two notes above.
There's a very interesting benchmark comparison below of Opus 4.7 run under three different harnesses : OpenCode, Cursor and Claude Code where it's not very close at all and Opus's native harness, Claude Code, performs worst of all three.
The pass@1 scores are 50/45/40 for OpenCode/Cursor/Claude Code respectively.
https://artificialanalysis.ai/agents/coding-agents#harness-c...
I've seen other benchmarks where Pi also outperforms Claude Code both in model performance and in much reduced token usage.