Useful comparison. One thing a single-shot, single-file test can't show is how much of the difference is the harness rather than the model. In day to day use the same model behaves pretty differently between harnesses on an existing codebase - tool calling style, how much context it pulls in, whether it verifies its own edits. Would be curious to see this matrix on a task that modifies a multi-file project instead of generating one self-contained HTML. Greenfield single-file output is close to what these models see in training, so it tends to flatter all of them.
The Astra version seems to have used three.js r170, which is from October 2024. Sol used an even earlier version. GLM's code used the latest version, but I think it's just getting three.js@latest from jsdelivr so it's unlikely to be writing code against that version. Qwen on OpenCode also fetches from jsdelivr, but using a pinned version at r160.
I don't think any of these examples are using things like tone mapping so they're stuck in sRGB (AgX or ACES look much better), they're not using the node materials (good for programmatic texture implementation), and they're not doing anything cool like baking shadow environments or using post-processing effect.
They're nice, but I think they're showing how far behind AI models are on this sort of project rather than how good they are.
Really appreciate this perspective. It’s tempting to stop at being amazed at what these models can do, but my experience matches yours - they need help with details to do well.
But I am annoyed at these GUIs implementing features I don’t care about. I want them to just wrap my harness and forward it to my iPhone, but they can’t help themselves from feature creep.
I did try this - I found it overly dev focused - e.g. git +/- changes in left sidebar. I wanted something close to what I am use to with Codex/Claude - just for open models.
I wish there was another column with the estimated cost for each, with a specific date.
Ideally also finding somehow (not sure what would be the right away) what is publicly available before running the test. It's quite a different outcome if there are competitions, e.g. js13k, live code examples from books, even templates, on specific that topic. Visually here the results looks very very similar to the point that I can't help but wonder if it's the result from the short yet relatively descriptive prompt or because some template was always found and relied on.
Nice. But these tests raise a question what results are reproducible, the final visual, time, tool calling or it's mostly noise. Like, have you tried same combination or model and harness multiple times?
It’s really interesting how much little choices make the result better or worse. Astra and one of the GLMs added bright lights, and thus looked so much better to my eye.
Genuinely happy with some of the Qwen 3.8 results (especially since I can run that model at Q8).
Interesting to see how much better (at this task) Pi (OMP) is over Opencode as a harness.
I’d love to see a few more with outcomes that are as easy to judge but less subjective.
I’ve got a toy project going to make a fun to watch battle simulator where an LLM (or two if playing vs) has to write programs that control multiple bots (each with their own line of sight and limited battle context) that have to coordinate and fight alongside each other. Goal is to have the LLM update the code based on current situations maybe 5-10 times in a 5 min simulated battle. Exploring even allow the bots to request new programming and score based on number of reprogram steps.
I love everything about OpenCode except for the stuff it outputs. On paper it has everything I want in a harness and more, but it tends to struggle to deliver the desired outcome. Last I used it the context was massive and tool calls were reliably unreliable. Is it worth revisiting as a daily driver?
Useful comparison. One thing a single-shot, single-file test can't show is how much of the difference is the harness rather than the model. In day to day use the same model behaves pretty differently between harnesses on an existing codebase - tool calling style, how much context it pulls in, whether it verifies its own edits. Would be curious to see this matrix on a task that modifies a multi-file project instead of generating one self-contained HTML. Greenfield single-file output is close to what these models see in training, so it tends to flatter all of them.
The Astra version seems to have used three.js r170, which is from October 2024. Sol used an even earlier version. GLM's code used the latest version, but I think it's just getting three.js@latest from jsdelivr so it's unlikely to be writing code against that version. Qwen on OpenCode also fetches from jsdelivr, but using a pinned version at r160.
I don't think any of these examples are using things like tone mapping so they're stuck in sRGB (AgX or ACES look much better), they're not using the node materials (good for programmatic texture implementation), and they're not doing anything cool like baking shadow environments or using post-processing effect.
They're nice, but I think they're showing how far behind AI models are on this sort of project rather than how good they are.
Really appreciate this perspective. It’s tempting to stop at being amazed at what these models can do, but my experience matches yours - they need help with details to do well.
What is DSH / PTC?
Btw in my quest for a good desktop codex/claude like app - https://github.com/openchamber/openchamber - this seems to be front-runner. I pair it with OMP via https://github.com/alvins82/omp-openchamber-server/.
I wanted a powerful GUI+harness setup for open models so I could use/test as they came out.
I like Paseo https://github.com/getpaseo/paseo
But I am annoyed at these GUIs implementing features I don’t care about. I want them to just wrap my harness and forward it to my iPhone, but they can’t help themselves from feature creep.
I did try this - I found it overly dev focused - e.g. git +/- changes in left sidebar. I wanted something close to what I am use to with Codex/Claude - just for open models.
I wish there was another column with the estimated cost for each, with a specific date.
Ideally also finding somehow (not sure what would be the right away) what is publicly available before running the test. It's quite a different outcome if there are competitions, e.g. js13k, live code examples from books, even templates, on specific that topic. Visually here the results looks very very similar to the point that I can't help but wonder if it's the result from the short yet relatively descriptive prompt or because some template was always found and relied on.
I've just pushed the transcripts of each if that helps. https://github.com/alvins82/hangar-harness-model-tests
Nice. But these tests raise a question what results are reproducible, the final visual, time, tool calling or it's mostly noise. Like, have you tried same combination or model and harness multiple times?
I have on Qwen3.8 27B. Results are close enough across multiple runs.
So it's not reproducible.
It’s really interesting how much little choices make the result better or worse. Astra and one of the GLMs added bright lights, and thus looked so much better to my eye.
Genuinely happy with some of the Qwen 3.8 results (especially since I can run that model at Q8).
Interesting to see how much better (at this task) Pi (OMP) is over Opencode as a harness.
I’d love to see a few more with outcomes that are as easy to judge but less subjective.
I’ve got a toy project going to make a fun to watch battle simulator where an LLM (or two if playing vs) has to write programs that control multiple bots (each with their own line of sight and limited battle context) that have to coordinate and fight alongside each other. Goal is to have the LLM update the code based on current situations maybe 5-10 times in a 5 min simulated battle. Exploring even allow the bots to request new programming and score based on number of reprogram steps.
Interesting also that Astra and Sol didn't check visually using the browser. I suspect they would have done even better if they had.
I love everything about OpenCode except for the stuff it outputs. On paper it has everything I want in a harness and more, but it tends to struggle to deliver the desired outcome. Last I used it the context was massive and tool calls were reliably unreliable. Is it worth revisiting as a daily driver?
Why not ohmypi?
So where did most of them get the idea of calling it hanger 07?
Should I be able to see the actual results? Because I don’t see a hangar anywhere.
File -> Open
If you don't have anything working check the console, maybe a WebGL issue.
Click Open in the file column
What I'm missing in all of those one-shot tests:
How different are the results between multiple runs of the same setup?
I have done multiple runs on Qwen3.8 27B. Results are close enough across multiple runs qualitatively.
Refreshing to see a post that gets straight to the point and not hide the data behind pages of prose.
Qwen with open code seemed like the best to me. It seemed like the best balance between performance and visuals