I keep seeing Astra make beautiful 3d stuff online, yet when I feed it some old school RuneScape assets (even tried with some very detailed guidelines) and asked it to generate some new plausible assets it failed horribly.
I think there’s still something really off with current (frontier) models when it comes to creating “novel” stuff? Even 2004 style graphics..
> That’s the problem, these tests can’t tell you how good a model is anymore because it’s trivial for labs to optimise for exactly these tests by the next release.
But they don't prove the claim. Are the models amazing at recreating Minecraft, but the second you swap the word Minecraft out with another game or a custom game, it shits the bed?
That's not what I see. My feed is full of people using Astra to recreate all sorts of games from Diablo to some random idea they came up with, in ridiculously polished detail like animations that would have taken me weeks of iteration in gpt-5.6-sol but it was a single shot by Astra.
Can you point me to one of these? The only Astra game I tried was the incredibly janky Mario Kart clone from OpenAI where you could fly by spamming jump
For every person using Astra to create a game from some random idea and sharing the impressive result, there are an unknown number who have tried the same thing and gave up in frustration.
This sounds unconvincing, because a) pelican test is subjective, there's simply nothing to leak as it has no available direct answers and maybe an extremely faint preference signal, and b) the same small models actually do perform well when you change the subject. Some models are genuinely trained to be better at some domain, such as 2D layouts or vector graphics in this case. It all depends on particular recipes and datasets. Which is the actual reason these tests are poor as vibe checks: they don't do anything to disentangle generalization, memorization, and training preference. One-shotting popular software in particular is definitely not a good test of anything as memorization is going to dominate it.
AAII is also not very useful, neither is any generic score/benchmark. If you want a weather forecast you aren't looking at the average temperature of Earth.
(actually when did the term "one-shot" get hijacked to mean something other than "one example"?..)
> Some models are genuinely trained to be better at some domain, such as 2D layouts or vector graphics in this case. It all depends on particular recipes and datasets.
I'd argue this is inherently true for every single model today, none of them have completely generalized to be able to solve any task, so whenever people come up with new evaluations and benchmarks, all the models score relatively poorly initially, until researchers start to tune the models to do well in the domains that the evaluations and benchmarks tests, and then we see strong improvements in that domain, which then tapers out to incremental improvements, and some other domain is chosen to be the new focus.
Models aren't better agents today merely by chance, but because it's explicitly part of the training data. They do well with software because we've talked so much about software on the internet until this point and that's part of the training data, but pit them against problems people don't talk so much about, and if the labs creating and training these models didn't consider those problems, then the model will pretty much suck at it.
I guess eventually they will literally cover every single task the model could ever come across, at least some variant/permutation of it, but until then every benchmark/evaluation will just uncover "did the labs consider this and who considered it most important before/during training?" basically.
Build a flowsheet based steady state chemical process simulator, then use it to simulate and optimize a full scale oil refinery.
1) Building a solver engine that works at this scale is not a trivial problem, and the successful ones rely more on heuristics than some categorically different solution approach.
2) Defining the engineering equations relevant to this task is relies on understanding what level of fidelity is required to answer the questions people ask of steady state process models.
3) Knowing the thermophysical properties of chemicals and crude oils is possible from the open literature, but the information is diffuse and different correlations are applicable in different situations.
4) Creating a GUI which converts a flowsheet into matrix math is non-trivial, although a sequential modular approach is a bit easier.
5) Defining large scale models in such a way that they solve robustly is as much art as science. For example, completely closed recycle loops like refrigeration systems are a nightmare for solvers, so it is often better to define them in an open-loop way.
6) Optimization involves knowing the relevant commodity prices, but more importantly how to define the constraints on the model so it doesn't just say to produce infinite gasoline.
7) Troubleshooting the inevitable convergence failures is also as much art as science. There are a large number of diagnostic techniques, but fundamentally you need to be able to relate what is happening during the solver iterations with the intent of your model because more often than not the problem is that you've asserted something impossible, redundant, or irrelevant.
I partially owe my career to benchmarking AI with Minecraft, so, I'm going to disagree with the author on this one. Games are a great way to test a model, it's not that deep.
Though I somewhat think some benchmarks are silly like the article says. I saw someone on YouTube recently take a picture of a building across the street from them (seemed like it was in NYC), and asked GPT 6 Astra to make it in blender. It did a surprisingly good job in 30 minutes. So though these benchmarks don't seem to mean much you could always add a touch of randomness to them like the person in the YouTube video did, but the problem with that is how would you compare the benchmarks in any clear way if they aren't even consistent? Regardless it appears LLMs are getting this good at the general task and not just at the particular instances of said task.
@notch Hey Markus, can I get a refund on my alpha distro of minecraft? I think the currency is worth more than it used to be considering how many versions there are now :)
Running Firefox here, no problems whatsoever even with UBO and Privacy Badger disabled (the twitter embeds get blocked at the DNS level, but I doubt those are the problem).
Mac here. There's nothing all that interesting going on with the JS on that page, so I would suspect a bug with Firefox and/or your desktop environment.
Most of my and my peers PortCos run their own eval and benchmark sets, simply because they know what they need best.
The reality is, capabilities have largely converged across foundation models over the last 18 months, and much of the value add is coming from the harness layer itself now.
This has been the operating assumption for me and my peers, and has largely played out that way.
That said, this has always been an issue with benchmarking since the very beginning. DB Benchmarks, compute benchmarks, and others that were external facing were always inherently a content and product marketing tool. The actual internal benchmarking used to model, understand, and enhance your product was always a closely held secret.
Most of these conversations are happening, but largely in person and not on HN.
It's not "models have stagnated" but "models released at the same time are on the same level". Improvements are still real but the relative gaps between OpenAI, Anthropic, Meta, Grok, Gemini and open models are closer than ever. That doesn't mean progress is slowing down, it's just more widely distributed.
I’ve used Astra for the past day and a half. My layperson’s review is that it is impressive at computer use and 3D reasoning, and fails in similar ways to 5.6 Sol at similar rates when it comes to coding. I have no idea how it scored so high on SWE benchmarks because so far it has been very “mid” as the kids say.
I keep seeing Astra make beautiful 3d stuff online, yet when I feed it some old school RuneScape assets (even tried with some very detailed guidelines) and asked it to generate some new plausible assets it failed horribly.
I think there’s still something really off with current (frontier) models when it comes to creating “novel” stuff? Even 2004 style graphics..
Or am promoting it wrong?
It's funny seeing this when I was going to mention runebench
https://maxbittker.github.io/runebench/
Did you try blender MCP and prompt for low-poly vertex colored assets?
> That’s the problem, these tests can’t tell you how good a model is anymore because it’s trivial for labs to optimise for exactly these tests by the next release.
But they don't prove the claim. Are the models amazing at recreating Minecraft, but the second you swap the word Minecraft out with another game or a custom game, it shits the bed?
That's not what I see. My feed is full of people using Astra to recreate all sorts of games from Diablo to some random idea they came up with, in ridiculously polished detail like animations that would have taken me weeks of iteration in gpt-5.6-sol but it was a single shot by Astra.
> ridiculously polished detail
Can you point me to one of these? The only Astra game I tried was the incredibly janky Mario Kart clone from OpenAI where you could fly by spamming jump
X.com is full of unverified posts, but here's a few:
- https://x.com/MengTo/status/2096213835460084184?s=20
- https://x.com/birdabo/status/2096101381925114327?s=20
Seems really impressive at first glance, but there's no guarantee that they are real
You are falling for selection bias.
For every person using Astra to create a game from some random idea and sharing the impressive result, there are an unknown number who have tried the same thing and gave up in frustration.
I call bullshit
This sounds unconvincing, because a) pelican test is subjective, there's simply nothing to leak as it has no available direct answers and maybe an extremely faint preference signal, and b) the same small models actually do perform well when you change the subject. Some models are genuinely trained to be better at some domain, such as 2D layouts or vector graphics in this case. It all depends on particular recipes and datasets. Which is the actual reason these tests are poor as vibe checks: they don't do anything to disentangle generalization, memorization, and training preference. One-shotting popular software in particular is definitely not a good test of anything as memorization is going to dominate it.
AAII is also not very useful, neither is any generic score/benchmark. If you want a weather forecast you aren't looking at the average temperature of Earth.
(actually when did the term "one-shot" get hijacked to mean something other than "one example"?..)
> Some models are genuinely trained to be better at some domain, such as 2D layouts or vector graphics in this case. It all depends on particular recipes and datasets.
I'd argue this is inherently true for every single model today, none of them have completely generalized to be able to solve any task, so whenever people come up with new evaluations and benchmarks, all the models score relatively poorly initially, until researchers start to tune the models to do well in the domains that the evaluations and benchmarks tests, and then we see strong improvements in that domain, which then tapers out to incremental improvements, and some other domain is chosen to be the new focus.
Models aren't better agents today merely by chance, but because it's explicitly part of the training data. They do well with software because we've talked so much about software on the internet until this point and that's part of the training data, but pit them against problems people don't talk so much about, and if the labs creating and training these models didn't consider those problems, then the model will pretty much suck at it.
I guess eventually they will literally cover every single task the model could ever come across, at least some variant/permutation of it, but until then every benchmark/evaluation will just uncover "did the labs consider this and who considered it most important before/during training?" basically.
The benchmark I want to see people adopt is:
Build a flowsheet based steady state chemical process simulator, then use it to simulate and optimize a full scale oil refinery.
1) Building a solver engine that works at this scale is not a trivial problem, and the successful ones rely more on heuristics than some categorically different solution approach.
2) Defining the engineering equations relevant to this task is relies on understanding what level of fidelity is required to answer the questions people ask of steady state process models.
3) Knowing the thermophysical properties of chemicals and crude oils is possible from the open literature, but the information is diffuse and different correlations are applicable in different situations.
4) Creating a GUI which converts a flowsheet into matrix math is non-trivial, although a sequential modular approach is a bit easier.
5) Defining large scale models in such a way that they solve robustly is as much art as science. For example, completely closed recycle loops like refrigeration systems are a nightmare for solvers, so it is often better to define them in an open-loop way.
6) Optimization involves knowing the relevant commodity prices, but more importantly how to define the constraints on the model so it doesn't just say to produce infinite gasoline.
7) Troubleshooting the inevitable convergence failures is also as much art as science. There are a large number of diagnostic techniques, but fundamentally you need to be able to relate what is happening during the solver iterations with the intent of your model because more often than not the problem is that you've asserted something impossible, redundant, or irrelevant.
I partially owe my career to benchmarking AI with Minecraft, so, I'm going to disagree with the author on this one. Games are a great way to test a model, it's not that deep.
Though I somewhat think some benchmarks are silly like the article says. I saw someone on YouTube recently take a picture of a building across the street from them (seemed like it was in NYC), and asked GPT 6 Astra to make it in blender. It did a surprisingly good job in 30 minutes. So though these benchmarks don't seem to mean much you could always add a touch of randomness to them like the person in the YouTube video did, but the problem with that is how would you compare the benchmarks in any clear way if they aren't even consistent? Regardless it appears LLMs are getting this good at the general task and not just at the particular instances of said task.
@notch Hey Markus, can I get a refund on my alpha distro of minecraft? I think the currency is worth more than it used to be considering how many versions there are now :)
I cant select text nor click links on this page with firefox.
Running Firefox here, no problems whatsoever even with UBO and Privacy Badger disabled (the twitter embeds get blocked at the DNS level, but I doubt those are the problem).
Works on phone but not on my linux desktop.
Mac here. There's nothing all that interesting going on with the JS on that page, so I would suspect a bug with Firefox and/or your desktop environment.
Who cares? Restart your browser maybe
Most of my and my peers PortCos run their own eval and benchmark sets, simply because they know what they need best.
The reality is, capabilities have largely converged across foundation models over the last 18 months, and much of the value add is coming from the harness layer itself now.
This has been the operating assumption for me and my peers, and has largely played out that way.
That said, this has always been an issue with benchmarking since the very beginning. DB Benchmarks, compute benchmarks, and others that were external facing were always inherently a content and product marketing tool. The actual internal benchmarking used to model, understand, and enhance your product was always a closely held secret.
Most of these conversations are happening, but largely in person and not on HN.
> capabilities have largely converged across foundation models over the last 18 months
For reference, in March '25 the models du jour were Sonnet 3.7, gpt o4 and gemini 2.5 pro. GPT5 was in august '25.
It's been a while since we've heard the old "models have stagnated". Oh well.
It's not "models have stagnated" but "models released at the same time are on the same level". Improvements are still real but the relative gaps between OpenAI, Anthropic, Meta, Grok, Gemini and open models are closer than ever. That doesn't mean progress is slowing down, it's just more widely distributed.
Ah, I see. I misunderstood then. The thing about "gains come from the harness" made me think about it in that way.
This, and depending on the workflow and usecase, you don't necessarily need the latest and greatest with the right kind of harness engineering.
Like everything in engineering, it's about tradeoffs and what works best for your specific problem.
I’ve used Astra for the past day and a half. My layperson’s review is that it is impressive at computer use and 3D reasoning, and fails in similar ways to 5.6 Sol at similar rates when it comes to coding. I have no idea how it scored so high on SWE benchmarks because so far it has been very “mid” as the kids say.