> I think I’m suspecting something is going “wrong” in the training process. The model is greatly rewarded for succeeding on long-horizon tasks, but presumably there is very little punishing going on for “shitty code.”
My suspicion is that both OpenAI and Anthropic moved their RL agendas from "being rated as useful according to human feedback" to "succeeds at long horizon tasks" in the last few months, resulting in agents that are closer to AGI in an autonomous task-completing sense, but strangely bad at communicating.
The result is that they are amazingly good at long horizon tasks, computer use, solving difficult math/ARC-AGI type problems, but becoming weirder and weirder to work with.
> I’m more and more convinced that all of AI engineering is Neijuan (内卷, meaning curl inwards). In China it describes a system that demands ever more effort and competition without improving output. The way in which it sometimes shows up in the West is the 996 nonsense. The English term for Neijuan is “Involution” from the book Agricultural Involution. Agricultural involution describes the intensification of farming that raises productivity per square meter while leaving productivity per head unchanged.
it reminds me of a thread I read on PTT, Taiwan's Reddit. AI finally achieved what humans could not. Managers must give exact context for what they want, must pay exact wages (tokens), and can't delay salary payments (which seems to be a problem in China).
51 comments so far, the vast majority panning Astra's coding abilities. An uninformed reader may come away with the impression that this isn't an absolutely revolutionary technology that with coding abilities many of us thought were not even going to be possible with language models as recently as a year ago.
Yeah, it's not perfect, but it's really good and extrapolating this rate of improvement for 6 months is rather terrifying (from a SWE perspective, at least).
> the models are also just not for me as a software engineer (...) these models increasingly are for other people. For lawyers, 3D artists, mathematicians
This is a good observation, perhaps AI will not completely replace humans ins software engineering because by the time it has the capability to do so like in write a prompt and get a CRM coded for you, tokens are so expensive that you are better off spending them to substitute other disciplines (what about automating the work of the customers that would become records in that CRM?).
Early lesson I learned from AI engineering was - there is no substitute to giving a groomed epic to an agent. Instead of simply saying 'implement themes in my product' you need to be specific, in fact more specific than usual. You need to say exactly what is in scope and what's not, even down to a buttons, events and layouts.
You can groom the epic with the help of AI, but final review must be done by someone who can take ownership of the specs and hence is responsible if something has fallen through the cracks. AI's response will be limited by the output tokens of that specific agent, and there will no repercussions for AI even if it accepts its mistakes.
I’ve observed exactly these patterns with Opus and Fable as well - for example, forgetting that they can edit files and instead use python scripts as a patching tool…
Sounds like you don't have enough experience with coding agents. Deterministic scripts must always be preferred instead of LLM tool calls.
In fact, you should instruct your agents to write code to execute instead of letting them call tools.
Sounds like you completely lack all reading comprehension ability
LLMs sometimes like to execute one-off Python scripts to make edits to files rather than just calling the edit tool directly. Both are tool calls so saying that you should have it write code instead of doing tool calls makes no sense because writing code is a tool call for it...
gpt-6-astra is a bitch, it constantly scope creeps itself with "yet another thing" to give it that darn polished lick. the results are eventually a little bit better but at what cost? let's do the math.
gpt-5.6-sol: 1x base
gpt-6-astra 2.5x base in subscription
then gpt-6-astra tends to spawn subagents a lot, often with all kinds of models such as gpt-5.6, 5.3-codex etc., which is neat. it's a good coordinator but even more cost.
and then it tends to run _full test suites_ over an over again (each costs like 15 minutes) just to verify that _one test_ was fixed etc., and does so for as long as until the test is fixed, eventually accumulating 2 hours or so.
yesterday I assigned it a task to rebase my changs in a repo onto the latest upstream changes. while gpt-5.6-sol consistently took like an hour to do so end-to-end, astra ran for more than 6 hours and still wasn't done. it kept finding "one more thing" that was goldplating that I didn't ask for.
Even better, use some kind of local-ci runner that does deterministic builds from a dependency graph. No changes, no build, massive parallelism if you want it
They don't always have a great concept of time so for something like running a full test suite that takes a long time you should just tell it not to do that
I've observed the same thing where the new models want to run obscene bash commands or python scripts which are completely unreadable and utilise every option flag that exists.
It's impossible to review. These commands are less readable than regex.
If these tools are as clever as they seem then why not just tell them to rewrite the code in a more review friendly style?
I only dabble in the use of LLMs to generate code for hobby programming (I'm retired from software development) so I don't use any specialised tools.
I almost always have to tell ChatGPT (via Duck AI usually) to rewrite several times even when it has produced a workable script just because it has often used some unnecessarily roundabout way of achieving something. Usually with extra prompting I can get something that is both more efficient and more readable.
I noticed that too so I appended to Claude Code’s system prompt a reminder to use the standard read/write tools, but since Claude Code switched to default auto-mode, I’ve seen it imply that the auto-mode tooling encourages the use of bash-only commands (sed, python, etc) which has a whole slew of negative side affects.
Yes.. this happened recently. I basically always use auto mode, and when I asked it why it kept editing code with python, it explained that this is part of its prompt when auto mode is turned on.
I can only assume that's because their safety verification model is better at such snippets or something, but it means that the whole write tool they have which actually shows you the changes as they happen is just unused and it makes it more annoying to follow along.
My own observations are that I used to target turn lengths of 10-15 minutes and these new models (since 5.6) extended that a bit to ~25 minutes, as they tend to do more tests and reviews. Targeting hours-long turns makes as much sense, as putting on cruise control and going to sleep.
In SWE I've found gpt-6-astra (high) inconsistent and oddly focused on overtly taking responsibility for mistakes it made rather than prioritizing concrete steps to rectify problems. Such steps once elicited are often either incomplete or beyond the scope.
Maybe it's just vibes but I've repeatedly felt like gpt-6-astra on its default setting of medium is less rigorous and thoughtful than gpt-5.6-sol on its default setting. What I am certainly not getting is any sense that we are at "AGI" yet.
> I’m more and more convinced that all of AI engineering is Neijuan (内卷, meaning curl inwards). In China it describes a system that demands ever more effort and competition without improving output
I don't know what to say, except that articles exactly like this one have been showing up constantly for the last three years, and literally all of them were obviously outdated and irrelevant within about a month.
> I actually don’t know if the model thinks someone is looking
It 'knows' (from simply training) with an extremely high degree of certainty when its prompt is written by an LLM/itself - and thus will change what it writes.
> And potentially as a byproduct of enabling all of this, you can now slop your way to a one-shot 3D game over the weekend which looks impressive.
I think we've finally reached a weird point where AI has effectively reduced the amount of competition that real game developers have to endure.
Nothing unravels faster than a game project being built with AI. You can achieve impressive results in a day, but you can't get much further than that without actual talent. LLMs will never be able to best a human environment artist at scene composition, especially if that composition needs to be directed with nuance over time.
There's a huge difference between a game that looks impressive and one that feels impressive. You can only achieve games that feel like counter strike, call of duty and overwatch with thousands of hours of human sacrifice. The AI is almost pointless once you get to play testing and balancing. Knowing how much to adjust magical integers isn't a conversation a chat bot can resolve with endless pontification tokens.
>> And potentially as a byproduct of enabling all of this, you can now slop your way to a one-shot 3D game over the weekend which looks impressive.
Do these games really look impressive? Everything I've seen has looked like someone completely new to Unity/Unreal has slapped together a bunch of premade scripts and very poor 3d assets.
Yep. Every single post I've seen about "game development is over" is yet another procedurally generated game. Not only it doesn't prove anything about Astra being "better" at making games (do people have any idea the sheer amount of open source games that do exactly that? You can find thousands of the same planet exploration games through repos, blog posts,etc. Game design schools have it as an exercise, that's how basic it is), but as you said: the wow factor of space exploration is cool, sure. It makes for a catastrophically boring game.
Leaves me to wonder whether the OpenAI glazers just never played games in their lives, or are just really superficial tech bros. Most likely, both.
These machines are doing some crazy things to get to the result. That said, I can't help but feel like this is the compilers argument all over again. Are the methods used to get to the result good? No. Is the code that it generates good? No. Does it achieve the goal. Yes. Is it likely to get better with time. Also yes. In my use cases, jobs that would have taken weeks to months are being done in minutes to hours. Involving complex testing and reasoning and experimentation. I'm no fanboy, but I can't argue against the speed gains. I'm sure we'll still have artisans who hand weave incredible code. But for me, I'm switching to the weaving loom for speed and efficiency.
It seems like we're not supposed to care about the code quality then? I guess that's the compilers argument. But I'm not ready to give up the code just yet.. These LLMs don't even have a stable interface, they change every few months in how they interpret our prompts and tasks.
Similar for me, I don't like the development for many reasons, but that's another discussion. I also can't deny the capabilities.
I use the tools with this "risk analysis":
- If performance doesn't improve I can just always switch back to whatever I've done for the past 10 years, so it's not really a risk to start exploring.
- If performance does improve, then I'm already familiar with it.
The compiler argument is great, if we turn it on its head.
To create professional products, compilers are great, when used by professional or passionate and technical amateurs. They're useless if you're neither.
LLMs are the next step up. They are quite useful if you are neither, and you can get a lot farther with them, which means that low quality software is much easier to create. But if, for whatever reason, you need to create higher quality software (like most software that's actually sold directly or through subscriptions or ads), you're back to the "be a professional or passionate and technical amateur".
Back in the day, the argument was that compilers produce unreadable assembly, so people used to writing assembly were arguing against the use of compilers.
Compilers also had bugs, so we still had to debug the assembly to understand how to fix the problem. Nowadays, almost nobody has to resort to those steps, except of course compiler developers. But that is just a testament to the quality of compilers.
Comparing LLMs to compilers is a take I often see, but I am not sure the comparison quite holds. The problem is that LLMs are inherently non-deterministic, so we always get a different output on the same prompt.
Maybe if LLMs are powerful enough it won't matter. I doubt it but we will see.
What you say is true, the comparison indeed doesn't hold.
But is it relevant? does it matter from a product perspective if LLMs are non-deterministic. You don't need to one shot the correct result, english is ambiguous and LLMs non-deterministic, but you can iterate.
If it's possible to iterate fast and cheap enough, even ambiguous language can produce the results you want, given enough iterations.
There are a lot of ifs and buts here, just a thought on the compiler argument.
I think it's that when some code compiles to say assembler, the compiler doesn't prioritize readability and maintainability of the assembler code, since people are not expected to read and maintain it directly
Lots of really silly people love to compare LLMs to compilers. "You don't look at the compiled code either" and "Back in the day, people also had negative reactions to compilers and wanted to keep writing assembly by hand" and other such nonsense.
Astra is indeed the pinnacle of "black box slop". It is overall a smarter software development agent for many things I do, but the code sometimes is indistinguishable from Brainfuck when writing things like GPU shaders. It doesn't even attempt to make it remotely formatted or readable.
Have you tried identifying exactly what is unreadable about it and telling it to make it more readable?
I had GTP-5.6 write some shader code recently and it wasn't very clear to me. I spent about an hour chatting with the until I understood the concepts and was able to express them back to the AI using math formulas and variables named in a way that made sense to me. The AI then rendered the code using the formula and variables I was familiar with and it was clear to me.
"But for how much more Fable costs, for how much more Astra costs, I do not feel like the results are there."
we are in the middle of the beginning. Its just a weird take to talk about the newest model like this while we are still in a R&D phase.
And these points don't matter if you let it search and analyse a bug, for example, or if you have good harness and a good architecture and let it do small PRs or if you do stuff no one needs to read (yes a software engineer also needs tools)
Ironically I burned out Fable usage early this week because of Astra using it to run inane full codebase reviews over one line changes, so I have been using Astra extensively.
We need a word for “potentially highly capable, but in reality an idiot savant” to describe certain models. No, I don’t need you to write a tmux emulator in bash to test your changes bro, just ask me to run the command.
I love this dance we are doing where when people write the "AI models are garbage machines that produce garbage and are no where close to the fantasy being pedalled by the Crypto bros who pivoted to AI" it always has to be caveated with "AI models are useful and I am highly productive with them"
It feels like people should just be able to say "This article comes with the standard disclaimer" and just dive into the meat of the article without wasting time.
I’ve asked Astra to build me an app for a prototype I created quickly using Sonnet.
It’s been 2 days and it made no real progress on the actual app. It created docs, scripts, workflows, and it’s doing a bunch of reviewing on every PR.
I told it that I just need an MVP.
I’m pretty sure an average senior engineer would have finished that task much quicker, and guaranteed with more readable, higher-quality code. Meanwhile, I think I’ve easily crossed 100k tokens so far on nothing.
Funny world we’re living in that this is “SOTA” and “AGI”.
I’m genuinely curious what these OAI and A/ engineers are working on that they praise these models so much. I did not see any improvement since Opus 4.5.
Also, I’m really unimpressed by any “one shot” demo that’s out there in the wild. It means nothing for serious software engineering.
This matches my experience with Astra so far too.
> I think I’m suspecting something is going “wrong” in the training process. The model is greatly rewarded for succeeding on long-horizon tasks, but presumably there is very little punishing going on for “shitty code.”
My suspicion is that both OpenAI and Anthropic moved their RL agendas from "being rated as useful according to human feedback" to "succeeds at long horizon tasks" in the last few months, resulting in agents that are closer to AGI in an autonomous task-completing sense, but strangely bad at communicating.
The result is that they are amazingly good at long horizon tasks, computer use, solving difficult math/ARC-AGI type problems, but becoming weirder and weirder to work with.
> I’m more and more convinced that all of AI engineering is Neijuan (内卷, meaning curl inwards). In China it describes a system that demands ever more effort and competition without improving output. The way in which it sometimes shows up in the West is the 996 nonsense. The English term for Neijuan is “Involution” from the book Agricultural Involution. Agricultural involution describes the intensification of farming that raises productivity per square meter while leaving productivity per head unchanged.
This resonates
it reminds me of a thread I read on PTT, Taiwan's Reddit. AI finally achieved what humans could not. Managers must give exact context for what they want, must pay exact wages (tokens), and can't delay salary payments (which seems to be a problem in China).
Yeah it's a funny thing - a lot of the things you need to feed the model are things that actually would have helped humans...
But doesn't this reduce the required amount of farmland?
Also known has the Red Queen's Race
51 comments so far, the vast majority panning Astra's coding abilities. An uninformed reader may come away with the impression that this isn't an absolutely revolutionary technology that with coding abilities many of us thought were not even going to be possible with language models as recently as a year ago.
Yeah, it's not perfect, but it's really good and extrapolating this rate of improvement for 6 months is rather terrifying (from a SWE perspective, at least).
> the models are also just not for me as a software engineer (...) these models increasingly are for other people. For lawyers, 3D artists, mathematicians
This is a good observation, perhaps AI will not completely replace humans ins software engineering because by the time it has the capability to do so like in write a prompt and get a CRM coded for you, tokens are so expensive that you are better off spending them to substitute other disciplines (what about automating the work of the customers that would become records in that CRM?).
Early lesson I learned from AI engineering was - there is no substitute to giving a groomed epic to an agent. Instead of simply saying 'implement themes in my product' you need to be specific, in fact more specific than usual. You need to say exactly what is in scope and what's not, even down to a buttons, events and layouts.
You can groom the epic with the help of AI, but final review must be done by someone who can take ownership of the specs and hence is responsible if something has fallen through the cracks. AI's response will be limited by the output tokens of that specific agent, and there will no repercussions for AI even if it accepts its mistakes.
> Instead of simply saying 'implement themes in my product' you need to be specific, in fact more specific than usual.
Around February you could get away with very vague prompts to Claude. I feel lime models have regressed since
I’ve observed exactly these patterns with Opus and Fable as well - for example, forgetting that they can edit files and instead use python scripts as a patching tool…
Why would you use a constrained edit tool when you are also allowed to use the complete power of python?
Because the complete power of Python also includes the power to fuck things up.
Simple is better than complex Complex is better than complicated
Or something, I don't remember...
Why even offer the edit tool in that case? Also, what kind of editing could they possible do what wouldn't be possible with POSIX ed?
this is intentional, afaik agents do better with python and alike than the harness tooling.
Sounds like you don't have enough experience with coding agents. Deterministic scripts must always be preferred instead of LLM tool calls. In fact, you should instruct your agents to write code to execute instead of letting them call tools.
Sounds like you completely lack all reading comprehension ability
LLMs sometimes like to execute one-off Python scripts to make edits to files rather than just calling the edit tool directly. Both are tool calls so saying that you should have it write code instead of doing tool calls makes no sense because writing code is a tool call for it...
gpt-6-astra is a bitch, it constantly scope creeps itself with "yet another thing" to give it that darn polished lick. the results are eventually a little bit better but at what cost? let's do the math.
gpt-5.6-sol: 1x base gpt-6-astra 2.5x base in subscription
then gpt-6-astra tends to spawn subagents a lot, often with all kinds of models such as gpt-5.6, 5.3-codex etc., which is neat. it's a good coordinator but even more cost.
and then it tends to run _full test suites_ over an over again (each costs like 15 minutes) just to verify that _one test_ was fixed etc., and does so for as long as until the test is fixed, eventually accumulating 2 hours or so.
yesterday I assigned it a task to rebase my changs in a repo onto the latest upstream changes. while gpt-5.6-sol consistently took like an hour to do so end-to-end, astra ran for more than 6 hours and still wasn't done. it kept finding "one more thing" that was goldplating that I didn't ask for.
> each costs like 15 minutes
I've got a custom agent loop that will reuse unit testing results if no apply patch operations occurred since the last invoke.
Wall clock time isn't something I would put on the AI provider. That's entirely a consequence of the system that you've brought to the party.
Even better, use some kind of local-ci runner that does deterministic builds from a dependency graph. No changes, no build, massive parallelism if you want it
have you ever worked for a big company where that's the status quo for any tiny change... hours on _full test suites_ over and over again.
They don't always have a great concept of time so for something like running a full test suite that takes a long time you should just tell it not to do that
I've observed the same thing where the new models want to run obscene bash commands or python scripts which are completely unreadable and utilise every option flag that exists.
It's impossible to review. These commands are less readable than regex.
If these tools are as clever as they seem then why not just tell them to rewrite the code in a more review friendly style?
I only dabble in the use of LLMs to generate code for hobby programming (I'm retired from software development) so I don't use any specialised tools.
I almost always have to tell ChatGPT (via Duck AI usually) to rewrite several times even when it has produced a workable script just because it has often used some unnecessarily roundabout way of achieving something. Usually with extra prompting I can get something that is both more efficient and more readable.
I noticed that too so I appended to Claude Code’s system prompt a reminder to use the standard read/write tools, but since Claude Code switched to default auto-mode, I’ve seen it imply that the auto-mode tooling encourages the use of bash-only commands (sed, python, etc) which has a whole slew of negative side affects.
Yes.. this happened recently. I basically always use auto mode, and when I asked it why it kept editing code with python, it explained that this is part of its prompt when auto mode is turned on.
I can only assume that's because their safety verification model is better at such snippets or something, but it means that the whole write tool they have which actually shows you the changes as they happen is just unused and it makes it more annoying to follow along.
My own observations are that I used to target turn lengths of 10-15 minutes and these new models (since 5.6) extended that a bit to ~25 minutes, as they tend to do more tests and reviews. Targeting hours-long turns makes as much sense, as putting on cruise control and going to sleep.
They are probably using Actual Indians. If it takes 25 minutes you can just type the code yourself.
In SWE I've found gpt-6-astra (high) inconsistent and oddly focused on overtly taking responsibility for mistakes it made rather than prioritizing concrete steps to rectify problems. Such steps once elicited are often either incomplete or beyond the scope.
Opus also does this and then writes comments in code or PR descriptions describing how it went wrong earlier in the session.
Just imagine how confused a human would have to be to do that. And we want to trust these clankers to build software.
The biggest issue with LLMs is that they still suck at general contextual awareness and ability to judge what is appropriate.
It does? Must be neat to no longer have Claude code constantly saying "your commit introduced the issue", right after committing itself
Maybe it's just vibes but I've repeatedly felt like gpt-6-astra on its default setting of medium is less rigorous and thoughtful than gpt-5.6-sol on its default setting. What I am certainly not getting is any sense that we are at "AGI" yet.
Most still applaud that as Sol has quite a reputation for over engineering. Not every software needs to go to the moon.
So your experience is that Astra doesn't over engineer? For more than twice the price of Sol I think most people will take the over engineering.
> I’m more and more convinced that all of AI engineering is Neijuan (内卷, meaning curl inwards). In China it describes a system that demands ever more effort and competition without improving output
I don't know what to say, except that articles exactly like this one have been showing up constantly for the last three years, and literally all of them were obviously outdated and irrelevant within about a month.
> I actually don’t know if the model thinks someone is looking
It 'knows' (from simply training) with an extremely high degree of certainty when its prompt is written by an LLM/itself - and thus will change what it writes.
> And potentially as a byproduct of enabling all of this, you can now slop your way to a one-shot 3D game over the weekend which looks impressive.
I think we've finally reached a weird point where AI has effectively reduced the amount of competition that real game developers have to endure.
Nothing unravels faster than a game project being built with AI. You can achieve impressive results in a day, but you can't get much further than that without actual talent. LLMs will never be able to best a human environment artist at scene composition, especially if that composition needs to be directed with nuance over time.
There's a huge difference between a game that looks impressive and one that feels impressive. You can only achieve games that feel like counter strike, call of duty and overwatch with thousands of hours of human sacrifice. The AI is almost pointless once you get to play testing and balancing. Knowing how much to adjust magical integers isn't a conversation a chat bot can resolve with endless pontification tokens.
> LLMs will never be able
That is a very bold claim, unless you meant "current LLMs".
>> And potentially as a byproduct of enabling all of this, you can now slop your way to a one-shot 3D game over the weekend which looks impressive.
Do these games really look impressive? Everything I've seen has looked like someone completely new to Unity/Unreal has slapped together a bunch of premade scripts and very poor 3d assets.
Discoverability has been the main challenge for smaller game dev companies for a while now and a flood of slop will only make it worse
Yep. Every single post I've seen about "game development is over" is yet another procedurally generated game. Not only it doesn't prove anything about Astra being "better" at making games (do people have any idea the sheer amount of open source games that do exactly that? You can find thousands of the same planet exploration games through repos, blog posts,etc. Game design schools have it as an exercise, that's how basic it is), but as you said: the wow factor of space exploration is cool, sure. It makes for a catastrophically boring game.
Leaves me to wonder whether the OpenAI glazers just never played games in their lives, or are just really superficial tech bros. Most likely, both.
There could be a large overlap between OpenAI glazers and Star Citizen dreamers.
There is something odd, I've got single astra session that's now running for... 4d 13h 10m and still going.
What are you having it do?
P=np...
These machines are doing some crazy things to get to the result. That said, I can't help but feel like this is the compilers argument all over again. Are the methods used to get to the result good? No. Is the code that it generates good? No. Does it achieve the goal. Yes. Is it likely to get better with time. Also yes. In my use cases, jobs that would have taken weeks to months are being done in minutes to hours. Involving complex testing and reasoning and experimentation. I'm no fanboy, but I can't argue against the speed gains. I'm sure we'll still have artisans who hand weave incredible code. But for me, I'm switching to the weaving loom for speed and efficiency.
It seems like we're not supposed to care about the code quality then? I guess that's the compilers argument. But I'm not ready to give up the code just yet.. These LLMs don't even have a stable interface, they change every few months in how they interpret our prompts and tasks.
Similar for me, I don't like the development for many reasons, but that's another discussion. I also can't deny the capabilities.
I use the tools with this "risk analysis":
- If performance doesn't improve I can just always switch back to whatever I've done for the past 10 years, so it's not really a risk to start exploring.
- If performance does improve, then I'm already familiar with it.
The compiler argument is great, if we turn it on its head.
To create professional products, compilers are great, when used by professional or passionate and technical amateurs. They're useless if you're neither.
LLMs are the next step up. They are quite useful if you are neither, and you can get a lot farther with them, which means that low quality software is much easier to create. But if, for whatever reason, you need to create higher quality software (like most software that's actually sold directly or through subscriptions or ads), you're back to the "be a professional or passionate and technical amateur".
What is the "compilers argument"?
Back in the day, the argument was that compilers produce unreadable assembly, so people used to writing assembly were arguing against the use of compilers.
Compilers also had bugs, so we still had to debug the assembly to understand how to fix the problem. Nowadays, almost nobody has to resort to those steps, except of course compiler developers. But that is just a testament to the quality of compilers.
Comparing LLMs to compilers is a take I often see, but I am not sure the comparison quite holds. The problem is that LLMs are inherently non-deterministic, so we always get a different output on the same prompt.
Maybe if LLMs are powerful enough it won't matter. I doubt it but we will see.
What you say is true, the comparison indeed doesn't hold.
But is it relevant? does it matter from a product perspective if LLMs are non-deterministic. You don't need to one shot the correct result, english is ambiguous and LLMs non-deterministic, but you can iterate.
If it's possible to iterate fast and cheap enough, even ambiguous language can produce the results you want, given enough iterations.
There are a lot of ifs and buts here, just a thought on the compiler argument.
I think it's that when some code compiles to say assembler, the compiler doesn't prioritize readability and maintainability of the assembler code, since people are not expected to read and maintain it directly
Lots of really silly people love to compare LLMs to compilers. "You don't look at the compiled code either" and "Back in the day, people also had negative reactions to compilers and wanted to keep writing assembly by hand" and other such nonsense.
People keep saying that "models are just compilers, and I don't see you complsining about compilers". Which is such a bullshit argument
> Is it likely to get better with time. Also yes.
There are no signs to show that. If anything, the new models produce worse code, only significantly faster
https://xkcd.com/1319/
The title text on that one is gold.
Astra is indeed the pinnacle of "black box slop". It is overall a smarter software development agent for many things I do, but the code sometimes is indistinguishable from Brainfuck when writing things like GPU shaders. It doesn't even attempt to make it remotely formatted or readable.
Have you tried identifying exactly what is unreadable about it and telling it to make it more readable?
I had GTP-5.6 write some shader code recently and it wasn't very clear to me. I spent about an hour chatting with the until I understood the concepts and was able to express them back to the AI using math formulas and variables named in a way that made sense to me. The AI then rendered the code using the formula and variables I was familiar with and it was clear to me.
Perhaps you could have just written it yourself.
"But for how much more Fable costs, for how much more Astra costs, I do not feel like the results are there."
we are in the middle of the beginning. Its just a weird take to talk about the newest model like this while we are still in a R&D phase.
And these points don't matter if you let it search and analyse a bug, for example, or if you have good harness and a good architecture and let it do small PRs or if you do stuff no one needs to read (yes a software engineer also needs tools)
Just switch back and wait a little bit?
I asked Astra for fully working code, and it gave me bad code.
But when I broke it down into function units, some parts were bad and some parts were good.
So I can't tell the difference
What’s more relevant is that apparently Astra can’t tell the difference.
At this point I am starting to wonder about the RLHF that is going on for programmig.
The quirks in fallbacks, defaults and ludicrous gold plating seems to get more and more intrusive with every model upgrade.
Ironically I burned out Fable usage early this week because of Astra using it to run inane full codebase reviews over one line changes, so I have been using Astra extensively.
We need a word for “potentially highly capable, but in reality an idiot savant” to describe certain models. No, I don’t need you to write a tmux emulator in bash to test your changes bro, just ask me to run the command.
I love this dance we are doing where when people write the "AI models are garbage machines that produce garbage and are no where close to the fantasy being pedalled by the Crypto bros who pivoted to AI" it always has to be caveated with "AI models are useful and I am highly productive with them"
It feels like people should just be able to say "This article comes with the standard disclaimer" and just dive into the meat of the article without wasting time.
I’ve asked Astra to build me an app for a prototype I created quickly using Sonnet.
It’s been 2 days and it made no real progress on the actual app. It created docs, scripts, workflows, and it’s doing a bunch of reviewing on every PR.
I told it that I just need an MVP.
I’m pretty sure an average senior engineer would have finished that task much quicker, and guaranteed with more readable, higher-quality code. Meanwhile, I think I’ve easily crossed 100k tokens so far on nothing.
Funny world we’re living in that this is “SOTA” and “AGI”.
I’m genuinely curious what these OAI and A/ engineers are working on that they praise these models so much. I did not see any improvement since Opus 4.5.
Also, I’m really unimpressed by any “one shot” demo that’s out there in the wild. It means nothing for serious software engineering.