i mean its not wrong, basically everyone learns this within a few weeks of actively working with the agentic loop.
but as usual with ai written content, the word bloat is roughly x5 of the words necessary to convey the message - with basically no effort on the meat proxy's part to clean it up in any way, shape of form
Bitter lesson means all these tricks will not be needed in a year or two. Either labs will abstract it in harness or models will become good enough that it can do it by itself
Models are already good enough. Last week I had Fable plan out a project that took approx 4 days end to end with each phase orchestrated by a supervisor agent delegating individual tasks to other agents, coordinating everything and checking status by simple text files in the repo. I didn’t have to tell the agent to do it that way, it just came up with it and set up the infra as part of the planning overview. Great that everyone’s posting their “my secret sauce” cookbooks just to jump on the hype train but it looks like the models have already figured it out for themselves.
If implemented correctly these tricks will still work on less expensive models delivering nearly identical value.
Today, a hefty amount of standard coding tasks can be completed with similar results to gpt astra using terra and a tailored harness around it.
Also the scale matters. One big, expensive session, with a frontier model paired with a dev-babysitter is ok. But make it a factory (kindergarden: few devs, many parallel streams) and you'll want to follow a strict protocol.
Yes it’ll eventually get built into the harness/model/interface; just like how “work it out step by step” became thinking, but if you have work to do today, it can be worth trying to improve it.
I’ll tell the missing part, what comes next: the team, with teammates and managers, starts making their product decisions. If the user was anywhere near the chat and wrote a mere “ok”, they will be recorded as “user's”.
They will write tests. Lots of tests. Instead of removing any code, there will be 3 layers of backward compatibility, and tests that test presence of tests that test that backward compatibility.
The reviews will find all possible edge cases, including those that can never happen, and make the UI gracefully handle them. With tests.
The diff from any integration PR from team work will be over 10K lines, half of them bureaucracy. Zero chance to review even one – they will churn half-a-dozen per day.
For anything outside of known shape, the original hard topics become quickly displaced with shortcuts and familiar patterns.
Next, the app will break under load, and you will find that it’s caused by a quadratic sweep over the whole DB on any insert to prevent something irrelevant that you specifically told not to do.
You will ask, “wtf? why is it there?”. “It’s load-bearing, you ruled it”.
(That's not a joke. That's how I spent the summer.)
It begins as “let’s give the agents a kanban board to track stuff”. Then, why not have one agent do the tracking while another does the development?
Then why not have a fleet of them - specialists doing their thing? Then why not have them communicate in a standard way? Then, oh now we have so much docs/messages that we are getting lost. Why not add memory and semantic search for the project?
The rabbit hole keeps going until you run the project and find silly stupid logical issues and wonder “is this what I burnt all those tokens for?! Why is it so over-engineered?!”
I have one agent now that I use to fill in at specific places in functions/modules that I have created and working on.
Slow down and use AI to just do the tightly scoped mundane work. It is nice and effective. We all don’t need to save the world.
I tried this approach and the agents just built tons of tests. The Agent in Charge ordered more and more tests. After two weeks it reported finished” and the end product was completely unusable.
I really hope future models will do a better job at this. As it works be useful (if it worked).
i mean its not wrong, basically everyone learns this within a few weeks of actively working with the agentic loop.
but as usual with ai written content, the word bloat is roughly x5 of the words necessary to convey the message - with basically no effort on the meat proxy's part to clean it up in any way, shape of form
Bitter lesson means all these tricks will not be needed in a year or two. Either labs will abstract it in harness or models will become good enough that it can do it by itself
Models are already good enough. Last week I had Fable plan out a project that took approx 4 days end to end with each phase orchestrated by a supervisor agent delegating individual tasks to other agents, coordinating everything and checking status by simple text files in the repo. I didn’t have to tell the agent to do it that way, it just came up with it and set up the infra as part of the planning overview. Great that everyone’s posting their “my secret sauce” cookbooks just to jump on the hype train but it looks like the models have already figured it out for themselves.
If implemented correctly these tricks will still work on less expensive models delivering nearly identical value.
Today, a hefty amount of standard coding tasks can be completed with similar results to gpt astra using terra and a tailored harness around it.
Also the scale matters. One big, expensive session, with a frontier model paired with a dev-babysitter is ok. But make it a factory (kindergarden: few devs, many parallel streams) and you'll want to follow a strict protocol.
So we should just ignore?
If you’re using models today, it’s worth it.
Yes it’ll eventually get built into the harness/model/interface; just like how “work it out step by step” became thinking, but if you have work to do today, it can be worth trying to improve it.
I’ll tell the missing part, what comes next: the team, with teammates and managers, starts making their product decisions. If the user was anywhere near the chat and wrote a mere “ok”, they will be recorded as “user's”.
They will write tests. Lots of tests. Instead of removing any code, there will be 3 layers of backward compatibility, and tests that test presence of tests that test that backward compatibility.
The reviews will find all possible edge cases, including those that can never happen, and make the UI gracefully handle them. With tests.
The diff from any integration PR from team work will be over 10K lines, half of them bureaucracy. Zero chance to review even one – they will churn half-a-dozen per day.
For anything outside of known shape, the original hard topics become quickly displaced with shortcuts and familiar patterns.
Next, the app will break under load, and you will find that it’s caused by a quadratic sweep over the whole DB on any insert to prevent something irrelevant that you specifically told not to do.
You will ask, “wtf? why is it there?”. “It’s load-bearing, you ruled it”.
(That's not a joke. That's how I spent the summer.)
+1 my last 2 months….
It begins as “let’s give the agents a kanban board to track stuff”. Then, why not have one agent do the tracking while another does the development? Then why not have a fleet of them - specialists doing their thing? Then why not have them communicate in a standard way? Then, oh now we have so much docs/messages that we are getting lost. Why not add memory and semantic search for the project?
The rabbit hole keeps going until you run the project and find silly stupid logical issues and wonder “is this what I burnt all those tokens for?! Why is it so over-engineered?!”
I have one agent now that I use to fill in at specific places in functions/modules that I have created and working on.
Slow down and use AI to just do the tightly scoped mundane work. It is nice and effective. We all don’t need to save the world.
Lol, this was my summer too!
I tried this approach and the agents just built tons of tests. The Agent in Charge ordered more and more tests. After two weeks it reported finished” and the end product was completely unusable.
I really hope future models will do a better job at this. As it works be useful (if it worked).