Could use a non-negative name. Just like it was not the ‘no heavyweight dev processes’ manifesto but Agile manifesto which was wsy catchier to reference, maybe yours could be named like the… Human-serving Engineering manifesto or something?
> 17. There will always be alpha in reviewing something.
What does alpha mean in this case? I only understand this term in from a finance perspective (excess returns). Is it used here to mean 'value'? E.g. there is always value in reviewing something?
This really connect with the latest conversations I had with coworkers regarding how can we review gigantic PR generated by AI. One way could be to ask the submitter to give their prompt as part of their contribution. This way it's easier for the reviewer to understand what was the thought process of the contributor and if this very process is sound.
Not saying that this way to review is a silver bullet, but at least it removes some weight.
> One way could be to ask the submitter to give their prompt as part of their contribution
Zed's delta (https://delta.dev/) seems to be solving this problem by extending the version control to the conversation layer. I don't know if I'd want all my conversation to be tracked, some stuff belongs to the draft sheet.
Look into [openspec](https://openspec.dev). My team is routinely passing 1-2-5k line PRs through review, with the following caveats:
1. PRs come with "the prompt". In openspec parlance this is a set of docs: a proposal, design, specs, task list.
2. I have a tiny skill to slice and dice PRs, so I can review each part of the system separately by rewriting commits in order of reviewal: specs/docs, production code, tests, evidence. I sometimes pull DB queries to a separate commit for focused review.
3. The evidence bit is built by another skill: go over all added spec scenarios and produce a markdown document with links to the code implementing the scenario and the tests proving the implementation works. Emacs helps me jump back and forth via those links and read the parts I'm interested in.
My next goal is to extend openspec schemas with custom workflow steps to make PRs land in my preferred review structure and contain the evidence report, so I don't have to build it.
Just ban large prs. They can always be split up, and usually as well as helping review it makes them less risky and easier to find bugs and roll back if there was a problem.
The discipline of splitting them up helps in almost every dimension.
To add to my openspec shilling above, I've been known to "fix" large PRs by reverse engineering specs from them and building each spec out in a separate PR.
Last week me and a coworker turned a 15k line PR into 10 x 1.5k line ones that went in smoothly.
Yes I do like to make a big pr first sometimes then it's usually pretty easy to split it into small ones and it's a helpful process in many ways. You can even present both together so people can see it either way, but I think in most systems it's better to have smaller prs merged one at a time.
For a rewrite, I guess sometimes it's better to have a spec then write to that, and PRs might be larger.
So what do you think of Discord rewriting in rust, and AWS shipping an emulator that was ai generated? Do you think those were done with small prs? Do you think that it's bad they did those things?
I, personally, would certainly argue that those should be done in small PRs. If you have a working application and want to rewrite in rust, I would expect it to be done by replacing individual components one at a time while making sure relevant tests keep passing. If you're making a new application, I would expect it to be built incrementally by adding parts incrementally.
Please add your own thoughts to the conversation rather than aggressively asking rhetorical questions. "Counterexamples exist" is hardly a valid criticism of another human's perspective.
If someone says PRs need to be small, and I say here are some examples of huge PRs that I think are good, do you think these are still bad? That is not remotely invalid.
Where did grey-area say they need to be small. As in an absolute? They didn't, you just assumed that's what they meant and then went off the rails in your response.
The first 4 words? "Just ban large prs." A ban is pretty absolute language. He would have banned my example PRs clearly. Or maybe he wouldn't which is why I asked about them as examples. Because that is the obvious follow up. If you ban large PRs what are you losing? You're loosing big swing aggressive efforts to make sweeping progress.
That was a comment written in a particular context, where the original commenter was concerned about large PRs.
>> how can we review gigantic PR generated by AI.
It was not a comment written in a vacuum saying that all PRs should be small (or not large, whatever that means). Though grey-area does provide some reasonable commentary for why it's good to shrink them (easier to roll back, easier to isolate bugs, etc.). Maybe grey-area does mean to ban all large PRs, but that's not evident in this thread where they wrote a response to a specific comment offering their take on how to resolve the problem raised in that first comment.
> You're loosing big swing aggressive efforts to make sweeping progress.
Just a tip: "losing" not "loosing". We loose arrows at our enemies, we let loose the dogs of war, we loosen our purse strings. But we do not "loose big swing aggressive efforts", we "lose big swing aggressive efforts". It's funny that you get it right once, but mix the two up in the very next sentence.
It's like responding to "you should wear a seatbelt, statistically they keep you safer in a crash" with "here are two people who got in a crash without a seatbelt and survived"
I think they could do those things with small PRs if they chose. If their agents are not intelligent enough to make small prs, I guess they can YOLO and assume everything is just fine with what the agents did. I do think small PRs would be better for agent review as well though, because they avoid any problems with context being overwhelmed and things being missed/forgotten.
What are the PRs you're thinking of though. If you mean a full clean rewrite, they'd probably proceed with spec first then just write it in bits to the spec. No reason it has to be large prs or smaller that's a design choice.
Often an exercise I find useful is do a quick big PR, then split it into small chunks that make sense independently for review. Or you can chain prs. Both those things can be done.
You're welcome to do large PRs, I'm not a fan of huge PRs though and would not pretend to review them (as pretending is what it would be), nor would I like to work somewhere where this is the culture.
Do most people generate PRs from a one-shot prompt? I have a whole pipeline of prompts, tests and evals, which loop back and potentially restart if the result is inadequate. How would I include that in the PR?
Good question, and I think that this is a problem that needs to be solved. Just like a commit repository, it would be lit to record the chain of thoughts of the agents you're using.
Yep true, I'm still thinking about this as of today so my thoughts on this are not really mature and there are definitely a lot of caveats in that approach I didn't see.
However, having a rigorous spec process which ends up giving lots of details to the agent seems to me the right way to build stuff with LLMs. Rather than iterating and piling changes after changes.
Because, unless you are changing some code by hand. Your prompt and the context your giving to your agent is the seed for the outcome of the agent itself. No agent as far as I know can generate content ex-nihilo. So having a look at what people are telling their agent to do can definitely help understanding the result of the same agent in my opinion.
Yeah true, actually my phrasing was poor. I came to that idea through debate with other coworkers. This is not a solution for gigantic PRs but rather a method on how people should be reviewing AI changes.
Why, yes, blame someone else, and not the person that shat out the PR :)
I had a recent case where I, being the lead, sliced up the work real good into an epic and what not and a guy on the team just did a "hurr durr, Claude, go wild with all that" and threw a huge PR over the wall linked to 5+ stories.
Well, we got to have "the conversation" after that and learned how to fix these kinds of PRs together...
> Slop is when we put less effort into producing something than we ask the consumer to spend understanding it.
This feels directionally right, but if you define slop this way then you put a cap on AI productivity. For example, I heard an interview with Terry Tao talking about how the work of mathematicians is now to understand the proofs generated by AI. He's, by definition, describing these groundbreaking proofs as slop (by the above definition).
Well maybe Tao is wrong and we should instead focus on making LLMs explain their ideas better. It really seems eminently achievable and would be better all round.
I think the problem with those Maths proofs was they were massively rushed and OpenAI just couldn't be bothered to do it properly or actually check them. They were more interested in the PR than in helping advance Mathematics.
Aren't you really trying to get at is a Human Flourishing manifesto? Your postions are pro-human, anti-AI. But what if AI did go away tomorrow because... um... Noosphere Prime Intellect awoke and humans started working with it? Your pro-human arguments probably wouldn't change but all the AI slop stuff would be slop itself.
While I am sympathetic to the authors, the reason that agile swept over the industry is that it was a demonstrably more effective when measured by business outcomes. "AI slop all the things" is miserable for human employees and introduces new kinds of dysfunctions called out or hinted at in this manifesto. However, if on balance it is more effective when measured by business outcomes, there is little chance of this manifesto gaining traction in the industry.
From my own personal experience, which may not reflect the broader industry although I suspect it does, the ones who embrace slopping with the least regard for their peers are also the ones perceived as the most effective by their employers. Those who take a principled stance get the worst of both worlds. They cannot combat the deluge of slop with their own clanker, and they lose the influence with their company leadership that would provide sponsorship to change how AI is being used. This creates competitive pressure to follow suit, and from there it is a race to the bottom.
> Comments and feedback on this document are welcome. What is slop? Why does slop happen? Why is slop worth fighting? How do we fight slop?
This sounds like slop. Like an engagement bait. Also a no slop manifesto should start by defining “slop” as the first thing. That’s not something for the comment section.
> AI cannot replace human judgement, it can only amplify our judgement (or lack thereof)
This ship has sailed. We first need to suffer the consequences (if there will be any, not arguing that) before action is taken to prevent the consequences from repeating in the future. It's just human nature. Even the agile manifesto happened after decades of doing it "wrong". Most people who don't go on the slop train will experience their colleagues doing it and getting rewarded for it. The HN crowd is in general an exception here, being above average (not counting me), but most people experience management expectations that are not too forward looking.
The fact of the matter is software development of the future is not going to look like AI slop, and its also not going to look like the hand crafted engineering of the past. It's going to be a whole new third thing of organically growing software with constraints and processes that result in quality. The degree to which different companies and people have figured it out varies wildly, and almost every best practice so far is wildly over confident and untested. We were still arguing orver OOP and thats been around for 50 years. People are having a meltdown over the changes in the last 2 years and reacting in wildly unpredictable ways.
> Slop is asymmetric. The cost of producing a thousand-line PR or a ten-page doc is approaching zero, while the cost of reviewing it remains constant.
I'd say the cost actually grows as people get laid off left and right, and on top of that, a 1000 line PR from a good human, give me a case of beer and I'll make me some sense of it, but if it's AI slop, get ready to hand me a crate of homemade Slivo or I won't touch that...
Plus, I find reading slop docs takes cognitively much more effort. When I find problems in human docs, it's easier for me to point out the issue, comment etc. But on these AI slop docs, I find it overwhelming to comment on, because the issues are often woven throughout the document in a way that makes it so I can't isolate the comment in a single place. It's hard for me to describe.
I had thought the same while reading. Every single letter/word/line needs a review and even that might not be enough to prevent slop. Use AI for reviewing, catching edge cases and improving the overall quality instead.
More than 20 apps developed in the last six months, all top quality, I haven't coded a single line, speed and output is what matters, results improving by the day and will be unrecognizable in a couple of years by any metric
AI is unstoppable and no matter how many manifests luddites write AI won't care
The authors should take the parent comment as a sign their manifesto's title is bad. Some will read the title and make a snap judgement that the manifesto must be totally against AI.
Seems like a good thought for pre-agentic era where ai assisted existing human workflows. We are fast approaching the end of that era. In the agentic future the past human-centric workflows (reviewing, validating, approving etc.) will be replaced with agent-to-agent workflows. There will be resistance from humans who would want to be in the comfort of familiar human-centric workflows. The survivors will see where we are heading and adapt.
LOL stop it already with the agentic crap. Besides the already automation-ready software use cases, agents are simply the wrong tool for the job, unless you bend and distort the definition of such job.
Y’all fully invested in this agentic bullshit that you sound delusional and talk in self-evident truths that nobody outside of your niche SWE bubble would ever share.
Could use a non-negative name. Just like it was not the ‘no heavyweight dev processes’ manifesto but Agile manifesto which was wsy catchier to reference, maybe yours could be named like the… Human-serving Engineering manifesto or something?
Here, let's trade manifestos :) https://johnoestmannmusic.com/manifesto-academy-of-creative-...
> 17. There will always be alpha in reviewing something.
What does alpha mean in this case? I only understand this term in from a finance perspective (excess returns). Is it used here to mean 'value'? E.g. there is always value in reviewing something?
In this context, "alpha" is knowledge that gives you a competitive advantage.
Is it a claudism?
> Improving the system over patching the output
This really connect with the latest conversations I had with coworkers regarding how can we review gigantic PR generated by AI. One way could be to ask the submitter to give their prompt as part of their contribution. This way it's easier for the reviewer to understand what was the thought process of the contributor and if this very process is sound. Not saying that this way to review is a silver bullet, but at least it removes some weight.
> One way could be to ask the submitter to give their prompt as part of their contribution
Zed's delta (https://delta.dev/) seems to be solving this problem by extending the version control to the conversation layer. I don't know if I'd want all my conversation to be tracked, some stuff belongs to the draft sheet.
Look into [openspec](https://openspec.dev). My team is routinely passing 1-2-5k line PRs through review, with the following caveats:
1. PRs come with "the prompt". In openspec parlance this is a set of docs: a proposal, design, specs, task list.
2. I have a tiny skill to slice and dice PRs, so I can review each part of the system separately by rewriting commits in order of reviewal: specs/docs, production code, tests, evidence. I sometimes pull DB queries to a separate commit for focused review.
3. The evidence bit is built by another skill: go over all added spec scenarios and produce a markdown document with links to the code implementing the scenario and the tests proving the implementation works. Emacs helps me jump back and forth via those links and read the parts I'm interested in.
My next goal is to extend openspec schemas with custom workflow steps to make PRs land in my preferred review structure and contain the evidence report, so I don't have to build it.
Your workflow looks really neat, I'll definitely give it a try.
So now I get to review the prompts and a slopped PR-description? Oh man, that sounds nice.
Just ban large prs. They can always be split up, and usually as well as helping review it makes them less risky and easier to find bugs and roll back if there was a problem.
The discipline of splitting them up helps in almost every dimension.
Fully agree with a twist - ban dumb large PRs.
To add to my openspec shilling above, I've been known to "fix" large PRs by reverse engineering specs from them and building each spec out in a separate PR.
Last week me and a coworker turned a 15k line PR into 10 x 1.5k line ones that went in smoothly.
Yes I do like to make a big pr first sometimes then it's usually pretty easy to split it into small ones and it's a helpful process in many ways. You can even present both together so people can see it either way, but I think in most systems it's better to have smaller prs merged one at a time.
For a rewrite, I guess sometimes it's better to have a spec then write to that, and PRs might be larger.
So what do you think of Discord rewriting in rust, and AWS shipping an emulator that was ai generated? Do you think those were done with small prs? Do you think that it's bad they did those things?
I, personally, would certainly argue that those should be done in small PRs. If you have a working application and want to rewrite in rust, I would expect it to be done by replacing individual components one at a time while making sure relevant tests keep passing. If you're making a new application, I would expect it to be built incrementally by adding parts incrementally.
Please add your own thoughts to the conversation rather than aggressively asking rhetorical questions. "Counterexamples exist" is hardly a valid criticism of another human's perspective.
If someone says PRs need to be small, and I say here are some examples of huge PRs that I think are good, do you think these are still bad? That is not remotely invalid.
Where did grey-area say they need to be small. As in an absolute? They didn't, you just assumed that's what they meant and then went off the rails in your response.
The first 4 words? "Just ban large prs." A ban is pretty absolute language. He would have banned my example PRs clearly. Or maybe he wouldn't which is why I asked about them as examples. Because that is the obvious follow up. If you ban large PRs what are you losing? You're loosing big swing aggressive efforts to make sweeping progress.
> The first 4 words? "Just ban large prs."
That was a comment written in a particular context, where the original commenter was concerned about large PRs.
>> how can we review gigantic PR generated by AI.
It was not a comment written in a vacuum saying that all PRs should be small (or not large, whatever that means). Though grey-area does provide some reasonable commentary for why it's good to shrink them (easier to roll back, easier to isolate bugs, etc.). Maybe grey-area does mean to ban all large PRs, but that's not evident in this thread where they wrote a response to a specific comment offering their take on how to resolve the problem raised in that first comment.
> You're loosing big swing aggressive efforts to make sweeping progress.
Just a tip: "losing" not "loosing". We loose arrows at our enemies, we let loose the dogs of war, we loosen our purse strings. But we do not "loose big swing aggressive efforts", we "lose big swing aggressive efforts". It's funny that you get it right once, but mix the two up in the very next sentence.
It's like responding to "you should wear a seatbelt, statistically they keep you safer in a crash" with "here are two people who got in a crash without a seatbelt and survived"
I think they could do those things with small PRs if they chose. If their agents are not intelligent enough to make small prs, I guess they can YOLO and assume everything is just fine with what the agents did. I do think small PRs would be better for agent review as well though, because they avoid any problems with context being overwhelmed and things being missed/forgotten.
What are the PRs you're thinking of though. If you mean a full clean rewrite, they'd probably proceed with spec first then just write it in bits to the spec. No reason it has to be large prs or smaller that's a design choice.
Often an exercise I find useful is do a quick big PR, then split it into small chunks that make sense independently for review. Or you can chain prs. Both those things can be done.
You're welcome to do large PRs, I'm not a fan of huge PRs though and would not pretend to review them (as pretending is what it would be), nor would I like to work somewhere where this is the culture.
Do most people generate PRs from a one-shot prompt? I have a whole pipeline of prompts, tests and evals, which loop back and potentially restart if the result is inadequate. How would I include that in the PR?
Good question, and I think that this is a problem that needs to be solved. Just like a commit repository, it would be lit to record the chain of thoughts of the agents you're using.
Would have to include the full conversion history. But who is gonna review that it's probably as big as some of these big PRs are in the first place.
Yep true, I'm still thinking about this as of today so my thoughts on this are not really mature and there are definitely a lot of caveats in that approach I didn't see.
However, having a rigorous spec process which ends up giving lots of details to the agent seems to me the right way to build stuff with LLMs. Rather than iterating and piling changes after changes.
How does reviewing the long discussion the author had with the LLM prove the code is correct?
Because, unless you are changing some code by hand. Your prompt and the context your giving to your agent is the seed for the outcome of the agent itself. No agent as far as I know can generate content ex-nihilo. So having a look at what people are telling their agent to do can definitely help understanding the result of the same agent in my opinion.
If a PR is gigantic it means the team lead screwed up slicing the work.
Yeah true, actually my phrasing was poor. I came to that idea through debate with other coworkers. This is not a solution for gigantic PRs but rather a method on how people should be reviewing AI changes.
It also means the engineer didn't check in with the lead as their PR became unwieldy.
Why, yes, blame someone else, and not the person that shat out the PR :)
I had a recent case where I, being the lead, sliced up the work real good into an epic and what not and a guy on the team just did a "hurr durr, Claude, go wild with all that" and threw a huge PR over the wall linked to 5+ stories.
Well, we got to have "the conversation" after that and learned how to fix these kinds of PRs together...
> Slop is when we put less effort into producing something than we ask the consumer to spend understanding it.
This feels directionally right, but if you define slop this way then you put a cap on AI productivity. For example, I heard an interview with Terry Tao talking about how the work of mathematicians is now to understand the proofs generated by AI. He's, by definition, describing these groundbreaking proofs as slop (by the above definition).
I think the issue lies partly in not understanding your product yourself, and then handing it off to someone else who has to make sense of it.
The comparison doesn’t work for the music industry, at least. Many hit songs are written and produced on a whim, while people analyze it for years.
Only human can publish the slop, which mean if Tao release something without reviewing, it will be called slop.
Yes, what I'm saying is some of these proofs may have had essentially zero marginal human effort. Should they not be reviewed at all?
Reviewing to make sure the proof is correct. Even the proof comes with lean, it could still proof the wrong thing.
In that context, it'd only be slop if Terry published those proofs and asked someone else to do that work.
Well maybe Tao is wrong and we should instead focus on making LLMs explain their ideas better. It really seems eminently achievable and would be better all round.
I think the problem with those Maths proofs was they were massively rushed and OpenAI just couldn't be bothered to do it properly or actually check them. They were more interested in the PR than in helping advance Mathematics.
> put a cap on AI productivity
Oh no, the horror
Aren't you really trying to get at is a Human Flourishing manifesto? Your postions are pro-human, anti-AI. But what if AI did go away tomorrow because... um... Noosphere Prime Intellect awoke and humans started working with it? Your pro-human arguments probably wouldn't change but all the AI slop stuff would be slop itself.
While I am sympathetic to the authors, the reason that agile swept over the industry is that it was a demonstrably more effective when measured by business outcomes. "AI slop all the things" is miserable for human employees and introduces new kinds of dysfunctions called out or hinted at in this manifesto. However, if on balance it is more effective when measured by business outcomes, there is little chance of this manifesto gaining traction in the industry.
From my own personal experience, which may not reflect the broader industry although I suspect it does, the ones who embrace slopping with the least regard for their peers are also the ones perceived as the most effective by their employers. Those who take a principled stance get the worst of both worlds. They cannot combat the deluge of slop with their own clanker, and they lose the influence with their company leadership that would provide sponsorship to change how AI is being used. This creates competitive pressure to follow suit, and from there it is a race to the bottom.
> However, if on balance it is more effective when measured by business outcomes...
That's a very big "if". So far, it appears that AI slop leads to worse business outcomes.
I signed, very balanced take.
> The more slop flying around a team, the more we lose the will to fight it.
So true, I feel guilty not fighting it at the place I work at, but it seems impossible.
This manifesto gives me a spark on hope I haven't felt in months.
> Comments and feedback on this document are welcome. What is slop? Why does slop happen? Why is slop worth fighting? How do we fight slop?
This sounds like slop. Like an engagement bait. Also a no slop manifesto should start by defining “slop” as the first thing. That’s not something for the comment section.
> AI cannot replace human judgement, it can only amplify our judgement (or lack thereof)
Why not? Did you mean should not.
I’ve seen first hand $40,000/year private schools assigning AI slop phonics handouts to their students.
It really boggles my mind. I mean, come on just reuse the curriculum from the past 20 years. You do have that don’t you? Don’t you?!
This ship has sailed. We first need to suffer the consequences (if there will be any, not arguing that) before action is taken to prevent the consequences from repeating in the future. It's just human nature. Even the agile manifesto happened after decades of doing it "wrong". Most people who don't go on the slop train will experience their colleagues doing it and getting rewarded for it. The HN crowd is in general an exception here, being above average (not counting me), but most people experience management expectations that are not too forward looking.
The fact of the matter is software development of the future is not going to look like AI slop, and its also not going to look like the hand crafted engineering of the past. It's going to be a whole new third thing of organically growing software with constraints and processes that result in quality. The degree to which different companies and people have figured it out varies wildly, and almost every best practice so far is wildly over confident and untested. We were still arguing orver OOP and thats been around for 50 years. People are having a meltdown over the changes in the last 2 years and reacting in wildly unpredictable ways.
> Slop is asymmetric. The cost of producing a thousand-line PR or a ten-page doc is approaching zero, while the cost of reviewing it remains constant.
I'd say the cost actually grows as people get laid off left and right, and on top of that, a 1000 line PR from a good human, give me a case of beer and I'll make me some sense of it, but if it's AI slop, get ready to hand me a crate of homemade Slivo or I won't touch that...
Plus, I find reading slop docs takes cognitively much more effort. When I find problems in human docs, it's easier for me to point out the issue, comment etc. But on these AI slop docs, I find it overwhelming to comment on, because the issues are often woven throughout the document in a way that makes it so I can't isolate the comment in a single place. It's hard for me to describe.
just ignore lol
There's no responsibility in not reviewing some code you ship, this manifesto is really more pro than against slop.
I had thought the same while reading. Every single letter/word/line needs a review and even that might not be enough to prevent slop. Use AI for reviewing, catching edge cases and improving the overall quality instead.
AI says: "we don't care, we will do it our way"
More than 20 apps developed in the last six months, all top quality, I haven't coded a single line, speed and output is what matters, results improving by the day and will be unrecognizable in a couple of years by any metric
AI is unstoppable and no matter how many manifests luddites write AI won't care
Link to them.
Amen
The authors should take the parent comment as a sign their manifesto's title is bad. Some will read the title and make a snap judgement that the manifesto must be totally against AI.
Seems like a good thought for pre-agentic era where ai assisted existing human workflows. We are fast approaching the end of that era. In the agentic future the past human-centric workflows (reviewing, validating, approving etc.) will be replaced with agent-to-agent workflows. There will be resistance from humans who would want to be in the comfort of familiar human-centric workflows. The survivors will see where we are heading and adapt.
Correct. This document is only relevant for a particular time period in the development of AI and will not "be relevant for years to come."
14. We own the intent, taste, and consequences of what we ship
It sounds to me like you're arguing against this point in particular.
More power to you. I'd like never to work at a place that sanctions slop-validated-slop.
LOL stop it already with the agentic crap. Besides the already automation-ready software use cases, agents are simply the wrong tool for the job, unless you bend and distort the definition of such job. Y’all fully invested in this agentic bullshit that you sound delusional and talk in self-evident truths that nobody outside of your niche SWE bubble would ever share.