> Almost a third of our source code was comments. I deleted all of them and asked an AI agent to build the same features on both versions of the codebase, three times each. The code it wrote was just as good with or without comments.
Seems like this might be another example of The Bitter Lesson [1]:
"The bitter lesson is the observation in artificial intelligence that, in the long run, general approaches that scale with available computational power tend to outperform ones based on domain-specific understanding because they are better at taking advantage of the falling cost of computation over time."
AIs don't need comments that are designed to work around human cognitive limitations. They're quite happy to just crunch the code and extract the meaning.
Makes sense if the comments were AI generated as the author writes since AI during reading the code has roughly the same context as with which it produces the comments. I would be more curious for a test with human written comments.
This is really interesting, my gut instinct was that comments would always be helpful to AI clarifying the broader picture of what code is doing, and defining a spec to police what code should be doing.
I think there are still contexts where comments would be superior like when they feed into an OpenAPI spec where the AI or person can't see the code, and information that bubbles up through Intellisense (for humans anyway).
This seems believable! And the n=3 of it all seems very wise (though n=3 alone isn't quite enough to speak confidently of "noise" AFAIU).
Two important caveats, though, before we rush to universalize this:
1. It's based on the smartest model ever released to the public in the history of humanity, Opus 5.5. So if this is true for your codebase, it probably only became so ~post-Mythos.
2. It seems like the hard task isn't really hard enough -- the "trick" is something about PHP, and sounds common enough that it would be well baked into each model. Regardless, there are far, far easier to miss gotchas than "fails on reload", especially in a large codebase where a post-mortem are written about correct code that just runs slower all of a sudden or something, much less cases where you have to rely on users (or worse, metrics) to uncover such issues.
When I joined Google, they gave me the baffling instruction to try to write as few comments as possible. I've since learned that's a semi-popular approach, based on the logic that "the code should speak for itself". I'm just a baby engineer (4 more months to make it onto 30-under-30!), but I find that to be more about self-esteem for seniors and feeling like a craftsman than it is about real, measured tradeoffs.
Plus, comments can A) be documentation[1] and B) be beautiful[2] ;)
> Almost a third of our source code was comments. I deleted all of them and asked an AI agent to build the same features on both versions of the codebase, three times each. The code it wrote was just as good with or without comments.
Seems like this might be another example of The Bitter Lesson [1]:
"The bitter lesson is the observation in artificial intelligence that, in the long run, general approaches that scale with available computational power tend to outperform ones based on domain-specific understanding because they are better at taking advantage of the falling cost of computation over time."
AIs don't need comments that are designed to work around human cognitive limitations. They're quite happy to just crunch the code and extract the meaning.
[1] https://en.wikipedia.org/wiki/Bitter_lesson
This is so interesting! I would like to see the same experiment, but with these variations:
- All the comments removed (the blog post)
- Half the comments removed
- 10% of the comments replaced with slightly incorrect comments
- 10% of the comments replaced with very incorrect comments
- 10% of the comments replaced with slightly incorrect comments
- 90% of the comments removed, 10% remain but are very incorrect
And variations like that
Test to see where the inflection point is for correctness, token usage and other things
Makes sense if the comments were AI generated as the author writes since AI during reading the code has roughly the same context as with which it produces the comments. I would be more curious for a test with human written comments.
This is really interesting, my gut instinct was that comments would always be helpful to AI clarifying the broader picture of what code is doing, and defining a spec to police what code should be doing.
I think there are still contexts where comments would be superior like when they feed into an OpenAPI spec where the AI or person can't see the code, and information that bubbles up through Intellisense (for humans anyway).
Were the comments accessible in the bit history? If so, I wonder if the agent accessed it.
This seems believable! And the n=3 of it all seems very wise (though n=3 alone isn't quite enough to speak confidently of "noise" AFAIU).
Two important caveats, though, before we rush to universalize this:
1. It's based on the smartest model ever released to the public in the history of humanity, Opus 5.5. So if this is true for your codebase, it probably only became so ~post-Mythos.
2. It seems like the hard task isn't really hard enough -- the "trick" is something about PHP, and sounds common enough that it would be well baked into each model. Regardless, there are far, far easier to miss gotchas than "fails on reload", especially in a large codebase where a post-mortem are written about correct code that just runs slower all of a sudden or something, much less cases where you have to rely on users (or worse, metrics) to uncover such issues.
When I joined Google, they gave me the baffling instruction to try to write as few comments as possible. I've since learned that's a semi-popular approach, based on the logic that "the code should speak for itself". I'm just a baby engineer (4 more months to make it onto 30-under-30!), but I find that to be more about self-esteem for seniors and feeling like a craftsman than it is about real, measured tradeoffs.
Plus, comments can A) be documentation[1] and B) be beautiful[2] ;)
[1]: https://github.com/doering-ai/my-basis/blob/main/docs/regex....
[2]: https://doering.ai/work/sublime-python-syntax/