What I find to be maybe the single most important part of code review is knowledge transfer.
Our entire small team thumbs up a PR before it's merged unless there's a big rush on it, and this gives everyone on the team a rough idea of the state of the codebase at any given time. There's no being blindsided like "this whole system I depend on is gone" like I had happen at far more siloed places I've worked.
Beyond that, it gives a forum to ask questions about how things work to further build understanding. On a high functioning team, every developer should have at least a modest understanding of the entire system, including parts they never touch.
Another important feature is just the institutional knowledge check. For instance recently I made a small change to a table and a coworker pointed out that there was a microservice I wasn't considering that wrote to that table that would break (yes, sharing tables is bad design, unrelated). I had no idea this microservice existed let alone had access to this table. The institutional knowledge check here though prevented a larger issue and potential data cleanup situation.
> For instance recently I made a small change to a table and a coworker pointed out that there was a microservice I wasn't considering that wrote to that table that would break
If code reviews are important, where does testing sit? Presumably if the coworker had not been part of the code review something would have stopped the breaking change making its way to prod?
We even find ourselves creating PRs in situations where the code is going to be merged immediately anyway, and tagging other devs, just so they have a convenient way to see what got merged and why. So people don't lose track of what is in the codebase.
It's a good practice. Worth mentioning also: the same can be done with ordinary git log, assuming everyone is using git well. A proper git log of yesterday's work can be like your work newspaper with coffee.
Becoming very comfortable with "rebase --interactive" and other cmds for editing your (local!) history before merging helps a lot. Once you are, it only adds 5m or so of extra work to most PRs. And while acquiring this knowledge used to be difficult, LLMs make it very easy these days.
My attitude has always been that code review is best thought of as the gate where code goes from being owned by the author to being owned by the team or project. The code I'm reviewing is not your code, it is code that is about to become our code.
Maintainability is a major factor in that, of course.
Yep, without a decent team culture this is what LLMs force, the slop deluge is just overwhelming without leadership asserting "no, stop"
Ultimately you just let bugs through because the alternative is spend an inordinate amount of time communicating with someones claude through PR comments about what the shape should be.
Career was fun while it lasted. I suppose its a blessing a to get to do a job that you enjoyed for as many years as I did.
> The primary purpose of code review is to find code that will be _hard to maintain_.
In some organizations, maintainability may be the biggest risk being mitigated in a code review. But for me, that's selling code reviews short.
In my experience, code reviews are the single most important quality control process in the entire development life cycle. Engineers often don't have a lot of influence over the quality of requirements. Engineers often don't have a lot of influence over the competence and thoroughness of the QA process (and it often doesn't exist at all). But engineers frequently have total control over code reviews.
If I can't depend on the rest of the organization for QC, code reviews are the first place I look to mitigate that risk. That means code reviews find bugs. That means code reviews identify code smells. That means code reviews pressure test requirements and whether the implementation matches the assignment. That means code reviews transfer knowledge and serve as a teacher for both the PR author and the reviewer. And so on.
Thorough and pedantic code reviews are challenging and tedious, at least at first, but the team adapts and both the code and the review process gets better.
The purpose of code review is multi-faceted. Hard to maintain? Yes. Might have bugs? Yes. Can be done simpler/cleaner? Yes. Is in line with project code style? Yes. Get someone else to also understand the code? Yes. Onboard junior team member? Yes. Sanity check design decisions? Yes.
This flippant note is mostly more self-justification for being a lazy code reviewer.
Indeed, we worked hard as an industry to move beyond "blame the author" to "blame the process/team".
Unfortunately this article is just bait, may as well say "People seem to think dinner is about eating food, but it's not about eating at all, actually it's about connecting with family and friends!". It's a specific type of poorly constructed reductionist argument that plays well on HN.
Agreed. There's a whole checklist of what to look out for:
- Does it functionally achieve what it sets out to (as per tacker issue or PR description)?
- Does it have extraneous code? Leftover debug prints, private API keys etc...
- Does it have any obvious defects? Memory leaks, un-handled edge cases, security flaws, obsolete API calls, etc...
- Could it be more understandable? Add/remove abstractions, better variable/method names, more/less functional etc...
- Is the style consistent with the codebase and/or style guidelines?
- Are there obvious performance improvements? Hashset instead of list, lazy evaluations, etc...
- Is it sufficiently well tested?
I'm not even sure I agree that if I can't understand the code then it shouldn't go through. Some code is just really hard to understand. The aim is to make it as easy as possible to understand, while being functionally correct.
> The primary purpose of code review is to find code that will be _hard to maintain_
Says who? I agree that it's a good purpose. But the main problem with code reviews is not having a in-house code review guidance. Doesn't need to be long. 1 page would do wonders. Then we're all on the same page.
The author is a mathematician, so when he says “it is not in general possible to find bugs by examining the code” he does not mean it is completely impossible to find bugs. He means only that it is not possible to find all bugs or even any particular bug.
I would add that (related to your "maintainability" point) ensuring the code is as simple as possible, and thus much more likely to be "debuggable by review", is a goal of review. Even that won't prevent bugs in the absolute sense, as you rightly say, but it boosts your probabilities.
True - the biggest thing I want to catch in an MR is "will this change lead us onto a path that is uglier, buggier, less maintenanable".
People will generally copy and follow existing patterns, so for example if you let somebody add a new internal date time format, then soon your codebase will bifurcate and there'll be multiple inconsistent versions roaming around.
The other stuff (minor bugs, overly verbose code) can easily be fixed. Paradigm rot cannot.
I work with someone who tends to rejects PR suggestions. I also work with someone else who accepts suggestions.
I think that the for the person who accepts suggestions, it's made me wonder if they accept them in part to share ownership with me. I feel like we both maintain and understand the code, and are on the same page.
For the person who rejects PR suggestions, it makes me less inclined to participate in those PRs. Why spend the time doing a thorough review if it's going to get rejected anyways.
> For the person who rejects PR suggestions, it makes me less inclined to participate in those PRs. Why spend the time doing a thorough review if it's going to get rejected anyways.
This is why you leave blocking suggestions and force the conversation if you think it is important enough.
Sure, ensuring maintainability is one of the benefits of code reviews, but I think it is a bold claim to say that's the solo purpose. For example, code reviews is also a tool that allows teams to get inform of the changes in the code and share responsibility of the whole code base.
It seems the form that code review takes depends on the structure of the organization. In practice, some places use code review as a way to assert hierarchy and tear someone down, while in others it's a friendly, knowledge-sharing exchange. Code review varies depending on the organization's shape and the manager.
And I agree to some extent with what the OP's tweeter said.
these days, bugs can look perfectly fine on the surface, but when combined with the existing system, entirely new types of bugs emerge. This is an especially common pattern in the AI era: the added code itself isn't the problem, but it becomes a bug once it interacts with the existing codebase.
It’s probably important to define what sort of code review you are talking about when making broad claims about it.
GitHub style asynchronous pull request review with inline comments is the norm now, but it’s not the only sort of review there is. I’m old enough to remember processes that include in person reviews that were more like a dissertation defense or conference presentation.
The literature around this that shows that code review is a useful quality practice (in fact one of the only useful quality practices) comes mostly from much more structured review processes than we see now.
My personal opinion is that before llms the GitHub style pr review was for making us feel better about our processes (or governance checkbox checking) and the age of llms will sweep them away as the cost/benefit is so much worse now.
On one of my first jobs, I had printed off change packages which had to be reviewed and signed. There was even a person owning the final copies in filing cabinets. This was more like traditional engineering and everyone had to think of software as more permanent.
The best writing on this is the "agent principal-agent" problem, which correctly frames the problem of agents and code review in terms of trust.
This is why the solutions for high-trust environments (small teams) and low-trust environments (big companies, open source projects) will be different.
Thanks, this articulates something that I've been struggling to put a finger on. You can't review agent generated code the same way you would review a PR, someone needs to fine comb it to make sure everything is fine. And doing that for something like 100,000 lines of code over a few weeks just doesn't sound realistic to me.
This post is inverted for high assurance domains. For example, DO-178C requires checks for compliance to requirements, coding standards, traceability, accuracy and verifiability.
Well, we are dumb and we make stupid mistakes, and code review can find and fix them sometimes. I am very surprised that the practice of "review" is not more widespread. For example when I see medical doctors going YOLO with their patients, ppl driving excavators, etc.
No, the real reason for the code review is to protect the moat of senior engineers/leaders that would nitpick on minute details of code while ignoring the big picture to make sure they can gatekeep any promotions and their competition.
> “Programs must be written for people to read, and only incidentally for machines to execute.”
people probably did not realise what it meant, but AI is bringing it to forefront. Huge amount of work we do is essentially to communicate decisions we want to take, are going to take or have taken. It is a cornerstone of our society when people are continuously exposed to the relevant decisions which are taken in order to build a shared understanding and move forward.
Programming was nothing but that. We did not have a good enough compiler till now and we definitely do not have a good enough language to describe what we need (mostly because thoughts and world in general are so much more complex than any language). And therefore, we used the same language for computer to execute our code and for us to read and understand it. But the reason we store our code in human readable language and not machine code is because we want to communicate the code to future self.
That's why Elon musks's statement about just directly getting a binary makes no sense, because the language used to specify that binary needs to be stored in some engineering records anyway, and then "that is the code".
Code review is also exactly the same, it is a signal which says, "I want to take this decision, are you okay with us taking this decision?" and everyone interested signs off.
Bugs finding are just people going, "I completely agree with the principle of the decision but this particular part of decision is anti-thetic to the principle, should we fix it?"
The author seens to misunderstand the purpose of code review. The purpose is, literally, review the code. Review means basically to think/talk about something again in order to make or not changes on it. When you review something (including code), you are basically asking for yourself: "Should it be changed or it's okay to stay like this?"
In order words, the purpose of code review is to or not ask for changes on the code.
This is largely my take as well. When I review code, I am checking for correctness. If I find something is not correct, that's a bug (or a bug waiting to happen). If I can't understand whether or not something is correct, that's a problem. If I don't know what the correct behavior should be, that's a problem.
Though I do think there is value in the original post. Re-framing is a powerful creative tool when you hit a mental dead end. And the responses let people share the other benefits that change management can bring.
Personally, I would tell you that whatever understanding you gain may still have bugs. Unless your understanding is as complete as a formal treatment of the code, then there may still be bugs in the code due to shared misunderstandings between author and reviewer. The biggest one is both having an incomplete understanding of what a library function does.
So while there may be some overlap, particularly if each person has full understanding of the code's dependencies, in the general case, understanding code and finding bugs are quite different aims.
> The primary purpose of code review is to find code that will be _hard to maintain_.
This makes me wonder if we all have a different primary purpose in mind when it comes to code reviews because that wouldn't be my number one. Talk within your teams would be my advice. Especially now with AI enabling more rapid changes.
As other's already pointed out, the author argues about the primary intent of code review. Of course you find bugs while doing it, and that's a nice side effect, but doing code reviews to assert correctness is maybe suboptimal QA. At least that is my take ...
I would add to this other purposes of code review:
* to share knowledge among the team — if code has been reviewed then at least two people know about it
* to ensure the changes won’t cause problems with other systems or future plans — maybe we will be rolling out a new logging system soon and we don’t want to solve the problem in this specific way
* a chance for the reviewer and reviewee to learn something about the system or the code via the review discussion
* team coherence — code that’s well reviewed is a team effort. Working together on small things like code reviews helps teammates work better together on other tasks in the future
My employer has the weirdest code reviews I have ever participated in. I kid you all not, we all get around and it's like team-wide show and tell. We basically demo anything cool we learned, found, etc.. There is actually no real review of any code at all. At least, not in terms of quality or security.
I don't know about many people, but the author for sure doesn't understand the primary purpose of code review.
If the primary purpose of code review is to assess maintainability, there is no need for review, that can be done by automated tooling (formatting, bad naming, cyclomatic complexity etc.)
One of my favorite little things to notice is when everybody thinks they know what something is, and they all agree about it, but they in fact don't agree. In this case we have the statement "Code review is a good idea". What right-minded software engineer could possibly disagree with that?
But then notice 1. the number of people jumping up to say "No, you don't understand the point of code review" and 2. how what follows "The point is..." varies between so many different people. I can't quite say it's a unique take per person, as I've seen before, there are some common threads, but they are also not all the same answer by any means either.
In this case, there isn't a "the" point of code review to discuss. It turns out that while we all may have thought we were doing it for the same reasons, we aren't. This is real. We don't have the same goals, we don't have the same methodology, and thus, the value we get from it may be different. And in fact it is perfectly reasonable to discuss the multiple cost/benefits ratios that differ across the various definitions, because the simplification "it's good, end of story" is destroying important distinctions.
In this situation, it is helpful to frame this as a matter of the costs and benefits of the various options available. Forget the statement "code review is good"; it is fallacious to start with that statement as an axiom and then argue about whether or not your definition of "code review" is or is not the "correct" definition so that your definition gets the "good" attribute applied to it. Consider the options directly.
(I have to admit I've used this effect in anger... in meetings where I can tell that everybody thinks they know what some project is but I can tell they all have a different definition of it in mind, but I also know it's not going to happen anyhow, I don't chase down the differences. Sometimes you can use this to your advantage to cut short what would otherwise be a quite interminable, yet ultimately pointless, meeting.)
I think you're missing the point of code review.
By the time when the PR is ready to merge, discussions around the architecture and how the code should be structured should already be part of the tech design of a given feature. So the discussion around whether a A feature is built and planned in a maintainable way, should be way before a PR is filed.
A PR review is making sure that you verify against the already agreed-upon structure, making sure everything matches the plan, and also find bugs and stuff that was missed, according to the plan.
Every team should follow a plan, fine on a side project, but if you work in a large codebase with a bunch of devs, you need to have some sort of workflow to avoid stepping on each other's toes.
bug fixes are supposed to be small, contained, if they're rearchitecting the codebase, then they're not _bugs_, but tech improvements, and need to be addressed differently and I agree that this should be flagged in the PR.
Well, the code review should also be reviewing the provided test code or test plan or whatever that will prove it does not have bugs.
You're not reviewing the code to confirm that the code is bug free... you're reviewing the additional code that confirms that the feature-code is bug free.
Any process that has a step of "we'll get to that later" is a failure. That includes testing. Until there is some provided content that will be able to provide evidence that that code is safe to merge, it's not done.
But yeah, I need to be able to understand what every line does.
Exactly. The procedure is to read the description of the change to understand its motivation, goals, and overall design. Then you read the tests, checking whether they are compatible with and cover all aspects of what was described. Then you can read the code under test but at that point you enjoy the assumption that it at least passed those tests.
Whatever purpose code review served pre-2002, post-2002 it serves as corporate audit coverage. A common (mis?) interpretation of SOX and SOC2 is essentially that the company must have a two-person sign-off on any system change that could damage the company or its reputation.
Because the reviewer is not magical. If there was something in the code the author couldn't see, the reviewer probably won't see it either.
The way to confirm that code does not have bugs is testing. So the reviewer is not looking at the code saying "this will work", they're looking at the code saying "I understand how this works, it makes sense."
Evidence that the code is safe is something that also should be provided in the PR, but it is not the main code. It is ideally test automation that is just as understandable as the feature code, but failing that ad-hoc test evidence or a specific step-by-step plan with evidence of execution is good too.
> many people misunderstand the purpose of code review
Oooh, I bet including the author? Yeah, right there, he fails to make any qualifications for his statement, making it factually incorrect.
There are plenty of reasons to do code review. If you force me to, I'll define it as information transfer. The point is to have a conversation about the code. To expand both people's understanding about the codebase. Everything on top of that is extra. I've found real and significant bugs doing code review. In large part because I understand the codebase better than the author. That's finding and preventing bugs.
I also read PRs looking for malicious code trying to sneak in, you never know, the person I've called my best friend, someone with opsec better than mine, may suddenly have turned evil and this is the PR where they're finally sneaking in the back door.... that's never happened yet, but fingers crossed!
> As everyone should know by now, it is not in general possible to find bugs by examining the code.
... I'd love to know what the author really meant to write here, because it certainly wasn't this.
He's a mathematician, so what he means by "in general", is "in every possible case", or "without exception", so what I think he means is, "not all bugs will be found by code review." I agree it probably could have been made more clear.
The primary purpose of code review is to maintain existing hierarchy by preventing junior SWEs from getting promoted by committing code that is smarter than what the senior architect can understand.
If the code is so smart that it's not easily understandable, it's not easily fixable. My transition from junior to senior was accompanied by the realization that simpler is nearly always better.
I don't know enough to speak about that particular domain, but if the junior is writing something the senior can't understand, that's always going to be a problem. That code becomes the team's responsibility, and that code needs to be able to be maintained by the entire team, not only by the junior with something to prove.
Who is getting called at 2 AM when something breaks? Not the junior.
What I find to be maybe the single most important part of code review is knowledge transfer.
Our entire small team thumbs up a PR before it's merged unless there's a big rush on it, and this gives everyone on the team a rough idea of the state of the codebase at any given time. There's no being blindsided like "this whole system I depend on is gone" like I had happen at far more siloed places I've worked.
Beyond that, it gives a forum to ask questions about how things work to further build understanding. On a high functioning team, every developer should have at least a modest understanding of the entire system, including parts they never touch.
Another important feature is just the institutional knowledge check. For instance recently I made a small change to a table and a coworker pointed out that there was a microservice I wasn't considering that wrote to that table that would break (yes, sharing tables is bad design, unrelated). I had no idea this microservice existed let alone had access to this table. The institutional knowledge check here though prevented a larger issue and potential data cleanup situation.
> For instance recently I made a small change to a table and a coworker pointed out that there was a microservice I wasn't considering that wrote to that table that would break
If code reviews are important, where does testing sit? Presumably if the coworker had not been part of the code review something would have stopped the breaking change making its way to prod?
> something would have stopped the breaking change making its way to prod?
or a prod outage causes the knowledge to be experienced.
We even find ourselves creating PRs in situations where the code is going to be merged immediately anyway, and tagging other devs, just so they have a convenient way to see what got merged and why. So people don't lose track of what is in the codebase.
It's a good practice. Worth mentioning also: the same can be done with ordinary git log, assuming everyone is using git well. A proper git log of yesterday's work can be like your work newspaper with coffee.
this seems like a chain of good practices. though I find it hard to stay disciplined about keeping commits well scoped and well described
Becoming very comfortable with "rebase --interactive" and other cmds for editing your (local!) history before merging helps a lot. Once you are, it only adds 5m or so of extra work to most PRs. And while acquiring this knowledge used to be difficult, LLMs make it very easy these days.
My attitude has always been that code review is best thought of as the gate where code goes from being owned by the author to being owned by the team or project. The code I'm reviewing is not your code, it is code that is about to become our code.
Maintainability is a major factor in that, of course.
Such a luxury, I am envious!
Our team started using AI, so I switched to a simple method: no comments, and a binary "is this batshit crazy or passable" approval decision rule.
Saving myself time and sanity.
Yep, without a decent team culture this is what LLMs force, the slop deluge is just overwhelming without leadership asserting "no, stop"
Ultimately you just let bugs through because the alternative is spend an inordinate amount of time communicating with someones claude through PR comments about what the shape should be.
Career was fun while it lasted. I suppose its a blessing a to get to do a job that you enjoyed for as many years as I did.
> The primary purpose of code review is to find code that will be _hard to maintain_.
In some organizations, maintainability may be the biggest risk being mitigated in a code review. But for me, that's selling code reviews short.
In my experience, code reviews are the single most important quality control process in the entire development life cycle. Engineers often don't have a lot of influence over the quality of requirements. Engineers often don't have a lot of influence over the competence and thoroughness of the QA process (and it often doesn't exist at all). But engineers frequently have total control over code reviews.
If I can't depend on the rest of the organization for QC, code reviews are the first place I look to mitigate that risk. That means code reviews find bugs. That means code reviews identify code smells. That means code reviews pressure test requirements and whether the implementation matches the assignment. That means code reviews transfer knowledge and serve as a teacher for both the PR author and the reviewer. And so on.
Thorough and pedantic code reviews are challenging and tedious, at least at first, but the team adapts and both the code and the review process gets better.
This just makes reviewers and authors lazier.
The purpose of code review is multi-faceted. Hard to maintain? Yes. Might have bugs? Yes. Can be done simpler/cleaner? Yes. Is in line with project code style? Yes. Get someone else to also understand the code? Yes. Onboard junior team member? Yes. Sanity check design decisions? Yes.
This flippant note is mostly more self-justification for being a lazy code reviewer.
Indeed, we worked hard as an industry to move beyond "blame the author" to "blame the process/team".
Unfortunately this article is just bait, may as well say "People seem to think dinner is about eating food, but it's not about eating at all, actually it's about connecting with family and friends!". It's a specific type of poorly constructed reductionist argument that plays well on HN.
Agreed. There's a whole checklist of what to look out for:
- Does it functionally achieve what it sets out to (as per tacker issue or PR description)?
- Does it have extraneous code? Leftover debug prints, private API keys etc...
- Does it have any obvious defects? Memory leaks, un-handled edge cases, security flaws, obsolete API calls, etc...
- Could it be more understandable? Add/remove abstractions, better variable/method names, more/less functional etc...
- Is the style consistent with the codebase and/or style guidelines?
- Are there obvious performance improvements? Hashset instead of list, lazy evaluations, etc...
- Is it sufficiently well tested?
I'm not even sure I agree that if I can't understand the code then it shouldn't go through. Some code is just really hard to understand. The aim is to make it as easy as possible to understand, while being functionally correct.
It's also important to help other members of the team to learn and understand.
> The primary purpose of code review is to find code that will be _hard to maintain_
Says who? I agree that it's a good purpose. But the main problem with code reviews is not having a in-house code review guidance. Doesn't need to be long. 1 page would do wonders. Then we're all on the same page.
The author is a mathematician, so when he says “it is not in general possible to find bugs by examining the code” he does not mean it is completely impossible to find bugs. He means only that it is not possible to find all bugs or even any particular bug.
User names match... are you the original author? Why commenting in the third person?
Because it was him... 10 months ago?
Ofc you're correct in that sense.
I would add that (related to your "maintainability" point) ensuring the code is as simple as possible, and thus much more likely to be "debuggable by review", is a goal of review. Even that won't prevent bugs in the absolute sense, as you rightly say, but it boosts your probabilities.
True - the biggest thing I want to catch in an MR is "will this change lead us onto a path that is uglier, buggier, less maintenanable".
People will generally copy and follow existing patterns, so for example if you let somebody add a new internal date time format, then soon your codebase will bifurcate and there'll be multiple inconsistent versions roaming around.
The other stuff (minor bugs, overly verbose code) can easily be fixed. Paradigm rot cannot.
I work with someone who tends to rejects PR suggestions. I also work with someone else who accepts suggestions.
I think that the for the person who accepts suggestions, it's made me wonder if they accept them in part to share ownership with me. I feel like we both maintain and understand the code, and are on the same page.
For the person who rejects PR suggestions, it makes me less inclined to participate in those PRs. Why spend the time doing a thorough review if it's going to get rejected anyways.
> For the person who rejects PR suggestions, it makes me less inclined to participate in those PRs. Why spend the time doing a thorough review if it's going to get rejected anyways.
This is why you leave blocking suggestions and force the conversation if you think it is important enough.
Sure, ensuring maintainability is one of the benefits of code reviews, but I think it is a bold claim to say that's the solo purpose. For example, code reviews is also a tool that allows teams to get inform of the changes in the code and share responsibility of the whole code base.
It seems the form that code review takes depends on the structure of the organization. In practice, some places use code review as a way to assert hierarchy and tear someone down, while in others it's a friendly, knowledge-sharing exchange. Code review varies depending on the organization's shape and the manager.
And I agree to some extent with what the OP's tweeter said. these days, bugs can look perfectly fine on the surface, but when combined with the existing system, entirely new types of bugs emerge. This is an especially common pattern in the AI era: the added code itself isn't the problem, but it becomes a bug once it interacts with the existing codebase.
It’s probably important to define what sort of code review you are talking about when making broad claims about it.
GitHub style asynchronous pull request review with inline comments is the norm now, but it’s not the only sort of review there is. I’m old enough to remember processes that include in person reviews that were more like a dissertation defense or conference presentation.
The literature around this that shows that code review is a useful quality practice (in fact one of the only useful quality practices) comes mostly from much more structured review processes than we see now.
My personal opinion is that before llms the GitHub style pr review was for making us feel better about our processes (or governance checkbox checking) and the age of llms will sweep them away as the cost/benefit is so much worse now.
On one of my first jobs, I had printed off change packages which had to be reviewed and signed. There was even a person owning the final copies in filing cabinets. This was more like traditional engineering and everyone had to think of software as more permanent.
"The purpose of code review is not for the reviewer to find bugs"
Well kinda - code review needs to identify any missing tests? And without the tests more likely a bug could exist.
The best writing on this is the "agent principal-agent" problem, which correctly frames the problem of agents and code review in terms of trust.
This is why the solutions for high-trust environments (small teams) and low-trust environments (big companies, open source projects) will be different.
https://crawshaw.io/blog/agent-principal-agent
Thanks, this articulates something that I've been struggling to put a finger on. You can't review agent generated code the same way you would review a PR, someone needs to fine comb it to make sure everything is fine. And doing that for something like 100,000 lines of code over a few weeks just doesn't sound realistic to me.
This post is inverted for high assurance domains. For example, DO-178C requires checks for compliance to requirements, coding standards, traceability, accuracy and verifiability.
Well, we are dumb and we make stupid mistakes, and code review can find and fix them sometimes. I am very surprised that the practice of "review" is not more widespread. For example when I see medical doctors going YOLO with their patients, ppl driving excavators, etc.
No, the real reason for the code review is to protect the moat of senior engineers/leaders that would nitpick on minute details of code while ignoring the big picture to make sure they can gatekeep any promotions and their competition.
I think when the the SICP authors said
> “Programs must be written for people to read, and only incidentally for machines to execute.”
people probably did not realise what it meant, but AI is bringing it to forefront. Huge amount of work we do is essentially to communicate decisions we want to take, are going to take or have taken. It is a cornerstone of our society when people are continuously exposed to the relevant decisions which are taken in order to build a shared understanding and move forward.
Programming was nothing but that. We did not have a good enough compiler till now and we definitely do not have a good enough language to describe what we need (mostly because thoughts and world in general are so much more complex than any language). And therefore, we used the same language for computer to execute our code and for us to read and understand it. But the reason we store our code in human readable language and not machine code is because we want to communicate the code to future self.
That's why Elon musks's statement about just directly getting a binary makes no sense, because the language used to specify that binary needs to be stored in some engineering records anyway, and then "that is the code".
Code review is also exactly the same, it is a signal which says, "I want to take this decision, are you okay with us taking this decision?" and everyone interested signs off.
Bugs finding are just people going, "I completely agree with the principle of the decision but this particular part of decision is anti-thetic to the principle, should we fix it?"
Nowadays the main point of PR's is to find out what Claude has been coding for me.
The author seens to misunderstand the purpose of code review. The purpose is, literally, review the code. Review means basically to think/talk about something again in order to make or not changes on it. When you review something (including code), you are basically asking for yourself: "Should it be changed or it's okay to stay like this?"
In order words, the purpose of code review is to or not ask for changes on the code.
I make code review and PR approval optional for my teams. A solid process should emerge from respect and peer pressure alone.
What if I told you that understanding what it is doing and finding bugs is actually the same problem?
This is largely my take as well. When I review code, I am checking for correctness. If I find something is not correct, that's a bug (or a bug waiting to happen). If I can't understand whether or not something is correct, that's a problem. If I don't know what the correct behavior should be, that's a problem.
Though I do think there is value in the original post. Re-framing is a powerful creative tool when you hit a mental dead end. And the responses let people share the other benefits that change management can bring.
Personally, I would tell you that whatever understanding you gain may still have bugs. Unless your understanding is as complete as a formal treatment of the code, then there may still be bugs in the code due to shared misunderstandings between author and reviewer. The biggest one is both having an incomplete understanding of what a library function does.
So while there may be some overlap, particularly if each person has full understanding of the code's dependencies, in the general case, understanding code and finding bugs are quite different aims.
> The primary purpose of code review is to find code that will be _hard to maintain_.
This makes me wonder if we all have a different primary purpose in mind when it comes to code reviews because that wouldn't be my number one. Talk within your teams would be my advice. Especially now with AI enabling more rapid changes.
Not all bugs are buffer overflows, many are just the code not doing what it claimed at a high level
100% this. This is why it's so hard to trust AI on code reviews, at least for now.
Found plenty of bugs by reading/doing code review.
As other's already pointed out, the author argues about the primary intent of code review. Of course you find bugs while doing it, and that's a nice side effect, but doing code reviews to assert correctness is maybe suboptimal QA. At least that is my take ...
I would add to this other purposes of code review:
* to share knowledge among the team — if code has been reviewed then at least two people know about it
* to ensure the changes won’t cause problems with other systems or future plans — maybe we will be rolling out a new logging system soon and we don’t want to solve the problem in this specific way
* a chance for the reviewer and reviewee to learn something about the system or the code via the review discussion
* team coherence — code that’s well reviewed is a team effort. Working together on small things like code reviews helps teammates work better together on other tasks in the future
My employer has the weirdest code reviews I have ever participated in. I kid you all not, we all get around and it's like team-wide show and tell. We basically demo anything cool we learned, found, etc.. There is actually no real review of any code at all. At least, not in terms of quality or security.
That's code bazaar!
It's a bizarre bazaar for sure.
I don't know about many people, but the author for sure doesn't understand the primary purpose of code review.
If the primary purpose of code review is to assess maintainability, there is no need for review, that can be done by automated tooling (formatting, bad naming, cyclomatic complexity etc.)
One of my favorite little things to notice is when everybody thinks they know what something is, and they all agree about it, but they in fact don't agree. In this case we have the statement "Code review is a good idea". What right-minded software engineer could possibly disagree with that?
But then notice 1. the number of people jumping up to say "No, you don't understand the point of code review" and 2. how what follows "The point is..." varies between so many different people. I can't quite say it's a unique take per person, as I've seen before, there are some common threads, but they are also not all the same answer by any means either.
In this case, there isn't a "the" point of code review to discuss. It turns out that while we all may have thought we were doing it for the same reasons, we aren't. This is real. We don't have the same goals, we don't have the same methodology, and thus, the value we get from it may be different. And in fact it is perfectly reasonable to discuss the multiple cost/benefits ratios that differ across the various definitions, because the simplification "it's good, end of story" is destroying important distinctions.
In this situation, it is helpful to frame this as a matter of the costs and benefits of the various options available. Forget the statement "code review is good"; it is fallacious to start with that statement as an axiom and then argue about whether or not your definition of "code review" is or is not the "correct" definition so that your definition gets the "good" attribute applied to it. Consider the options directly.
(I have to admit I've used this effect in anger... in meetings where I can tell that everybody thinks they know what some project is but I can tell they all have a different definition of it in mind, but I also know it's not going to happen anyhow, I don't chase down the differences. Sometimes you can use this to your advantage to cut short what would otherwise be a quite interminable, yet ultimately pointless, meeting.)
I think you're missing the point of code review. By the time when the PR is ready to merge, discussions around the architecture and how the code should be structured should already be part of the tech design of a given feature. So the discussion around whether a A feature is built and planned in a maintainable way, should be way before a PR is filed. A PR review is making sure that you verify against the already agreed-upon structure, making sure everything matches the plan, and also find bugs and stuff that was missed, according to the plan.
Not every codebase project etc use such workflow
Also such approach doesnt work with bug fixes / regressions
Every team should follow a plan, fine on a side project, but if you work in a large codebase with a bunch of devs, you need to have some sort of workflow to avoid stepping on each other's toes.
bug fixes are supposed to be small, contained, if they're rearchitecting the codebase, then they're not _bugs_, but tech improvements, and need to be addressed differently and I agree that this should be flagged in the PR.
a PR review is the final defence line before a QA
Well, the code review should also be reviewing the provided test code or test plan or whatever that will prove it does not have bugs.
You're not reviewing the code to confirm that the code is bug free... you're reviewing the additional code that confirms that the feature-code is bug free.
Any process that has a step of "we'll get to that later" is a failure. That includes testing. Until there is some provided content that will be able to provide evidence that that code is safe to merge, it's not done.
But yeah, I need to be able to understand what every line does.
Exactly. The procedure is to read the description of the change to understand its motivation, goals, and overall design. Then you read the tests, checking whether they are compatible with and cover all aspects of what was described. Then you can read the code under test but at that point you enjoy the assumption that it at least passed those tests.
Whatever purpose code review served pre-2002, post-2002 it serves as corporate audit coverage. A common (mis?) interpretation of SOX and SOC2 is essentially that the company must have a two-person sign-off on any system change that could damage the company or its reputation.
Uh.. why not both?
Because the reviewer is not magical. If there was something in the code the author couldn't see, the reviewer probably won't see it either.
The way to confirm that code does not have bugs is testing. So the reviewer is not looking at the code saying "this will work", they're looking at the code saying "I understand how this works, it makes sense."
Evidence that the code is safe is something that also should be provided in the PR, but it is not the main code. It is ideally test automation that is just as understandable as the feature code, but failing that ad-hoc test evidence or a specific step-by-step plan with evidence of execution is good too.
> many people misunderstand the purpose of code review
Oooh, I bet including the author? Yeah, right there, he fails to make any qualifications for his statement, making it factually incorrect.
There are plenty of reasons to do code review. If you force me to, I'll define it as information transfer. The point is to have a conversation about the code. To expand both people's understanding about the codebase. Everything on top of that is extra. I've found real and significant bugs doing code review. In large part because I understand the codebase better than the author. That's finding and preventing bugs.
I also read PRs looking for malicious code trying to sneak in, you never know, the person I've called my best friend, someone with opsec better than mine, may suddenly have turned evil and this is the PR where they're finally sneaking in the back door.... that's never happened yet, but fingers crossed!
> As everyone should know by now, it is not in general possible to find bugs by examining the code.
... I'd love to know what the author really meant to write here, because it certainly wasn't this.
[delayed]
He's a mathematician, so what he means by "in general", is "in every possible case", or "without exception", so what I think he means is, "not all bugs will be found by code review." I agree it probably could have been made more clear.
The primary purpose of code review is to maintain existing hierarchy by preventing junior SWEs from getting promoted by committing code that is smarter than what the senior architect can understand.
If the code is so smart that it's not easily understandable, it's not easily fixable. My transition from junior to senior was accompanied by the realization that simpler is nearly always better.
Good programmers write code that’s so simple and obvious it looks like anyone could have written it.
Bad programmers make the simplest things really complicated.
Write me simple AI inference code that has high performance in big batches.
I don't know enough to speak about that particular domain, but if the junior is writing something the senior can't understand, that's always going to be a problem. That code becomes the team's responsibility, and that code needs to be able to be maintained by the entire team, not only by the junior with something to prove.
Who is getting called at 2 AM when something breaks? Not the junior.
Does tinygrad not count for some reason?