Why don't we just pagerank github contributors? Merged PRs approved by other quality contributors improves rank. New PRs tagged by a bot with the rank of the submitter. Add more scoring features (account age? employer?) as desired.
The last paragraph is interesting: "Overall I think we're going to see a much higher quality of software, ironically around the same level than before 2000 when the net became usable by everyone to download fixes. When the software had to be pressed to CDs or written to millions of floppies, it had to survive an amazing quantity of tests that are mostly neglected nowadays since updates are easy to distribute."
Was software made before 2000 better? And, if so, was it because of better testing or lower complexity?
It is hard to say which of the 2 is the reason, more likely both, i.e. lower complexity enabled more exhaustive testing.
In any case some of the software from before 2000 was definitely better than today, i.e. it behaved like being absolutely foolproof, i.e. nothing that you could do could cause any crash or corrupted data or any other kind of unpredictable behavior.
However, the computers to which most people had access at that time had only single-threaded CPUs. Even if you used a preemptive multitasking operating system and a heavily multi-threaded application, executing it on a single-threaded CPU was unlikely to expose subtle bugs due to race conditions, that might have been exposed on a multi-core CPU.
While nowadays there exists no standard operating system that I fully trust to never fail in any circumstance, unlike before 2003, I wonder whether this is caused by a better quality of the older programs or by the fact that it is much harder to implement software concurrency correctly on systems with hardware parallelism.
I was a developer at Microsoft in the 90s (Visual Studio (Boston) and Windows teams). I won't claim that software back then was "better," but what is definitely true is that we had to think about everything at a much lower level.
For example, you had to know which Win32 functions caused ring-3 -> ring-0 transitions because those transitions could be incredibly costly. You couldn't just "find the right function" and move on. You had to find the right function that wouldn't bring your app (and entire system) to its knees.
I specifically remember hating my life whenever we ran into a KiUserExceptionDispatcher [0] issue, because even something as simple as an exception could kill your app's performance.
Additionally, we didn't get to just patch flaws as they arose. We either had to send out patches on floppy disks, post them to BBSs, or even send them to PC Magazine.
At the time of release, yes. They had to ensure the software worked before printing CDs and floppies. Nowadays they release buggy versions that users essentially test for them.
Also in terms of security, there was generally a much smaller potential attack surface and those surfaces were harder to reach because we were much less constantly connected.
Software has gotten drastically more secure than it was in 2000. It's hard to comprehend how bad the security picture was in 2000. This very much, extremely includes Linux.
I remember opening a webpage and being hacked seemed more likely. Adobe Flash and Java had more vulnerabilities and weaker (if any) sandboxes than JavaScript.
But there was much less awareness of buffer overflows and none of the countermeasures that are widespread today. It was almost defining of the Win95 era that applications (eg. Word) frequently crashed because of improper and unsafe memory management.
Yes. The incentives for writing reliable, robust code were much higher. The internet existed so you could, in theory, get a patch out for people to download - but a sizeable part of any user base might have limited access, so would require something physical shipped to them (a floppy or CD). Making sure that your code worked and worked well at time of shipping was important. Large corporate customers were not going to appreciate having to distribute an update across their tens of thousands of machines,
No. The world wasn't as connected as it is today, which meant that the attack surface to reasonably consider was much smaller. A lot of the issues that we had back then were due to designs and implementations that assumed a closed system overall - but often allowed very open interoperability between components (programs or machines) within the system. For example, Outlook was automatable, so that it could be part of larger systems and send mail in an automated way. This makes sense within an individual organisation's "system", but isn't wise at a global level. Email worms ran rampant until Microsoft was forced to reduce that functionality via patches, which were costly for their customers to apply. It damaged their reputation considerably.
An extreme version of this was openness was SQL Slammer - a worm which attacked SQL Servers and development machines. Imagine that - enough organisations had their SQL Servers or developer machines directly accessible that an actual worm could thrive on a relational database system. Which is mindboggling to think about these days, but it really happened - see https://en.wikipedia.org/wiki/SQL_Slammer for details.
I wouldn't say that the evidence points to software being better in the way that we would think of "better" today. I'd say that the environment it had to exist in was simpler, and that the costs of shipping & updating were higher - so it made more sense to spend time creating robust software. Also nobody was thinking about the possible misuse or abuse of their software except in very limited ways. These days we have to protect against much more ingenious use & abuse of programs.
Furthermore today patching is quick and easy (by historical comparison), and a company might even be offering its own hosted solution, which makes the cost of patching very low for them. In such an environment it can seem more reasonable to focus on shipping features quickly over shipping robust code slowly. I'd argue that's a mistake, but a lot of software development managers disagree with me, and their pay packet often depends on that view, so they're not going to change their minds any time soon.
In a way this is best viewed as the third age of computing. The first was the mainframe age - centralised computer usage, with controlled access and oversight, so mistakes were costly but could be quickly recovered from. The second was the desktop PC age - distributed computer usage, with less access control, so mistakes were often less costly but recovering from them was potentially very expensive. The third is the cloud & device age, with a mix of centralised and distributed computer use, a mix of access control, and potentially much lower costs of recovery. In this third age if you make the wrong decisions on what to prioritise (robustness vs speed of shipping), it can be the worst of both the previous ages. But it doesn't have to be.
I hope that makes sense, and is a useful perspective for you.
Just think of 8 and 16 bit video console games. Those cartridges were expensive so just how sure they had to be they were bug free before making millions of them?
It was a simpler time. Not better. Not worse. Programs still had bugs, but they weren't sloppy UI bugs, they were logic bugs and memory leaks. If software was better back then, we'd still be using it!
Literally the moment everyone got on the internet, pretty much every computer program and operating system in the world was besieged by viruses and security flaws, so no.
“Reversing was already mostly a speed-bump even for entry-level teams, who lift binaries into IR or decompile them all the way back to source. Agents can do this too, but they can also reason directly from assembly. If you want a problem better suited to LLMs than bug hunting, program translation is a good place to start.”
Huh. Direct debugging, in assembly. At that point, why not jump down to machine code?
Interesting that it's been higher than forecast since 2023. Personally I'd expect that trend to continue given that LLMs both increase bugs written as well as bugs discovered.
> I don't know how long this pace will last. I suspect that bugs are reported faster than they are written, so we could in fact be purging a long backlog
Hopefully these same tools will also help catch security bugs at the point they're written. Maybe one day we'll reach a point where the discovery of new, live vulnerabilities is extremely rare?
Around 70% of security vulnerabilities are about memory safety and only exist because software is written in C and C++. Because most vulnerabilities are in newly written code, Google has found that simply starting writing new code in Rust (rather than trying to rewrite existing codebases) quickly brings the number of found vulnerabilities down drastically.
No, this is false. For Rust codebases that aren't doing high-peformance data structures, C interop, or bare-metal stuff, it's typical to write no unsafe code at all. I'm not sure who told you otherwise, but they have no idea what they're talking about.
It's interesting to hear from people directly in the thick of it that these bug reports are apparently gaining value and are no longer just slop. Maybe there is hope for a world where AI helps create bug free software and doesn't just overload maintainers.
An AI enthusiast having a breathless and predictive position on the future of the technology? No way! It's almost like Wall Street is about to sour on the whole stack and there is a concerted effort to artificially push these views into the conversation to get people on board.
Then again, I'm a known crank and aggressive cynic, but you never really see any gathered data backing these points up.
Anyone who says anything good about AI must be an AI shill from the start, not someone who is genuinely observing reality or had their mind changed, don't you know?
Sort of a tautology to just assert that someone saying good things about AI is an AI enthusiast and therefore their opinion should be dismissed. He also happens to have been a kernel maintainer, his experience as he's describing it should count for something.
"On the kernel security list we've seen a huge bump of reports. We were between 2 and 3 per week maybe two years ago, then reached probably 10 a week over the last year with the only difference being only AI slop, and now since the beginning of the year we're around 5-10 per day depending on the days (fridays and tuesdays seem the worst). Now most of these reports are correct, to the point that we had to bring in more maintainers to help us."
Is there a reason you’ve copy pasted the first paragraph from the link? It doesn’t add anything to the discussion, and also doesn’t help as a tl;dr because it’s literally the first paragraph. Genuine question!
Reports being written faster than bugs being created? Better quality software than before the 2000s?
Oh my sweet summer child.
This is some seriously delusional cope from someone who drank the entire jug of kool-aid.
I’d love to be proven wrong but the current trajectory is pretty plain as day from current outcomes. Everything is getting worse, and everyone is getting overwhelmed and we are under attack even more and the attacks are getting substantially more sophisticated and the blast radius is much bigger.
Why don't we just pagerank github contributors? Merged PRs approved by other quality contributors improves rank. New PRs tagged by a bot with the rank of the submitter. Add more scoring features (account age? employer?) as desired.
The last paragraph is interesting: "Overall I think we're going to see a much higher quality of software, ironically around the same level than before 2000 when the net became usable by everyone to download fixes. When the software had to be pressed to CDs or written to millions of floppies, it had to survive an amazing quantity of tests that are mostly neglected nowadays since updates are easy to distribute."
Was software made before 2000 better? And, if so, was it because of better testing or lower complexity?
It is hard to say which of the 2 is the reason, more likely both, i.e. lower complexity enabled more exhaustive testing.
In any case some of the software from before 2000 was definitely better than today, i.e. it behaved like being absolutely foolproof, i.e. nothing that you could do could cause any crash or corrupted data or any other kind of unpredictable behavior.
However, the computers to which most people had access at that time had only single-threaded CPUs. Even if you used a preemptive multitasking operating system and a heavily multi-threaded application, executing it on a single-threaded CPU was unlikely to expose subtle bugs due to race conditions, that might have been exposed on a multi-core CPU.
While nowadays there exists no standard operating system that I fully trust to never fail in any circumstance, unlike before 2003, I wonder whether this is caused by a better quality of the older programs or by the fact that it is much harder to implement software concurrency correctly on systems with hardware parallelism.
I was a developer at Microsoft in the 90s (Visual Studio (Boston) and Windows teams). I won't claim that software back then was "better," but what is definitely true is that we had to think about everything at a much lower level.
For example, you had to know which Win32 functions caused ring-3 -> ring-0 transitions because those transitions could be incredibly costly. You couldn't just "find the right function" and move on. You had to find the right function that wouldn't bring your app (and entire system) to its knees.
I specifically remember hating my life whenever we ran into a KiUserExceptionDispatcher [0] issue, because even something as simple as an exception could kill your app's performance.
Additionally, we didn't get to just patch flaws as they arose. We either had to send out patches on floppy disks, post them to BBSs, or even send them to PC Magazine.
[0]: https://doar-e.github.io/blog/2013/10/12/having-a-look-at-th...
> Was software made before 2000 better?
At the time of release, yes. They had to ensure the software worked before printing CDs and floppies. Nowadays they release buggy versions that users essentially test for them.
Also in terms of security, there was generally a much smaller potential attack surface and those surfaces were harder to reach because we were much less constantly connected.
Software has gotten drastically more secure than it was in 2000. It's hard to comprehend how bad the security picture was in 2000. This very much, extremely includes Linux.
I remember opening a webpage and being hacked seemed more likely. Adobe Flash and Java had more vulnerabilities and weaker (if any) sandboxes than JavaScript.
But there was much less awareness of buffer overflows and none of the countermeasures that are widespread today. It was almost defining of the Win95 era that applications (eg. Word) frequently crashed because of improper and unsafe memory management.
Yes and no.
Yes. The incentives for writing reliable, robust code were much higher. The internet existed so you could, in theory, get a patch out for people to download - but a sizeable part of any user base might have limited access, so would require something physical shipped to them (a floppy or CD). Making sure that your code worked and worked well at time of shipping was important. Large corporate customers were not going to appreciate having to distribute an update across their tens of thousands of machines,
No. The world wasn't as connected as it is today, which meant that the attack surface to reasonably consider was much smaller. A lot of the issues that we had back then were due to designs and implementations that assumed a closed system overall - but often allowed very open interoperability between components (programs or machines) within the system. For example, Outlook was automatable, so that it could be part of larger systems and send mail in an automated way. This makes sense within an individual organisation's "system", but isn't wise at a global level. Email worms ran rampant until Microsoft was forced to reduce that functionality via patches, which were costly for their customers to apply. It damaged their reputation considerably.
An extreme version of this was openness was SQL Slammer - a worm which attacked SQL Servers and development machines. Imagine that - enough organisations had their SQL Servers or developer machines directly accessible that an actual worm could thrive on a relational database system. Which is mindboggling to think about these days, but it really happened - see https://en.wikipedia.org/wiki/SQL_Slammer for details.
I wouldn't say that the evidence points to software being better in the way that we would think of "better" today. I'd say that the environment it had to exist in was simpler, and that the costs of shipping & updating were higher - so it made more sense to spend time creating robust software. Also nobody was thinking about the possible misuse or abuse of their software except in very limited ways. These days we have to protect against much more ingenious use & abuse of programs.
Furthermore today patching is quick and easy (by historical comparison), and a company might even be offering its own hosted solution, which makes the cost of patching very low for them. In such an environment it can seem more reasonable to focus on shipping features quickly over shipping robust code slowly. I'd argue that's a mistake, but a lot of software development managers disagree with me, and their pay packet often depends on that view, so they're not going to change their minds any time soon.
In a way this is best viewed as the third age of computing. The first was the mainframe age - centralised computer usage, with controlled access and oversight, so mistakes were costly but could be quickly recovered from. The second was the desktop PC age - distributed computer usage, with less access control, so mistakes were often less costly but recovering from them was potentially very expensive. The third is the cloud & device age, with a mix of centralised and distributed computer use, a mix of access control, and potentially much lower costs of recovery. In this third age if you make the wrong decisions on what to prioritise (robustness vs speed of shipping), it can be the worst of both the previous ages. But it doesn't have to be.
I hope that makes sense, and is a useful perspective for you.
Just think of 8 and 16 bit video console games. Those cartridges were expensive so just how sure they had to be they were bug free before making millions of them?
It was a simpler time. Not better. Not worse. Programs still had bugs, but they weren't sloppy UI bugs, they were logic bugs and memory leaks. If software was better back then, we'd still be using it!
> Was software made before 2000 better?
Literally the moment everyone got on the internet, pretty much every computer program and operating system in the world was besieged by viruses and security flaws, so no.
I think there was a period where things got better but I don’t think it was pre-internet.
There was a point in time where both windows wasn’t constantly bsoding and Microsoft’s primary objectives weren't telemetry and slop coding.
Important to note that this is a comment on this article: https://lwn.net/Articles/1065586/.
“Reversing was already mostly a speed-bump even for entry-level teams, who lift binaries into IR or decompile them all the way back to source. Agents can do this too, but they can also reason directly from assembly. If you want a problem better suited to LLMs than bug hunting, program translation is a good place to start.”
Huh. Direct debugging, in assembly. At that point, why not jump down to machine code?
Probably related to this (genuinely interesting) talk given by an entropic researcher https://youtu.be/1sd26pWhfmg?si=j2AWyCfbNbOxU4MF
Anecdotally, I've been seeing a higher rate of CVEs tracked by a few dependabot projects.
Seems supported by this as well: https://www.first.org/blog/20260211-vulnerability-forecast-2...
Interesting that it's been higher than forecast since 2023. Personally I'd expect that trend to continue given that LLMs both increase bugs written as well as bugs discovered.
> I don't know how long this pace will last. I suspect that bugs are reported faster than they are written, so we could in fact be purging a long backlog
Hopefully these same tools will also help catch security bugs at the point they're written. Maybe one day we'll reach a point where the discovery of new, live vulnerabilities is extremely rare?
Around 70% of security vulnerabilities are about memory safety and only exist because software is written in C and C++. Because most vulnerabilities are in newly written code, Google has found that simply starting writing new code in Rust (rather than trying to rewrite existing codebases) quickly brings the number of found vulnerabilities down drastically.
And to a good approximation all real world Rust uses unsafe everywhere.
So we now have a new code base in an undefined language which still has memory bugs.
This is progress.
No, this is false. For Rust codebases that aren't doing high-peformance data structures, C interop, or bare-metal stuff, it's typical to write no unsafe code at all. I'm not sure who told you otherwise, but they have no idea what they're talking about.
It's interesting to hear from people directly in the thick of it that these bug reports are apparently gaining value and are no longer just slop. Maybe there is hope for a world where AI helps create bug free software and doesn't just overload maintainers.
An AI enthusiast having a breathless and predictive position on the future of the technology? No way! It's almost like Wall Street is about to sour on the whole stack and there is a concerted effort to artificially push these views into the conversation to get people on board.
Then again, I'm a known crank and aggressive cynic, but you never really see any gathered data backing these points up.
Could you back up your assertion that Willy Tarreau — who used to maintain the Linux kernel — is “an AI enthusiast”? I can’t find anything about it.
Also one of the initial creator of haproxy, a well known reverse proxy. To imply somebody like as a simple "AI shill" is just ignorant.
Anyone who says anything good about AI must be an AI shill from the start, not someone who is genuinely observing reality or had their mind changed, don't you know?
> but you never really see any gathered data backing these points up.
https://www.anthropic.com/news/mozilla-firefox-security
?
Sort of a tautology to just assert that someone saying good things about AI is an AI enthusiast and therefore their opinion should be dismissed. He also happens to have been a kernel maintainer, his experience as he's describing it should count for something.
"On the kernel security list we've seen a huge bump of reports. We were between 2 and 3 per week maybe two years ago, then reached probably 10 a week over the last year with the only difference being only AI slop, and now since the beginning of the year we're around 5-10 per day depending on the days (fridays and tuesdays seem the worst). Now most of these reports are correct, to the point that we had to bring in more maintainers to help us."
Is there a reason you’ve copy pasted the first paragraph from the link? It doesn’t add anything to the discussion, and also doesn’t help as a tl;dr because it’s literally the first paragraph. Genuine question!
Reports being written faster than bugs being created? Better quality software than before the 2000s?
Oh my sweet summer child.
This is some seriously delusional cope from someone who drank the entire jug of kool-aid.
I’d love to be proven wrong but the current trajectory is pretty plain as day from current outcomes. Everything is getting worse, and everyone is getting overwhelmed and we are under attack even more and the attacks are getting substantially more sophisticated and the blast radius is much bigger.