Before the "rewrite it in Rust" comments take over the thread:
It is worth noting that the class of bugs described here (logic errors in highly concurrent state machines, incorrect hardware assumptions) wouldn't necessarily be caught by the borrow checker. Rust is fantastic for memory safety, but it will not stop you from misunderstanding the spec of a network card or writing a race condition in unsafe logic that interacts with DMA.
That said, if we eliminated the 70% of bugs that are memory safety issues, the SNR ratio for finding these deep logic bugs would improve dramatically. We spend so much time tracing segfaults that we miss the subtle corruption bugs.
> It is worth noting that the class of bugs described here (logic errors in highly concurrent state machines, incorrect hardware assumptions)
While the bugs you describe are indeed things that aren't directly addressed by Rust's borrow checker, I think the article covers more ground than your comment implies.
For example, a significant portion (most?) of the article is simply analyzing the gathered data, like grouping bugs by subsystem:
Subsystem Bug Count Avg Lifetime
drivers/can 446 4.2 years
networking/sctp 279 4.0 years
networking/ipv4 1,661 3.6 years
usb 2,505 3.5 years
tty 1,033 3.5 years
netfilter 1,181 2.9 years
networking 6,079 2.9 years
memory 2,459 1.8 years
gpu 5,212 1.4 years
bpf 959 1.1 years
Or by type:
Bug Type Count Avg Lifetime Median
race-condition 1,188 5.1 years 2.6 years
integer-overflow 298 3.9 years 2.2 years
use-after-free 2,963 3.2 years 1.4 years
memory-leak 2,846 3.1 years 1.4 years
buffer-overflow 399 3.1 years 1.5 years
refcount 2,209 2.8 years 1.3 years
null-deref 4,931 2.2 years 0.7 years
deadlock 1,683 2.2 years 0.8 years
And the section describing common patterns for long-lived bugs (10+ years) lists the following:
> 1. Reference counting errors
> 2. Missing NULL checks after dereference
> 3. Integer overflow in size calculations
> 4. Race conditions in state machines
All of which cover more ground than listed in your comment.
Furthermore, the 19-year-old bug case study is a refcounting error not related to highly concurrent state machines or hardware assumptions.
It depends what they mean by some of these: are the state machine race conditions logic races (which Rust won’t trivially solve) or data races? If they are data races, are they the kind of ones that Rust will catch (missing atomics/synchronization) or the ones it won’t (bad atomic orderings, etc.).
It’s also worth noting that Rust doesn’t prevent integer overflow, and it doesn’t panic on it by default in release builds. Instead, the safety model assumes you’ll catch the overflowed number when you use it to index something (a constant source of bugs in unsafe code).
I’m bullish about Rust in the kernel, but it will not solve all of the kinds of race conditions you see in that kind of context.
> are the state machine race conditions logic races (which Rust won’t trivially solve) or data races? If they are data races, are they the kind of ones that Rust will catch (missing atomics/synchronization) or the ones it won’t (bad atomic orderings, etc.).
The example given looks like a generalized example:
spin_lock(&lock);
if (state == READY) {
spin_unlock(&lock);
// window here where another thread can change state
do_operation(); // assumes state is still READY
}
So I don't think you can draw strong conclusions from it.
> I’m bullish about Rust in the kernel, but it will not solve all of the kinds of race conditions you see in that kind of context.
Sure, all I'm trying to say is that "the class of bugs described here" covers more than what was listed in the parentheses.
I'd argue, that while null ref and those classes of bugs may decrease, logic errors will increase. Rust is not an extraordinary readable language in my opinion, especially in the kernel where the kernel has its own data structures. IMHO Apple did it right in their kernel stack, they have a restricted subset of C++ that you can write drivers with.
Which is also why in my opinion Zig is much more suitable, because it actually addresses the readability aspect without bring huge complexity with it.
> I'd argue, that while null ref and those classes of bugs may decrease, logic errors will increase.
To some extent that argument only makes sense; if you can find a way to greatly reduce the incidence of non-logic bugs while not addressing other bugs then of course logic bugs would make up a greater proportion of what remains.
I think it's also worth considering the fact that while Rust doesn't guarantee that it'll catch all logic bugs, it (like other languages with more "advanced" type systems) gives you tools to construct systems that can catch certain kinds of logic bugs. For example, you can write lock types in a way that guarantees at compile time that you'll take locks in the correct order, avoiding deadlocks [0]. Another example is the typestate pattern [1], which can encode state machine transitions in the type system to ensure that invalid transitions and/or operations on invalid states are caught at compile time.
These, in turn, can lead to higher-order benefits as offloading some checks to the compiler means you can devote more attention to things the compiler can't check (though to be fair this does seem to be more variable among different programmers).
> Rust is not an extraordinary readable language in my opinion, especially in the kernel where the kernel has its own data structures.
The above notwithstanding, I'd imagine it's possible to think up scenarios where Rust would make some logic bugs more visible and others less so; only time will tell which prevails in the Linux kernel, though based on what we know now I don't think there's strong support for the notion that logic bugs in Rust are a substantially more common than they have been in C, let alone because of readability issues.
Of course there's the fact that readability is very much a personal thing and is a multidimensional metric to boot (e.g., a property that makes code readable in one context may simultaneously make code less readable in another). I don't think there would be a universal answer here.
Maybe increase as a ratio, but not absolute. There are various benefits of Rust that affect other classes of issues: fancy enums, better errors, ability to control overflow behaviour and others. But for actual experience, check out what the kernel code developer has to say: https://xcancel.com/linaasahi/status/1577667445719912450
> Zig is much more suitable, because it actually addresses the readability aspect
How? It doesn't look very different from Rust. In terms of readability Swift does stand out among LLVM frontends, don't know if it is or can be used for systems programming though.
I think they are right in that claim, but in making it so, at least some of the code loses some of the readability of Swift. For truly low-level code, you’ll want to give up on classes, may not want to have copy-on-write collections, and may need to add quite a few some annotations.
Swift is very slow relative to rust or c though. You can also cause seg faults in swift with a few lines. I Don't find any of these languages particularly difficult to read, so I'm not sure why this is listed as a discriminator between them.
You can make a fully safe segfault the same way you can in go. Swapping a base reference between two child types. The data pointer and vft pointer aren't updated atomically, so a thread safety issue becomes a memory safety one.
The default Mutex struct in Rust makes it impossible to modify the data it protects without holding the lock.
"Each mutex has a type parameter which represents the data that it is protecting. The data can only be accessed through the RAII guards returned from lock and try_lock, which guarantees that the data is only ever accessed when the mutex is locked."
Even if used with more complex operations, the RAII approach means that the example you provided is much less likely to happen.
> Furthermore, the 19-year-old bug case study is a refcounting error
It always surprised me how the top-of-the line analyzers, whether commercial or OSS, never really implemented C-style reference count checking. Maybe someone out there has written something that works well, but I haven’t seen it.
This is I think an under-appreciated aspect, both for detractors and boosters. I take a lot more “risks” with Rust, in terms of not thinking deeply about “normal” memory safety and prioritizing structuring my code to make the logic more obviously correct. In C++, modeling things so that the memory safety is super-straightforward is paramount - you’ll almost never see me store a std::string_view anywhere for example. In Rust I just put &str wherever I please, if I make a mistake I’ll know when I compile.
> It is worth noting that the class of bugs described here (logic errors in highly concurrent state machines, incorrect hardware assumptions) wouldn't necessarily be caught by the borrow checker. Rust is fantastic for memory safety, but it will not stop you from misunderstanding the spec of a network card or writing a race condition in unsafe logic that interacts with DMA.
Rust is not just about memory safety. It also have algebraic data types, RAII, among other things, which will greatly help in catching this kind of silly logic bugs.
Yeah, Rust gives you much better tools to write highly concurrent state machines than C does, and most of those tools are in the type system and not the borrow checker per se. This is exactly what the Typestate pattern (https://docs.rust-embedded.org/book/static-guarantees/typest...) is good at modeling.
I’ve seen too many embedded drivers written by well known companies not use spinlocks for data shared with an ISR.
At one point, I found serious bugs (crashing our product) that had existed for over 15 years. (And that was 10 years ago).
Rust may not be perfect but it gives me hope that some classes of stupidity will be either be avoided or made visible (like every function being unsafe because the author was a complete idiot).
The concurrent state machine example looks like a locking error? If the assumption is that it shouldn't change in the meantime, doesn't it mean the lock should continue to be held? In that case rust locks can help, because they can embed the data, which means you can't even touch it if it's not held.
Rust has more features than just the borrow checker. For example, it has a a more featured type system than C or C++, which a good developer can use to detect some logic mistakes at compile time. This doesn't eliminate bugs, but it can catch some very early.
> But unsafe Rust, which is generally more often used in low-level code, is more difficult than C and C++.
I think "is" is a bit too strong. "Can be", sure, but I'm rather skeptical that all uses of unsafe Rust will be more difficult than writing equivalent C/C++ code.
People who make that kind of remarks should be called out and shunned. The Rust community is tired of discrimination and being the butt of jokes. All the other inferior languages prey on its minority status, despite Rust being able to solve all their problems. I take offense to these remarks, I don't want my kids to grow up as Rustaceans in such a caustic society.
> It’s hilarious that you feel the need to preemptively take control of the narrative in anticipation of the Rust people that you fear so much.
> Is this an irrational fear, I wonder? Reminds me of methods used in the political discourse.
In a sad sort of way, I think its hilarious that hn users have been so completely conditioned to expect rust evangelism any time a topic like this comes up that they wanted to get ahead of it.
Not sure who it says more about, but it sure does say a whole lot.
> race condition in unsafe logic that interacts with DMA
It's worth noting that if you write memory safe code but mis-program a DMA transfer, or trigger a bug in a PCIe device, it's possible for the hardware to give you memory-safety problems by splatting invalid data over a region that's supposed to contain something else.
Using the data provided, memory safety issues (use-after-free, memory-leak, buffer-overflow, null-deref) account for 67% of their bugs. If we include refcount It is just over 80%.
Eh... Removing concurrence bugs is one of the main selling points for Rust. And algebraic types are a really boost for situations where you have lots of assumptions.
Yes, I saw this last night and was confused because only one comment mentioned Rust, and it was deleted I think. I nearly replied "you're about to prompt 1,000 rust replies with this" and here's what I woke up to lol
Expensive because of: 1/ a re-write is never easy 2/ rust is specifically tough (because it catches error and forces you to think about it for real, because it makes some contruct (linked list) really hard to implement) for kernel/close to kernel code ?
Both I'd say. Rust imposes more constraints on the structure of code than most languages. The borrow checker really likes ownership trees whereas most languages allow any ownership graph no matter how spaghetti it is.
As far as I know that's why Microsoft rewrote Typescript in Go instead of Rust.
Thanks for raising this. It feels like evangelists paint a picture of Rust basically being magic which squashes all bugs. My personal experience is rather different. When I gave Rust a whirl a few years ago, I happened to play with mio for some reason I can't remember yet. Had some basic PoC code which didn't work as expected. So while not being a Rust expert, I am still too much fan of the scratch your own itch philosophy, so I started to read the mio source code. And after 5 minutes, I found the logic bug. Submitted a PR and moved on. But what stayed with me was this insight that if someone like me can casually find and fix a Rust library bug, propaganda is probably doing more work then expected. The Rust craze feels a bit like Java. Just because a language baby-sits the developer doesn't automatically mean better quality. At the end of the day, the dev needs to juggle the development process. Sure, tools are useful, but overstating safety is likely a route better avoided.
Interesting! We did a similar analysis on Content Security Policy bugs in Chrome and Firefox some time ago, where the average bug-to-report time was around 3 years and 1 year, respectively.
https://www.usenix.org/conference/usenixsecurity23/presentat...
Our bug dataset was way smaller, though, as we had to pinpoint all bug introductions unfortunately. It's nice to see the Linux project uses proper "Fixes: " tags.
Is the intention of the author to use the number of years bugs stay "hidden" as a metric of the quality of the kernel codebase or of the performance of the maintainers? I am asking because at some point the articles says "We're getting faster".
IMHO a fact that a bug hides for years can also be indication that such bug had low severity/low priority and therefore that the overall quality is very good. Unless the time represents how long it takes to reproduce and resolve a known bug, but in such case I would not say that "bug hides" in the kernel.
> IMHO a fact that a bug hides for years can also be indication that such bug had low severity/low priority
Not really true. A lot of very severe bugs have lurked for years and even decades. Heartbleed comes to mind.
The reason these bugs often lurk for so long is because they very often don't cause a panic, which is why they can be really tricky to find.
For example, use after free bugs are really dangerous. However, in most code, it's a pretty safe bet that nothing dangerous happens when use after free is triggered. Especially if the pointer is used shortly after the free and dies shortly after it. In many cases, the erroneous read or write doesn't break something.
The same is true of the race condition problems (which are some of the longest lived bugs). In a lot of cases, you won't know you have a race condition because in many cases the contention on the lock is low so the race isn't exposed. And even when it is, it can be very tricky to reproduce as the race isn't likely to be done the same way twice.
> …lurked for years and even decades. Heartbleed comes to mind.
I don’t know much about Heartbleed, but Wikipedia says:
> Heartbleed is a security bug… It was introduced into the software in 2012 and publicly disclosed in April 2014.
Two years doesn’t sound like “years or even decades” to me? But again, I don’t know much about Heartbleed so I may be missing something. It does say it was also patched in 2014, not just discovered then.
This may just be me misremembering, but as I recall, the bug of Heartbleed was ultimately a very complex macro system which supported multiple very old architectures. The bug, IIRC, was the interaction between that old macro system and the new code which is what made it hard to recognize as a bug.
Part of the resolution to the problem was I believe they ended up removing a fair number of unsupported platforms. It also ended up spawning alternatives to openssl like boring ssl which tried to remove as much as possible to guard against this very bug.
> IMHO a fact that a bug hides for years can also be indication that such bug had low severity/low priority and therefore that the overall quality is very good.
It doesn't seem to indicate that. It indicates the bug just isn't in tested code or isn't reached often. It could still be a very severe bug.
The issue with longer lived bugs is that someone could have been leveraging it for longer.
But it matters for detection time, because there's a lot more "normal" use of any given piece of code than intentional attempts to break it. If a bug can't be triggered unintentionally it'll never get detected through normal use, which can lead to it staying hidden for longer.
The state machine race pattern resonates beyond kernel work. I've seen similar bugs hide for years in application code - transaction state edge cases that only trigger under specific sequences of user actions that nobody tests for.
The median lifetimes are fascinating. Race conditions at 5.1 years vs null-deref at 2.2 years makes intuitive sense - the former needs specific timing to manifest, while the latter will crash obviously once you hit the code path. The ones that need rare conditions to trigger are the ones that survive longest.
Yea, it's pretty common. We had a customer years ago that was having a rare and random application crash under load. Never could figure out where it was from. Quite some time later a batch load interface was added to the app and with the rate things were input with it the crash could be triggered reliably.
It's something else that's added/changed in the application that eventually makes the bug stand out.
One of the iOS 26 Core Audio bug (CVE-2025-31200) is about synchronizing two different arrays with each other and the assumption mistakes that were made trusting dimensional information which could be coming from the user.
It may be just my system, but the times look like hyperlinks but aren't for some reason. It is especially disappointing that the commit hashes don't link to the actual commit in the kernel repo.
They're <strong> tags with color:#79635c on hover in the CSS. A really weird style choice for sure, but semantically they aren't meant to be links at all.
Why? A sample size of 28% is positively huge compared to what most statistical studies have to work with. The accuracy of an extrapolation is mostly determined by underlying sampling bias, not the amount of data. If you have any basis to suggest that capturing "only bugs with fixes tags" creates a skewed sample, that would be grounds to distrust the extrapolation, but simply claiming "it's only 28%" does not make it worth noting.
This implies (or states, hard to say) that they don't upstream specifically in order to profit. That is nonsense.
1. Tons of bugs are reported upstream by grsecurity historically.
2. Tons of critical security mitigations in the kernel were outright invented by that team. ASLR, SMAP, SMEP, NX, etc.
3. They were completely FOSS until very recently.
4. They have always maintained that they are entirely willing to upstream patches but that it's a lot of work and would require funding. Upstream has always been extremely hostile towards attempts to take small pieces of Grsecurity and upstream them.
Profiting from selling their patchset is not the whole story, though. grsec was public and free for a long time and there were many effects at play preventing the kernel from adopting it.
One of my favorite Firefox bugs was some I don’t quite remember the details of, but went something like this:
“There’s a crash while using this config file.” Something more complex than that, but ultimately a crash of some kind.
Years later, like 20 years later, the bug was closed. You see, they re-wrote the config parser in Rust, and now this is fixed.”
That’s cool but it’s not the part I remember. The part I always think about is, imagine responding to the bug right after it was opened with “sorry, we need to go off and write our own programming language before this bug is fixed. Don’t worry, we’ll be back, it’s just gonna take some time.”
Nobody would believe you. But yet, it’s what happened.
I don't think the problem is the kernel. Kernel bugs stay hidden because no one runs recent Kernels.
My Pixel 8 runs kernel a stable minor from 6.1, which was released more than 4 years ago. Yes, fixes get backported to it, but the new features in 6.2->6.19 stay unused on that hardware. All the major distros suffer from the same problem, most people are not running them in production
Most hyperscalers are running old kernel versions on which they do backports. If you go Linux conferences you hear folks from big companies mentioning 4.xx, 3.xx kernels, in 2025.
Might be obviously, but there is definitely a lot of biases in the data here. It's unavoidable. E.g. many bugs will not be detected, but they will be removed when the code is rewritten. So code that is refactored more often will have lower age of fixed bugs. Components/subsystems that are heavily used will detect bugs faster. Some subsystems by their very nature can tolerate bugs more, while some by necessity will need to be more correct (like bpf).
I do have to somewhat trust Xen, but Qubes' isolation relies on hardware virtualization (VT-d), which statistically has much less security issues than Xen itself. Most Xen advisories do not affect Qubes: https://www.qubes-os.org/security/xsa/
Only tangentially related but maybe someone here can help me.
I have a server which has many peripherals and multiple GPUs. Now, I can use vfio and vfio-pcio to memory map and access their registers in user space. My question is, how could I start with kernel driver development? And I specifically mean the dev setup.
Would it be a good idea to use vfio with or without a vm to write and test drivers? How to best debug, reload and test changing some code of an existing driver?
It's an easter egg on the website that usually goes unnoticed. It's our first time on the front page of HN, so it's a little overutilized right now. Capital-C clears it.
This is fascinating stuff, especially the per-subsystem data. I've worked with CAN in several different professional and amateur settings, I'm not surprised to see it near the bottom of this list. That's not a dig against the kernel or the folks who work on it... more of a heavy sigh about the state of the industries that use CAN.
On a related note, I'm seeing a correlation between "level of hoopla" and a "level of attention/maintenance." While it's hard to distinguish that correlation from "level of use," the fact that CAN is so far down the list suggests to me that hoopla matters; it's everywhere but nobody talks about it. If a kernel bug takes down someone's datacenter, boy are we gonna hear about it. But if a kernel bug makes a DeviceNet widget freak out in a factory somewhere? Probably not going to make the front page of HN, let alone CNN.
There is a general rule on bugs is that the more devices they are on, the more apt they are to trigger.
A CAN with 10,000 machines total and relatively fixed applications is either going to trigger the bug right off the bat and then work around it, or trigger the bug so rarely it won't be recognized as a kernel issue.
General purpose systems running millions and millions of units with different workloads are an evolutionary breeding ground for finding bugs and exploits.
The lesson here is that people have an unrealistic view of how complex it is to write correct and safe multithreaded code on multi-core, multi-thread, assymmetric core, out-of-order processors. This is no shade to kernel developers. Rather, I direct this at people who seem to you can just create a thread pool in C++ and solve all your concurrency problems.
One criticism of Rust (and, no, I'm not saying "rewrite it in Rust", to be clear) is that the borrow checker can be hard to use whereas many C++ engineers (in particular, for some reason) seem to argue that it's easier to write in C++. I have two things to say about that:
1. It's not easier in C++. Nothing is. C++ simply allows you to make mistakes without telling you. GEtting things correct in C++ is just as difficult as any other language if not more so due to the language complexity; and
2. The Rust borrow checker isn't hard or difficult to use. What you're doing is hard and difficult to do correctly.
This is I favor cooperative multitasking and using battle-tested concurrency abstractions whenever possible. For example the cooperative async-await of Hack and the model of a single thread responding to a request then discarding everything in PHP/Hack is virtually ideal (IMHO) for serving Web traffic.
I remember reading about Google's work on various C++ tooling including valgrind and that they exposed concurrency bugs in their own code that had lain dormant for up to a decade. That's Google with thousands of engineers and some very talented engineers at that.
> The Rust borrow checker isn't hard or difficult to use. What you're doing is hard and difficult to do correctly.
There are entire classes of structures that no, aren't hard to do properly, but the borrow checker makes artificially hard due to design limitations that are known to be sub-optimal.
No, two-directional linked lists and partially editable data structures aren't inherently hard. It's a Rust limitation that a piece of code can't take enough ownership of them to edit they safely.
The implementations of sort in Rust are filled with unsafe.[0]
Another example is that of doubly linked lists.[1] It is possible to implement a doubly linked list correctly in C++ without much trouble. In Rust, it can be significantly more challenging.
In C++, pointers are allowed to alias if their types, roughly said, are compatible. In Rust, there are stricter rules, and getting those rules wrong in an unsafe block, or code outside unsafe blocks that code inside unsafe blocks rely on, will result in breakage of memory safety.
This has been discussed by others.[2]
Based on that, do you agree that there are algorithms and data structures that are significantly easier to implement efficiently and correctly in C++ than in Rust? And thus that you are being completely wrong in your claim?
> The implementations of sort in Rust are filled with unsafe.
Strictly speaking, the mere presence of `unsafe` says nothing on its own about whether "it" is easier in C++. Not only does `unsafe` on its own say nothing about the "difficulty" of the code it contains, but that is just one factor of one side of a comparison - very much insufficient for a complete conclusion.
Furthermore, "just" writing a sorting algorithm is pretty straightforwards both in Rust and C++; it's the more interesting properties that tend to make for equally interesting implementations, and one would need to procure Rust and C++ implementations with equivalent properties, preferably from the same author(s), for a proper comparison.
Past research has shown that Rust's current sorting algorithms have different properties than C++ implementations from the time (e.g., the "X safety" results in [0]), so if nothing substantial has changed since then there's going to be some work to do for a proper comparison.
Also, the Linux kernel developers turned off strict aliasing in the C compilers they use, because they found strict aliasing too difficult. Yet unsafe Rust has more difficult rules than even strict aliasing in C and C++.
> Also, the Linux kernel developers turned off strict aliasing in the C compilers they use, because they found strict aliasing too difficult.
I'm not sure "they found strict aliasing too difficult" is an entirely correct characterization? From this rather (in)famous email from Linus [0]:
The fact is, using a union to do type punning is the traditional AND
STANDARD way to do type punning in gcc. In fact, it is the
*documented* way to do it for gcc, when you are a f*cking moron and
use "-fstrict-aliasing" and need to undo the braindamage that that
piece of garbage C standard imposes.
[snip]
This is why we use -fwrapv, -fno-strict-aliasing etc. The standard
simply is not *important*, when it is in direct conflict with reality
and reliable code generation.
The *fact* is that gcc documents type punning through unions as the
"right way". You may disagree with that, but putting some theoretical
standards language over the *explicit* and long-time documentation of
the main compiler we use is pure and utter bullshit.
From the stats we see that most bugs effectively come from the limitations of the language.
Impressive results on the model, I'm surprised they improved it with very simple heuristics. Hopefully this tool will be made available to the kernel developers and integrated to the workflow.
My conclusion is that microkernels offer some protection from random reboots, but not much against hacking
Say the USB system runs in its own isolated process. Great, but if someone pwns the USB process they can change disk contents, intercept and inject keystrokes, etc. You can usually leverage that into a whole system compromise.
Same with most subsystems: GPU, network, file system process compromises are all easily leveraged to pwn the whole system.
Of course by now processor manufacturers decided that blowing holes into the CPUs security model to make it go faster was the way to go. So your micro kernel is stuck on a hardware security model that looks like swiss cheese and smells like Surströmming.
Mach is not a very good microkernel at all, because the overhead is much higher than necessary. The L4 family’s IPC design is substantially more efficient, and that’s why they’re used in actual systems. Fuchsia/Zircon have improved on the model further.
Someone will of course bring up XNU, but the microkernel aspect of it died when they smashed the FreeBSD kernel into the codebase. DriverKit has brought some userspace drivers back, but they use shared memory for all the heavy lifting.
> Mach is not a very good microkernel at all, because the overhead is much higher than necessary. The L4 family’s IPC design is substantially more efficient, and that’s why they’re used in actual systems.
As opposed to Mach, which is not used in any actual systems
I mentioned XNU below. It doesn’t really count as a microkernel if you, you know, don’t actually use the microkernel part. At least for the 30 years between the FreeBSD collision and the introduction of DriverKit, which does most of its IPC through shared memory (because the mach ports are not efficient enough, I would assume).
Windows NT 3.x was a true microkernel. Microsoft ruined it but the design was quite good and the driver question was irrelevant, until they sidestepped HAL.
You’re saying they improved the design. I know they added user-privilege device driver support for USB (etc).; did they revert the display compromise/mess as well?
Mind you I don't have access to Microsoft code, so this is all indirect, and a lot of this knowledge was when I was fledgling developer.
The Windows NT code was engineered to be portable across many different architectures--not just X86--so it has a hardware abstraction layer. The kernel only ever communicated to the device-driver implementation through this abstraction layer; so the kernel code itself was isolated.
That doesn't mean the device drivers were running in user-land privilege, but it does mean that the kernel code is quite stable and easy to reason about.
When Microsoft decided to compromise on this design, I remember senior engineers--when I first started my career--being abuzz about it for Windows NT 4.0 (or apparently earlier?).
I think NTFS get a bit of crap from the OS above it adding limitations. If you read up on what NTFS allows, it is far better than what Windows and the explorer allows you to do with it.
NTFS is a beast of a filesystem and has been nothing but solid for 25+ years. The performance grievances ignore the warranties that NTFS offers vs many antiquated POSIX filesystems.
- mandatory byte-range locks enforced by the kernel
- explicit sharing modes
- guarantees around write ordering and durability
- safe delete-on-close
- first-class cache coherency contracts for networked access
POSIX aims for portability while NTFS/Win32 aims for explicit contracts and enforced behavior. For apps assuming POSIX semantics (e.g. git) NTFS feels rigid and weird. Coming the other way from NTFS, POSIX looks "optimistic" if not downright sloppy.
Of course ZFS et al. are more theoretically more robust than EXT4 but are still limited by the lowest common denominator POSIX API. Maybe you can detect that you're dealing with a ZFS backed volume and use extra ioctls to improve things but its still a messy business.
These are pretty much all about mandatory locking. Which giveth and taketh away in my experience. I’ve had substantially fewer weird file handling bugs in my Linux code than my Windows code. POSIX is very loosey-goosey in general, but Linux’s VFS system + ext4 has a much stronger model than the general POSIX guarantees.
`FILE_FLAG_DELETE_ON_CLOSE`’s equivalent on Posix is just rm. Windows doesn’t let you open new handles to `FILE_FLAG_DELETE_ON_CLOSE`ed files anyway, so it’s effectively the same. The inode will get deleted when the last file description is removed.
NFS is a disaster though, I’ll give you that one. Though mandatory locks on SMB shares hanging is also very aggravating in its own right.
Also to point out that outside UNIX, surviving mainframe and micros, the filesystems are closer to NTFS than UNIX world, in regards to what is enforced.
There is also the note that some of them are more database like, than classical filesystems.
Ah, and modern Windows also has Resilient File System (ReFS), which is used on Dev Drives.
North Korea is called a “democratic people's republic”. Just because one thing that really isn't <whatever> is called <whatever> by the people in coontrol of it, doesn't mean that it is or that incorrectly calling other things <whatever> is correct.
Speaking of nasty kernel bugs although on another platform, there's a nasty one in either Microsoft's Win 11 nwifi.sys handling of deadlock conditions or Qualcomm's QCNCM865 FastConnect 7800 WCN785x driver that panics because of a watchdog failure in nwifi!MP6SendNBLInternal+0x4b guarded by a deadlocked ndis!NdisAcquireRWLockRead+0x8b. It "BSODs" the system rather than doing something sane like dropping a packet or retransmitting.
Am I the only unreasonable maniac who wants a very long-term stable, seL4-like capability-based, ubiquitous, formally-verified μkernel that rarely/never crashes completely* because drivers are just partially-elevated programs sprinkled with transaction guards and rollback code for critical multiple resource access coordination patterns? (I miss hacking on MINIX 2.)
* And never need to reboot or interrupt server/user desktop activities because the core μkernel basically never changes since it's tiny and proven correct.
Before the "rewrite it in Rust" comments take over the thread:
It is worth noting that the class of bugs described here (logic errors in highly concurrent state machines, incorrect hardware assumptions) wouldn't necessarily be caught by the borrow checker. Rust is fantastic for memory safety, but it will not stop you from misunderstanding the spec of a network card or writing a race condition in unsafe logic that interacts with DMA.
That said, if we eliminated the 70% of bugs that are memory safety issues, the SNR ratio for finding these deep logic bugs would improve dramatically. We spend so much time tracing segfaults that we miss the subtle corruption bugs.
> It is worth noting that the class of bugs described here (logic errors in highly concurrent state machines, incorrect hardware assumptions)
While the bugs you describe are indeed things that aren't directly addressed by Rust's borrow checker, I think the article covers more ground than your comment implies.
For example, a significant portion (most?) of the article is simply analyzing the gathered data, like grouping bugs by subsystem:
Or by type: And the section describing common patterns for long-lived bugs (10+ years) lists the following:> 1. Reference counting errors
> 2. Missing NULL checks after dereference
> 3. Integer overflow in size calculations
> 4. Race conditions in state machines
All of which cover more ground than listed in your comment.
Furthermore, the 19-year-old bug case study is a refcounting error not related to highly concurrent state machines or hardware assumptions.
It depends what they mean by some of these: are the state machine race conditions logic races (which Rust won’t trivially solve) or data races? If they are data races, are they the kind of ones that Rust will catch (missing atomics/synchronization) or the ones it won’t (bad atomic orderings, etc.).
It’s also worth noting that Rust doesn’t prevent integer overflow, and it doesn’t panic on it by default in release builds. Instead, the safety model assumes you’ll catch the overflowed number when you use it to index something (a constant source of bugs in unsafe code).
I’m bullish about Rust in the kernel, but it will not solve all of the kinds of race conditions you see in that kind of context.
> are the state machine race conditions logic races (which Rust won’t trivially solve) or data races? If they are data races, are they the kind of ones that Rust will catch (missing atomics/synchronization) or the ones it won’t (bad atomic orderings, etc.).
The example given looks like a generalized example:
So I don't think you can draw strong conclusions from it.> I’m bullish about Rust in the kernel, but it will not solve all of the kinds of race conditions you see in that kind of context.
Sure, all I'm trying to say is that "the class of bugs described here" covers more than what was listed in the parentheses.
I'd argue, that while null ref and those classes of bugs may decrease, logic errors will increase. Rust is not an extraordinary readable language in my opinion, especially in the kernel where the kernel has its own data structures. IMHO Apple did it right in their kernel stack, they have a restricted subset of C++ that you can write drivers with.
Which is also why in my opinion Zig is much more suitable, because it actually addresses the readability aspect without bring huge complexity with it.
> I'd argue, that while null ref and those classes of bugs may decrease, logic errors will increase.
To some extent that argument only makes sense; if you can find a way to greatly reduce the incidence of non-logic bugs while not addressing other bugs then of course logic bugs would make up a greater proportion of what remains.
I think it's also worth considering the fact that while Rust doesn't guarantee that it'll catch all logic bugs, it (like other languages with more "advanced" type systems) gives you tools to construct systems that can catch certain kinds of logic bugs. For example, you can write lock types in a way that guarantees at compile time that you'll take locks in the correct order, avoiding deadlocks [0]. Another example is the typestate pattern [1], which can encode state machine transitions in the type system to ensure that invalid transitions and/or operations on invalid states are caught at compile time.
These, in turn, can lead to higher-order benefits as offloading some checks to the compiler means you can devote more attention to things the compiler can't check (though to be fair this does seem to be more variable among different programmers).
> Rust is not an extraordinary readable language in my opinion, especially in the kernel where the kernel has its own data structures.
The above notwithstanding, I'd imagine it's possible to think up scenarios where Rust would make some logic bugs more visible and others less so; only time will tell which prevails in the Linux kernel, though based on what we know now I don't think there's strong support for the notion that logic bugs in Rust are a substantially more common than they have been in C, let alone because of readability issues.
Of course there's the fact that readability is very much a personal thing and is a multidimensional metric to boot (e.g., a property that makes code readable in one context may simultaneously make code less readable in another). I don't think there would be a universal answer here.
[0]: https://lwn.net/Articles/995814/
[1]: https://cliffle.com/blog/rust-typestate/
Maybe increase as a ratio, but not absolute. There are various benefits of Rust that affect other classes of issues: fancy enums, better errors, ability to control overflow behaviour and others. But for actual experience, check out what the kernel code developer has to say: https://xcancel.com/linaasahi/status/1577667445719912450
> Zig is much more suitable, because it actually addresses the readability aspect
How? It doesn't look very different from Rust. In terms of readability Swift does stand out among LLVM frontends, don't know if it is or can be used for systems programming though.
Apple claims Swift can be used for systems programming, and is (partly) eating its own dogfood by using it in FoundationDB (https://news.ycombinator.com/item?id=38444876) and by providing examples of embedded projects (https://www.swift.org/get-started/embedded/)
I think they are right in that claim, but in making it so, at least some of the code loses some of the readability of Swift. For truly low-level code, you’ll want to give up on classes, may not want to have copy-on-write collections, and may need to add quite a few some annotations.
Swift is very slow relative to rust or c though. You can also cause seg faults in swift with a few lines. I Don't find any of these languages particularly difficult to read, so I'm not sure why this is listed as a discriminator between them.
But those segfaults will either be memory memory safe or your lines will contain “unsafe” or “unchecked” somewhere.
You can make a fully safe segfault the same way you can in go. Swapping a base reference between two child types. The data pointer and vft pointer aren't updated atomically, so a thread safety issue becomes a memory safety one.
I would argue logic errors would decrease because you aren't spending as much time worrying about and fixing null ref and other errors.
can you prove that?
Rust is a lot more explicit. I suspect logic bugs will be much less common. It's far easier to model complexity in Rust.
I would expect the opposite. C requires you to deal with extreme design complexity in large systems because the language offers nothing to help.
The default Mutex struct in Rust makes it impossible to modify the data it protects without holding the lock.
"Each mutex has a type parameter which represents the data that it is protecting. The data can only be accessed through the RAII guards returned from lock and try_lock, which guarantees that the data is only ever accessed when the mutex is locked."
Even if used with more complex operations, the RAII approach means that the example you provided is much less likely to happen.
I don’t think that the parent comment is saying all of the bugs would have been prevented by using Rust.
But in the listed categories, I’m equally skeptical that none of them would have benefited from Rust even a bit.
That’s not my point - just that “state machine races” is a too-broad category to say much about how Rust would or wouldn’t help.
Why doesn't it surprise me that the CAN bus driver bugs have the longest average lifetime?
> Furthermore, the 19-year-old bug case study is a refcounting error
It always surprised me how the top-of-the line analyzers, whether commercial or OSS, never really implemented C-style reference count checking. Maybe someone out there has written something that works well, but I haven’t seen it.
This is I think an under-appreciated aspect, both for detractors and boosters. I take a lot more “risks” with Rust, in terms of not thinking deeply about “normal” memory safety and prioritizing structuring my code to make the logic more obviously correct. In C++, modeling things so that the memory safety is super-straightforward is paramount - you’ll almost never see me store a std::string_view anywhere for example. In Rust I just put &str wherever I please, if I make a mistake I’ll know when I compile.
> It is worth noting that the class of bugs described here (logic errors in highly concurrent state machines, incorrect hardware assumptions) wouldn't necessarily be caught by the borrow checker. Rust is fantastic for memory safety, but it will not stop you from misunderstanding the spec of a network card or writing a race condition in unsafe logic that interacts with DMA.
Rust is not just about memory safety. It also have algebraic data types, RAII, among other things, which will greatly help in catching this kind of silly logic bugs.
Yeah, Rust gives you much better tools to write highly concurrent state machines than C does, and most of those tools are in the type system and not the borrow checker per se. This is exactly what the Typestate pattern (https://docs.rust-embedded.org/book/static-guarantees/typest...) is good at modeling.
I’ve seen too many embedded drivers written by well known companies not use spinlocks for data shared with an ISR.
At one point, I found serious bugs (crashing our product) that had existed for over 15 years. (And that was 10 years ago).
Rust may not be perfect but it gives me hope that some classes of stupidity will be either be avoided or made visible (like every function being unsafe because the author was a complete idiot).
The concurrent state machine example looks like a locking error? If the assumption is that it shouldn't change in the meantime, doesn't it mean the lock should continue to be held? In that case rust locks can help, because they can embed the data, which means you can't even touch it if it's not held.
Rust has more features than just the borrow checker. For example, it has a a more featured type system than C or C++, which a good developer can use to detect some logic mistakes at compile time. This doesn't eliminate bugs, but it can catch some very early.
But unsafe Rust, which is generally more often used in low-level code, is more difficult than C and C++.
https://rust-unofficial.github.io/too-many-lists/
https://chadaustin.me/2024/10/intrusive-linked-list-in-rust/
> But unsafe Rust, which is generally more often used in low-level code, is more difficult than C and C++.
I think "is" is a bit too strong. "Can be", sure, but I'm rather skeptical that all uses of unsafe Rust will be more difficult than writing equivalent C/C++ code.
It’s hilarious that you feel the need to preemptively take control of the narrative in anticipation of the Rust people that you fear so much.
Is this an irrational fear, I wonder? Reminds me of methods used in the political discourse.
People who make that kind of remarks should be called out and shunned. The Rust community is tired of discrimination and being the butt of jokes. All the other inferior languages prey on its minority status, despite Rust being able to solve all their problems. I take offense to these remarks, I don't want my kids to grow up as Rustaceans in such a caustic society.
> It’s hilarious that you feel the need to preemptively take control of the narrative in anticipation of the Rust people that you fear so much.
> Is this an irrational fear, I wonder? Reminds me of methods used in the political discourse.
In a sad sort of way, I think its hilarious that hn users have been so completely conditioned to expect rust evangelism any time a topic like this comes up that they wanted to get ahead of it.
Not sure who it says more about, but it sure does say a whole lot.
I don’t think evangelism is necessary anymore. Rust adoption is now a matter of time.
> race condition in unsafe logic that interacts with DMA
It's worth noting that if you write memory safe code but mis-program a DMA transfer, or trigger a bug in a PCIe device, it's possible for the hardware to give you memory-safety problems by splatting invalid data over a region that's supposed to contain something else.
I don't think 70% of bugs are memory safety issues.
In my experience it's closer to 5%.
I believe this is where that fact comes from [1]
Basically, 70% of high severity bugs are memory safety.
[1] https://www.chromium.org/Home/chromium-security/memory-safet...
High severity security issues.
Right, which is a measure which is heavily biased towards memory safety bugs.
Using the data provided, memory safety issues (use-after-free, memory-leak, buffer-overflow, null-deref) account for 67% of their bugs. If we include refcount It is just over 80%.
70% of security vulnerabilities are due to memory safety. Not all bugs.
That's the figure that Microsoft and Google found in their code bases.
probably quite a bit less than 5%, however, they tend to be quite serious when they happen
Only serious if you care about protecting from malicious actors running code on the same host.
you dont? I would imagine people that runs for example a browser would have quite an interest in that
Eh... Removing concurrence bugs is one of the main selling points for Rust. And algebraic types are a really boost for situations where you have lots of assumptions.
You're fighting air
No other top-level comments have since mentioned Rust[1] and TFA mentions neither Rust nor topics like memory safety. It’s just plain bugs.
The Rust phantom zealotry is unfortunately real.
[1] Aha, but the chilling effect of dismissing RIR comments before they are even posted...
Yes, I saw this last night and was confused because only one comment mentioned Rust, and it was deleted I think. I nearly replied "you're about to prompt 1,000 rust replies with this" and here's what I woke up to lol
Rust has other features that help prevent logic errors. It's not just C plus a borrow checker.
Rust would prevent a number of bugs, as it can model state machine guarantees as well.
Rewriting it all in Rust is extremely expensive, so it won't be done (soon).
Expensive because of: 1/ a re-write is never easy 2/ rust is specifically tough (because it catches error and forces you to think about it for real, because it makes some contruct (linked list) really hard to implement) for kernel/close to kernel code ?
Both I'd say. Rust imposes more constraints on the structure of code than most languages. The borrow checker really likes ownership trees whereas most languages allow any ownership graph no matter how spaghetti it is.
As far as I know that's why Microsoft rewrote Typescript in Go instead of Rust.
Thanks for raising this. It feels like evangelists paint a picture of Rust basically being magic which squashes all bugs. My personal experience is rather different. When I gave Rust a whirl a few years ago, I happened to play with mio for some reason I can't remember yet. Had some basic PoC code which didn't work as expected. So while not being a Rust expert, I am still too much fan of the scratch your own itch philosophy, so I started to read the mio source code. And after 5 minutes, I found the logic bug. Submitted a PR and moved on. But what stayed with me was this insight that if someone like me can casually find and fix a Rust library bug, propaganda is probably doing more work then expected. The Rust craze feels a bit like Java. Just because a language baby-sits the developer doesn't automatically mean better quality. At the end of the day, the dev needs to juggle the development process. Sure, tools are useful, but overstating safety is likely a route better avoided.
Interesting! We did a similar analysis on Content Security Policy bugs in Chrome and Firefox some time ago, where the average bug-to-report time was around 3 years and 1 year, respectively. https://www.usenix.org/conference/usenixsecurity23/presentat...
Our bug dataset was way smaller, though, as we had to pinpoint all bug introductions unfortunately. It's nice to see the Linux project uses proper "Fixes: " tags.
> It's nice to see the Linux project uses proper "Fixes: " tags.
Sort of. They often don't.
Is the intention of the author to use the number of years bugs stay "hidden" as a metric of the quality of the kernel codebase or of the performance of the maintainers? I am asking because at some point the articles says "We're getting faster".
IMHO a fact that a bug hides for years can also be indication that such bug had low severity/low priority and therefore that the overall quality is very good. Unless the time represents how long it takes to reproduce and resolve a known bug, but in such case I would not say that "bug hides" in the kernel.
> IMHO a fact that a bug hides for years can also be indication that such bug had low severity/low priority
Not really true. A lot of very severe bugs have lurked for years and even decades. Heartbleed comes to mind.
The reason these bugs often lurk for so long is because they very often don't cause a panic, which is why they can be really tricky to find.
For example, use after free bugs are really dangerous. However, in most code, it's a pretty safe bet that nothing dangerous happens when use after free is triggered. Especially if the pointer is used shortly after the free and dies shortly after it. In many cases, the erroneous read or write doesn't break something.
The same is true of the race condition problems (which are some of the longest lived bugs). In a lot of cases, you won't know you have a race condition because in many cases the contention on the lock is low so the race isn't exposed. And even when it is, it can be very tricky to reproduce as the race isn't likely to be done the same way twice.
> …lurked for years and even decades. Heartbleed comes to mind.
I don’t know much about Heartbleed, but Wikipedia says:
> Heartbleed is a security bug… It was introduced into the software in 2012 and publicly disclosed in April 2014.
Two years doesn’t sound like “years or even decades” to me? But again, I don’t know much about Heartbleed so I may be missing something. It does say it was also patched in 2014, not just discovered then.
This may just be me misremembering, but as I recall, the bug of Heartbleed was ultimately a very complex macro system which supported multiple very old architectures. The bug, IIRC, was the interaction between that old macro system and the new code which is what made it hard to recognize as a bug.
Part of the resolution to the problem was I believe they ended up removing a fair number of unsupported platforms. It also ended up spawning alternatives to openssl like boring ssl which tried to remove as much as possible to guard against this very bug.
Maybe you are thinking of ShellShock
https://en.wikipedia.org/wiki/Shellshock_(software_bug)
The bug was introduced into the code in 1989, and only found and exploited in 2014.
> IMHO a fact that a bug hides for years can also be indication that such bug had low severity/low priority and therefore that the overall quality is very good.
It doesn't seem to indicate that. It indicates the bug just isn't in tested code or isn't reached often. It could still be a very severe bug.
The issue with longer lived bugs is that someone could have been leveraging it for longer.
Worst case is that it doesn't even cause correctness issues in normal use, only when misused in a way that is unlikely to happen unintentionally.
I guess because I work in security the "unintentionally" doesn't matter much to me.
But it matters for detection time, because there's a lot more "normal" use of any given piece of code than intentional attempts to break it. If a bug can't be triggered unintentionally it'll never get detected through normal use, which can lead to it staying hidden for longer.
That's not really contested? The statement was that longer detection time indicates lower severity.
The state machine race pattern resonates beyond kernel work. I've seen similar bugs hide for years in application code - transaction state edge cases that only trigger under specific sequences of user actions that nobody tests for.
The median lifetimes are fascinating. Race conditions at 5.1 years vs null-deref at 2.2 years makes intuitive sense - the former needs specific timing to manifest, while the latter will crash obviously once you hit the code path. The ones that need rare conditions to trigger are the ones that survive longest.
>hide for years in application code
Yea, it's pretty common. We had a customer years ago that was having a rare and random application crash under load. Never could figure out where it was from. Quite some time later a batch load interface was added to the app and with the rate things were input with it the crash could be triggered reliably.
It's something else that's added/changed in the application that eventually makes the bug stand out.
One of the iOS 26 Core Audio bug (CVE-2025-31200) is about synchronizing two different arrays with each other and the assumption mistakes that were made trusting dimensional information which could be coming from the user.
https://youtu.be/nTO3TRBW00E
Thank goodness for reader mode. The transparent background where the text is with the wiggly line background is... challenging.
It may be just my system, but the times look like hyperlinks but aren't for some reason. It is especially disappointing that the commit hashes don't link to the actual commit in the kernel repo.
They're <strong> tags with color:#79635c on hover in the CSS. A really weird style choice for sure, but semantically they aren't meant to be links at all.
I know, I am saying they should be links, as it is what one would expect from an article like this.
Their section on "Dataset limitations" says that the study "Only captures bugs with Fixes: tags (~28% of fix commits)."
Just worth noting that it is a significant extrapolation from only "28%" of fix commits to assume that the average is 2 years.
Why? A sample size of 28% is positively huge compared to what most statistical studies have to work with. The accuracy of an extrapolation is mostly determined by underlying sampling bias, not the amount of data. If you have any basis to suggest that capturing "only bugs with fixes tags" creates a skewed sample, that would be grounds to distrust the extrapolation, but simply claiming "it's only 28%" does not make it worth noting.
grsecurity project has fixed many security bugs but did not contribute back, as they're profiting from selling the patchset.
It's not uncommon for the bugs they found to be rediscovered 6-7 years later.
https://xcancel.com/spendergrsec
This implies (or states, hard to say) that they don't upstream specifically in order to profit. That is nonsense.
1. Tons of bugs are reported upstream by grsecurity historically.
2. Tons of critical security mitigations in the kernel were outright invented by that team. ASLR, SMAP, SMEP, NX, etc.
3. They were completely FOSS until very recently.
4. They have always maintained that they are entirely willing to upstream patches but that it's a lot of work and would require funding. Upstream has always been extremely hostile towards attempts to take small pieces of Grsecurity and upstream them.
But the patchset should use the same license as the original code, shouldn't?
It is: https://grsecurity.net/faq
> as they're profiting from selling the patchset
Profiting from selling their patchset is not the whole story, though. grsec was public and free for a long time and there were many effects at play preventing the kernel from adopting it.
Firefox bugs stay in the open for that long.
One of my favorite Firefox bugs was some I don’t quite remember the details of, but went something like this:
“There’s a crash while using this config file.” Something more complex than that, but ultimately a crash of some kind.
Years later, like 20 years later, the bug was closed. You see, they re-wrote the config parser in Rust, and now this is fixed.”
That’s cool but it’s not the part I remember. The part I always think about is, imagine responding to the bug right after it was opened with “sorry, we need to go off and write our own programming language before this bug is fixed. Don’t worry, we’ll be back, it’s just gonna take some time.”
Nobody would believe you. But yet, it’s what happened.
To be fair, any rewrite could have fixed it, didn't have to wait for Rust.
No, Graydon Hoare took one look at the config code, went “fuck this” and decided to create a new language instead.
But that take ruins all the intrigue of their comment... But youre spot on. They fabricated a story.
I didn’t say otherwise. Rust is not the point here.
All software has long-lived bugs. None are bug-free, at any point in their existance, so it's almost inevitable. Have you seen Windows' bug tracker?
The anti-Firefox mob really is striving to take shots at it.
The point of the article isn't a criticism of Linux, but an analysis that leads to more productive code review.
I don't think the problem is the kernel. Kernel bugs stay hidden because no one runs recent Kernels.
My Pixel 8 runs kernel a stable minor from 6.1, which was released more than 4 years ago. Yes, fixes get backported to it, but the new features in 6.2->6.19 stay unused on that hardware. All the major distros suffer from the same problem, most people are not running them in production
Most hyperscalers are running old kernel versions on which they do backports. If you go Linux conferences you hear folks from big companies mentioning 4.xx, 3.xx kernels, in 2025.
Still probably a lot better than Windows.
Might be obviously, but there is definitely a lot of biases in the data here. It's unavoidable. E.g. many bugs will not be detected, but they will be removed when the code is rewritten. So code that is refactored more often will have lower age of fixed bugs. Components/subsystems that are heavily used will detect bugs faster. Some subsystems by their very nature can tolerate bugs more, while some by necessity will need to be more correct (like bpf).
The kernel this speaks of is probably linux. Does windows have a similar round time?
I mean, yea.
Here is a device driver bug that was around 11 years.
https://www.bitdefender.com/en-us/blog/hotforsecurity/google...
Deep bugs, particularly in kernels, can go unnoticed for years, according to analyses I keep seeing. Decades at times. ~
That seems frightening at first. However, the more I consider it, the more it seems... predictable.
The mental model that I find useful:
Users discover surface bugs.
Deep bugs only appear in infrequent combinations.
For some bugs to show up, new context is required.
I've observed a few patterns:
Undefined behavior-related bugs are permanently hidden.
Logic errors are less important than uncommon hardware or timing conditions.
Long before they can be exploited, security flaws frequently exist.
I'm curious what other people think of this:
Do persistent bugs indicate stability or failure?
What typically leads to their discovery?
To what extent do you trust "well-tested" code?
> Undefined behavior-related bugs are permanently hidden.
No they are often found and fixed.
> To what extent do you trust "well-tested" code?
I don't, which is why I use Qubes OS providing security through compartmentalization.
Then the question becomes: to what extent do you trust Xen and Qubes RPC?
I do have to somewhat trust Xen, but Qubes' isolation relies on hardware virtualization (VT-d), which statistically has much less security issues than Xen itself. Most Xen advisories do not affect Qubes: https://www.qubes-os.org/security/xsa/
It's interesting to consider that the same phenomenon may also hold true for humanity's psychological software.
Only tangentially related but maybe someone here can help me.
I have a server which has many peripherals and multiple GPUs. Now, I can use vfio and vfio-pcio to memory map and access their registers in user space. My question is, how could I start with kernel driver development? And I specifically mean the dev setup.
Would it be a good idea to use vfio with or without a vm to write and test drivers? How to best debug, reload and test changing some code of an existing driver?
What's with the odd scribbles in the background?
It's an easter egg on the website that usually goes unnoticed. It's our first time on the front page of HN, so it's a little overutilized right now. Capital-C clears it.
A bug is a piece of code that doesn't agree with requirements or architecture. The misalignment can not be attributed to code alone.
I'd find this article a bit more compelling if it was used to find current introduced bugs, instead of just using a holdout set
This is fascinating stuff, especially the per-subsystem data. I've worked with CAN in several different professional and amateur settings, I'm not surprised to see it near the bottom of this list. That's not a dig against the kernel or the folks who work on it... more of a heavy sigh about the state of the industries that use CAN.
On a related note, I'm seeing a correlation between "level of hoopla" and a "level of attention/maintenance." While it's hard to distinguish that correlation from "level of use," the fact that CAN is so far down the list suggests to me that hoopla matters; it's everywhere but nobody talks about it. If a kernel bug takes down someone's datacenter, boy are we gonna hear about it. But if a kernel bug makes a DeviceNet widget freak out in a factory somewhere? Probably not going to make the front page of HN, let alone CNN.
There is a general rule on bugs is that the more devices they are on, the more apt they are to trigger.
A CAN with 10,000 machines total and relatively fixed applications is either going to trigger the bug right off the bat and then work around it, or trigger the bug so rarely it won't be recognized as a kernel issue.
General purpose systems running millions and millions of units with different workloads are an evolutionary breeding ground for finding bugs and exploits.
The lesson here is that people have an unrealistic view of how complex it is to write correct and safe multithreaded code on multi-core, multi-thread, assymmetric core, out-of-order processors. This is no shade to kernel developers. Rather, I direct this at people who seem to you can just create a thread pool in C++ and solve all your concurrency problems.
One criticism of Rust (and, no, I'm not saying "rewrite it in Rust", to be clear) is that the borrow checker can be hard to use whereas many C++ engineers (in particular, for some reason) seem to argue that it's easier to write in C++. I have two things to say about that:
1. It's not easier in C++. Nothing is. C++ simply allows you to make mistakes without telling you. GEtting things correct in C++ is just as difficult as any other language if not more so due to the language complexity; and
2. The Rust borrow checker isn't hard or difficult to use. What you're doing is hard and difficult to do correctly.
This is I favor cooperative multitasking and using battle-tested concurrency abstractions whenever possible. For example the cooperative async-await of Hack and the model of a single thread responding to a request then discarding everything in PHP/Hack is virtually ideal (IMHO) for serving Web traffic.
I remember reading about Google's work on various C++ tooling including valgrind and that they exposed concurrency bugs in their own code that had lain dormant for up to a decade. That's Google with thousands of engineers and some very talented engineers at that.
> The Rust borrow checker isn't hard or difficult to use. What you're doing is hard and difficult to do correctly.
There are entire classes of structures that no, aren't hard to do properly, but the borrow checker makes artificially hard due to design limitations that are known to be sub-optimal.
No, two-directional linked lists and partially editable data structures aren't inherently hard. It's a Rust limitation that a piece of code can't take enough ownership of them to edit they safely.
> It's not easier in C++. Nothing is.
The implementations of sort in Rust are filled with unsafe.[0]
Another example is that of doubly linked lists.[1] It is possible to implement a doubly linked list correctly in C++ without much trouble. In Rust, it can be significantly more challenging.
In C++, pointers are allowed to alias if their types, roughly said, are compatible. In Rust, there are stricter rules, and getting those rules wrong in an unsafe block, or code outside unsafe blocks that code inside unsafe blocks rely on, will result in breakage of memory safety.
This has been discussed by others.[2]
Based on that, do you agree that there are algorithms and data structures that are significantly easier to implement efficiently and correctly in C++ than in Rust? And thus that you are being completely wrong in your claim?
[0] https://github.com/rust-lang/rust/blob/main/library/core/src...
[1] https://rust-unofficial.github.io/too-many-lists/
[2] https://chadaustin.me/2024/10/intrusive-linked-list-in-rust/
> The implementations of sort in Rust are filled with unsafe.
Strictly speaking, the mere presence of `unsafe` says nothing on its own about whether "it" is easier in C++. Not only does `unsafe` on its own say nothing about the "difficulty" of the code it contains, but that is just one factor of one side of a comparison - very much insufficient for a complete conclusion.
Furthermore, "just" writing a sorting algorithm is pretty straightforwards both in Rust and C++; it's the more interesting properties that tend to make for equally interesting implementations, and one would need to procure Rust and C++ implementations with equivalent properties, preferably from the same author(s), for a proper comparison.
Past research has shown that Rust's current sorting algorithms have different properties than C++ implementations from the time (e.g., the "X safety" results in [0]), so if nothing substantial has changed since then there's going to be some work to do for a proper comparison.
Edit: forgot to add the reference [0]: https://github.com/Voultapher/sort-research-rs/blob/main/wri...
Also, the Linux kernel developers turned off strict aliasing in the C compilers they use, because they found strict aliasing too difficult. Yet unsafe Rust has more difficult rules than even strict aliasing in C and C++.
> Also, the Linux kernel developers turned off strict aliasing in the C compilers they use, because they found strict aliasing too difficult.
I'm not sure "they found strict aliasing too difficult" is an entirely correct characterization? From this rather (in)famous email from Linus [0]:
[0]: https://lkml.org/lkml/2018/6/5/769From the stats we see that most bugs effectively come from the limitations of the language.
Impressive results on the model, I'm surprised they improved it with very simple heuristics. Hopefully this tool will be made available to the kernel developers and integrated to the workflow.
Millions of lines of code, all running in supervisor mode.
One bug is all it takes to compromise the entire system.
The monolithic UNIX kernel was a good design in the 60s; Today, we should know better[0][1].
0. https://sel4.systems/
1. https://genode.org/
My conclusion is that microkernels offer some protection from random reboots, but not much against hacking
Say the USB system runs in its own isolated process. Great, but if someone pwns the USB process they can change disk contents, intercept and inject keystrokes, etc. You can usually leverage that into a whole system compromise.
Same with most subsystems: GPU, network, file system process compromises are all easily leveraged to pwn the whole system.
Year of HURD on the desktop?
Highly unrealistic rewrite disease
Of course by now processor manufacturers decided that blowing holes into the CPUs security model to make it go faster was the way to go. So your micro kernel is stuck on a hardware security model that looks like swiss cheese and smells like Surströmming.
How are SEL4 and Genode going for you in your day-to-day compute usage?
Yeah cause windows is amazing Or maybe macos? Ignore their freebsd parts of course.
Yes. As far as kernels go, NT was pretty damn good.
So is Mach, by the way, if you can afford the microkernel performance overhead.
Mach is not a very good microkernel at all, because the overhead is much higher than necessary. The L4 family’s IPC design is substantially more efficient, and that’s why they’re used in actual systems. Fuchsia/Zircon have improved on the model further.
Someone will of course bring up XNU, but the microkernel aspect of it died when they smashed the FreeBSD kernel into the codebase. DriverKit has brought some userspace drivers back, but they use shared memory for all the heavy lifting.
> Mach is not a very good microkernel at all, because the overhead is much higher than necessary. The L4 family’s IPC design is substantially more efficient, and that’s why they’re used in actual systems.
As opposed to Mach, which is not used in any actual systems
I mentioned XNU below. It doesn’t really count as a microkernel if you, you know, don’t actually use the microkernel part. At least for the 30 years between the FreeBSD collision and the introduction of DriverKit, which does most of its IPC through shared memory (because the mach ports are not efficient enough, I would assume).
XNU monolith-ized itself over time, even over some microkernel-esque boundaries.
If you include all the drivers too (which surely makes the comparison more accurate), is that still the case?
Windows NT 3.x was a true microkernel. Microsoft ruined it but the design was quite good and the driver question was irrelevant, until they sidestepped HAL.
The Linux kernel was and is a monstrosity.
This is outdated since Windows Vista, and even more so in Windows 11.
Windows Vista isn't Windows NT 3.x. In the internal versioning, it's not even 4.0.
Indeed, it is something better, Windows NT 6.0.
And it is irrelevant anyway, given that this comment was written from 10.0.26100.
Oh, I see.
You’re saying they improved the design. I know they added user-privilege device driver support for USB (etc).; did they revert the display compromise/mess as well?
What do you meant by them sidestepping the HAL?
Mind you I don't have access to Microsoft code, so this is all indirect, and a lot of this knowledge was when I was fledgling developer.
The Windows NT code was engineered to be portable across many different architectures--not just X86--so it has a hardware abstraction layer. The kernel only ever communicated to the device-driver implementation through this abstraction layer; so the kernel code itself was isolated.
That doesn't mean the device drivers were running in user-land privilege, but it does mean that the kernel code is quite stable and easy to reason about.
When Microsoft decided to compromise on this design, I remember senior engineers--when I first started my career--being abuzz about it for Windows NT 4.0 (or apparently earlier?).
I think the biggest one is that the whole GDI library was moved into the Kernel in 3.5x because the performance was terrible at the time.
I don't think they ever intended to keep all drivers strictly userland, though. Just the service side.
Apple at least has been making a concerted effort to kick more macOS/iOS functionality out into userland in the past several years.
Just like Windows since Vista.
NT is actually a pretty good kernel. NTFS and the userland is what is shit.
I think NTFS get a bit of crap from the OS above it adding limitations. If you read up on what NTFS allows, it is far better than what Windows and the explorer allows you to do with it.
Userland peaked in Windows 2000
NTFS is a beast of a filesystem and has been nothing but solid for 25+ years. The performance grievances ignore the warranties that NTFS offers vs many antiquated POSIX filesystems.
What warranties? I assume you’re comparing it to ext4 and not e.g. ZFS?
Here are a few
- mandatory byte-range locks enforced by the kernel
- explicit sharing modes
- guarantees around write ordering and durability
- safe delete-on-close
- first-class cache coherency contracts for networked access
POSIX aims for portability while NTFS/Win32 aims for explicit contracts and enforced behavior. For apps assuming POSIX semantics (e.g. git) NTFS feels rigid and weird. Coming the other way from NTFS, POSIX looks "optimistic" if not downright sloppy.
Of course ZFS et al. are more theoretically more robust than EXT4 but are still limited by the lowest common denominator POSIX API. Maybe you can detect that you're dealing with a ZFS backed volume and use extra ioctls to improve things but its still a messy business.
These are pretty much all about mandatory locking. Which giveth and taketh away in my experience. I’ve had substantially fewer weird file handling bugs in my Linux code than my Windows code. POSIX is very loosey-goosey in general, but Linux’s VFS system + ext4 has a much stronger model than the general POSIX guarantees.
`FILE_FLAG_DELETE_ON_CLOSE`’s equivalent on Posix is just rm. Windows doesn’t let you open new handles to `FILE_FLAG_DELETE_ON_CLOSE`ed files anyway, so it’s effectively the same. The inode will get deleted when the last file description is removed.
NFS is a disaster though, I’ll give you that one. Though mandatory locks on SMB shares hanging is also very aggravating in its own right.
Also to point out that outside UNIX, surviving mainframe and micros, the filesystems are closer to NTFS than UNIX world, in regards to what is enforced.
There is also the note that some of them are more database like, than classical filesystems.
Ah, and modern Windows also has Resilient File System (ReFS), which is used on Dev Drives.
Imagine if no one outside a select circle ever got to examine the code.
Everything is open source if you're skilled with Ghidra.
We call AI models "open source" if you can download the binary and not the source. Why not programs?
> We call AI models "open source" if you can download the binary and not the source.
Who's "we"? There's been quite a lot of pushback on this naming scheme from the OSS community, with many preferring the term "open weights".
>We call AI models "open source" if you can download the binary and not the source. Why not programs?
the weights of a model aren't equivalent to the binary output of source code, no matter how you try to stretch the metaphor.
>why not
because we aren't beholden to change all definitions and concepts because some guy at some corp said so.
Unless that corp is OSI, right?
Binaries and AI models can be inscrutable. They're meant to be interpreted by machines.
We want human readable, comprehensible, reproducible and maintainable sources at minimum when we say open source.
North Korea is called a “democratic people's republic”. Just because one thing that really isn't <whatever> is called <whatever> by the people in coontrol of it, doesn't mean that it is or that incorrectly calling other things <whatever> is correct.
Speaking of nasty kernel bugs although on another platform, there's a nasty one in either Microsoft's Win 11 nwifi.sys handling of deadlock conditions or Qualcomm's QCNCM865 FastConnect 7800 WCN785x driver that panics because of a watchdog failure in nwifi!MP6SendNBLInternal+0x4b guarded by a deadlocked ndis!NdisAcquireRWLockRead+0x8b. It "BSODs" the system rather than doing something sane like dropping a packet or retransmitting.
Am I the only unreasonable maniac who wants a very long-term stable, seL4-like capability-based, ubiquitous, formally-verified μkernel that rarely/never crashes completely* because drivers are just partially-elevated programs sprinkled with transaction guards and rollback code for critical multiple resource access coordination patterns? (I miss hacking on MINIX 2.)
* And never need to reboot or interrupt server/user desktop activities because the core μkernel basically never changes since it's tiny and proven correct.