> One thing that concerns me is the possibility of on-die ECC interacting with ECC on the motherboard and reducing it’s effectiveness
DDR5 on-die ECC detects and corrects one-bit errors. It cannot detect two-bit errors, so it will miscorrect some of them into three-bit errors. My understanding is that the on-die error correction scheme is specifically specially designed such that the resulting three-bit errors are mathematically guaranteed to be detected as uncorrectable two-bit errors by a standard full system-level ECC running on top of the on-die ECC. But I've never found a real authoritative reference that directly says that.
The people designing RAM and motherboards and the people working at JEDEC will of course know more about these things than almost everyone who reads hacker news. But I'd still prefer not to just blindly trust that they do the best things for us and have some documentation available.
Is anyone here good at navigating patent records? I would be surprised if there isn't something to be revealed in a patent but I don't know how to find it.
Very good point. It’s nontrivial but not impossible to determine this, I am very curious!
One nightmare scenario is a 3-bit error being detected as a correctable 1-bit error upstream, resulting in data corruption. I agree it’d be really nice to know that could never happen!
I've always thought it odd that we don't use ECC as a standard on all computers. I think people really downplay the impact of memory issues. They can be devastating, especially over time. They will slowly corrupt your file system, documents, binary files, code, everything. You notice when things like your compressed files start giving CRC/checksum errors or your downloads don't match SHA-512. You are then left with this "icky" feeling. What parts of your work are compromised? Obviously this is far worse with things like binary files and structures than those built on readable text. I have normally just rebuilt my system from scratch in these instances (after fixing the issue of course), but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time. You might not find them until years later.
You need the ECC RAM for your filesystem with integrity only then can you be confident you aren't going to be familiar with a concept like this happening 9 or 10 times.
Systems with "consumer amounts of RAM" are often run with what you might call "consumer amounts of sysadmin knowledge" and not notice the errors.
One time I took a working workstation from my home to my parents' home to give them. When I booted it up it gave a lot of strange kernel errors which I googled and discovered were reports of ECC correction. I reseated all DIMMs and it was fine. If that gave errors a few times a year instead of thousands of times on the first boot I wouldn't have noticed.
...No? Faulty storage will almost always show up as I/O errors, modern storage generally doesn't want to send a block of data to the CPU if there is an error in it. If your bits are flipping, chances are it's RAM and not storage. Your storage already has ECC.
Some crappy flash drives might not do this, though. Seems to behave like a floppy disk at times.
The majority of storage errors I've seen since using BTRFS and ZFS everywhere have shown up only as the storage returning corrupt data and claiming it to be good. I've seen drives and SSDs return dozens of errors per month for months on end without any SAS or SATA errors or any SMART errors.
I've seen this in DC grade SSDs, NAS grade hard drives, and SSDs sold in high end laptops.
AMD has historically supported ECC on all it's chips. That's changed in the last gen or two where some of the lower end APUs don't support ECC but the Pro APUs do and all the ryzen stuff (5, 7, 9) support ECC if you get a motherboard that supports it.
It's very important IMO that the ECC failures get reported and logged. The DDR5 on-die correction is not true ECC; the correction happens silently and is not reported or logged.
I have DDR4 ECC memory on my home server. Last week I got a crash... logged into my BMC and saw that one of the modules had "uncorrectable ECC". Then I looked over the logs and saw about 200 or so errors for the exact same module but "correctable ECC". They would happen over a short period of time followed by 4 months of no issues. Thanks to ECC and logging it was quick and easy to catch the issue right on the first occurrence, no data corruption. I have 4 modules in total, and only this one was having issues, so clearly a hardware fault.
Now imagine it was a regular non-ECC memory. There won’t be any crashes due to uncorrectable ECC; another 3 or more years would pass before corruption would become frequent enough for the issue to be noticed. At that point, there would be hundreds of corrupted files.
> I think people really downplay the impact of memory issues.
I don't think so. I've got my important data backed up on offline HDDs and SSDs and my most important data lives on a RAID system (ZFS) on a Xeon with ECC RAM but...
... there are simply a huge lot of checks everywhere making it so that bit flips on non ECC systems simply aren't that serious.
We live in a world of checksums and retries at every layer and distributed systems that are just about everywhere.
I went on for decades without ECC memory and nothing bad happened. Yes, I had the occasional bitflips. I did found out faulty mem using memtest (or weirder stuff, like some RAM stick identical to the other three, same chip everywhere, same sticker: but reported by memstick as being some other mem).
But I don't think that the "one bitflip cascading into corruption of your life's data" is something necessary to freak about.
It's not happening. Never happened to anyone.
If you (or other) are really concerned, you can use my scheme:
- dsc09874.jpg
becomes:
- dsc09874-b3-73b0835221.jpg
which means "that file's Blake3 checksum begins with 73b0835221" (use something else than Blake3 if so desired).
About 1/3rd of all my files are "tagged" this way, directly in the filename: this survive different medium and different OSes.
I then have my custom rsync backuping script doing a dry-run and warning me in a "stop the world!" fashion if it detects a bogus checksum (either on the source or on the dest: during the dry-run any modification to a file with a checksum is verified and any checksum mismatch means "stop the world").
I then also have, in addition to my custom backuping script, another script that allows me to verify either all files with that naming convention or a percentage I pick of those files (so basically some random sampling).
It's not mutually exclusive with ZFS / a system with ECC RAM.
> but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time
I hear you but... How much do you really need, years later, a file that you haven't needed for years? That one file is unlikely to be a big loss.
Now I'm not saying you're wrong: I ended up with my naming scheme (appending a partial cryptographic hash to the filename) after finding a series of about 20 pictures that were corrupted. And on all the backups I first checked, they were also corrupted. I ended up finding a very old data CD on which that series of .jpg files were fine. And I decided "never again, I want such SNAFU to be detected in the future".
But truth is: with 40 000 pictures, life without those 20 pictures would still be fine.
Something important not discussed here is that you need not only error correction (when possible) but also error reporting, so you can know you RAM is failing. Sadly, is not possible to get DDR5 on-die ECC error counts reported to the OS. Alternative solution, use IBECC with DDR5, where the CPU reserves a portion of the RAM for parity bits and does the check / reporting.
Google reports 8% of DIMMs being affected by errors. While the differences in metrics prevent comparing the two things directly it seems that DRAM errors are more common than CPU errors.
Does this imply that the all plastic gamer PCs (that are more designed to be art projects than serious computers) are bad and that everyone should be using systems with solid steel cases to block radiation?
I would think most servers and workstations would be RDIMM (Registered DIMM) by now and consumer stuff uses soldered down memory. Memory failing because it's old is definitely a thing, and very possible in this scenario, but I feel like I haven't seen errors due to physical insertion, that were not caught immediately by POST, in years.
Curious to learn more here - why would altitude play a role here? And when you say high altitude - are we talking La Paz, Bolivia (~12k feet) or Denver, CO (~5k feet)?
Engineering computer memory to work in space is a whole field of study, see Radiation Hardened memory. The main physical difference about space is the radiation profile and radiation induced memory upset.
Are you disputing that the Earth's atmosphere absorbs and reduces the amount of ambient cosmic radiation? Or where is the skepticism coming from?
The word "uncorrelated" is doing a lot of heavy lifting in that claim. Most DRAM error sources are pretty clearly correlated, if you're paying any attention.
Cosmic radiation is always a source of errors for any kind of DRAM.
Whether it is the primary source of errors, depends on the design of the DRAM die.
In a badly designed DRAM there could be many other error causes.
The designers of a DRAM certainly attempt to minimize all the error causes that they can control. If they succeed, the cosmic radiation would remain the primary error source.
Without access to internal documents of the DRAM vendors, we cannot know if they have indeed minimized all other error causes.
In modern DRAM chips, a major error cause can also be the disturbance of some memory cells when some other cells are accessed in their neighborhood, which is the basis of the Rowhammer attacks, but which can also happen during the normal use of the memory, when certain access patterns happen by chance.
The DRAM vendors do not provide details about the behavior of their devices, with the hope that this makes harder the task for someone who wants to design a variant of the Rowhammer attacks, but this also makes impossible for the owner of the memory to predict whether an application program will not perform by chance an access pattern that will unintentionally flip some memory bits, which without ECC will not be detected and corrected.
Companies making DRAM have no incentive to make the best products theoretically possible, they just want them good enough to not get many returns and avoid the risk of a class action lawsuit if a significant pattern of problems appears.
Avoiding returns for ECC DIMMs is a harder challenge when they are used in an environment where the server will have a management system that tracks errors over all time and when there might be a network management system tracking all errors over all systems.
Avoiding returns for non-ECC DIMMs is easy when a few errors per month won't be noticed by most users who assume that Windows always crashes anyway.
Has any DRAM company tested chips and made non-ECC DIMMs from the ones that have lower quality?
As an aside are the 2^N*3 sizes of DDR5 DIMMs (like 24G and 48G) made from chips that had errors in one section and got reconfigured to have 3/4 the capacity to not use the bad parts?
> Corolary: submarine datacenters are much better than orbital datacenters
Yea, except datacenters also need connectivity. And submarine connectivity, while not particularly expensive, is quite vulnerable - even more so than on land.
People outside the US have a significant lack of Internet functionality when there are problems with submarine cables, but it hardly ever fails. Apart from when certain countries sabotage things.
If you only need 100M deep water then you don't need to go far offshore, there are places where it's only about 1KM out. 12M of water is used for storing spent nuclear fuel assemblies and less than half of that is needed for protection so 5M should be adequate for stopping cosmic rays and that depth is common in harbours for recreational boating, such harbours shouldn't have issues of cable breakage.
But the real issue for comparing submarine and orbital computers is cooling. Space is cold but doesn't allow easy transmission of heat.
If only they were as easy to detect as morons. One is easily detectable with a single glass of beer and nothing more. The other, however, needs dry ice, or a pint of beer on a slab of dry ice.
Got ECC udimm for my ddr4 server and was surprised to see it picking up errors occasionally (once every couple months). Likely from a weakness in one of the sticks. This far it’s always corrected it though so opted to keep them anyway (nobody wants to be minus 32gb in these trying memory times)
With normal sticks I’d not have know that there is a potential issue.
You should be able to get the stick replaced under warranty. Every memory manufacturer I have ever seen has a lifetime warranty. As long as you RMA it before replacements become unavailable.
Depending on your server's CPU sockets, reseating the CPU might help if they were ever swapped before (esp if you got the server used). I have a marginally bad RAM slot on one of my Supermicro X11 boards because my dumb ass dropped the CPU into the socket and bent a few pins. I bent them back, but you can only do so much and the furthest RAM slot won't work without throwing errors daily if I run it at 2400MHz.
I have a HP Z640 that only has 3/4 DIMM slots working. Reseating the CPU didn't help and the same CPU works perfectly in another Z640. I never found out the cause. It was sold as "unknown condition for parts" so $50 for a server that works perfectly apart from one DIMM slot is still a good deal.
There should not be errors during normal use at rated speeds.
With non-ECC RAM running a memory test and getting any errors means it’s not stable. The memory tests run at maximum bandwidth constantly so a few days of full bandwidth memory testing might produce as many transmissions as your months of regular server use. Seeing bit errors (even if corrected) means something is not right.
The ECC can correct it for a while but I would want to RMA that stick if it’s in warranty.
Alas this is white label mem off eBay and fair bit of time has elapsed so not confident exchange would work. Minor miracle the ECC works at all given that
At some point I should try running it at slightly lower speed i guess. It’s very hard to troubleshoot something that only happens every month or two though
Never seen it fail a memtest but I can see it in the ECC stats since those cover weeks/months.
If you bought it from one of those ebay outfits that specialize in memory modules, they also often offer a lifetime warranty. Usually those sellers put their own sticker on the module. Worth a check.
"I have used more than a few systems with ECC UDIMMs and I recommend avoiding them if convenient as ECC UDIMMs are expensive on the second hand market while ECC RDIMMs can get very cheap."
Boy did I learn that the hard way. I found an ASRock server motherboard that fit my needs for a NAS perfectly. I just assumed all ECC memory were registered so I didn't even bother to check. I had already put everything together minus the memory. I had actually purchased the memory and had to return them because they were RDIMMs and get the ECC UDIMMs for quite a bit more. Luckily this was just before the AI induced price spike.
A very large portion of people who go from working on consumer grade systems to getting enterprise grade stuff for their home lab make a mistake like that, I did some similar things, my biggest mistake was spending $300 on non-refundable ECC RDIMMs without realising that my server (the cheapest Dell Txxx server) took ECC UDIMMs.
I hope that my blog posts on these topics will save some people some money.
The problem I have with ECC discussions is that it’s always anecdotal. There is one Google study finding memory flips with some probability in their datacenters, but it was awhile ago. Everyone repeats the statement DDR5 on-die ECC isn’t a replacement for regular ECC, but nobody ever shows data on DDR5 bitflips. That’s not a knock on this article per se, but it’d be nice to have actual data here.
>> Everyone repeats the statement DDR5 on-die ECC isn’t a replacement for regular ECC, but nobody ever shows data on DDR5 bitflips.
That's because the on-die ECC cannot detect errors in the transmission of data from the DIMMs to the CPU. It seems like on-die ECC is meant to "hide" some level of errors to make less reliable RAM chips appear good. I have no idea if that's a reasonable way to do it, but it seems like it could be misused. Like TFA suggests, we need to know how the two levels of ECC interact.
Right. Everyone agrees that real ECC is more robust, and production servers of course run it. But it seems to me that if most RAM bitflips are in the module itself (rather than in transmission to CPU), then on-die ECC could in theory make the probability of a bitflip in practice so low that for things like home servers it isn’t worth the price premium for real ECC. But without data on this, everyone is just speculating.
This is the attitude that needs to change. If RAM is 25% of the system cost (say for a home server in your example), then the cost for ECC is 3% of the system cost. There is no world where accepting data corruption in order to save 3% cost is a reasonable tradeoff. ECC should be standard, period.
When I last looked at this (a couple of years ago), it wasn't just the cost difference—the ECC memory available to purchase was also slower than non-ECC offerings, or at least cost significantly more if I actually wanted the same speed. At least for me, going ECC simply didn't make sense.
Caveat that I was building a gaming PC at that time, I might have gone differently for a home server. And, this was before the memory market went nuts for unrelated reasons.
Intel and AMD segmented the market. Workstations and servers use registered memory. Desktops use unregistered memory. Most desktop motherboards or CPUs do not support ECC. And most people who want ECC want reliable more than fast. So ECC unregistered above standard speed memory is made and priced for a niche. But this would change if ECC was standard.
There are absolutely fast DDR5 ECC DIMMs[1]. Just break out your wallet (and piggy bank, and rainy day fund, and..)
The main reason there aren’t more options here is that a) high-frequency memory usually isn’t sufficiently stable, unless binned, and b) there’s very, very, very little demand.
Yes, ECC DIMMs are available only for the memory speeds guaranteed to work by the manufacturers of CPUs, like Intel and AMD, e.g. for up to DDR5-5600 in UDIMMs or DDR5-6400 in RDIMMs.
The "gaming" DIMMs are available in much greater speeds, which are "overclocked" speeds, i.e. there exists absolutely no guarantee from Intel or AMD that they work at the advertised speed, even if they frequently do work.
However, with such overclocked DIMMs, you do not know how well they work. They might work most of the time, but e.g. in a hotter room they might have an error or two per day without you noticing, especially if the computer is used mainly for games.
For someone doing anything professional on a computer, overclocked DIMMs are not an acceptable choice.
> ECC DIMMs are available only for the memory speeds guaranteed to work by the manufacturers of CPUs, like Intel and AMD, e.g. for up to DDR5-5600 in UDIMMs or DDR5-6400 in RDIMMs.
This is not true[1]. These DIMMs are not guaranteed to work with all Threadripper CPUs, but they are guaranteed themselves to be stable at their advertised frequencies, which are higher than JEDEC. (And from my experience, 9000-series threadripper is perfectly happy to drive 8-channel 1Rx8 at 7200MT/s.)
AMD (and possibly Intel, I don’t have any experience with their recent offerings) also lets you overclock memory completely on WRX90. So if you buy JEDEC sticks, you’re welcome to try to push them as hard as you can.
So as I was overclocking my AMD Ryzen 7 2700 I discovered that the Infinity Fabric has ECC because if I tried to run my fancy DDR-3466 ram at full speed I started getting logged errors from it. Those failed transactions slow down the system because the data needs to be corrected or even re-sent. The RAM runs fine overclocked at DDR4-4200 in a newer system, but my old Ryzen 7 2700 couldn't handle anything past DDR4-3266 without the errors creeping in.
Without the reported ECC on Infinity Fabric I would have had no idea that the errors were happening and affecting performance.
AMD, Intel, and motherboard manufacturers do qualify the CPUs and motherboards for "overclocked" (above JEDEC) memory speeds. The systems are designed to work at the overclocked transmission speeds. They need to be paired with memory modules that are capable of functioning at those speeds. With ECC presently being a datacenter thing, companies only make it according to JEDEC specs. If ECC were standard for everyone, then the pre-qualified overclocked "gaming" memory modules would be available in ECC configuration as well.
Without ECC, nobody knows if their memory is working or not. Regardless whether it is running a standard JEDEC speed or an overclocked one.
The motherboard manufacturers test their motherboards, to ensure that they typically work with overclocked memory modules.
If you buy a complete gaming computer which includes overclocked memory, then hopefully the vendor has done a burn-in and has tested for some time the computer, as sold.
However, if you buy the CPU yourself, neither Intel nor AMD provides any guarantee that the CPU will work with overclocked memory.
It is upon you to test your assembled computer, but true tests would require a very long time and a climatic chamber, to offer any kind of certainty that the combination CPU-DIMMs works at the desired speed.
If you have ECC memory, you can try to overclock it and at least in this case you will know for sure if the memory works at the higher speed, or not.
Overclocking ECC memory is actually a method frequently used to verify whether the hardware ECC support and also the operating system EDAC driver work OK.
Both AMD and Intel publish on their site, for each CPU model, the maximum supported memory speed.
If the CPU does not work at that speed, you are entitled to a replacement or a refund.
For instance, AMD Ryzen™ 9 9950X3D is guaranteed to work with DDR5-5600, but with nothing faster. No AMD non-server CPU goes above this limit for DDR5.
Some of the Intel CPUs are guaranteed to work with faster memories, e.g. Core Ultra 9 285K with up to DDR5-6400, and the very recently launched Core Ultra 7 270K PLUS with up to DDR5-7200.
Let me kmow where I can get ECC RAM that's only 12% more than the price of non-ECC and I'll strongly consider it the next time I do a build. (Otoh, prices are high, so I'm not building anything for a while unless stuff breaks)
Some years ago, it was easy to find ECC modules with a so small price difference.
From that time, I have a few old computers with 64 GB or 128 GB of DDR4 ECC memory.
Unfortunately, after the passage to DDR5, DDR5 ECC UDIMMs have become hard to find, and even when you could find them, the price difference could be as high as 30% to 50%.
> There is no world where accepting data corruption in order to save 3% cost is a reasonable tradeoff.
It is if you're doing something where corruption is detectable after the fact, and happens rarely enough that redoing affected work adds less than 3% overhead.
DRAM is just one level in the memory hierarchy and it can absolutely be tested by the other levels in the system. It won't be continually monitored, and there will be a performance penalty, and it's a PITA, but it can definitely be done without hardware ECC. Even some of that can be minimized as most tests are statistical in nature anyway.
If you do not monitor it continuously, you achieve nothing, because the memory errors happen seldom and unpredictably.
Without ECC, you need a custom operating system, which will compute some error detection code, e.g. a CRC, for each read-only memory page, i.e. including all code pages and constant pages, and all cached but not dirty file system pages, and which will check the CRC codes periodically in the background.
That would still leave the read-write data segments unprotected, though on those some errors may be benign, if the locations are overwritten later without being read again.
On-die ECC was added in DDR5 because the error frequency has increased too much.
Its only effect was that now DDR5, as seen externally, has a reliability similar to that of the older memory generations. On-die ECChas not brought any serious improvement in reliability. It has just prevented the degradation of the reliability.
The errors generated internally are caused mostly by the ionizing radiation from the environment, which discharges the storage capacitors.
However there are many errors that appear during the communication between memories and the CPU, which are caused by the electrical noise from the environment.
These external errors are much less likely to appear for soldered DRAM. Because of this, for soldered LPDDRx memory it is less important to have end-to-end ECC than for socketed modules, which are much more vulnerable to noise.
The DDR5 standard has some options for using some error detection for the communication link, but it is impossible to know whether a given computer or motherboard has implemented such options or if they are enabled by the firmware when the motherboard has the physical support.
The only way to be certain that the memory works fine is to use ECC that covers completely the circuit from the CPU memory controller to the values stored in the DRAM and back to the CPU memory controller and your operating system has the appropriate device driver for obtaining the error reports.
Besides electrical noise, there are mechanical issues as well. I've seen systems fail memtest86+ after being moved, and the issue being fixed after reseating the DIMMs. ECC can catch mis-seated DIMMs very quickly.
The mechanical issues also act through increasing the sensitivity to electrical noise.
The imperfect contacts are equivalent to adding series resistors, possibly in parallel with parasitic capacitors, on the link traces, allowing a greater amplitude for the noise pulses.
As TFA says, it is very likely that the hyperscalers continue to make studies about DRAM reliability in their servers, but they do not publish them, either because they believe that such data could be useful for competitors, or, more likely, they might have NDAs with the memory vendors.
When you do not own a great number of servers, you cannot provide anything else except anecdotal evidence.
Because the vast majority of PCs do not have ECC memory, even the companies that have a great number of PCs have no idea about the reliability of the memory used in those PCs, because it is very difficult to distinguish memory defects from the huge number of software bugs.
Moreover, the frequency of memory errors is proportional with the amount of memory you have, so those with tiny amounts of memory, like 8 GB, are much less likely to encounter memory errors than those who have 32 GB or 64 GB of DRAM in their PCs.
I have tried to always use ECC memory in my computers, whenever possible. Even my laptop is an older Dell Precision, with ECC memory.
But I can also offer only anecdotal testimony.
I have also seen cases like that mentioned in TFA, where the existence of ECC allowed to discover that some modules were not seated well in their sockets, so reseating them prevented the reappearance of errors.
I have also seen a case when a laptop was not used for a long time and it was stored in a rather humid place, so the contacts in the SODIMM sockets had oxidized, which resulted in frequent memory errors. Scrubbing vigorously the contacts and reinserting the memory modules solved the problem.
I have also seen many cases where certain memory modules degraded after many years of use, e.g. 5 or more years of 24/7 use, and they began to have frequent errors, e.g. multiple errors per day, even if when they were new the error rate could have been of one error per year or even less.
In such cases ECC was extremely useful, because the DIMM that had become worn out could be identified and replaced and the server could work fine some more years.
i have years and terabytes of ECC DDR4 and DDR5 memories, UDIMM and RDIMM, with edac enabled all this time, and i have literally never once observed a single error correction. however, (1) i built all these computers myself (2) the servers are located on the middle floor a thick, concrete multi-story warehouse-to-office conversion. as others have said, on a correctly built computer, the source of bit flips is cosmic radiation, and there is a lot of mass between these machines and the sky.
ECC end-to-end means carrying sufficient parity along the whole way including at-rest (memory and storage hierarchy), links, buffering, and computation. Skipping ECC in transposer/buffering/bus circuitry creates a source of silent errors. Silent errors are way, way too prevalent across billions of users and the vast majority never understand the source of such malfunctions or know they happen.
One thing I also did not know until recently was that there actually is ECC unified memory.
I noticed the difference because in a Hardware forum someone wrote that the framework desktop does not have ECC while an HP G2 Workstation also featuring the Ryzen 395 Max but in the ECC supporting variant paying a premium of +600 bucks.
Does encrypted memory give ECC-like detection benefit, along with rowhammer protection, for cheaper cost (only additional computation), and 100% reliable?
It's preferred but not having it still works if you're willing to leave some potential avenues for errors to get in. However, it's not like not having ECC is a recipe for a corrupted FS.
I run a 9800x3d with 96GB ECC (2x 48GB DDR5-5800).
It was a pain to select a compatible board and source the RAM but glad I did, in June 2024... the RAM prices went up immediately after.
non-ECC should be a kernel taint in Linux. Newer versions of Windows (12?) should also refuse to boot without ECC. It should be mandatory in DDR6. The insanity must end.
> We need ECC RAM to be more widely used. Ideally we would have some government action to force this given the ongoing cost....
The rest of this article is great. This, however, is absolute garbage and nonsense. The last thing you want is more regulations and "government certified systems". Take a look at the bullshit with age verification bills and operating systems.
How is “all RAM sold in consumer PCs must be registered ECC” equivalent to age verification laws? I’m not saying I agree with the premise but that is a bit of a leap
"This, however, is absolute garbage and nonsense. The last thing you want is more regulations and government certified systems." — the guy who designed the Ford Pinto
Claiming that every issue you've experienced is from seating errors? Come on man, at least deliberately reproduce that scenario once. I've tortured motherboards for fun and never managed that on purpose. We did pry every cap off of an old K6 board while it was on and... it just kept running. It didn't come back after we turned it off
The amount of time it takes to reseat memory, even if the case is already open and ready, is also more than enough time for it to cool back down.
If you have decent memory running at what it's rated for and you're seeing more than a couple per year, good odds are it's getting too hot or the load line calibration is not aggressive enough.
E: I have never, and I mean literally never, seen age or abuse play a factor.
Interesting article, and I agree that ECC needs to be more widely available. But I think this is a major overreach:
> We need ECC RAM to be more widely used. Ideally we would have some government action to force this given the ongoing cost to society in corrupted data and lost time due to RAM hardware errors. I think that at minimum we need sufficient taxes on non-ECC RAM (and EC4 RAM for DDR5) to make it more expensive when bought new than ECC RAM.
ECC RAM is not needed for every application and mandating it for every gaming rig and machine used to run Chrome will just increase prices unnecessarily. 99% of the time it's also not a "cost to society" but a cost to whoever owns/uses that machine. If they want to eat that cost, it should be up to them.
> ECC RAM is not needed for every application and mandating it for every gaming rig and machine used to run Chrome will just increase prices unnecessarily.
The price increase will be very small if manufacturers are no longer able to segment the market to make a higher margin on those who need it. No one wants even games or web browsers to crash randomly.
> If they want to eat that cost, it should be up to them.
They need to be given the choice. Most people do not know they even have the choice, and they are not offered it at a reasonable price.
Without quantifying the cost to society that someone's web-browser crashes sometimes, it's hard to make a rational case that this issue is so dire that it needs specific regulation.
I'm not sure making it illegal to sell consumer grade hardware is the boon for the people you imagine
No-one ever proposed making non-ECC RAM illegal, that's a straw-man.
Restricting risky items is standard practice for all governments. All the "use by" dates on food etc are from government regulation. The current experiments with "raw milk" shows that you can get away with flouting food safety regulations and not have immediate problems.
>The price increase will be very small if manufacturers are no longer able to segment the market to make a higher margin on those who need it.
AMD doesn't have this segmentation. Where's the hoard of people building systems with ECC ram? At best you have a few people on HN saying they're doing it, but in practice nobody actually does it.
Important data should probably be worked on and stored with error-correcting codes, in addition to the hardware-provided ones. The Btrfs mention in TFA suggests that maintaining integrity of the Btrfs metadata is not considered part of the job description of Btrfs, but perhaps it should be.
BTRFS has checksums on all metadata. It isn't designed to cope with bad RAM but has been shown to be somewhat resistant to it to the degree that BTRFS developers can recognise symptoms in problem reports.
Whether a filesystem could be designed to operate in the face of random memory changes would be an interesting computer science research topic. But for getting working systems having RAM just not fail is a better option.
I had a conversation with chatgpt about that very thing, whether it was a bubble or not and how the big players could absorb the cost and where it would all land and the interesting thing was if China gets its foothold and grabs enough share because the prices are too high even being subsidized, that could cause real trouble. But the point was its not a bubble in the conventional sense. Have your own conversation about that with an AI.
RAM without ECC is not faulty. It still does exactly what the fuck it is intended to do - act as a rapidly-modifiable store of referenceable information.
Anyone considering non-ECC RAM to be faulty or defective is a fool that needs to go back to school and re-take some basic Computer Science courses.
That particular quote did stand out to me. I think the counter claim is that if everyone uses software everywhere for everything, there is a very real societal cost for damaged data and if it's hard to detect then there is no way to prevent or measure harm. I think the "invisible hand" free market is showing some serious strain in modern times. People stopped using leaded-gas because it was causing measurable decreases in IQ for generations. I don't care how much you want to save a few dollars if it harms a lot of people and is difficult to measure. Everything else seems like a race to the bottom price so pushing the market in this manner doesn't bother me in the least. If you are running a server with others' data at stake, it seems like an obvious benefit for society on the whole to address the problem rather than serve some cheap datacenter operators.
This would be a consumer protection law like any other such law.
Forcing the vendors to sell only products that do what they are advertised cannot be considered in any way as "forcing unnecessary things upon people".
Any computer without ECC memory is a computer that does not compute from time to time and you have no idea when that happens.
Originally, any IBM PC or compatible had memory error detection.
All this problem of the computers without ECC memory has been created by some evil Intel employees, who have conceived around 1994 a scheme for market segmentation between "Pentium" and "Pentium Pro".
So they removed memory error detection support from the Pentium chipsets (Intel Triton), so that this would remain a marketing feature exclusive to the "Pro" Intel products.
Obviously the removal of this feature from "consumer" products was done without any price reduction, which made happy also the memory vendors, because this trick increased their profits.
So in the next years this market segmentation method became the "standard" business practice, preying on the naivety of most computer users.
A consumer protection law should force that any computer or motherboard must support ECC, which adds a completely negligible cost.
The end users may buy non-ECC modules, if they choose so, but non-ECC modules should be sold like cigarettes, with a visible advertisement that using such modules may cause unpredictable errors in your computer.
"All this problem of the computers without ECC memory has been created by some evil Intel employees, who have conceived around 1994 a scheme for market segmentation between "Pentium" and "Pentium Pro"."
I hate to break it to you but the computer I had in the 80s didn't come with ECC. Non-ECC computing was a thing WELL BEFORE Intel tried to think of market segmentation. Revisionist history is not welcomed here.
The original IBM PC/XT had parity RAM. In the early PC/XT and PC/AT clones you had banks of 9 DIP sockets for memory chips with the 9th being for parity.
The DDR4 ECC RAM has 72 bits total and 64bits of data which means the same 9/8 ratio of data storage to usable data as the original PC/XT.
The Z80 and 6502 based computers in the 80s typically didn't have parity RAM. The Amiga and Macintosh were higher performance systems in the 80s which also didn't have parity RAM.
We could and should have gone straight from parity RAM to ECC RAM for everything.
I think it's a pattern that reinforces itself, as a nanny state develops it's opinion of it's people worsens and it's more willing to use force to compel them.
Right, the opinion shifting to the people being unworthy of uncorrupted data unless they’re wealthy enough to pay the artificial extortion price for integrity.
What you're actually saying is that I shouldn't be allowed to determine what kind of RAM my application needs. Instead, you are saying that someone with a gun should dictate this choice to me.
That's what we mean when we say that people with your point of view are promoting nanny-statism. It amounts to infantilization of adult consumers and subsequent disempowerment.
Occasionally such legislation can be justified, as when genuinely hazardous products are involved, or products whose use involves wide-ranging externalities, but usually not. People can and should be educated -- and expected -- to make their own decisions about things like whether they need ECC RAM.
How do YOU propose doing this when most computer "experts" have never used ECC RAM and when it's only available in high end specialist systems that aren't on sale in regular computer stores?
I think my knowledge of the topic is pretty good but I learned from this discussion that almost all AMD CPUs support ECC RAM and you apparently just need a motherboard to match. When accompanying people who were buying "gamer" PCs (after they rejected my recommendation for a second hand workstation grade computer with ECC RAM) I never witnessed store staff saying "hey you could get a motherboard with ECC support".
It's simply none of my business whether or not someone does the homework to understand what kind of RAM they should buy. I bothered to inform myself; they can, too.
Next up. CPUs that will spit out the wrong calculations 1/1M times. You’ll have to pay more for the enterprise version to get uncorrupted processing. The spare traces cost 5 cents, but we can charge 20% more for the privilege instead.
Just because you can allow it, doesn’t mean you should. I don’t want to live in a world where you need a PhD to avoid being scammed by increasingly sophisticated ploys. Data isn’t free anymore and the right questions are becoming harder to ask.
> One thing that concerns me is the possibility of on-die ECC interacting with ECC on the motherboard and reducing it’s effectiveness
DDR5 on-die ECC detects and corrects one-bit errors. It cannot detect two-bit errors, so it will miscorrect some of them into three-bit errors. My understanding is that the on-die error correction scheme is specifically specially designed such that the resulting three-bit errors are mathematically guaranteed to be detected as uncorrectable two-bit errors by a standard full system-level ECC running on top of the on-die ECC. But I've never found a real authoritative reference that directly says that.
The people designing RAM and motherboards and the people working at JEDEC will of course know more about these things than almost everyone who reads hacker news. But I'd still prefer not to just blindly trust that they do the best things for us and have some documentation available.
Is anyone here good at navigating patent records? I would be surprised if there isn't something to be revealed in a patent but I don't know how to find it.
Very good point. It’s nontrivial but not impossible to determine this, I am very curious!
One nightmare scenario is a 3-bit error being detected as a correctable 1-bit error upstream, resulting in data corruption. I agree it’d be really nice to know that could never happen!
What about rowhammer attacks ?
Wonder if they could be used to introduce “correctable” errors…
I've always thought it odd that we don't use ECC as a standard on all computers. I think people really downplay the impact of memory issues. They can be devastating, especially over time. They will slowly corrupt your file system, documents, binary files, code, everything. You notice when things like your compressed files start giving CRC/checksum errors or your downloads don't match SHA-512. You are then left with this "icky" feeling. What parts of your work are compromised? Obviously this is far worse with things like binary files and structures than those built on readable text. I have normally just rebuilt my system from scratch in these instances (after fixing the issue of course), but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time. You might not find them until years later.
> You notice when things like your compressed files start giving CRC/checksum errors or your downloads don't match SHA-512.
And 9 times out of 10 (probably much more) it will be faulty storage, not RAM.
I'd say you are way overestimating impact of no-ECC.
You need the ECC RAM for your filesystem with integrity only then can you be confident you aren't going to be familiar with a concept like this happening 9 or 10 times.
You can read online how often correctable errors happen to people with ECC and reporting.
Single digit - sometimes none - errors per year in systems with consumer amount of RAM.
Systems with "consumer amounts of RAM" are often run with what you might call "consumer amounts of sysadmin knowledge" and not notice the errors.
One time I took a working workstation from my home to my parents' home to give them. When I booted it up it gave a lot of strange kernel errors which I googled and discovered were reports of ECC correction. I reseated all DIMMs and it was fine. If that gave errors a few times a year instead of thousands of times on the first boot I wouldn't have noticed.
...No? Faulty storage will almost always show up as I/O errors, modern storage generally doesn't want to send a block of data to the CPU if there is an error in it. If your bits are flipping, chances are it's RAM and not storage. Your storage already has ECC.
Some crappy flash drives might not do this, though. Seems to behave like a floppy disk at times.
The majority of storage errors I've seen since using BTRFS and ZFS everywhere have shown up only as the storage returning corrupt data and claiming it to be good. I've seen drives and SSDs return dozens of errors per month for months on end without any SAS or SATA errors or any SMART errors.
I've seen this in DC grade SSDs, NAS grade hard drives, and SSDs sold in high end laptops.
This is/was classic market segmentation. Want ECC? Pay up for Xeon (server/workstation CPU)
AMD has historically supported ECC on all it's chips. That's changed in the last gen or two where some of the lower end APUs don't support ECC but the Pro APUs do and all the ryzen stuff (5, 7, 9) support ECC if you get a motherboard that supports it.
It's very important IMO that the ECC failures get reported and logged. The DDR5 on-die correction is not true ECC; the correction happens silently and is not reported or logged.
I have DDR4 ECC memory on my home server. Last week I got a crash... logged into my BMC and saw that one of the modules had "uncorrectable ECC". Then I looked over the logs and saw about 200 or so errors for the exact same module but "correctable ECC". They would happen over a short period of time followed by 4 months of no issues. Thanks to ECC and logging it was quick and easy to catch the issue right on the first occurrence, no data corruption. I have 4 modules in total, and only this one was having issues, so clearly a hardware fault.
Now imagine it was a regular non-ECC memory. There won’t be any crashes due to uncorrectable ECC; another 3 or more years would pass before corruption would become frequent enough for the issue to be noticed. At that point, there would be hundreds of corrupted files.
> I think people really downplay the impact of memory issues.
I don't think so. I've got my important data backed up on offline HDDs and SSDs and my most important data lives on a RAID system (ZFS) on a Xeon with ECC RAM but...
... there are simply a huge lot of checks everywhere making it so that bit flips on non ECC systems simply aren't that serious.
We live in a world of checksums and retries at every layer and distributed systems that are just about everywhere.
I went on for decades without ECC memory and nothing bad happened. Yes, I had the occasional bitflips. I did found out faulty mem using memtest (or weirder stuff, like some RAM stick identical to the other three, same chip everywhere, same sticker: but reported by memstick as being some other mem).
But I don't think that the "one bitflip cascading into corruption of your life's data" is something necessary to freak about.
It's not happening. Never happened to anyone.
If you (or other) are really concerned, you can use my scheme:
becomes: which means "that file's Blake3 checksum begins with 73b0835221" (use something else than Blake3 if so desired).About 1/3rd of all my files are "tagged" this way, directly in the filename: this survive different medium and different OSes.
I then have my custom rsync backuping script doing a dry-run and warning me in a "stop the world!" fashion if it detects a bogus checksum (either on the source or on the dest: during the dry-run any modification to a file with a checksum is verified and any checksum mismatch means "stop the world").
I then also have, in addition to my custom backuping script, another script that allows me to verify either all files with that naming convention or a percentage I pick of those files (so basically some random sampling).
It's not mutually exclusive with ZFS / a system with ECC RAM.
> but not all our files are replaceable, and you forever and always might have corrupted files you are unaware of that got written during this time
I hear you but... How much do you really need, years later, a file that you haven't needed for years? That one file is unlikely to be a big loss.
Now I'm not saying you're wrong: I ended up with my naming scheme (appending a partial cryptographic hash to the filename) after finding a series of about 20 pictures that were corrupted. And on all the backups I first checked, they were also corrupted. I ended up finding a very old data CD on which that series of .jpg files were fine. And I decided "never again, I want such SNAFU to be detected in the future".
But truth is: with 40 000 pictures, life without those 20 pictures would still be fine.
Something important not discussed here is that you need not only error correction (when possible) but also error reporting, so you can know you RAM is failing. Sadly, is not possible to get DDR5 on-die ECC error counts reported to the OS. Alternative solution, use IBECC with DDR5, where the CPU reserves a portion of the RAM for parity bits and does the check / reporting.
My Linux installation came with an AMD MCE driver that reports them to dmesg. It logs a few corrected errors a day.
Check if those are all coming from DIMMs, as opposed to other subsystems (on-die caches).
https://www.theregister.com/on-prem/2021/06/04/fyi-todays-co...
Google and Facebook report a few unreliable CPU cores out of thousands of systems. Meaning maybe a 1/10,000 rate of repeatable CPU errors.
https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf
Google reports 8% of DIMMs being affected by errors. While the differences in metrics prevent comparing the two things directly it seems that DRAM errors are more common than CPU errors.
A few errors per day seems much too frequent, unless you live at a high altitude.
Normal good DIMMs, at least when new, should not have errors more frequently than one error per many months.
Frequent errors may appear with memories that are not seated well in their sockets, or which are old, at least several years old.
Frequent errors may also be caused by more general computer problems, like a bad power supply unit.
Does this imply that the all plastic gamer PCs (that are more designed to be art projects than serious computers) are bad and that everyone should be using systems with solid steel cases to block radiation?
Are DIMMs (a.k.a. UDIMM) still a thing?
I would think most servers and workstations would be RDIMM (Registered DIMM) by now and consumer stuff uses soldered down memory. Memory failing because it's old is definitely a thing, and very possible in this scenario, but I feel like I haven't seen errors due to physical insertion, that were not caught immediately by POST, in years.
But maybe it's just me. Happy to be lucky. :)
The client/consumer desktop market hasn't actually disappeared, and soldered memory is almost unheard-of in that market segment.
Curious to learn more here - why would altitude play a role here? And when you say high altitude - are we talking La Paz, Bolivia (~12k feet) or Denver, CO (~5k feet)?
The primary source of uncorrelated bit-flips is, effectively, cosmic radiation.
(Correlated bit-flips are likely from a manufacturing error... or, at least in one case, from excessively radioactive ceramic packaging.)
The more atmosphere you have to randomly absorb high-energy photons, the better.
Higher density memory drops fewer electrons in each well, which means a lower-energy photon can change the state.
Corolary: submarine datacenters are much better than orbital datacenters. Cheaper to cool, construct, shield and access.
>The primary source of uncorrelated bit-flips is, effectively, cosmic radiation.
This gets said a lot but with little evidence. The mario speedrun bit flip has been replicated with marginal connector insertion.
https://youtu.be/vj8DzA9y8ls
Engineering computer memory to work in space is a whole field of study, see Radiation Hardened memory. The main physical difference about space is the radiation profile and radiation induced memory upset.
Are you disputing that the Earth's atmosphere absorbs and reduces the amount of ambient cosmic radiation? Or where is the skepticism coming from?
I am disputing that cosmic ray showers are the obvious dominant source of terrestrial DRAM bit flips.
AFAIK it's been documented by Los Alamos since they're at high altitude.
There is a lot of direct evidence from increased error rates in computers on airplanes.
That's a much weaker statement than "the primary source of uncorrelated bit flips is cosmic radiation".
The word "uncorrelated" is doing a lot of heavy lifting in that claim. Most DRAM error sources are pretty clearly correlated, if you're paying any attention.
Cosmic radiation is always a source of errors for any kind of DRAM.
Whether it is the primary source of errors, depends on the design of the DRAM die.
In a badly designed DRAM there could be many other error causes.
The designers of a DRAM certainly attempt to minimize all the error causes that they can control. If they succeed, the cosmic radiation would remain the primary error source.
Without access to internal documents of the DRAM vendors, we cannot know if they have indeed minimized all other error causes.
In modern DRAM chips, a major error cause can also be the disturbance of some memory cells when some other cells are accessed in their neighborhood, which is the basis of the Rowhammer attacks, but which can also happen during the normal use of the memory, when certain access patterns happen by chance.
The DRAM vendors do not provide details about the behavior of their devices, with the hope that this makes harder the task for someone who wants to design a variant of the Rowhammer attacks, but this also makes impossible for the owner of the memory to predict whether an application program will not perform by chance an access pattern that will unintentionally flip some memory bits, which without ECC will not be detected and corrected.
Companies making DRAM have no incentive to make the best products theoretically possible, they just want them good enough to not get many returns and avoid the risk of a class action lawsuit if a significant pattern of problems appears.
Avoiding returns for ECC DIMMs is a harder challenge when they are used in an environment where the server will have a management system that tracks errors over all time and when there might be a network management system tracking all errors over all systems.
Avoiding returns for non-ECC DIMMs is easy when a few errors per month won't be noticed by most users who assume that Windows always crashes anyway.
Has any DRAM company tested chips and made non-ECC DIMMs from the ones that have lower quality?
As an aside are the 2^N*3 sizes of DDR5 DIMMs (like 24G and 48G) made from chips that had errors in one section and got reconfigured to have 3/4 the capacity to not use the bad parts?
> Corolary: submarine datacenters are much better than orbital datacenters
Yea, except datacenters also need connectivity. And submarine connectivity, while not particularly expensive, is quite vulnerable - even more so than on land.
People outside the US have a significant lack of Internet functionality when there are problems with submarine cables, but it hardly ever fails. Apart from when certain countries sabotage things.
If you only need 100M deep water then you don't need to go far offshore, there are places where it's only about 1KM out. 12M of water is used for storing spent nuclear fuel assemblies and less than half of that is needed for protection so 5M should be adequate for stopping cosmic rays and that depth is common in harbours for recreational boating, such harbours shouldn't have issues of cable breakage.
But the real issue for comparing submarine and orbital computers is cooling. Space is cold but doesn't allow easy transmission of heat.
FYI, it's not so much photons (gamma rays) as muons.
Muons are nasty little buggers.
If only they were as easy to detect as morons. One is easily detectable with a single glass of beer and nothing more. The other, however, needs dry ice, or a pint of beer on a slab of dry ice.
Isn't that a lot? I see them only when heavily over clocking .
Got ECC udimm for my ddr4 server and was surprised to see it picking up errors occasionally (once every couple months). Likely from a weakness in one of the sticks. This far it’s always corrected it though so opted to keep them anyway (nobody wants to be minus 32gb in these trying memory times)
With normal sticks I’d not have know that there is a potential issue.
You should be able to get the stick replaced under warranty. Every memory manufacturer I have ever seen has a lifetime warranty. As long as you RMA it before replacements become unavailable.
Depending on your server's CPU sockets, reseating the CPU might help if they were ever swapped before (esp if you got the server used). I have a marginally bad RAM slot on one of my Supermicro X11 boards because my dumb ass dropped the CPU into the socket and bent a few pins. I bent them back, but you can only do so much and the furthest RAM slot won't work without throwing errors daily if I run it at 2400MHz.
I have a HP Z640 that only has 3/4 DIMM slots working. Reseating the CPU didn't help and the same CPU works perfectly in another Z640. I never found out the cause. It was sold as "unknown condition for parts" so $50 for a server that works perfectly apart from one DIMM slot is still a good deal.
There should not be errors during normal use at rated speeds.
With non-ECC RAM running a memory test and getting any errors means it’s not stable. The memory tests run at maximum bandwidth constantly so a few days of full bandwidth memory testing might produce as many transmissions as your months of regular server use. Seeing bit errors (even if corrected) means something is not right.
The ECC can correct it for a while but I would want to RMA that stick if it’s in warranty.
Alas this is white label mem off eBay and fair bit of time has elapsed so not confident exchange would work. Minor miracle the ECC works at all given that
At some point I should try running it at slightly lower speed i guess. It’s very hard to troubleshoot something that only happens every month or two though
Never seen it fail a memtest but I can see it in the ECC stats since those cover weeks/months.
If you bought it from one of those ebay outfits that specialize in memory modules, they also often offer a lifetime warranty. Usually those sellers put their own sticker on the module. Worth a check.
"I have used more than a few systems with ECC UDIMMs and I recommend avoiding them if convenient as ECC UDIMMs are expensive on the second hand market while ECC RDIMMs can get very cheap."
Boy did I learn that the hard way. I found an ASRock server motherboard that fit my needs for a NAS perfectly. I just assumed all ECC memory were registered so I didn't even bother to check. I had already put everything together minus the memory. I had actually purchased the memory and had to return them because they were RDIMMs and get the ECC UDIMMs for quite a bit more. Luckily this was just before the AI induced price spike.
A very large portion of people who go from working on consumer grade systems to getting enterprise grade stuff for their home lab make a mistake like that, I did some similar things, my biggest mistake was spending $300 on non-refundable ECC RDIMMs without realising that my server (the cheapest Dell Txxx server) took ECC UDIMMs.
I hope that my blog posts on these topics will save some people some money.
The problem I have with ECC discussions is that it’s always anecdotal. There is one Google study finding memory flips with some probability in their datacenters, but it was awhile ago. Everyone repeats the statement DDR5 on-die ECC isn’t a replacement for regular ECC, but nobody ever shows data on DDR5 bitflips. That’s not a knock on this article per se, but it’d be nice to have actual data here.
>> Everyone repeats the statement DDR5 on-die ECC isn’t a replacement for regular ECC, but nobody ever shows data on DDR5 bitflips.
That's because the on-die ECC cannot detect errors in the transmission of data from the DIMMs to the CPU. It seems like on-die ECC is meant to "hide" some level of errors to make less reliable RAM chips appear good. I have no idea if that's a reasonable way to do it, but it seems like it could be misused. Like TFA suggests, we need to know how the two levels of ECC interact.
Right. Everyone agrees that real ECC is more robust, and production servers of course run it. But it seems to me that if most RAM bitflips are in the module itself (rather than in transmission to CPU), then on-die ECC could in theory make the probability of a bitflip in practice so low that for things like home servers it isn’t worth the price premium for real ECC. But without data on this, everyone is just speculating.
This is the attitude that needs to change. If RAM is 25% of the system cost (say for a home server in your example), then the cost for ECC is 3% of the system cost. There is no world where accepting data corruption in order to save 3% cost is a reasonable tradeoff. ECC should be standard, period.
When I last looked at this (a couple of years ago), it wasn't just the cost difference—the ECC memory available to purchase was also slower than non-ECC offerings, or at least cost significantly more if I actually wanted the same speed. At least for me, going ECC simply didn't make sense.
Caveat that I was building a gaming PC at that time, I might have gone differently for a home server. And, this was before the memory market went nuts for unrelated reasons.
Intel and AMD segmented the market. Workstations and servers use registered memory. Desktops use unregistered memory. Most desktop motherboards or CPUs do not support ECC. And most people who want ECC want reliable more than fast. So ECC unregistered above standard speed memory is made and priced for a niche. But this would change if ECC was standard.
There are absolutely fast DDR5 ECC DIMMs[1]. Just break out your wallet (and piggy bank, and rainy day fund, and..)
The main reason there aren’t more options here is that a) high-frequency memory usually isn’t sufficiently stable, unless binned, and b) there’s very, very, very little demand.
[1] https://v-color.net/products/ddr5-ocrdimm-amd-wrx90-workstat...
Right, my point was that it's no longer 3% of the system cost.
Yes, ECC DIMMs are available only for the memory speeds guaranteed to work by the manufacturers of CPUs, like Intel and AMD, e.g. for up to DDR5-5600 in UDIMMs or DDR5-6400 in RDIMMs.
The "gaming" DIMMs are available in much greater speeds, which are "overclocked" speeds, i.e. there exists absolutely no guarantee from Intel or AMD that they work at the advertised speed, even if they frequently do work.
However, with such overclocked DIMMs, you do not know how well they work. They might work most of the time, but e.g. in a hotter room they might have an error or two per day without you noticing, especially if the computer is used mainly for games.
For someone doing anything professional on a computer, overclocked DIMMs are not an acceptable choice.
> ECC DIMMs are available only for the memory speeds guaranteed to work by the manufacturers of CPUs, like Intel and AMD, e.g. for up to DDR5-5600 in UDIMMs or DDR5-6400 in RDIMMs.
This is not true[1]. These DIMMs are not guaranteed to work with all Threadripper CPUs, but they are guaranteed themselves to be stable at their advertised frequencies, which are higher than JEDEC. (And from my experience, 9000-series threadripper is perfectly happy to drive 8-channel 1Rx8 at 7200MT/s.)
AMD (and possibly Intel, I don’t have any experience with their recent offerings) also lets you overclock memory completely on WRX90. So if you buy JEDEC sticks, you’re welcome to try to push them as hard as you can.
[1] https://v-color.net/products/ddr5-ocrdimm-amd-wrx90-workstat...
So as I was overclocking my AMD Ryzen 7 2700 I discovered that the Infinity Fabric has ECC because if I tried to run my fancy DDR-3466 ram at full speed I started getting logged errors from it. Those failed transactions slow down the system because the data needs to be corrected or even re-sent. The RAM runs fine overclocked at DDR4-4200 in a newer system, but my old Ryzen 7 2700 couldn't handle anything past DDR4-3266 without the errors creeping in.
Without the reported ECC on Infinity Fabric I would have had no idea that the errors were happening and affecting performance.
AMD, Intel, and motherboard manufacturers do qualify the CPUs and motherboards for "overclocked" (above JEDEC) memory speeds. The systems are designed to work at the overclocked transmission speeds. They need to be paired with memory modules that are capable of functioning at those speeds. With ECC presently being a datacenter thing, companies only make it according to JEDEC specs. If ECC were standard for everyone, then the pre-qualified overclocked "gaming" memory modules would be available in ECC configuration as well.
Without ECC, nobody knows if their memory is working or not. Regardless whether it is running a standard JEDEC speed or an overclocked one.
The motherboard manufacturers test their motherboards, to ensure that they typically work with overclocked memory modules.
If you buy a complete gaming computer which includes overclocked memory, then hopefully the vendor has done a burn-in and has tested for some time the computer, as sold.
However, if you buy the CPU yourself, neither Intel nor AMD provides any guarantee that the CPU will work with overclocked memory.
It is upon you to test your assembled computer, but true tests would require a very long time and a climatic chamber, to offer any kind of certainty that the combination CPU-DIMMs works at the desired speed.
If you have ECC memory, you can try to overclock it and at least in this case you will know for sure if the memory works at the higher speed, or not.
Overclocking ECC memory is actually a method frequently used to verify whether the hardware ECC support and also the operating system EDAC driver work OK.
The ECC on the GTX 1080 let me figure out the optimal overclock with the only feedback mechanism being benchmarked throughout.
> However, if you buy the CPU yourself, neither Intel nor AMD provides any guarantee that the CPU will work with overclocked memory.
There's no guarantee that it works with standard clocked memory either.
Both AMD and Intel publish on their site, for each CPU model, the maximum supported memory speed.
If the CPU does not work at that speed, you are entitled to a replacement or a refund.
For instance, AMD Ryzen™ 9 9950X3D is guaranteed to work with DDR5-5600, but with nothing faster. No AMD non-server CPU goes above this limit for DDR5.
Some of the Intel CPUs are guaranteed to work with faster memories, e.g. Core Ultra 9 285K with up to DDR5-6400, and the very recently launched Core Ultra 7 270K PLUS with up to DDR5-7200.
The same reason your utility trucks isn't rated for 200KM/H speeds.
Let me kmow where I can get ECC RAM that's only 12% more than the price of non-ECC and I'll strongly consider it the next time I do a build. (Otoh, prices are high, so I'm not building anything for a while unless stuff breaks)
For second hand gear RDIMMs are significantly cheaper than UDIMMs. As all RDIMMs have ECC that means generally ECC is cheaper for second hand systems.
Aliexpress had $50 single socket motherboards for Xeon E5-26xx CPUs and $150 dual socket motherboards last time I checked that use DDR4 RDIMMs.
Before the prices went up due to AI foolishness DDR4 RDIMMs were $1.20AU/G means means something like $US0.80/G.
Some years ago, it was easy to find ECC modules with a so small price difference.
From that time, I have a few old computers with 64 GB or 128 GB of DDR4 ECC memory.
Unfortunately, after the passage to DDR5, DDR5 ECC UDIMMs have become hard to find, and even when you could find them, the price difference could be as high as 30% to 50%.
> There is no world where accepting data corruption in order to save 3% cost is a reasonable tradeoff.
It is if you're doing something where corruption is detectable after the fact, and happens rarely enough that redoing affected work adds less than 3% overhead.
If you don't have ECC, you don't even know that your CPU is executing the correct code. There goes the "detectable" part.
There's a decent amount of distributed stuff that has a design principle of not trusting the hardware it's running on.
DRAM is just one level in the memory hierarchy and it can absolutely be tested by the other levels in the system. It won't be continually monitored, and there will be a performance penalty, and it's a PITA, but it can definitely be done without hardware ECC. Even some of that can be minimized as most tests are statistical in nature anyway.
If you do not monitor it continuously, you achieve nothing, because the memory errors happen seldom and unpredictably.
Without ECC, you need a custom operating system, which will compute some error detection code, e.g. a CRC, for each read-only memory page, i.e. including all code pages and constant pages, and all cached but not dirty file system pages, and which will check the CRC codes periodically in the background.
That would still leave the read-write data segments unprotected, though on those some errors may be benign, if the locations are overwritten later without being read again.
No.
On-die ECC was added in DDR5 because the error frequency has increased too much.
Its only effect was that now DDR5, as seen externally, has a reliability similar to that of the older memory generations. On-die ECChas not brought any serious improvement in reliability. It has just prevented the degradation of the reliability.
The errors generated internally are caused mostly by the ionizing radiation from the environment, which discharges the storage capacitors.
However there are many errors that appear during the communication between memories and the CPU, which are caused by the electrical noise from the environment.
These external errors are much less likely to appear for soldered DRAM. Because of this, for soldered LPDDRx memory it is less important to have end-to-end ECC than for socketed modules, which are much more vulnerable to noise.
The DDR5 standard has some options for using some error detection for the communication link, but it is impossible to know whether a given computer or motherboard has implemented such options or if they are enabled by the firmware when the motherboard has the physical support.
The only way to be certain that the memory works fine is to use ECC that covers completely the circuit from the CPU memory controller to the values stored in the DRAM and back to the CPU memory controller and your operating system has the appropriate device driver for obtaining the error reports.
Besides electrical noise, there are mechanical issues as well. I've seen systems fail memtest86+ after being moved, and the issue being fixed after reseating the DIMMs. ECC can catch mis-seated DIMMs very quickly.
The mechanical issues also act through increasing the sensitivity to electrical noise.
The imperfect contacts are equivalent to adding series resistors, possibly in parallel with parasitic capacitors, on the link traces, allowing a greater amplitude for the noise pulses.
In addition to reducing the energy content of the actual desired signal.
Correct.
Also to note, asides from the physical differences, LPDDR5+ also features link ECC.
As TFA says, it is very likely that the hyperscalers continue to make studies about DRAM reliability in their servers, but they do not publish them, either because they believe that such data could be useful for competitors, or, more likely, they might have NDAs with the memory vendors.
When you do not own a great number of servers, you cannot provide anything else except anecdotal evidence.
Because the vast majority of PCs do not have ECC memory, even the companies that have a great number of PCs have no idea about the reliability of the memory used in those PCs, because it is very difficult to distinguish memory defects from the huge number of software bugs.
Moreover, the frequency of memory errors is proportional with the amount of memory you have, so those with tiny amounts of memory, like 8 GB, are much less likely to encounter memory errors than those who have 32 GB or 64 GB of DRAM in their PCs.
I have tried to always use ECC memory in my computers, whenever possible. Even my laptop is an older Dell Precision, with ECC memory.
But I can also offer only anecdotal testimony.
I have also seen cases like that mentioned in TFA, where the existence of ECC allowed to discover that some modules were not seated well in their sockets, so reseating them prevented the reappearance of errors.
I have also seen a case when a laptop was not used for a long time and it was stored in a rather humid place, so the contacts in the SODIMM sockets had oxidized, which resulted in frequent memory errors. Scrubbing vigorously the contacts and reinserting the memory modules solved the problem.
I have also seen many cases where certain memory modules degraded after many years of use, e.g. 5 or more years of 24/7 use, and they began to have frequent errors, e.g. multiple errors per day, even if when they were new the error rate could have been of one error per year or even less.
In such cases ECC was extremely useful, because the DIMM that had become worn out could be identified and replaced and the server could work fine some more years.
i have years and terabytes of ECC DDR4 and DDR5 memories, UDIMM and RDIMM, with edac enabled all this time, and i have literally never once observed a single error correction. however, (1) i built all these computers myself (2) the servers are located on the middle floor a thick, concrete multi-story warehouse-to-office conversion. as others have said, on a correctly built computer, the source of bit flips is cosmic radiation, and there is a lot of mass between these machines and the sky.
It's not just cosmic radiation or improperly built computers that result in bit flips. Row hammer is a thing :)
ECC end-to-end means carrying sufficient parity along the whole way including at-rest (memory and storage hierarchy), links, buffering, and computation. Skipping ECC in transposer/buffering/bus circuitry creates a source of silent errors. Silent errors are way, way too prevalent across billions of users and the vast majority never understand the source of such malfunctions or know they happen.
One thing I also did not know until recently was that there actually is ECC unified memory.
I noticed the difference because in a Hardware forum someone wrote that the framework desktop does not have ECC while an HP G2 Workstation also featuring the Ryzen 395 Max but in the ECC supporting variant paying a premium of +600 bucks.
Does encrypted memory give ECC-like detection benefit, along with rowhammer protection, for cheaper cost (only additional computation), and 100% reliable?
ECC is essentially mandatory for ZFS. Assuming that is you prefer your data uncorrupted.
> ECC is essentially mandatory for ZFS
Please stop repeating this old myth, it is - and always has been - BS and needs to die.
See for example: https://kldload.com/zfs-wiki/myths
It's preferred but not having it still works if you're willing to leave some potential avenues for errors to get in. However, it's not like not having ECC is a recipe for a corrupted FS.
I run a 9800x3d with 96GB ECC (2x 48GB DDR5-5800).
It was a pain to select a compatible board and source the RAM but glad I did, in June 2024... the RAM prices went up immediately after.
non-ECC should be a kernel taint in Linux. Newer versions of Windows (12?) should also refuse to boot without ECC. It should be mandatory in DDR6. The insanity must end.
Yes a kernel taint is a really good idea!
> We need ECC RAM to be more widely used. Ideally we would have some government action to force this given the ongoing cost....
The rest of this article is great. This, however, is absolute garbage and nonsense. The last thing you want is more regulations and "government certified systems". Take a look at the bullshit with age verification bills and operating systems.
How is “all RAM sold in consumer PCs must be registered ECC” equivalent to age verification laws? I’m not saying I agree with the premise but that is a bit of a leap
We need any RAM at this point.
"This, however, is absolute garbage and nonsense. The last thing you want is more regulations and government certified systems." — the guy who designed the Ford Pinto
This is not a very good post.
Claiming that every issue you've experienced is from seating errors? Come on man, at least deliberately reproduce that scenario once. I've tortured motherboards for fun and never managed that on purpose. We did pry every cap off of an old K6 board while it was on and... it just kept running. It didn't come back after we turned it off
The amount of time it takes to reseat memory, even if the case is already open and ready, is also more than enough time for it to cool back down.
If you have decent memory running at what it's rated for and you're seeing more than a couple per year, good odds are it's getting too hot or the load line calibration is not aggressive enough.
E: I have never, and I mean literally never, seen age or abuse play a factor.
I can only report what I have seen. Feel free to write your own blog post reporting what you have seen.
Interesting article, and I agree that ECC needs to be more widely available. But I think this is a major overreach:
> We need ECC RAM to be more widely used. Ideally we would have some government action to force this given the ongoing cost to society in corrupted data and lost time due to RAM hardware errors. I think that at minimum we need sufficient taxes on non-ECC RAM (and EC4 RAM for DDR5) to make it more expensive when bought new than ECC RAM.
ECC RAM is not needed for every application and mandating it for every gaming rig and machine used to run Chrome will just increase prices unnecessarily. 99% of the time it's also not a "cost to society" but a cost to whoever owns/uses that machine. If they want to eat that cost, it should be up to them.
> ECC RAM is not needed for every application and mandating it for every gaming rig and machine used to run Chrome will just increase prices unnecessarily.
The price increase will be very small if manufacturers are no longer able to segment the market to make a higher margin on those who need it. No one wants even games or web browsers to crash randomly.
> If they want to eat that cost, it should be up to them.
They need to be given the choice. Most people do not know they even have the choice, and they are not offered it at a reasonable price.
A cost to people in general is a cost to society.
Without quantifying the cost to society that someone's web-browser crashes sometimes, it's hard to make a rational case that this issue is so dire that it needs specific regulation.
I'm not sure making it illegal to sell consumer grade hardware is the boon for the people you imagine
Crash is the best-case outcome for a browser bit flip. Here's bitsquatting:
https://ripe92.ripe.net/programme/meeting-plan/sessions/112/...
No-one ever proposed making non-ECC RAM illegal, that's a straw-man.
Restricting risky items is standard practice for all governments. All the "use by" dates on food etc are from government regulation. The current experiments with "raw milk" shows that you can get away with flouting food safety regulations and not have immediate problems.
>The price increase will be very small if manufacturers are no longer able to segment the market to make a higher margin on those who need it.
AMD doesn't have this segmentation. Where's the hoard of people building systems with ECC ram? At best you have a few people on HN saying they're doing it, but in practice nobody actually does it.
ECC UDIMMs are crazy expensive, presumably because they don't have economy of scale. Mandating ECC would solve that.
Important data should probably be worked on and stored with error-correcting codes, in addition to the hardware-provided ones. The Btrfs mention in TFA suggests that maintaining integrity of the Btrfs metadata is not considered part of the job description of Btrfs, but perhaps it should be.
BTRFS has checksums on all metadata. It isn't designed to cope with bad RAM but has been shown to be somewhat resistant to it to the degree that BTRFS developers can recognise symptoms in problem reports.
Whether a filesystem could be designed to operate in the face of random memory changes would be an interesting computer science research topic. But for getting working systems having RAM just not fail is a better option.
16GB ECC DDR5 was $50 early 2025
it is now $500
when the AI/datacenter bubble finally bursts we are going to have some amazing hardware deals
I had a conversation with chatgpt about that very thing, whether it was a bubble or not and how the big players could absorb the cost and where it would all land and the interesting thing was if China gets its foothold and grabs enough share because the prices are too high even being subsidized, that could cause real trouble. But the point was its not a bubble in the conventional sense. Have your own conversation about that with an AI.
I'd rather just have the conversation here. I don't need some hallucinated random seed response to tell me what China can and cannot do.
Of course an AI is going to tell you AI isn't a bubble.
I'm planning some RAM purchases for the end of this year, I expect the crash to happen in a few months.
"Ideally we would have some government action to force this given the ongoing cost to society in corrupted data"
Why is it so often Australia with the "We must force unnecessary things upon people" attitude?
How is it unnecessary. Australia has tons of laws about faulty products. All developed countries do
"We need to force ECC on everyone"
RAM without ECC is not faulty. It still does exactly what the fuck it is intended to do - act as a rapidly-modifiable store of referenceable information.
Anyone considering non-ECC RAM to be faulty or defective is a fool that needs to go back to school and re-take some basic Computer Science courses.
That particular quote did stand out to me. I think the counter claim is that if everyone uses software everywhere for everything, there is a very real societal cost for damaged data and if it's hard to detect then there is no way to prevent or measure harm. I think the "invisible hand" free market is showing some serious strain in modern times. People stopped using leaded-gas because it was causing measurable decreases in IQ for generations. I don't care how much you want to save a few dollars if it harms a lot of people and is difficult to measure. Everything else seems like a race to the bottom price so pushing the market in this manner doesn't bother me in the least. If you are running a server with others' data at stake, it seems like an obvious benefit for society on the whole to address the problem rather than serve some cheap datacenter operators.
This would be a consumer protection law like any other such law.
Forcing the vendors to sell only products that do what they are advertised cannot be considered in any way as "forcing unnecessary things upon people".
Any computer without ECC memory is a computer that does not compute from time to time and you have no idea when that happens.
Originally, any IBM PC or compatible had memory error detection.
All this problem of the computers without ECC memory has been created by some evil Intel employees, who have conceived around 1994 a scheme for market segmentation between "Pentium" and "Pentium Pro".
So they removed memory error detection support from the Pentium chipsets (Intel Triton), so that this would remain a marketing feature exclusive to the "Pro" Intel products.
Obviously the removal of this feature from "consumer" products was done without any price reduction, which made happy also the memory vendors, because this trick increased their profits.
So in the next years this market segmentation method became the "standard" business practice, preying on the naivety of most computer users.
A consumer protection law should force that any computer or motherboard must support ECC, which adds a completely negligible cost.
The end users may buy non-ECC modules, if they choose so, but non-ECC modules should be sold like cigarettes, with a visible advertisement that using such modules may cause unpredictable errors in your computer.
> Any computer without ECC memory is a computer that does not compute from time to time
So are the computers with ECC memory. It's not an absolute protection either.
"All this problem of the computers without ECC memory has been created by some evil Intel employees, who have conceived around 1994 a scheme for market segmentation between "Pentium" and "Pentium Pro"."
I hate to break it to you but the computer I had in the 80s didn't come with ECC. Non-ECC computing was a thing WELL BEFORE Intel tried to think of market segmentation. Revisionist history is not welcomed here.
The original IBM PC/XT had parity RAM. In the early PC/XT and PC/AT clones you had banks of 9 DIP sockets for memory chips with the 9th being for parity.
The DDR4 ECC RAM has 72 bits total and 64bits of data which means the same 9/8 ratio of data storage to usable data as the original PC/XT.
The Z80 and 6502 based computers in the 80s typically didn't have parity RAM. The Amiga and Macintosh were higher performance systems in the 80s which also didn't have parity RAM.
We could and should have gone straight from parity RAM to ECC RAM for everything.
Because despite geographical location, Australia is a western country, and all western countries are like that.
I think it's a pattern that reinforces itself, as a nanny state develops it's opinion of it's people worsens and it's more willing to use force to compel them.
Right, the opinion shifting to the people being unworthy of uncorrupted data unless they’re wealthy enough to pay the artificial extortion price for integrity.
When you are used to thinking of citizens as children who need a nanny, all problems become bedtime shaped.
What you're actually saying is that I shouldn't be allowed to determine what kind of RAM my application needs. Instead, you are saying that someone with a gun should dictate this choice to me.
That's what we mean when we say that people with your point of view are promoting nanny-statism. It amounts to infantilization of adult consumers and subsequent disempowerment.
Occasionally such legislation can be justified, as when genuinely hazardous products are involved, or products whose use involves wide-ranging externalities, but usually not. People can and should be educated -- and expected -- to make their own decisions about things like whether they need ECC RAM.
How do YOU propose doing this when most computer "experts" have never used ECC RAM and when it's only available in high end specialist systems that aren't on sale in regular computer stores?
I think my knowledge of the topic is pretty good but I learned from this discussion that almost all AMD CPUs support ECC RAM and you apparently just need a motherboard to match. When accompanying people who were buying "gamer" PCs (after they rejected my recommendation for a second hand workstation grade computer with ECC RAM) I never witnessed store staff saying "hey you could get a motherboard with ECC support".
It's simply none of my business whether or not someone does the homework to understand what kind of RAM they should buy. I bothered to inform myself; they can, too.
Next up. CPUs that will spit out the wrong calculations 1/1M times. You’ll have to pay more for the enterprise version to get uncorrupted processing. The spare traces cost 5 cents, but we can charge 20% more for the privilege instead.
Just because you can allow it, doesn’t mean you should. I don’t want to live in a world where you need a PhD to avoid being scammed by increasingly sophisticated ploys. Data isn’t free anymore and the right questions are becoming harder to ask.
Data isn’t free anymore and the right questions are becoming harder to ask.
Ridiculous. Access to knowledge has never been as widely and freely available as it is now (no thanks to the nanny-statists.)