One key point about retention which is not often mentioned, and indeed neither does this article, is that retention is inversely proportional to program/erase cycles and decreases exponentially with increasing temperature. Hence why retention specs are usually X amount of time after Y cycles at Z temperature. Even a QLC SSD that has only been written to once, and kept in a freezer at -40, may hold data for several decades.
Manufacturers have been playing this game with DWPD/TBW numbers too --- by reducing the retention spec, they can advertise a drive as having a higher endurance with the exact same flash. But if you compare the numbers over the years, it's clear that NAND flash has gotten significantly worse; the only thing that has gone up, multiplicatively, is capacity, while endurance and rentention have both gone down by a few orders of magnitude.
For a long time, 10 years after 100K cycles was the gold standard of SLC flash.
Now we are down to several months after less than 1K cycles for QLC.
I also seem to remember reading retention is proportional to temperature at time of write. Ie, best case scenario = write data when drive is hot, and store in freezer. Would be happy if someone can confirm or deny this.
As far as I'm aware flash got a bit of a size boost when it went 3D and hasn't shrunk much since then. If you use the same number of bits per cell, I don't know if I would expect 2010 and 2020 or 2025 flash to vary much in endurance.
For logic and DRAM the biggest factors are how far they're being pushed with voltage and heat, which is a thing that trends back and forth over the years. So I could see that go either way.
So on the off-chance that there's a firmware engineer in here, how does this actually work?
Like does a SSD do some sort of refresh on power-on, or every N hours, or you have to access the specific block, or...? What if you interrupt the process, eg, having a NVMe in an external case that you just plug once a month for a few minutes to just use it as a huge flash drive, is that a problem?
What about the unused space, is a 4 TB drive used to transport 1 GB of stuff going to suffer anything from the unused space decaying?
It's all very unclear about what all of this means in practice and how's an user supposed to manage it.
SSD firmware engineer here. I work on enterprise stuff, so ymmv on consumer grade internals.
Generally, the data refresh will all happen in the background when the system is powered (depending on the power state). Performance is probably throttled during those operations, so you just see a slightly slower copy while this is happening behind the scenes.
The unused space decaying is probably not an issue, since the internal filesystem data is typically stored on a more robust area of media (an SLC location) which is less susceptible to data loss over time.
As far as how a user is supposed to manage it, maybe do an fsck every month or something? Using an SSD like that is probably ok most of the time, but might not be super great as a cold storage backup.
So say I have a 4TB USB SSD from a few years ago, that's been sitting unpowered in a drawer most of that time. How long would it need to be powered on (ballpark) for the full disk refresh to complete? Assume fully idle.
(As a note: I do have a 4TB USB SSD which did sit in a drawer without being touched for a couple of years. The data was all fine when I plugged it back in. Of course, this was a new drive with very low write cycles and stored climate controlled. Older worn out drive would probably have been an issue.) Just wondering how long I should keep it plugged in if I ever have a situation like that so I can "reset the fade clock" per se.
So you need to do an fsck? My big question after reading this article (and others like it) is whether it is enough to just power up the device (for how long?), or if each byte actually needs to be read.
The case an average user is worried about is where they have an external SSD that they back stuff up to on a relatively infrequent schedule. In that situation, the question is whether just plugging it and copying some stuff to it is enough to ensure that all the data on the drive is refreshed, or if there's some explicit kind of "maintenance" that needs to be done.
>Generally, the data refresh will all happen in the background when the system is powered (depending on the power state).
How does the SSD know when to run the refresh job? AFAIK SSDs don't have an internal clock so it can't tell how long it's been powered off. Moreover does doing a read generate some sort of telemetry to the controller indicating how strong/weak the signal is, thereby informing whether it should refresh? Or does it blindly refresh on some sort of timer?
Pretty much, but it depends a lot on the vendor and how much you spent on the drive. A lot of the assumptions about enterprise SSDs is that they’re powered pretty much all the time, but are left in a low power state when not in use. So, data can still be refreshed on a timer, as long as it happens within the power budget.
There are several layers of data integrity that are increasingly expensive to run. Once the drive tries to read something that requires recovery, it marks that block as requiring a refresh and rewrites it in the background.
How long does the data refresh take, approx? Let's say I have an external portable SSD that I keep stored data on. Would plugging the drive into my computer and running
A full read would do it, but I think the safer recommendation is to just use a small hdd for external storage. Anything else is just dealing with mitigating factors
Typically unused empty space is a good thing, as it will allow drives to run in MLC or SLC mode instead of their native QLC. (At least, this seems to be the obvious implication from performance testing, given the better performance of SLC/MLC compared to QLC.) And the data remanence of SLC/MLC can be expected to be significantly better than QLC.
>as it will allow drives to run in MLC or SLC mode instead of their native QLC
That depends on the SSD controller implementation, specifically whether it proactively moves stuff from the SLC cache to the TLC/QLC area. I expect most controllers to do this, given that if they don't, the drive will quickly lose performance as it fills up. There's basically no reason not proactively move stuff over.
Is there a real source that confirms this with data? I generally like xda, but the quality of their articles is uneven and they trend towards click bait headlines that try to shock/surprise you with thin content underneath. There have been a string of "Here is the one piece of software you didn't know you needed for your NAS" and it turns out to be something extremely popular like home assistant.
This article just seems to link to a series of other xda articles with no primary source. I wouldn't ever trust any single piece of hardware to store my data forever but this feels like clickbait- At one point they even state "...but you shouldn't really worry about it..."
So, product idea: A powered "cold storage box" for M.2 SSDs. 2 to 8 M.2 slots. Periodically, an internal computer connects one of the slots, reads every byte, waits for some period of time, then powers off. Maybe shows a little green light next to each drive when the last read was successful. Could be battery-powered.
What about powered SSDs that contain files that are rarely read?
My desktop computer is generally powered except when there is a power failure, but among the million+ files on its SSD there are certainly some that I do not read or write for years.
Does the SSD controller automatically look for used blocks that need to have their charge refreshed and do so, or do I need to periodically do something like "find / -type f -print0 | xargs -0 cat > /dev/null" to make sure every file gets read occasionally?
no, the firmware does any maintenance.
good firmware should do gradual scrub whenever it's idle.
unfortunately, there's no real way to know whether the firmware is good, or doing anything.
I wonder if there's some easy way to measure power consumed by a device - to detect whether it's doing housekeeping.
Honestly this is one of my favorite things about ZFS. I know that a disk scan is performed every week/month (whatever schedule). And I also know that it has verified the contents of each block. It is very reassuring in that way.
Is there any type of flash-based storage (preferably accessible to end users) that focuses on long term data retention?
If not, that feels like a substantial hole in the market. Non-flash durable storage tend to be annoying or impractical for day to day use. I want to be able to find a 25 year old SD card hiding in some crevice and unearth an unintentional time capsule, much like how one can pick up 20+ year old MiniDiscs and be able to play the last thing their former owners recorded to them perfectly.
We really don't know. One thing I wish some of these sites would do is actually test how long it takes for the drives to decay and also do a retest after they have been left powered for say 10 minutes to an hour, read completely, written to a bit etc and see if they can determine what a likely requirement is.
The problem is the test will take years, be out of date by the time its released and new controllers will be out with potentially different needs/algorithms.
The data on this SSD, which hadn't been used or powered up for two years, was 100% good on initial inspection. All the data hashes verified, but it was noted that the verification time took a smidgen longer than two years previously. HD Sentinel tests also showed good, consistent performance for a SATA SSD.
Digging deeper, all isn't well, though. Firing up Crystal Disk Info, HTWingNut noted that this SSD had a Hardware ECC Recovered value of over 400. In other words, the disk's error correction had to step in to fix hundreds of data-based parity bits.
...
As the worn SSD's data was being verified, there were already signs of performance degradation. The hashing audit eventually revealed that four files were corrupt (hash not matching). Looking at the elapsed time, it was observed that this operation astonishingly took over 4x longer, up from 10 minutes and 3 seconds to 42 minutes and 43 seconds.
Further investigations in HD Sentinel showed that three out of 10,000 sectors were bad and performance was 'spiky.' Returning to Crystal Disk Info, things look even worse. HTWingNut notes that the uncorrectable sectors count went from 0 to 12 on this drive, and the hardware ECC recovered value went from 11,745 before to 201,273 after tests on the day.
Note that the SSD that showed corrupted files was the one that had been worn out well beyond the manufacturer's max TBW rating (4× the TBW or so). There was a difference of two-to-three orders of magnitude in the ECC count between the "fresh" and the worn-out SSD; I'd call that very significant. It will be interesting to see if there's any update for late 2025.
I'd imagine full read of the whole device might trigger any self-preservation, but I'd also imagine it's heavily dependent on manufacturer and firmware
> Backing up your data is the simplest strategy to counteract the limitations of storage media. Having multiple copies of your data on different types of storage ensures that any unexpected incidents protect your data from vanishing forever. This is exactly what the 3-2-1 backup rule talks about: 3 copies of data on at least 2 different storage media, with 1 copy stored off-site.
Um. Backups seem like exactly why I might have data on an unpowered SSD.
I use HDDs right now because they're cheaper, but that might not be true some day. Also, I would expect someone less technically inclined than I am to just use whatever they have lying around, which may well be an SSD.
Powering the SSD on isn't enough. You need to read every bit occasionally in order to recharge the cell. If you have them in a NAS, then using a monthly full volume check is probably sufficient.
It would surely depend on the SSD and the firmware it's running. I don't think you can entirely count on it. Even if it were working perfectly, and your strategy was to power the SSD on periodicially to refresh the cells, how would you know when it had finished?
NVMe has read recovery levels (RRLs) and two different self-test modes (short and long) but what both of those modes do is entirely up to the manufacturer. So I'd think the only way to actually do this is to have host software do it, no? Or would even that not be enough? I mean, in theory the firmware could return anything to the host but... That feels too much like a conspiracy to me?
Huh. I wonder if this is why I'd sometimes get random corruption on my laptop's SSD. I'd reboot after a while and fsck would find issues in random files I haven't touched in a long time.
If you're getting random corruption like that, you should replace the SSD. SSDs (and also hard drives) already have built-in ECC, so if you're getting errors on top, it not just random cosmic rays. It's your SSD being extra broken, and doesn't bode too well for the health of the SSD as a whole.
I bought a replacement but never bothered swapping it. The weird thing is the random corruption stopped happening a few years ago (confirmed against old backups, so it's not like I'm just not noticing).
It's quite possible. Some SSDs are worse offenders for this than others. I have some Samsung 870 EVOs that lost data the way you described. Samsung knew about the issue and quietly swept it under the rug with a firmware update, but once the data was lost, it was gone for good.
I ran into this firmware bug with the two drives in my computer. They randomly failed after a while -- and by "a while" I mean less than a year of usage. Took two replacements before I finally realized that I should check for an fw update
It found problems in the tree - lost files, wrong node counts, other stuff - which led to me finding files that didn't match previous backups (and when opened were obviously corrupted, like the bottom half of an image being just noise). Once I found this was a problem I've also caught ones that couldn't be read (IOError) that fsck would delete on the next run.
I may not have noticed had fsck not alerted me something was wrong.
But metadata is data too, right? I guess the next question is, would it be possible for parts of the FS metadata to remain untouched for a time long enough for the SSD data corruption process to occur.
The article implies this is not a concern for "regular" people. That is absolutely false. How many people get their family photos when they finally decide to recycle that 15 year old PC in the basement?
How many people have a device that they may only power up ever few years, like on vacation. In fact, I have a device that I've only used on rare occasions these days (an arcade machine) that now I suspect I'll have to reinstall since It's been 2 or 3 years since I've last used it.
This is a pretty big deal that they don't put on the box.
Flash is programmed by increasing the probability that electrons will tunnel onto the floating gate and erased by increasing the probability they will tunnel back off. Those probabilities are never zero. Multiply that by time and the number of cells, and the probability you don’t end up with bit errors gets quite low.
The difference between slc and mlc is just that mlc has four different program voltages instead of two, so reading back the data you have to distinguish between charge levels that are closer together. Same basic cell design. Honestly I can’t quite believe mlc works at all, let alone qlc. I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
All the big 3D NAND makers have already switched from floating gate to charge trapping. Basically the same as what you describe but basically the electrons get stuck in a non-conductive region instead of on an insulated gate.
> I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
You can run an error-correcting code on top of the regular blocks of memory, storing, for example (really an example; I don’t know how large the ‘blocks’ that you can erase are in flash memory), 4096 bits in every 8192 bits of memory, and recovering those 4096 bits from each block of 8192 bits that you read in the disk driver. I think that would be better than a simple “map low levels to 0, high levels to 1” scheme.
> I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
Loads of drives do this(or SLC) internally. Though it would be handy if a physical format could change the provisioning at the kernel accessible layer.
> Honestly I can’t quite believe mlc works at all, let alone qlc. I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
Manufacturers often do sell such pMLC or pSLC (p = pseudo) cells as "high endurance" flash.
tlc/qlc works just fine, it's really difficult to consume the erase cycles unless you really are writing 24/7 to the disk at hundred of megabytes a second
I've got some old SSDs just to test this myself, the old 256gb corsairs I tested previously were fine after a year and a half, but I might have misplaced them...(they only had 10% write life left, so no huge loss) the 512gb samsungs on my desk should be getting pretty ripe soon though, I'll have to check those too.
Hmm, so what about these modern high density hard drives which store track parameters for their servos in on-board flash (aka OptiNAND)? Do we get "spinning rust" which might loose the information where exactly it stored the data?
I thought that was an ancient issue with Samsung 740? I had that one and it was slowly losing speed when unpowered due to an accumulation of errors and rewriting the individual sectors once for the whole drive made it work fine for a year.
AIUI, informal tests have demonstrated quite a bit of data corruption in Flash drives that are literally so worn out that they might as well be about to fail altogether - well beyond any manufacturer's actual TBW specs - but not otherwise, least of all in new drives that are only written once over for the test. It seems that if you don't wear out your drive all that much you'll have far less to worry about.
dd if=$1 of=/dev/null iflag=direct bs=16M status="progress"
smartctl -a $1
If someone wants to properly study SSD data-retention they could encrypt the drive using plain dm-crypt and fill the encrypted volume with zeroes and check at some time point afterwards to see if there are any non-zero blocks. This is an accessible way (no programming involved) to let you write random data to the SSD and save it without actually saving the entire thing - just the key. It will also ensure maximum variance in charge levels of all the cells. Will also prevent the SSD from potentially playing tricks such as compression.
I had to search around and feel like a dork not knowing this. I have my data backed up, but I keep the SSDs because it's nice to have the OS running like it was... I guess I need to be cloning the drives to ISOs and storing on spinning rust.
I learned this when both my old laptops would no longer boot after extended off power time (couple years). They were both stored in a working state and later both had SSDs that were totally dead.
I could be wrong, but I believe the general consensus is along the lines of "SSDs for in-use data, it's quicker and wants to be powered on often. HDDs for long-term storage, as they don't degrade when not in use nearly as fast as SSDs do.
I'd imagine HDDs also don't like not spinning for years(as mechanical elements generally like to be used from time to time). But at least platters itself are intact
I've been going through stack of external USB drives with laptop disks in them. They're all failing in some form or another. I'm going to have to migrate it all to a NAS with server-class drives I guess
At the very least, you can usually still get the data off of them. Most SSDs I've encountered with defects failed catastrophically, rendering the data completely inaccessible.
why would it not? it's a low level tool to do exactly that. you could "of" it to somewhere else if you're worried it's not. I like to | hexdump -C, on an xterm set to a green font on a black background for a real matrix movie kind of feel.
Quick note to not store any valuable data on a single drive. And when you store it on two drives, don't use the same kind of drive. (Speaking from bitter experience using spinning drives in servers that had a firmware bug where they all died at the time number of seconds of power-on time).
I assume "couple of" was figurative, to indicate the cost is substantially less than managing your own bank of SSDs and ensuring it is periodically powered etc.
[Edit: LOL, I see someone else posted literally the same example within the same minute. Funny coincidences.]
That said, they could also be storing relatively small amounts. For example, I back up to Backblaze B2, advertised at $6/TB/month, so ~300 MB at rest will be a "couple" bucks.
Recently started fiddling with restic and B2, it worked fairly seamlessly once I stopped trying too hard being fancy with permissions and capabilities (cap_dac_read_search). There were some conflicts trying to have both "the way that works interactively" [0] versus "the way that works well with systemd". [AmbientCapabilities=]
One concern I have is B2's downloading costs means verifying remote snapshots could get expensive. I suppose I could use `restic check --read-data-subset X` to do a random spot-check of smaller portions of the data, but I'm not sure how valuable that would be.
I like how it resembles LUKS encryption, where I can have one key for the automated backup process, and a separate memorize-only passphrase for if things go Very Very Wrong.
Do we actually know the clouds will do this? S3 is just about coming to its 20th anniversary.
Long enough to experience data rot to a small degree but realistically what proportion of users have archived things away for 10+ years then audited the fidelity of their data on retrieval after fetching it from Glacier
> Even the cheapest SSDs, say those with QLC NAND, can safely store data for about a year of being completely unpowered. More expensive TLC NAND can retain data for up to 3 years, while MLC and SLC NAND are good for 5 years and 10 years of unpowered storage, respectively.
This is somewhat confused writing. Consumer SSDs usually do not have a data retention spec, even in this very detailed Micron datasheet you won't find it: https://advdownload.advantech.com/productfile/PIS/96FD25-S2T...
Meanwhile the data retention spec for enterprise SSDs is at the end of their rated life, which is usually a DPWD/TBW intensity you won't reach in actual use anyway - that's where numbers like "3 months @ 50 °C" or whatever come from.
In practice, SSDs don't tend to loose data over realistic time frames. Don't hope for a "guaranteed by design" spec on that though, some pieces of silicon are more equal than others.
Any given TBW/DWPD values are irrelevant for unpowered data retention. Afaik, nobody gives these values in their datasheet and I'm wondering where their numbers are from, because I've never seen anything official. At this point I'd need to be convinced that the manufacturers even know themselves internally, because it's never been mentioned by them and it seems to be outside the intended use cases for SSDs
One key point about retention which is not often mentioned, and indeed neither does this article, is that retention is inversely proportional to program/erase cycles and decreases exponentially with increasing temperature. Hence why retention specs are usually X amount of time after Y cycles at Z temperature. Even a QLC SSD that has only been written to once, and kept in a freezer at -40, may hold data for several decades.
Manufacturers have been playing this game with DWPD/TBW numbers too --- by reducing the retention spec, they can advertise a drive as having a higher endurance with the exact same flash. But if you compare the numbers over the years, it's clear that NAND flash has gotten significantly worse; the only thing that has gone up, multiplicatively, is capacity, while endurance and rentention have both gone down by a few orders of magnitude.
For a long time, 10 years after 100K cycles was the gold standard of SLC flash.
Now we are down to several months after less than 1K cycles for QLC.
I also seem to remember reading retention is proportional to temperature at time of write. Ie, best case scenario = write data when drive is hot, and store in freezer. Would be happy if someone can confirm or deny this.
That's probably this: https://www.sciencedirect.com/science/article/abs/pii/S00262...
I definitely remember seeing exactly this.
Endurance going down is hardly a surprise given that the feature size has gone down too. The same goes for logic and DRAM memory.
I suspect that 2035 years time, hardware from 2010 will work, while that from 2020 will be less reliable.
Completely anecdotal, and mostly unrelated, but my NES from 1990 is still going strong. Two PS3’s that I have owned simply broke.
CRTs from 1994 and 2002 still going strong. LCD tvs from 2012 and 2022 just went kaput for no reason.
Old hardware rocks.
As far as I'm aware flash got a bit of a size boost when it went 3D and hasn't shrunk much since then. If you use the same number of bits per cell, I don't know if I would expect 2010 and 2020 or 2025 flash to vary much in endurance.
For logic and DRAM the biggest factors are how far they're being pushed with voltage and heat, which is a thing that trends back and forth over the years. So I could see that go either way.
> Even a QLC SSD that has only been written to once, and kept in a freezer at -40, may hold data for several decades.
So literally put your data in cold storage.
So on the off-chance that there's a firmware engineer in here, how does this actually work?
Like does a SSD do some sort of refresh on power-on, or every N hours, or you have to access the specific block, or...? What if you interrupt the process, eg, having a NVMe in an external case that you just plug once a month for a few minutes to just use it as a huge flash drive, is that a problem?
What about the unused space, is a 4 TB drive used to transport 1 GB of stuff going to suffer anything from the unused space decaying?
It's all very unclear about what all of this means in practice and how's an user supposed to manage it.
SSD firmware engineer here. I work on enterprise stuff, so ymmv on consumer grade internals.
Generally, the data refresh will all happen in the background when the system is powered (depending on the power state). Performance is probably throttled during those operations, so you just see a slightly slower copy while this is happening behind the scenes.
The unused space decaying is probably not an issue, since the internal filesystem data is typically stored on a more robust area of media (an SLC location) which is less susceptible to data loss over time.
As far as how a user is supposed to manage it, maybe do an fsck every month or something? Using an SSD like that is probably ok most of the time, but might not be super great as a cold storage backup.
So say I have a 4TB USB SSD from a few years ago, that's been sitting unpowered in a drawer most of that time. How long would it need to be powered on (ballpark) for the full disk refresh to complete? Assume fully idle.
(As a note: I do have a 4TB USB SSD which did sit in a drawer without being touched for a couple of years. The data was all fine when I plugged it back in. Of course, this was a new drive with very low write cycles and stored climate controlled. Older worn out drive would probably have been an issue.) Just wondering how long I should keep it plugged in if I ever have a situation like that so I can "reset the fade clock" per se.
So you need to do an fsck? My big question after reading this article (and others like it) is whether it is enough to just power up the device (for how long?), or if each byte actually needs to be read.
The case an average user is worried about is where they have an external SSD that they back stuff up to on a relatively infrequent schedule. In that situation, the question is whether just plugging it and copying some stuff to it is enough to ensure that all the data on the drive is refreshed, or if there's some explicit kind of "maintenance" that needs to be done.
>Generally, the data refresh will all happen in the background when the system is powered (depending on the power state).
How does the SSD know when to run the refresh job? AFAIK SSDs don't have an internal clock so it can't tell how long it's been powered off. Moreover does doing a read generate some sort of telemetry to the controller indicating how strong/weak the signal is, thereby informing whether it should refresh? Or does it blindly refresh on some sort of timer?
Pretty much, but it depends a lot on the vendor and how much you spent on the drive. A lot of the assumptions about enterprise SSDs is that they’re powered pretty much all the time, but are left in a low power state when not in use. So, data can still be refreshed on a timer, as long as it happens within the power budget.
There are several layers of data integrity that are increasingly expensive to run. Once the drive tries to read something that requires recovery, it marks that block as requiring a refresh and rewrites it in the background.
How long does the data refresh take, approx? Let's say I have an external portable SSD that I keep stored data on. Would plugging the drive into my computer and running
work to refresh any bad blocks internally?A full read would do it, but I think the safer recommendation is to just use a small hdd for external storage. Anything else is just dealing with mitigating factors
Typically unused empty space is a good thing, as it will allow drives to run in MLC or SLC mode instead of their native QLC. (At least, this seems to be the obvious implication from performance testing, given the better performance of SLC/MLC compared to QLC.) And the data remanence of SLC/MLC can be expected to be significantly better than QLC.
>as it will allow drives to run in MLC or SLC mode instead of their native QLC
That depends on the SSD controller implementation, specifically whether it proactively moves stuff from the SLC cache to the TLC/QLC area. I expect most controllers to do this, given that if they don't, the drive will quickly lose performance as it fills up. There's basically no reason not proactively move stuff over.
I assume this blog is a re-hash of the JDEC retention standards[1].
The more interesting thing to note from those standards is that the required retention period differs between "Client" and "Enterprise" category.
Enterprise category only has power-off retention requirement of 3 months.
Client category has power-off retention requirement of 1 year.
Of course there are two sides to every story...
Enterprise category standard has a power-on active use of 24 hours/day, but Client category only intended for 8 hours/day.
As with many things in tech.... its up to the user to pick which side they compromise on.
[1]https://files.futurememorystorage.com/proceedings/2011/20110...
> I assume this blog is a re-hash of the JDEC retention standards[1].
Specifically in JEDEC JESD218. (Write endurance in JESD219.)
With 1 year power-off retention you still loose data, so still a compromise on data retention
Is there a real source that confirms this with data? I generally like xda, but the quality of their articles is uneven and they trend towards click bait headlines that try to shock/surprise you with thin content underneath. There have been a string of "Here is the one piece of software you didn't know you needed for your NAS" and it turns out to be something extremely popular like home assistant.
This article just seems to link to a series of other xda articles with no primary source. I wouldn't ever trust any single piece of hardware to store my data forever but this feels like clickbait- At one point they even state "...but you shouldn't really worry about it..."
So, product idea: A powered "cold storage box" for M.2 SSDs. 2 to 8 M.2 slots. Periodically, an internal computer connects one of the slots, reads every byte, waits for some period of time, then powers off. Maybe shows a little green light next to each drive when the last read was successful. Could be battery-powered.
here's a better idea, buy a mechanical hard disk.
How much are you saving vs an always-on ARM board with M.2 slots? Is it worth the pennies?
> Could be battery-powered.
How often does it need to run? If it could be solar powered you could probably avoid a whole bunch of complexity per unit longevity.
What about powered SSDs that contain files that are rarely read?
My desktop computer is generally powered except when there is a power failure, but among the million+ files on its SSD there are certainly some that I do not read or write for years.
Does the SSD controller automatically look for used blocks that need to have their charge refreshed and do so, or do I need to periodically do something like "find / -type f -print0 | xargs -0 cat > /dev/null" to make sure every file gets read occasionally?
no, the firmware does any maintenance. good firmware should do gradual scrub whenever it's idle. unfortunately, there's no real way to know whether the firmware is good, or doing anything.
I wonder if there's some easy way to measure power consumed by a device - to detect whether it's doing housekeeping.
Honestly this is one of my favorite things about ZFS. I know that a disk scan is performed every week/month (whatever schedule). And I also know that it has verified the contents of each block. It is very reassuring in that way.
I also need an answer to this.
It's fine. But the whole drive can turn to dust at any time, of course.
You should absolutely be doing a full block read of your disk, dd if=/dev/disk of=/dev/null every couple weeks
pointless to do it by the default 512b block size though.
pretty accurate username
Amusing profile too. For some reason he calls us here on HN a "congress".
Is there any type of flash-based storage (preferably accessible to end users) that focuses on long term data retention?
If not, that feels like a substantial hole in the market. Non-flash durable storage tend to be annoying or impractical for day to day use. I want to be able to find a 25 year old SD card hiding in some crevice and unearth an unintentional time capsule, much like how one can pick up 20+ year old MiniDiscs and be able to play the last thing their former owners recorded to them perfectly.
what is the exact protocol to "recharge" an ssd which was offline for months?
do I just plug it in and let the computer on for a few minutes? does it needs to stay on for hours?
do I need to run a special command or TRIM it?
We really don't know. One thing I wish some of these sites would do is actually test how long it takes for the drives to decay and also do a retest after they have been left powered for say 10 minutes to an hour, read completely, written to a bit etc and see if they can determine what a likely requirement is.
The problem is the test will take years, be out of date by the time its released and new controllers will be out with potentially different needs/algorithms.
There was one guy who tested this
https://www.tomshardware.com/pc-components/storage/unpowered...
Note that the SSD that showed corrupted files was the one that had been worn out well beyond the manufacturer's max TBW rating (4× the TBW or so). There was a difference of two-to-three orders of magnitude in the ECC count between the "fresh" and the worn-out SSD; I'd call that very significant. It will be interesting to see if there's any update for late 2025.
I'd imagine full read of the whole device might trigger any self-preservation, but I'd also imagine it's heavily dependent on manufacturer and firmware
I too wonder about this. I'd love to see someone build a simulated "fast decay" SSD which can show how various firmware actually behaves.
Back it up. 1 2 3.
I would run something like CHKDSK, or write a script to calculate a hash of every file on disk.
No idea if that's enough, but it seems like a reasonable place to start.
> But, most people don't need to worry about it
> You should always have a backup anyway
> Backing up your data is the simplest strategy to counteract the limitations of storage media. Having multiple copies of your data on different types of storage ensures that any unexpected incidents protect your data from vanishing forever. This is exactly what the 3-2-1 backup rule talks about: 3 copies of data on at least 2 different storage media, with 1 copy stored off-site.
Um. Backups seem like exactly why I might have data on an unpowered SSD.
I use HDDs right now because they're cheaper, but that might not be true some day. Also, I would expect someone less technically inclined than I am to just use whatever they have lying around, which may well be an SSD.
Powering the SSD on isn't enough. You need to read every bit occasionally in order to recharge the cell. If you have them in a NAS, then using a monthly full volume check is probably sufficient.
Isn't that the SSD controller's job?
It would surely depend on the SSD and the firmware it's running. I don't think you can entirely count on it. Even if it were working perfectly, and your strategy was to power the SSD on periodicially to refresh the cells, how would you know when it had finished?
NVMe has read recovery levels (RRLs) and two different self-test modes (short and long) but what both of those modes do is entirely up to the manufacturer. So I'd think the only way to actually do this is to have host software do it, no? Or would even that not be enough? I mean, in theory the firmware could return anything to the host but... That feels too much like a conspiracy to me?
Do you know any firmware engineers?
Huh. I wonder if this is why I'd sometimes get random corruption on my laptop's SSD. I'd reboot after a while and fsck would find issues in random files I haven't touched in a long time.
If you're getting random corruption like that, you should replace the SSD. SSDs (and also hard drives) already have built-in ECC, so if you're getting errors on top, it not just random cosmic rays. It's your SSD being extra broken, and doesn't bode too well for the health of the SSD as a whole.
I bought a replacement but never bothered swapping it. The weird thing is the random corruption stopped happening a few years ago (confirmed against old backups, so it's not like I'm just not noticing).
It's quite possible. Some SSDs are worse offenders for this than others. I have some Samsung 870 EVOs that lost data the way you described. Samsung knew about the issue and quietly swept it under the rug with a firmware update, but once the data was lost, it was gone for good.
Huh, I thought I got some faulty one, mine died shortly after warranty ended (and had a bunch of media errors before that)
I ran into this firmware bug with the two drives in my computer. They randomly failed after a while -- and by "a while" I mean less than a year of usage. Took two replacements before I finally realized that I should check for an fw update
Unless your setup is a very odd Linux box, fsck will never check the consistency of file contents.
It found problems in the tree - lost files, wrong node counts, other stuff - which led to me finding files that didn't match previous backups (and when opened were obviously corrupted, like the bottom half of an image being just noise). Once I found this was a problem I've also caught ones that couldn't be read (IOError) that fsck would delete on the next run.
I may not have noticed had fsck not alerted me something was wrong.
A ZFS scrub (default scheduled monthly) will do it.
But metadata is data too, right? I guess the next question is, would it be possible for parts of the FS metadata to remain untouched for a time long enough for the SSD data corruption process to occur.
The article implies this is not a concern for "regular" people. That is absolutely false. How many people get their family photos when they finally decide to recycle that 15 year old PC in the basement?
How many people have a device that they may only power up ever few years, like on vacation. In fact, I have a device that I've only used on rare occasions these days (an arcade machine) that now I suspect I'll have to reinstall since It's been 2 or 3 years since I've last used it.
This is a pretty big deal that they don't put on the box.
shameless plug of my anti-bitrot tool, which I am actually enhancing with a --daemon mode currently
https://github.com/pmarreck/bitrot_guard
Flash is programmed by increasing the probability that electrons will tunnel onto the floating gate and erased by increasing the probability they will tunnel back off. Those probabilities are never zero. Multiply that by time and the number of cells, and the probability you don’t end up with bit errors gets quite low.
The difference between slc and mlc is just that mlc has four different program voltages instead of two, so reading back the data you have to distinguish between charge levels that are closer together. Same basic cell design. Honestly I can’t quite believe mlc works at all, let alone qlc. I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
All the big 3D NAND makers have already switched from floating gate to charge trapping. Basically the same as what you describe but basically the electrons get stuck in a non-conductive region instead of on an insulated gate.
> I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
You can run an error-correcting code on top of the regular blocks of memory, storing, for example (really an example; I don’t know how large the ‘blocks’ that you can erase are in flash memory), 4096 bits in every 8192 bits of memory, and recovering those 4096 bits from each block of 8192 bits that you read in the disk driver. I think that would be better than a simple “map low levels to 0, high levels to 1” scheme.
I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
There is a way to turn QLC into SLC: https://news.ycombinator.com/item?id=40405578
Thanks! I missed this the first time around!
> I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
Loads of drives do this(or SLC) internally. Though it would be handy if a physical format could change the provisioning at the kernel accessible layer.
> Honestly I can’t quite believe mlc works at all, let alone qlc. I do wonder why there’s no way to operate qlc as if it were mlc, other than the manufacturer not wanting to allow it.
Manufacturers often do sell such pMLC or pSLC (p = pseudo) cells as "high endurance" flash.
the market demands mostly higher capacity
tlc/qlc works just fine, it's really difficult to consume the erase cycles unless you really are writing 24/7 to the disk at hundred of megabytes a second
I have a MLC SSD with TBW/GB much higher than the specified TBW/GB guarantee of usual qlc SSDs
I've got some old SSDs just to test this myself, the old 256gb corsairs I tested previously were fine after a year and a half, but I might have misplaced them...(they only had 10% write life left, so no huge loss) the 512gb samsungs on my desk should be getting pretty ripe soon though, I'll have to check those too.
Hmm, so what about these modern high density hard drives which store track parameters for their servos in on-board flash (aka OptiNAND)? Do we get "spinning rust" which might loose the information where exactly it stored the data?
https://blog.westerndigital.com/optinand-explained/
I wonder too. I don't trust SSDs/ flash for my archives, hence I'm stuck on max. 18TB drives atm.
shouldn't really be a problem, since the capacity required for this use is so low.
the real issue here is QLC in which the flash cell's margins are being squeezed enthusiastically...
I thought that was an ancient issue with Samsung 740? I had that one and it was slowly losing speed when unpowered due to an accumulation of errors and rewriting the individual sectors once for the whole drive made it work fine for a year.
I don't use my drive much. I still boot it up snd write some data, just not the long term one. Am I in risk?
AIUI, informal tests have demonstrated quite a bit of data corruption in Flash drives that are literally so worn out that they might as well be about to fail altogether - well beyond any manufacturer's actual TBW specs - but not otherwise, least of all in new drives that are only written once over for the test. It seems that if you don't wear out your drive all that much you'll have far less to worry about.
My scrub script:
If someone wants to properly study SSD data-retention they could encrypt the drive using plain dm-crypt and fill the encrypted volume with zeroes and check at some time point afterwards to see if there are any non-zero blocks. This is an accessible way (no programming involved) to let you write random data to the SSD and save it without actually saving the entire thing - just the key. It will also ensure maximum variance in charge levels of all the cells. Will also prevent the SSD from potentially playing tricks such as compression.I had to search around and feel like a dork not knowing this. I have my data backed up, but I keep the SSDs because it's nice to have the OS running like it was... I guess I need to be cloning the drives to ISOs and storing on spinning rust.
I learned this when both my old laptops would no longer boot after extended off power time (couple years). They were both stored in a working state and later both had SSDs that were totally dead.
Were the SSDs toasted, or were you able to reinstall to them?
I could be wrong, but I believe the general consensus is along the lines of "SSDs for in-use data, it's quicker and wants to be powered on often. HDDs for long-term storage, as they don't degrade when not in use nearly as fast as SSDs do.
I'd imagine HDDs also don't like not spinning for years(as mechanical elements generally like to be used from time to time). But at least platters itself are intact
I've been going through stack of external USB drives with laptop disks in them. They're all failing in some form or another. I'm going to have to migrate it all to a NAS with server-class drives I guess
At the very least, you can usually still get the data off of them. Most SSDs I've encountered with defects failed catastrophically, rendering the data completely inaccessible.
or you could power them on 1-2x /year.
Power them on and run something to exercise the read function over every bit. Thats why a ZFS filesystem integrity check/scrub is the useful model.
I'm unsure if dd if=/the/disk of=/dev/null does the read function.
why would it not? it's a low level tool to do exactly that. you could "of" it to somewhere else if you're worried it's not. I like to | hexdump -C, on an xterm set to a green font on a black background for a real matrix movie kind of feel.
Not having a (verified) backup is driving without a seatbelt.
Quick note to not store any valuable data on a single drive. And when you store it on two drives, don't use the same kind of drive. (Speaking from bitter experience using spinning drives in servers that had a firmware bug where they all died at the time number of seconds of power-on time).
That's kicking the can down the road for double the cost. Only a backup on spinning rust is actually a backup.
Furthermore, replication isn't a backup.
What is the best way to store data for a long time then?
Depending on your time requirements, carving it into a stone tablet is generally a good choice
all the major players say "tape". (but that's partly for practical issues like scaling and history)
This is why I would rather pay someone a couple of dollars per year to handle all this for me. If need be pay two providers to have a backup.
Who do you pay for this? (To rephrase : which cloud storage vendors do you use?) interested in the $2/month price point :)
I assume "couple of" was figurative, to indicate the cost is substantially less than managing your own bank of SSDs and ensuring it is periodically powered etc.
[Edit: LOL, I see someone else posted literally the same example within the same minute. Funny coincidences.]
That said, they could also be storing relatively small amounts. For example, I back up to Backblaze B2, advertised at $6/TB/month, so ~300 MB at rest will be a "couple" bucks.
tell me about this $2/week filestore option. I'm interested.
continuing the bizarre trend, I'm here for the $2/day deal
That would be Tarsnap. Cool product and the owner is a good dude, but good Lord is it expensive. I would love to support him but just can't afford it.
I'D love to be paying $2/minute!
Backblaze B2 is $6TB/mo, so if you have around 300GB... stuff like restic or kopia backups nicely to it
Recently started fiddling with restic and B2, it worked fairly seamlessly once I stopped trying too hard being fancy with permissions and capabilities (cap_dac_read_search). There were some conflicts trying to have both "the way that works interactively" [0] versus "the way that works well with systemd". [AmbientCapabilities=]
One concern I have is B2's downloading costs means verifying remote snapshots could get expensive. I suppose I could use `restic check --read-data-subset X` to do a random spot-check of smaller portions of the data, but I'm not sure how valuable that would be.
I like how it resembles LUKS encryption, where I can have one key for the automated backup process, and a separate memorize-only passphrase for if things go Very Very Wrong.
[0] https://restic.readthedocs.io/en/latest/080_examples.html#ba...
$72/yr is somewhere around 3x purchase price (per year). BB seems to be pretty smart about managing their exposure, infrastructure overhead, etc.
Do we actually know the clouds will do this? S3 is just about coming to its 20th anniversary.
Long enough to experience data rot to a small degree but realistically what proportion of users have archived things away for 10+ years then audited the fidelity of their data on retrieval after fetching it from Glacier
good to know but apart from some edge cases this doesnt matter that much
> Even the cheapest SSDs, say those with QLC NAND, can safely store data for about a year of being completely unpowered. More expensive TLC NAND can retain data for up to 3 years, while MLC and SLC NAND are good for 5 years and 10 years of unpowered storage, respectively.
This is somewhat confused writing. Consumer SSDs usually do not have a data retention spec, even in this very detailed Micron datasheet you won't find it: https://advdownload.advantech.com/productfile/PIS/96FD25-S2T... Meanwhile the data retention spec for enterprise SSDs is at the end of their rated life, which is usually a DPWD/TBW intensity you won't reach in actual use anyway - that's where numbers like "3 months @ 50 °C" or whatever come from.
In practice, SSDs don't tend to loose data over realistic time frames. Don't hope for a "guaranteed by design" spec on that though, some pieces of silicon are more equal than others.
Any given TBW/DWPD values are irrelevant for unpowered data retention. Afaik, nobody gives these values in their datasheet and I'm wondering where their numbers are from, because I've never seen anything official. At this point I'd need to be convinced that the manufacturers even know themselves internally, because it's never been mentioned by them and it seems to be outside the intended use cases for SSDs