It's crazy to me that rsync.net is buying mission-critical enterprise drives on Amazon.
I don't buy drives on Amazon for my 9 year old's laptop because of the rampant fraud and counterfeiting, I'm shocked that they're trusted for any business use-cases by anyone moderately savvy. I'm even more shocked that the takeaway is to blame the individual seller, rather than the marketplace that makes it possible.
As I pointed out in another post if you are using ZFS RAID Z2 you can use literal used garbage hard drives safely without risking data loss.
ZFS helped me discover that my motherboard SATA chip can't handle 6 drives; I had to purchase a cheap Chinese PCI Express SATA controller to communicate with my drives reliably and error-free.
This is how we operate at my job. We go through our trusted and reliable vendor, who gets us good pricing but doesn't always control shipping times. If it's urgent, Amazon will be delivered within 48 hrs.
> I'm shocked that they're trusted for any business use-cases by anyone moderately savvy.
I buy drives on Amazon all the time. I check them all. Never had any problems.
The mistake they made was buying not from Amazon, but from "Maestro Technology" listing on Amazon. If you understand that Amazon is a marketplace and you take 10 seconds to read who you're buying from, it's not a problem.
Amazon returns are also extremely easy. I once gambled on a sketchy seller and received a bad product (not computer related). A couple clicks and it was on its way back for a refund.
The problems with inventory commingling are virtually a thing of the past. I went through the process of selling a product on Amazon and understanding their evolved inventory labeling and commingling procedures so I'm not worried. Many of the tech community are anchored to news articles from years ago, though.
If you have a highly trusted vendor who can deliver at great prices and have products in stock that show up at your door when you need them, then use that. For the rest of us, using Amazon to buy common parts isn't really the problem that it's made out to be in HN comments. I think a lot of people here only understand Amazon through the occasional article that makes it to the top of HN and they don't understand what it's really like because they've been too scared to use it for years.
I have never gone through the process of selling on Amazon. For those of us anchored to news articles from decades in the past, is there any public documentation of these procedures?
I buy used enterprise hard drives that have been pushed hard. My current biggest NAS runs six used 14 terabyte enterprise hard drives and three have failed so far within a year. Each time I was able to get a warranty replacement and the replacement was in much better condition than the original ones I had. Zero data loss because of ZFS RAID Z2. I was able to measure the condition of the surface of the platters and other useful metadata using Victoria https://hdd.by/victoria/ included on Hiren's BootCD PE.
It almost doesn't matter. There's three choices for spinning drives and all storage devices fail (even SSDs), and you need a plan for that. There's variance in failure rates and warranty periods, but most (new) drives last the warranty period, and used drives is more complex than just manufacturer --- how were they used, stored, and handled before resale probably makes more difference than who made them.
IMHO, it's more important to consider correlated failures rather than worry about getting the best or avoiding the worst drives. Try to avoid running an array that's built from same model, same firmware, same build time, same power on hours, same workload. Every so often, you get things like drive disappears when power on time overflows, or a manufacturing error that makes most drives fail after N weeks of use, having all of your drives in the same part of their lifecycle makes you more likely to experience a catastrophic failure that failure rate analysis wouldn't pick up.
Picking SSDs carefully bears more fruit, there are many more makers and wider variance in performance and reliability as well as characteristics during failure: everyone says SSDs go read only during failure, but my experience has been that lots of SSDs disappear from their interfaces during failure; you might reasonably have less redundancy if you have confidence the SSD will remain readable for recovery if it fails.
I just get whatever is cheapest with a warranty of one or more years. RMA ability is most important when buying these kinds of heavily used decommissioned drives. ZFS RAID Z2 is absolutely key to preventing data loss here.
> For those of you tracking the stats closely, you’ll notice that the Seagate ST8000NM000A (8TB) is a frequent flier on this list. The last time it had a failure was in Q3 2024—and it was just a single failure for the whole quarter!
> blame the individual seller, rather than the marketplace
I'd have thought the fraud problems from "commingling" were well-enough known by now to avoid wanting to blame any specific Amazon Marketplace vendor, but perhaps not.
I mean, if you're a storage business, hopefully you've designed your architecture such that you assume drives will go bad, so you characterize the models of drive to make sure that not all the copies are on one manufacturer, and then you can take liberties finding the cheapest storage on the market. This only comes back to bite you when you didn't account for (because you didn't know) that there was decreased longevity, so your TCO calculation was off and you might not make as much money.
However, an even more fundamental philosophy behind any work that we do is "defense in depth" which means that even after building the fault-tolerant, anti-fragile system, we also spend time and resources qualifying the inputs ...
... and then spend time and resources monitoring the outputs (error rates, failures, correlation, etc.).
Any one of those pieces is, theoretically, sufficient. Layering the pieces in a defense in depth strategy is what gives us the highly confident posture we enjoy.
Jumping on two add I've now had 2 experiences of buying counterfeit drives on Amazon from different sellers. I've probably ordered only 4 hard drives off of Amazon ever, so that's a 50% counterfeit rate.
Both sellers issued refunds without trouble because I bought them under Prime. One seller seemed genuinely surprised they had a counterfeit in stock.
Both of these counterfeit drives look very, very convincingly authentic, except the serial numbers don't match real ones and don't validate as real ones with the OEM.
The first time, I actually argued with Seagate over it being real, until they pointed out that aside from the serial number not being in their databases, it's not even in the correct format for any of their drives.
If you care about your drives and you're buying on Amazon, only buy under Prime. And when they're delivered, check the serial numbers with the OEM first thing (usually via warranty validation). Don't buy anything not on Prime.
"Other than returning the four parts for a refund (which we did) and documenting this behavior here, our only other recourse was to guarantee that these four specific parts were never sold as new again:"
Alas, one can completely remove Sharpie writing from metal with 99% isopropyl alcohol. Did they make a better choice? This looks like Sharpie writing to me.
You can also remove sharpie writing using sharpies and a wet wipe - write over and wipe while its still wet. The dry pigment will dissolve in the solvent in the fresh ink.
One spray of brake cleaner and it's gone. Cheaper to use a carbide scribe or simply get a shard of glass or ceramic and physically scribe into the drive. Now they're permanently used drives, no debate is possible.
I bought a stack of WD gold drives several years ago that had several thousand hours on them as well. I believe I got those off Newegg. When I asked, they said something about initial testing, but why didn't they reset the counters before selling them? Who knows.
> I bought a stack of WD gold drives several years ago that had several thousand hours on them as well. I believe I got those off Newegg. When I asked, they said something about initial testing, but why didn't they reset the counters before selling them? Who knows.
Thousands of hours doesn't pass the smell test. There's no way a specific SSD goes through months of testing prior to sale. A couple of hours seems reasonable though. And I'd rather it not be easy to reset the counters, so they don't reset the counters after testing during manufacturing/burn-in.
Back in my NOC tech/datacenter days, we grew to trust drives with a combination of 10000+ poweron hrs + {zero SMART errors / zero reallocated sectors / zero pending sectors} actually more than a random unknown new drive.
Can you flash fake SMART data to drives? I suspect that's exactly what Maestro will start doing now (although it's possible it's not worth the effort for the small number of customers who will actually check this stuff).
Is there any way to tell if the SMART has been reset/tampered with in any way? If you have a drive that claims to have 0 hours of use, but it quickly starts to rack up failure indicators… how can you tell if it’s spurious failures or a fraudulent tampered drive?
Seagate has a proprietary version of SMART called FARM. It’s supposed to be more tamper resistant than SMART, but it appears the fraudsters have figured out how to manipulate it too [1].
The best you can do is check FARM if available and perform a long burn-in with something like badblocks. Then compare the SMART data before and after the burn in. Checking the serial number against the manufacturers database if available is also a good precaution.
These are probably things you should be doing whether or not the drive is allegedly new.
That was my first thought too. Apparently they’re “phasing it out” by “the end of this year” [0]
I did not know, per that article, that Amazon had for some time now offered motivated third-party sellers a means to avoid commingling by applying a “fulfillment network SKU” barcode to their goods. And that they estimate merchants spend $600mm a year on that type of “restickering.” Expensive, but possible.
It is WILD that anyone in tech assumes this will come as new. Simply no one makes the same model of "consumable" for 7 years. Intel doesn't even sell Intel-branded SSDs anymore, that division was spinned off.
It's also WILD that you would trust something as sketchy sounding as "Maestro Technologies" for a mission-critical task.
First of all, neither of those WILD facts seem that wild to me.
Intel did last orders for that drive Dec 30 2022. The article was written in April, so the author was conceivably purchasing drives that had sat on a shelf for a year and a half. That doesn't tickle alarm bells in my head.
Secondly, maybe my scam detector isn't well tuned enough, but "Maestro Technologies" doesn't seem that much stranger than "Apple" or "Micro soft" or "Zoom" or "Snap." If it were XBBHHZZZAA, LLC, maybe I'd have more room for pause.
The takeaway lesson here is that Amazon has become less and less reliable as a source for items. It's especially bad if it's purchased from a third party (something Amazon seems keen not to highlight on the purchase page), but even FBA is not free of trash. They straight up sell pirated N64 cartridges for example: https://www.amazon.com/Cartridge-Nintendo-Smash-64-Video-Ver...
I got bit by this exact seller on Amazon earlier this year.
* I reported it to Amazon's fraud team (with evidence): No response
* I wrote the seller a bad review (also with evidence!): Taken down by Amazon
https://x.com/johnboiles/status/1879713174427214131?s=20
It's crazy to me that rsync.net is buying mission-critical enterprise drives on Amazon.
I don't buy drives on Amazon for my 9 year old's laptop because of the rampant fraud and counterfeiting, I'm shocked that they're trusted for any business use-cases by anyone moderately savvy. I'm even more shocked that the takeaway is to blame the individual seller, rather than the marketplace that makes it possible.
First two lines of the article:
>At rsync.net we have trusted suppliers with verified supply chains and a long history of providing reliable service.
>However, from time to time, it is expedient to purchase parts from Amazon - something we do with care and suspicion.
That seems like a very reasonable and non-crazy approach to using Amazon.
As I pointed out in another post if you are using ZFS RAID Z2 you can use literal used garbage hard drives safely without risking data loss.
ZFS helped me discover that my motherboard SATA chip can't handle 6 drives; I had to purchase a cheap Chinese PCI Express SATA controller to communicate with my drives reliably and error-free.
Not for special VDEVs, which is the explicit purpose here.
You can mirror a slog... l2arc doesn't need a mirror because the data is already on disk. I believe a split metadata can also be mirrored.
That's correct. I ran a special metadata VDEV 3-way mirror using this NVMe PLX card for a while https://imgur.com/a/xiwzkA6
You can dump any vdev you want in a special. I have 2x 3 way mirrors in both of my nas boxes right now
I'd be more worried about a supply chain attack with malicious devices.
What kind of supply chain attack can one mount with a disk?
A malicious device plugged into your machine? The only limit is your imagination.
This is how we operate at my job. We go through our trusted and reliable vendor, who gets us good pricing but doesn't always control shipping times. If it's urgent, Amazon will be delivered within 48 hrs.
> I'm shocked that they're trusted for any business use-cases by anyone moderately savvy.
I buy drives on Amazon all the time. I check them all. Never had any problems.
The mistake they made was buying not from Amazon, but from "Maestro Technology" listing on Amazon. If you understand that Amazon is a marketplace and you take 10 seconds to read who you're buying from, it's not a problem.
Amazon returns are also extremely easy. I once gambled on a sketchy seller and received a bad product (not computer related). A couple clicks and it was on its way back for a refund.
The problems with inventory commingling are virtually a thing of the past. I went through the process of selling a product on Amazon and understanding their evolved inventory labeling and commingling procedures so I'm not worried. Many of the tech community are anchored to news articles from years ago, though.
If you have a highly trusted vendor who can deliver at great prices and have products in stock that show up at your door when you need them, then use that. For the rest of us, using Amazon to buy common parts isn't really the problem that it's made out to be in HN comments. I think a lot of people here only understand Amazon through the occasional article that makes it to the top of HN and they don't understand what it's really like because they've been too scared to use it for years.
I have never gone through the process of selling on Amazon. For those of us anchored to news articles from decades in the past, is there any public documentation of these procedures?
The problem is that you might not be able to tell a used drive from a new drive if the scammer bothered to reset the SMART data.
I buy used enterprise hard drives that have been pushed hard. My current biggest NAS runs six used 14 terabyte enterprise hard drives and three have failed so far within a year. Each time I was able to get a warranty replacement and the replacement was in much better condition than the original ones I had. Zero data loss because of ZFS RAID Z2. I was able to measure the condition of the surface of the platters and other useful metadata using Victoria https://hdd.by/victoria/ included on Hiren's BootCD PE.
Which manfucaturer is the best and worst one?
It almost doesn't matter. There's three choices for spinning drives and all storage devices fail (even SSDs), and you need a plan for that. There's variance in failure rates and warranty periods, but most (new) drives last the warranty period, and used drives is more complex than just manufacturer --- how were they used, stored, and handled before resale probably makes more difference than who made them.
IMHO, it's more important to consider correlated failures rather than worry about getting the best or avoiding the worst drives. Try to avoid running an array that's built from same model, same firmware, same build time, same power on hours, same workload. Every so often, you get things like drive disappears when power on time overflows, or a manufacturing error that makes most drives fail after N weeks of use, having all of your drives in the same part of their lifecycle makes you more likely to experience a catastrophic failure that failure rate analysis wouldn't pick up.
Picking SSDs carefully bears more fruit, there are many more makers and wider variance in performance and reliability as well as characteristics during failure: everyone says SSDs go read only during failure, but my experience has been that lots of SSDs disappear from their interfaces during failure; you might reasonably have less redundancy if you have confidence the SSD will remain readable for recovery if it fails.
I just get whatever is cheapest with a warranty of one or more years. RMA ability is most important when buying these kinds of heavily used decommissioned drives. ZFS RAID Z2 is absolutely key to preventing data loss here.
https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-...
> For those of you tracking the stats closely, you’ll notice that the Seagate ST8000NM000A (8TB) is a frequent flier on this list. The last time it had a failure was in Q3 2024—and it was just a single failure for the whole quarter!
Check the Backblaze quarterly reports.
Marketplaces only work when the participants maintain a reputation. The buyer here is doing his part.
See also: https://news.ycombinator.com/item?id=45896707 (HDD shortage)
> blame the individual seller, rather than the marketplace
I'd have thought the fraud problems from "commingling" were well-enough known by now to avoid wanting to blame any specific Amazon Marketplace vendor, but perhaps not.
"It's crazy to me that rsync.net is buying mission-critical enterprise drives on Amazon."
We don't.
"At rsync.net we have trusted suppliers with verified supply chains and a long history of providing reliable service."
...
"However, from time to time, it is expedient to purchase parts from Amazon - something we do with care and suspicion."
... and that care and suspicion takes the form of physical and logical inspections and extended part burn-in.
As you can see, this QC process caught these mis-labeled parts.
I mean, if you're a storage business, hopefully you've designed your architecture such that you assume drives will go bad, so you characterize the models of drive to make sure that not all the copies are on one manufacturer, and then you can take liberties finding the cheapest storage on the market. This only comes back to bite you when you didn't account for (because you didn't know) that there was decreased longevity, so your TCO calculation was off and you might not make as much money.
I agree with this.
However, an even more fundamental philosophy behind any work that we do is "defense in depth" which means that even after building the fault-tolerant, anti-fragile system, we also spend time and resources qualifying the inputs ...
... and then spend time and resources monitoring the outputs (error rates, failures, correlation, etc.).
Any one of those pieces is, theoretically, sufficient. Layering the pieces in a defense in depth strategy is what gives us the highly confident posture we enjoy.
from time to time your trusted supplier might be out of stock and you need drivers quickly
even backblaze bought drives in supermarket when there was HDD shortage
Jumping on two add I've now had 2 experiences of buying counterfeit drives on Amazon from different sellers. I've probably ordered only 4 hard drives off of Amazon ever, so that's a 50% counterfeit rate.
Both sellers issued refunds without trouble because I bought them under Prime. One seller seemed genuinely surprised they had a counterfeit in stock.
Both of these counterfeit drives look very, very convincingly authentic, except the serial numbers don't match real ones and don't validate as real ones with the OEM.
The first time, I actually argued with Seagate over it being real, until they pointed out that aside from the serial number not being in their databases, it's not even in the correct format for any of their drives.
If you care about your drives and you're buying on Amazon, only buy under Prime. And when they're delivered, check the serial numbers with the OEM first thing (usually via warranty validation). Don't buy anything not on Prime.
"Other than returning the four parts for a refund (which we did) and documenting this behavior here, our only other recourse was to guarantee that these four specific parts were never sold as new again:"
Alas, one can completely remove Sharpie writing from metal with 99% isopropyl alcohol. Did they make a better choice? This looks like Sharpie writing to me.
You can also remove sharpie writing using sharpies and a wet wipe - write over and wipe while its still wet. The dry pigment will dissolve in the solvent in the fresh ink.
My favourite trick is to write over Sharpie with a dry erase marker and erase it all.
https://www.amazon.com/Janlaugh-Resistant-Laboratory-Permane...
Alcohol/solvent resistant markers.
One spray of brake cleaner and it's gone. Cheaper to use a carbide scribe or simply get a shard of glass or ceramic and physically scribe into the drive. Now they're permanently used drives, no debate is possible.
But are they genuine and unused? !
They can do it but they likely won't bother at scale.
That’s your takeaway?
Yep. They should've engraved it.
I bought a stack of WD gold drives several years ago that had several thousand hours on them as well. I believe I got those off Newegg. When I asked, they said something about initial testing, but why didn't they reset the counters before selling them? Who knows.
> I bought a stack of WD gold drives several years ago that had several thousand hours on them as well. I believe I got those off Newegg. When I asked, they said something about initial testing, but why didn't they reset the counters before selling them? Who knows.
Thousands of hours doesn't pass the smell test. There's no way a specific SSD goes through months of testing prior to sale. A couple of hours seems reasonable though. And I'd rather it not be easy to reset the counters, so they don't reset the counters after testing during manufacturing/burn-in.
Back in my NOC tech/datacenter days, we grew to trust drives with a combination of 10000+ poweron hrs + {zero SMART errors / zero reallocated sectors / zero pending sectors} actually more than a random unknown new drive.
Might've worked for HDDs, but SSDs have a cap on lifetime writes.
Pre-tested, as it were?
https://en.wikipedia.org/wiki/Bathtub_curve
Whats stopping a more clever company from resetting the smart data on an ssd and reselling?
Can you flash fake SMART data to drives? I suspect that's exactly what Maestro will start doing now (although it's possible it's not worth the effort for the small number of customers who will actually check this stuff).
Yes, most of the used Amazon SATA Rotated Drive its do this with "fresh" data.
Is there any way to tell if the SMART has been reset/tampered with in any way? If you have a drive that claims to have 0 hours of use, but it quickly starts to rack up failure indicators… how can you tell if it’s spurious failures or a fraudulent tampered drive?
Seagate has a proprietary version of SMART called FARM. It’s supposed to be more tamper resistant than SMART, but it appears the fraudsters have figured out how to manipulate it too [1].
The best you can do is check FARM if available and perform a long burn-in with something like badblocks. Then compare the SMART data before and after the burn in. Checking the serial number against the manufacturers database if available is also a good precaution.
These are probably things you should be doing whether or not the drive is allegedly new.
[1] https://www.heise.de/en/news/Hard-disk-fraud-Larger-disks-wi...
does amazon still do inventory co-mingling ?
That was my first thought too. Apparently they’re “phasing it out” by “the end of this year” [0]
I did not know, per that article, that Amazon had for some time now offered motivated third-party sellers a means to avoid commingling by applying a “fulfillment network SKU” barcode to their goods. And that they estimate merchants spend $600mm a year on that type of “restickering.” Expensive, but possible.
[0] https://www.geekwire.com/2025/after-years-of-backlash-amazon...
Alcohol or lacquer thinner will remove Sharpie in no time. I use Sharpies as temporary markers on smooth metal for this reason.
This is how it's done - name and shame!
Without it, there isn't enough incentive to try and just eat the cost of a refund in the rare case they get caught.
Good to see someone name and shame shady suppliers.
This drive model is 7 years old.
It is WILD that anyone in tech assumes this will come as new. Simply no one makes the same model of "consumable" for 7 years. Intel doesn't even sell Intel-branded SSDs anymore, that division was spinned off.
It's also WILD that you would trust something as sketchy sounding as "Maestro Technologies" for a mission-critical task.
I bet they were cheap though.
First of all, neither of those WILD facts seem that wild to me.
Intel did last orders for that drive Dec 30 2022. The article was written in April, so the author was conceivably purchasing drives that had sat on a shelf for a year and a half. That doesn't tickle alarm bells in my head.
Secondly, maybe my scam detector isn't well tuned enough, but "Maestro Technologies" doesn't seem that much stranger than "Apple" or "Micro soft" or "Zoom" or "Snap." If it were XBBHHZZZAA, LLC, maybe I'd have more room for pause.
The takeaway lesson here is that Amazon has become less and less reliable as a source for items. It's especially bad if it's purchased from a third party (something Amazon seems keen not to highlight on the purchase page), but even FBA is not free of trash. They straight up sell pirated N64 cartridges for example: https://www.amazon.com/Cartridge-Nintendo-Smash-64-Video-Ver...
New old stock is a thing.