Data Hoarders Thread - The Scriptatorium of the Modern Age

[tehpope]

What was the issue with your WD drives?

I had them die on me. I was able to get the data recovered. But those might have been seagate drives. I can't remember. I just checked all my drives and only one of them is WD. Two are toshiba. the others are seagate.

 

[Sperg_rancher]

I had them die on me. I was able to get the data recovered. But those might have been seagate drives. I can't remember. I just checked all my drives and only one of them is WD. Two are toshiba. the others are seagate.

Lucky for you, it's one of 3 companies, and you named 2 of em. Unless you had hitachi drives. yuck.

 

[LatinasAreTheFuture]

All data needs to be collected and archived here locally. Then all that data is part of the site data and is bundled together in the event of site outages or attacks.

 

[Womble of Wimbledon]

What was the issue with your WD drives?

I'm hoping to eventually move over to SSDs but WD storage drives have been my go-to for the past couple of years after having issues with Seagate.

Same here. I've had two Seagate harddrives shit themselves for no apparent reason but I've got WDs that have been in regular use for nearly 15 years still going strong.

 

[Sperg_rancher]

What resources do you guys recommend to use to learn ZFS and BTRFS? I'm still using EXT4 on all my stuff. I've heard awesome things about both, but I'm not sure why I'd switch. Every resource I've come across jumps straight into inodes and mmaps, when what I really need is to be able to compare the capabilities of all 3

 

[seri0us]

I've said it before and I'll say it again: LTO drives are utter folly unless you're an actual enterprise with an actual budget and a full-time staff to keep you trotting along on the planned-obsolescence treadmill. Hope your cold-storage bunker contains multiple backup copies of the (expensive, specialized) hardware too.

If "The Big One" does hit, you'll probably always be able to find something in working condition that can read an optical disc, SATA, or even IDE drive. But tape? Forget about it. Things are better than they were - you probably don't need to preserve an entire computer workstation, hardware and software, in amber along with your tapes anymore - but it's just not a cold-storage solution. Tape is for migrating data from Facegoog Datacenter 2020 to Facegoog Datacenter 2025.

What do you use for cold storage?

 

[Kosher Dill]

What do you use for cold storage?

Just hard drives retired from NAS service, nothing fancy.

 

[King Vidiot]

I had them die on me. I was able to get the data recovered. But those might have been seagate drives. I can't remember. I just checked all my drives and only one of them is WD. Two are toshiba. the others are seagate.

Lucky for you, it's one of 3 companies, and you named 2 of em. Unless you had hitachi drives. yuck.

The Hitachi Deathstar days ended a while ago and their more recent stuff was pretty good. Hitachi tends to have some of the lower failure rates in the Backblaze drive stats. I had 4TB HGST drives in my file server and they worked great until I retired them for bigger drives. Hitachi storage doesn't really exist anymore as a brand since they were purchased by Western Digital in 2012 and all their new stuff is branded as WD products.

Also the Backblaze drive stats link above is the best site I know of for choosing which cheap consumer grade drives to use in a fileserver.

 

[HootersMcBoobies]

I had them die on me.

I've had two Seagate harddrives shit themselves

The Hitachi Deathstar days ende

I've had every single brand of hard drive die out on me over the past 20 years. Maxtor, Western Digital, Fujitsu, HGST, Seagate. Some after a few years. A couple after one.

Now my most recent drives (6TB up to 20TB) haven't had any big issues, but I also have regular backups. I use ZFS snapshots and physically put the backups in my NAS (hot swap SATA bays) once a month or so, to sync incremental snapshots. Then I take the drives out and but them in a fire resistant box. Hard drives have gotten a lot more reliable, but there is no replacement for backups.

Don't get caught up on "I won't buy x or y brands," because of something that happened in the past. You're talking about things that constantly spin at 5k to 10k rpm. None of them last forever. Always get the most TB per €/$/£ and always budget for two (primary/backup). Always use encryption (ZFS native encryption, LUKS, Geli, VeraCrypt, whatever). You should always have to manually unlock your drives on reboots. You may not think you even have anything bad, but all it takes is one copy of a wikileaks archive and one fed who needs to meet a quota and you can get Jan6ed.

At some point I plan on digging electricity conduits out to my shed so I can put a rack in there. That way I can have real "off-site" backups, in case of fire/flood/etc.

 

[Q !!Hs1Jq13jV6]

What resources do you guys recommend to use to learn ZFS and BTRFS? I'm still using EXT4 on all my stuff. I've heard awesome things about both, but I'm not sure why I'd switch.

Create a VM and give it a bunch of virtual disks to fuck around with it.
Generally the usecases are something like this:
ext4: stable, supported by fucking everything
btrfs: has snapshots and compression, not stable when using the more advanced features
zfs: very stable, installation can be annoying, has every feature you can think of, nicest way of handling multiple disks, should officially support single disk vdev extension
my personal guidance (not universal, this is an opinion) is to use btrfs for my desktop/iscsi volumes, ext4 for vms/k8s-mounted iscsi volumes and zfs for your nas
the command line tools for zfs and btrfs aren't too complicated so just read the arch/gentoo wiki pages on them and quickly look through the command man pages

 

[GeorgeFloydFentanylStash]

What was the issue with your WD drives?

I'm hoping to eventually move over to SSDs but WD storage drives have been my go-to for the past couple of years after having issues with Seagate.

WD has been my go to since external drives. Need to replace my 8TB drive i use for Plex here soon because i’ve had it since 2019.

 

[Boolean Bitch]

Same here. I've had two Seagate harddrives shit themselves for no apparent reason but I've got WDs that have been in regular use for nearly 15 years still going strong.

Interesting. Were they manufactured around 2018 or so? I had bought 2 Seagate external HDDs (different days, different stores), from that time period, and had BOTH committed seppuku one day apart.... and right after that my laptop HDD crapped out (I fortunately had a 3rd backup drive - SSD - so no data loss, just a lot of swearing). And I've heard of a lot of other people having problems with their HDDs from that time unexpectedly dying (though not all necessarily Seagates). I wonder if a large warehouse of drives ended up getting flooded, and the cheeky bastards just dried them out in rice and shipped them out for sale anyway.

I was also wondering if anyone here could give any advice about optical discs for archiving. I curently have a relatively small amount of data I maintain on drives - less than half a TB - and about 20-40GB of critical data I don't want damaged or lost because I can't replace it. I was planning to get a bluray/DVD burner to start burning this to optical discs at regular intervals (saving older copies as backups to backups of backups), but it's ... confusing. Verbatim M-Discs don't look like they are actually M-Discs anymore (but rather rebranded MABL hard-coat BD-Rs at an inflated price). I can't find any listings for M-Disc DVDs. Local stores only sell trash not even worth using for coasters. And every Verbatim product on Amazon is boasting "100 year archival life" - even their non-AZO blue-wrapper CD-Rs and those vinyl coated ones are listed as "archival quality". Are there any products left that actually ARE archival quality?

Do I just get standard Verbatim BD-Rs and hope they don't self-destruct?

 

[Pee Cola]

I'm only a small time data hoarder. I still use external HDDs like a poor.

To mitigate the risk, I keep no fewer than two exact copies of each HDD. Same capacity, different manufacturers (usually 1 Seagate and 1 WD), different ages. I usually buy one new external HDD a year; my oldest one is 12 years old but has relatively low hours and still passes all SMART tests. It should be noted that this is now a secondary backup and is no longer in my active backup rotation i.e. it's effectively a third copy of some of my older media.

I also avoid buying multiple HDDs at once, just in case I end up buying from a dud batch (which happens from time to time). I've had a higher failure rate with Seagates (2) than WD (0). Whilst I've had a higher failure rate with Seagate, it doesn't put me off buying them as I know WD have had bad batches in the past too, and I may have been just very unlucky with Seagate.

 

[Womble of Wimbledon]

Interesting. Were they manufactured around 2018 or so? I had bought 2 Seagate external HDDs (different days, different stores), from that time period, and had BOTH committed seppuku one day apart.... and right after that my laptop HDD crapped out (I fortunately had a 3rd backup drive - SSD - so no data loss, just a lot of swearing). And I've heard of a lot of other people having problems with their HDDs from that time unexpectedly dying (though not all necessarily Seagates). I wonder if a large warehouse of drives ended up getting flooded, and the cheeky bastards just dried them out in rice and shipped them out for sale anyway.

No, mine would have been purchased quite a few years before that, and bought at separate times. That's what makes me feel like Seagate is just less reliable generally, although I might just be very unlucky.

 

[ZMOT]

I wonder if a large warehouse of drives ended up getting flooded, and the cheeky bastards just dried them out in rice and shipped them out for sale anyway.

if they did it didn't pay off.
iirc it was just a somewhat questionable batch but the damage to the brand reputation persists to this day.

I was also wondering if anyone here could give any advice about optical discs for archiving. I curently have a relatively small amount of data I maintain on drives - less than half a TB - and about 20-40GB of critical data I don't want damaged or lost because I can't replace it. I was planning to get a bluray/DVD burner to start burning this to optical discs at regular intervals (saving older copies as backups to backups of backups), but it's ... confusing. Verbatim M-Discs don't look like they are actually M-Discs anymore (but rather rebranded MABL hard-coat BD-Rs at an inflated price). I can't find any listings for M-Disc DVDs. Local stores only sell trash not even worth using for coasters. And every Verbatim product on Amazon is boasting "100 year archival life" - even their non-AZO blue-wrapper CD-Rs and those vinyl coated ones are listed as "archival quality". Are there any products left that actually ARE archival quality?

Do I just get standard Verbatim BD-Rs and hope they don't self-destruct?

does it have to be optical? otherwise grab a HDD and just store them somewhere else, with a few old and that size you could probably use 2-3 at once, this would greatly reduce the chance all dying at the same time.

 

[Boolean Bitch]

i...does it have to be optical? otherwise grab a HDD and just store them somewhere else, with a few old and that size you could probably use 2-3 at once, this would greatly reduce the chance all dying at the same time.

Yes, I would like it to be optical. The data is already backed up onto 2xHDD and 1xSSD (and I'm not replacing any of these for optical storage). I was already considering making a 4th backup using a SD card/flash drive, or even a low capacity SSD - Kingston sells 240G 2.5" SATA III SSDs for only $25-30 USD and seem reliable (I've had one of these budget Kingstons in my computer for years now). I could even be a bit ridiculous and copy the crtical stuff half a dozen or more times. Even with "bit-rot", the likelyhood of every copy of a particular file being corrupted would be highly unlikely - especially taking into consideration that this backup would only be needed if every other drive failed at the exact same time.

The problem I was seeing was that all of these storage types have the exact same vulnerabilities. Especially that they need to be plugged in periodically or could lose data due to losing 'charge' - which makes it more difficult when dealing with offsite backups. Optical seems to eliminate the problem of needing to connect them to power, or worrying about static, or magnetic fields, or needing stable temperatures, etc. Humidity, scratches, and light expsosure can easily be managed when dealing with something as small as a few optical discs. I'm not looking for ridiculous claims like 100 years - it's just that it's not worth the effort if the disc stands a good chance to break down and be unusable within a couple of years. Tthat's just too unreliable for offsite cold-storage backup. I'm looking for something that when stored properly (not "ideal", but reasonably well), stands an excellent chance (nothing is 100%) of being readable after 10 years without having to be constantly babysat.

 

[Betonhaus]

I have a ThinkServer TS430 with 22tb worth of SAS storage that I use as a media server. There's some stuff that was difficult to find and archive, and I could run a script that gives me a list of everything so I can share things on request, but free anonymous file sharing sites are limited to 5gb and I rather not publish my IP address as I also run a webserver

 

[ToroidalBoat]

Something that can be quite annoying is when people claim there's no need to download stuff, and even that it's childish to do so.

It's potentially infuriating BS as stuff is taken down from the internet all the time. Even before the censorship of Current Year.

 

[Officer Barbrady]

Something that can be quite annoying is when people claim there's no need to download stuff, and even that it's childish to do so.

It's potentially infuriating BS as stuff is taken down from the internet all the time. Even before the censorship of Current Year.

And they'll be the same people running to fan forums and subreddits like r/lostmedia to beg for a saved copy of it when it's ultimately wiped from the internet.

 

[lichen]

I've toyed with the idea of data hoarding in the past (NAS sounds like a fun project for someone with light tech skills), but I always get stuck on the question of "for what purpose?" So I'd be curious to hear from the thread why they hoard, and how if at all you expect the hoard to be useful.

I keep an archive of certain things with a copy on dropbox, a copy on PC and a copy on an external hard drive. Nothing that tops 2 TB so can't say I play the data hoarder game. I could think of some things I'd be interested in using a larger storage array for, but I can't shake the idea that no one but me would ever know the archive exists (even if it would somehow be useful to them). And that I would personally be unlikely to ever go digging through 1% of the archive if it were built.

Preservation seems like a noble activity, but ideally there would be some hope of it being practically useful in addition to being a hobby.

Also curious how people organize their shit. I have a hard enough time keeping the folder names of clean for 800 GB of music.