Portable Library of Alexandria - Taking my digital hoarding habit on the go

[tehpope]

Was talking with some friends about this. Shit is gonna hit the fan someday. And people will probably have their phones or other mobile devices still. But internet connectivity might be spotty or non existant. Someone mentioned using something called PirateBox to do it, but its a dead project. There's also Y U NO Host. Looks good, but I've had issue installing some software.

But here's what I came up with

The Odroid M1 would be the heart of the project. A low power SBC with a buch of IO. The most important, to me, is built-in m.2 NVME slot. You can put a 4tb SSD in there. It also has 2 usb 3 ports on board and a sata connector. So a number of mechanical HDDs could be used. I want to pair it with at least a single 5tb drive. Maybe a second one. This isn't finalized. I could use something like a Rock 5B. Its way more powerful, includes a NVME slot. But I'm still wating on mine.

To serve content, I plan to pair it with a portable router. gl.inet gl-mt1300. Has wifi 5g connectivity and an SD card slot for extra storage. Up to 512gb. I don't plan on using it but its nice to have. And for keeping it safe, I thought about using a Pelican case. They're super durable and crush / water proof. Possibly a Pelican 1450. But I've been considering a Pelican 1400. $60 cheaper and smaller.

As for content, I plan on loading it up with content. Music, Movies, ebooks, site backups, etc. Jellyfin with another audio software (Possibly Navidrone or airsonic advance) to serve up most of it. Kiwix for serving backups of Wikipedia, Project Gutenberg, and other sites Kiwix can use. ebooks would probably be a web version of Calibre or just serving them with a file share. And then hosting some communication method like a imageboard, fluxpbb, matrix, and/or an irc server with a web interface.

I've got a dl of Wikipedia and PG downloaded. They only take up like <250gb total.

 

[TenMilesWide]

For full authenticity it should spontaneously combust.

 

[Ultrapenguin]

I'm really glad that there seem to be so many nerds hoarding info on disk drives for after the fall of the internet. That way it'll be super convenient to beat them up and take it when the time comes.

 

[Blasterisk]

For full authenticity it should spontaneously combust.

I rated you "drink" because the loss of the Library makes me cry. It was probably 75% smut, but still!

 

[The Mass Shooter Ron Soye]

It would be nice if we ever see something like this in the hands of plebs, as long as it can be read from at least as easily as an HDD or Blu-ray. It might end up with slow random reads compared to SSDs, and no rewritability, but if it can store hundreds of terabytes indefinitely with no data loss, it's a much more resilient Library of Alexandria.

Otherwise, we'll eventually see 8-32 terabytes in the M.2 2280 form factor or one of its replacements. There's a larger M.2 25110 coming to compensate for PCIe 5.0 heat issues, and M.2 could be replaced by something new in the consumer space in a decade.

If you trim the fat off of Wikipedia and compress it, it should be around 20 gigabytes or less. How large is Project Gutenberg? I don't think you should be anywhere near 250 gigabytes with just those two. Although if you're putting 10+ terabytes in this thing, I guess it doesn't matter.

 

[byuu]

but if it can store hundreds of terabytes indefinitely with no data loss, it's a much more resilient Library of Alexandria.

That's just marketing wank about some theoretical absolute maximum only achievable with a technology that doesn't exist. They ended up storing an MB worth of text on that crystal.

 

[tehpope]

If you trim the fat off of Wikipedia and compress it, it should be around 20 gigabytes or less. How large is Project Gutenberg? I don't think you should be anywhere near 250 gigabytes with just those two. Although if you're putting 10+ terabytes in this thing, I guess it doesn't matter.

I was guessing. Looking at my files, Wikipedia with lq pics is 86.6gb and PG is 66.3gb. So more like 153gb

 

[The Mass Shooter Ron Soye]

That's just marketing wank about some theoretical absolute maximum only achievable with a technology that doesn't exist. They ended up storing an MB worth of text on that crystal.

There's multiple different companies, universities, and other organizations like Arch working on the same kind of optical storage technology. Microsoft's Project Silica for example. If the Arch Mission Foundation wants to achieve its goals, they need to scale it up to multiple terabytes and get it to mass production.

HDDs and SSDs have problems and we could use better options.

 

[byuu]

If the Arch Mission Foundation wants to achieve its goals

They don't seem to actually develop any tech. For the crystals they went to some university to get it done.
In their later project, they used something completely different (etched nickel sheets).

 

[Big Stank Dick Dad]

low power SBC

How low? How will you keep it from becoming a paperweight when the power goes out?

 

[tehpope]

How low? How will you keep it from becoming a paperweight when the power goes out?

Shit, I forgot to talk about that part. With a 30k mAh usb battery, you can power it, the sbc and router, for about 3 days. As long as it has usb c fast charging, a dc to usb c adapter should power it. I'm using a usb c fast charger to power my Odroid N2+ atm and it should use the same barrel jack for power. It might be closer to 2 days if the portable hdd uses more power.

 

[Cringe newfag]

Wouldn't you want to use as many smaller storage devices as possible? Like fit 4 250gb drives instead of one 1tb?

I know that's very size inefficient, but the whole idea is that modern technology favours form factor and convenience over durability, so you want something else that is more durable.

 

[AmpleApricots]

flash storage is not good viable long term storage. Flash forgets. (storage cells keep losing their charge with time, refreshing them ages them) The controllers are overtly complicated and the memory many have is also subject to aging and not that high quality to begin with. While Flash memory actually likes it somewhat warm, the other components don't, along the typical stressors from things like a fitted heatsink slightly warping the board and breaking soldering balls, making parts fail. (I have seen such things happen in industrial settings inside of years, they definitively do) In a scenario like this, we're not talking for the computer to last for the next 5 years or so, we're probably talking decades. Might become a family heirloom even. Flash is the wrong type of memory for that. A lot of them also have very complex firmware that might have bugs you'll only find out about when physical properties of the device changes, like capacitors drifting or the controller starting to map out the broken down cells. You don't wanna find out that your controller suddenly null'd all data because of a bug some korean engineer did 12 years ago.

Same goes for your fancy computer. An intel Atom chip some years ago was subject to an internal aging process that would eventually end up with the chip dying prematurely. It could be shaken back to life by soldering a part externally. Would you be able to figure something like that out? Are there bugs in the firmware of the controller that won't be able to handle something? How can you know in advance? You might be able to scavenge another computer. You might not. I'm pretty sure people will hysterically hoard them and getting one might prove very expensive, difficult and even dangerous. You might also find many of them subject to similar aging processes and unforseen problems. Many of these computers also have prorpietary firmware they need in order to run. That firmware is saved in, you guessed right, flash memory. Yes, these computers will eventually "forget" how they work. A lot of modern eeproms are programmed with very low charges. They won't last as long without a bit flip as an eeprom programmed e.g. 20 years ago. (Most manufacturers guarantee a data retention of around 20 years for eeproms)

At least for your data, find something magnetic, tapes are preferable. Similar problems with eventually being destroyed by the magnetic field of the earth but you have one decisive advantage: you can read a bit over and over and over again until you get it eventually right and manage to restore it. If flash is dead, it's dead, the data has entropied out of existence.

If you get commercial tape drives take them apart, look if their mechanical side is repairable, if you could make one good drive out of two broken down ones. With some it's easy, with others it's impossible. Some will have rubber parts, these will eventually decay. Some have thin plastic parts, these will eventually wear down with mechanical movement. Can you make new ones with an e.g. machining table? Could the motor of the device handle a part of different weight and slightly different tolerance? Or Will that cause the failure of the drive? Could you replace those parts? If you found a simple enough drive (fat chance of that nowadays) get several of them and make sure they're actually internally the same. (Companies like to make things cheaper over the course of the lifetime of a device, your son doesn't want to find out you can't sell your secrets of farming and printouts of 70s nudies to Jim bob a village over twenty years down the road as he has no software drivers for the leftover tape drive because it turns out it has a slightly different controller internally)

You see, there are a lot of problems. I'd go with something a lot more primtive than a modern computer. I'd take something like a Z180 microcontroller (they cost next to nothing, buy 200 of them) and some SRAM. (again, just buy a thousand chips, it's not that expensive, if you're clever buy them big enough so you can just map out stuck bits in the OS if they're somewhat defective with age but you wanna keep using them) With these things, you have already the basic parts for your own computer. If I wana be fancy I'd go with EPROMs (not EEPROMs) and build my own programmer for the system, so I can build more down the road and make changes to the OS, but also leave a way open to enter the OS mechanically via switches. All the parts required for this you can buy dozens of boxes of and it wouldn't even set you that far back. Learn to put them together blind and by memory, they're not complicated. Make many human-readable printouts of the OS. Hell, have yourself a few fancy books made. Not expensive. Who knows? Maybe you'll eventually be able to trade copies of this system for something.

These things can run on a pittance of power that'd made any ARM blush. (self made batteries charged via hand-cranking is definitively possible with a modern Z180 and modern SRAM, yes, you'd need to come up with batteries because your fancy batteries will be dead and irrepairable in a few years - if they last that long - same for all the surviving mobile devices people will probably start killing each other over while you'll be left alone) Then I'd figure out an archiving system that works with tape drives, a way to query that archiving system via mechanical input, a way for the system to give you feedback via LED and a way to print the required information out. (Here same thing goes as for the tape drives Re: part availability, and reliance, might wanna look into thermal printing this is more difficult because you're gonna need more electricity here, just hope that some parts of society will stabilize enough otherwise nobody will care about what you can give them anyways) There's also dedicated chips that can do things like mp3 decoding and video decoding. If you have the time and want to have some extra features that might awe your local warlord and make sure you'll get everything what you want, you might buy a few hundreds of those too (again, they cost cents) and store them for programming later. I leave that part to your imagination.

This is not what I'd do. I'd just blow my brains out because I'm not gonna spend my retirement years sustenance farming only to die of an infected leg but hey, it's a fun scenario to think about.

(Edit: Wow, I have way too much time on my hands rn)

 

[Manul Otocolobus]

The Arch format sounds good to me, assuming they can expand it as far as they think they can.

I've heard of a different format that inscribes data in a 3D matrix inside of a diamond. The data density is incredible. However, the technology to read the contents is advanced requiring two different lasers with specific properties. But, as far as durability, chemical stability, etc... nothing really beats diamond.

 

[Overly Serious]

Sounds like a great project. Please keep posting your progress. I'd quite like to follow along.

I'm really interested in the data-getting side of this. How do you go about getting a complete download of Wikipedia or Project Gutenberg? I think that would be cool to get hold of myself.

 

[tehpope]

Sounds like a great project. Please keep posting your progress. I'd quite like to follow along.

I'm really interested in the data-getting side of this. How do you go about getting a complete download of Wikipedia or Project Gutenberg? I think that would be cool to get hold of myself.

https://download.kiwix.org/zim/

 

[Kosher Dill]

https://download.kiwix.org/zim/

The Kiwix extracts are particularly good because they have topical extracts. You can just take the math articles without the politisperging.

 

[Doppelmonger]

I was about to jokingly say "get an e-reader with SD storage" but frankly this sounds more interesting than I tought. I take my joke back fren. Godspeed!

 

[tehpope]

flash storage is not good viable long term storage. Flash forgets. (storage cells keep losing their charge with time, refreshing them ages them) The controllers are overtly complicated and the memory many have is also subject to aging and not that high quality to begin with. While Flash memory actually likes it somewhat warm, the other components don't, along the typical stressors from things like a fitted heatsink slightly warping the board and breaking soldering balls, making parts fail. (I have seen such things happen in industrial settings inside of years, they definitively do) In a scenario like this, we're not talking for the computer to last for the next 5 years or so, we're probably talking decades. Might become a family heirloom even. Flash is the wrong type of memory for that. A lot of them also have very complex firmware that might have bugs you'll only find out about when physical properties of the device changes, like capacitors drifting or the controller starting to map out the broken down cells. You don't wanna find out that your controller suddenly null'd all data because of a bug some korean engineer did 12 years ago.

Same goes for your fancy computer. An intel Atom chip some years ago was subject to an internal aging process that would eventually end up with the chip dying prematurely. It could be shaken back to life by soldering a part externally. Would you be able to figure something like that out? Are there bugs in the firmware of the controller that won't be able to handle something? How can you know in advance? You might be able to scavenge another computer. You might not. I'm pretty sure people will hysterically hoard them and getting one might prove very expensive, difficult and even dangerous. You might also find many of them subject to similar aging processes and unforseen problems. Many of these computers also have prorpietary firmware they need in order to run. That firmware is saved in, you guessed right, flash memory. Yes, these computers will eventually "forget" how they work. A lot of modern eeproms are programmed with very low charges. They won't last as long without a bit flip as an eeprom programmed e.g. 20 years ago. (Most manufacturers guarantee a data retention of around 20 years for eeproms)

At least for your data, find something magnetic, tapes are preferable. Similar problems with eventually being destroyed by the magnetic field of the earth but you have one decisive advantage: you can read a bit over and over and over again until you get it eventually right and manage to restore it. If flash is dead, it's dead, the data has entropied out of existence.

If you get commercial tape drives take them apart, look if their mechanical side is repairable, if you could make one good drive out of two broken down ones. With some it's easy, with others it's impossible. Some will have rubber parts, these will eventually decay. Some have thin plastic parts, these will eventually wear down with mechanical movement. Can you make new ones with an e.g. machining table? Could the motor of the device handle a part of different weight and slightly different tolerance? Or Will that cause the failure of the drive? Could you replace those parts? If you found a simple enough drive (fat chance of that nowadays) get several of them and make sure they're actually internally the same. (Companies like to make things cheaper over the course of the lifetime of a device, your son doesn't want to find out you can't sell your secrets of farming and printouts of 70s nudies to Jim bob a village over twenty years down the road as he has no software drivers for the leftover tape drive because it turns out it has a slightly different controller internally)

You see, there are a lot of problems. I'd go with something a lot more primtive than a modern computer. I'd take something like a Z180 microcontroller (they cost next to nothing, buy 200 of them) and some SRAM. (again, just buy a thousand chips, it's not that expensive, if you're clever buy them big enough so you can just map out stuck bits in the OS if they're somewhat defective with age but you wanna keep using them) With these things, you have already the basic parts for your own computer. If I wana be fancy I'd go with EPROMs (not EEPROMs) and build my own programmer for the system, so I can build more down the road and make changes to the OS, but also leave a way open to enter the OS mechanically via switches. All the parts required for this you can buy dozens of boxes of and it wouldn't even set you that far back. Learn to put them together blind and by memory, they're not complicated. Make many human-readable printouts of the OS. Hell, have yourself a few fancy books made. Not expensive. Who knows? Maybe you'll eventually be able to trade copies of this system for something.

These things can run on a pittance of power that'd made any ARM blush. (self made batteries charged via hand-cranking is definitively possible with a modern Z180 and modern SRAM, yes, you'd need to come up with batteries because your fancy batteries will be dead and irrepairable in a few years - if they last that long - same for all the surviving mobile devices people will probably start killing each other over while you'll be left alone) Then I'd figure out an archiving system that works with tape drives, a way to query that archiving system via mechanical input, a way for the system to give you feedback via LED and a way to print the required information out. (Here same thing goes as for the tape drives Re: part availability, and reliance, might wanna look into thermal printing this is more difficult because you're gonna need more electricity here, just hope that some parts of society will stabilize enough otherwise nobody will care about what you can give them anyways) There's also dedicated chips that can do things like mp3 decoding and video decoding. If you have the time and want to have some extra features that might awe your local warlord and make sure you'll get everything what you want, you might buy a few hundreds of those too (again, they cost cents) and store them for programming later. I leave that part to your imagination.

This is not what I'd do. I'd just blow my brains out because I'm not gonna spend my retirement years sustenance farming only to die of an infected leg but hey, it's a fun scenario to think about.

(Edit: Wow, I have way too much time on my hands rn)

50 Euros for eZ80 sbc.

AgonLight2 - Open Source Hardware Board

AgonLight2 Single Board BBC Basic Z80 Retro style Computer

www.olimex.com

Nothing else to update atm.

 

[AmpleApricots]

Fun board, and there are many kits like this. It's just such a simple CPU to wrap your head around that I'd come up with a solution for myself to really understand it. Depending on speed you don't really even need professionally-made PCBs to build computers with it. It's upside (easy to program) is also it's downside though, the Z80 is not a clever CPU and not fast in many things. Something like the 6502 (Apple II/II, C64 (kind of), many others) is more performant clock-for-clock but harder to program for. The deluxe would be the 68k which also has quite a few modern iterations and you could also do some simple graphical output more easily for without wanting to kill yourself or it feeling like pulling teeth. These architectures are harder to wrap your head around programming-wise, though. I'm sure there are cheap kits for them too.