Using Kiwix and Zim to archive the forum - Mr Moon, tear down this wall!

[Harvey Danger]

Inspired by the Portable Library of Alexandria idea, I started toying around with the aptly named Kiwix software. This is an archive reader program used to browse offline, packaged archives of popular wikis and websites. The idea is to have offline versions of critical information repositories, available on cheap Wifi hotspots without Internet. It reads compressed "zim" files, offering a pre-packaged library of things like Wikipedia, Stack Overflow, RationalWiki, Linux install wikis, etc.

What does this have to do with KF? Well, you can request a generated zim file of any website, for free, with some limitations. I can't think of a more appropriate and important site to have eternal offline access than Kiwi Farms. But while Zimit accepts KF for scanning, it gets stopped cold by kiwiflare. The resulting zim file only has 1 page to browse.



This is not the only way to create a zim archive, there are several more, but it's the quickest and least complicated route.

So my general question is, has anyone else played around with Kiwix/zim and found a good way to grab KF? And to @Null, can we (when not under active DDOS) get a temporary alternate URL to the site that bypasses Kiwiflare, so we can try this tool? Or maybe allow its user agent to bypass kiwiflare, though that probably requires more work.

I've got my beans bullets and bullion, but I don't want to go through Doomsday without my KF fix too.

 

[The Mass Shooter Ron Soye]

Null has said he plans to make a lightly scrubbed version of the site available by e.g. torrent if it goes offline forever.

 

[community member]

Ah fuck, I've been working on trying to archive sites too and was hoping HTTrack would work but haven't tested it yet. Totally forgot about kiwiflare

 

[Jeff Q. Anime]

Null has said he plans to make a lightly scrubbed version of the site available by e.g. torrent if it goes offline forever.

but what if he dies tho

 

[HyperboreanRightsActivist]

but what if he dies tho

Nigger

 

[Don Yagon]

This is not the only way to create a zim archive, there are several more, but it's the quickest and least complicated route.

One of the suggested ways there is using wget-2-zim wrapper script, with which it shouls be pretty easy to just pass the kiwiflare cookie from the browser. How long it'll last is another question, though.

 

[ENDTOPII]

I have never made kiwix's search function work, either it's too slow for me or it really is broken. I had the english Wikipedia (~100GB) and it wouldn't work. But I still would love a Kiwifarms zim file, just incase.

 

[Harvey Danger]

Null has said he plans to make a lightly scrubbed version of the site available by e.g. torrent if it goes offline forever.

Yeah, this is partially an experiment, and partially a way to see if we can make it easier for Null to generate a forever copy. Also I like having an easy-to-deploy format like this, makes it more likely people will keep a copy around.

One of the suggested ways there is using wget-2-zim wrapper script, with which it shouls be pretty easy to just pass the kiwiflare cookie from the browser. How long it'll last is another question, though.

This is probably a better solution if I wanted to keep the zim archive updated, like running it on a weekly schedule. I think the guys working on Sockchat made libraries for solving Kiwiflare and doing logins, that might keep from timing out (when we aren't in DDOS mode).

 

[community member]

but what if he dies tho

I'd say he could have something set up on a timer to automatically deploy it or through an agent or something but who knows if the inner potato farmer has him not caring enough about that. Regardless I would really like a way to rip images out of big threads like the tranny sideshows one

 

[Kosher Dill]

Scraping seems like a pointlessly complicated solution if the site owner is willing to cooperate. Why not just take a structured extract directly from the forum database and then just write up some scripts to turn that into whatever you want? Could be a ZIM file, could be just HTML or whatever else.