Archival Tools - How to archive anything.

Null

Ooperator
kiwifarms.net
Joined
Nov 14, 2012
This website is great because it archives content before people delete it.
Your natural instinct should be to archive, never to link directly. The moment you link something directly, it can be taken down.

Web Archives
archive.is
creates a full, living backup of a website with all its resources. (example)

Full page screenshots can be done with browser extensions or built-in tools. FireFox now has this built in using the right-click context menu and it is extremely useful, but do not hotlink content from the firefoxusercontent domain. Download it and re-upload it, or use the clipboard option to paste it directly into your post. I always encourage people to use Brave, but I still have FireFox installed for the screenshot tool.

Modern Chrome and Brave can capture screenshots that are larger than the screen by using inspect element.

1662032258164.png

Hosting
You should always want to host on the site. The upload limit is 200MiB and most formats are supported.

Mega is possibly the only OK off-site host. Nothing else can be trusted - both due to their susceptibility to delete things, and for smaller file sites to disappear overnight.

If your file is very large or very important, PM me or email me at null@kiwifarms.net and I will archive it to https://kiwifarms.net/archive - however, I will usually want you to find a way to host it in your post.

Links to 3rd party websites are rarely a good idea unless you're actually trying to show people the URL, and not its content.

Never, ever, ever rely on social media embeds or direct links. Take care when posting images saved from Facebook. The download URL includes a token which identifies the account that downloaded it. Change the filename, or just copy+paste the image directly.


Video Archives
You must learn how to use youtube-dl, which is currently called yt-dlp because of Google aggressively trying to prevent people from downloading their videos. yt-dlp is an omnitool for archiving web video content.

Uploading Videos to the Kiwi Farms
The file size limit is currently 200MiB. This is sufficient for most videos. I appreciate optimizing efforts, and only certain formats play on all sites, so I will teach you how to use a different program to optimize the video.

First, if an audio+video MP4 format is available on youtube-dl (check by adding -F), get that. See below:
Code:
$ youtube-dl -F https://www.youtube.com/watch?v=NxW_sLNQKsI
[youtube] NxW_sLNQKsI: Downloading webpage
[info] Available formats for NxW_sLNQKsI:
format code  extension  resolution note
249          webm       audio only tiny   46k , webm_dash container, opus @ 46k (48000Hz), 59.95MiB
250          webm       audio only tiny   64k , webm_dash container, opus @ 64k (48000Hz), 83.33MiB
251          webm       audio only tiny  116k , webm_dash container, opus @116k (48000Hz), 148.92MiB
140          m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2@129k (44100Hz), 166.00MiB
160          mp4        256x144    144p   61k , mp4_dash container, avc1.4d400c@  61k, 30fps, video only, 78.47MiB
278          webm       256x144    144p   68k , webm_dash container, vp9@  68k, 30fps, video only, 87.54MiB
242          webm       426x240    240p   94k , webm_dash container, vp9@  94k, 30fps, video only, 120.79MiB
133          mp4        426x240    240p  136k , mp4_dash container, avc1.4d4015@ 136k, 30fps, video only, 174.81MiB
243          webm       640x360    360p  164k , webm_dash container, vp9@ 164k, 30fps, video only, 210.94MiB
134          mp4        640x360    360p  249k , mp4_dash container, avc1.4d401e@ 249k, 30fps, video only, 319.61MiB
244          webm       854x480    480p  255k , webm_dash container, vp9@ 255k, 30fps, video only, 328.21MiB
135          mp4        854x480    480p  427k , mp4_dash container, avc1.4d401f@ 427k, 30fps, video only, 548.62MiB
247          webm       1280x720   720p  484k , webm_dash container, vp9@ 484k, 30fps, video only, 621.01MiB
302          webm       1280x720   720p60  616k , webm_dash container, vp9@ 616k, 60fps, video only, 789.87MiB
136          mp4        1280x720   720p  779k , mp4_dash container, avc1.64001f@ 779k, 30fps, video only, 999.44MiB
298          mp4        1280x720   720p60 1063k , mp4_dash container, avc1.640020@1063k, 60fps, video only, 1.33GiB
303          webm       1920x1080  1080p60  759k , webm_dash container, vp9@ 759k, 60fps, video only, 974.28MiB
299          mp4        1920x1080  1080p60 1932k , mp4_dash container, avc1.64002a@1932k, 60fps, video only, 2.42GiB
18           mp4        640x360    360p  322k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz), 413.64MiB
22           mp4        1280x720   720p  912k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)

See 22? That's the magic number, but 18 also works. You want an MP4 format that has both audio and video, so don't get one that says "video only" like 299.
Then use youtube-dl -f mp4 or youtube-dl -f 22 to grab it as mp4.

If it's under 200MiB, you're done. Upload it.


P.S. youtube-dl likes to name video files from Twitter the entire content of a tweet. If you get an error, try adding -o a_less_stupid_name.mp4 somewhere.

Dealing with Large Videos and Irregular Formats
There's two steps with videos > 200 MiB. If you got it as an MP4 off YouTube using the above step, you can skip reformatting the video.

There's a magic command for taking any video of any format. Got a MOV? Got a weird MP4?

1. Download ffmpeg.

2. Use the magic command. This works on Linux (if installed via package manager) and Windows (as .exe if you've moved your command line to the same directory).
ffmpeg -hwaccel auto -i YOUR_VIDEO_DOT_ANYTHING -c:v libx264 -pix_fmt yuv420p -profile:v baseline -level 3.0 -crf 22 -preset medium -c:a aac -strict experimental -movflags +faststart -threads 0 OUTPUT.mp4

You can also use the -vf scale=-2:720 parameter to scale it to a certain size. 1080, 720, 480, 360 are common ones. The -2 scales it to the nearest even number (which is a requirement for the MP4 container). You can swap the dimensions, i.e. -vf scale=1280:-2

The less important the video is, the smaller you can scale the video. If it's totally irrelevant just make it 360p and call it a day. This is the easiest way to reduce video size. Don't fuck with the audio, audio is more important almost always.

Full version with scaling:
ffmpeg -hwaccel auto -i yourvid.any -c:v libx264 -pix_fmt yuv420p -profile:v baseline -level 3.0 -crf 22 -vf scale=-2:720 -preset medium -c:a aac -strict experimental -movflags +faststart -threads 0 output.mp4

Under 200MiB now? Upload it, you're done.

3. Splitting the video. Splitting is a last resort, so please don't just split and upload large, unoptimized shit.

ffmpeg doesn't natively split by size, it splits by time. Since the upload limit is 200MiB, take the file size in MB and divide it by 200. For instance, if you have a 2 hour, 53 minute long 443MiB file, you'll need 3 chunks. So divide the total runtime by 3.

It is better to take a video that is 300MB large and split in half, rather than trying to make it 200MB+100MB.

(If your video is close to being a factor of 200, split once more than you need, because not all time segments are made equal. If you have a video that's exactly 400MiB and you try to split it in half, you are very unlikely to get two videos that are 199MiB.)

Adjust the -segment_time as needed. This is set for 1 hour each. This part takes very little time.
ffmpeg -hwaccel auto -i input.mp4 -c copy -map 0 -segment_time 01:00:00 -f segment -reset_timestamps 1 -movflags +faststart -threads 0 output%03d.mp4

At the end, I would have have input.mp4, and then 3 videos called output_0.mp4, output_1.mp4, and output_2.mp4. Those are the files to upload to the Kiwi Farms.


CLIPPING FOR RETARDS

tl;dw: download kdenlive and export as mp4 at 720p
 

Attachments

  • full page screen capture.png
    full page screen capture.png
    400.5 KB · Views: 6,102
  • How to Clip.mp4
    30.7 MB
Last edited:
For Firefox (I know it works on Nightly) press SHIFT+F2 and a console comes up.
The command "screenshot" screenshots the current section of the page.
--fullpage screenshots the entire page.
--delay (x) delays the screenshot.
--clipboard saves the image to the clipboard instead of a file.
After those, you can type in a custom filename (the default filename is the date you took the photo, put into the downloads folder).
If you wanted to take the entire page and put it onto your clipboard, you would do "screenshot --fullpage --clipboard"
To save the page to a custom filename, do "screenshot --fullpage Test"
Here's an example of this page taken with this method:
brHKBUs.jpg

Edit Note: This doesn't work on all websites (example, 4chan with 4chanx gives an error)
 
Last edited:
The easiest way to take small (read: non-fullscreen) screenshots in Windows is to click Start > type "snippingtool" into the run box > draw a box around what you want to snip and it'll take a screenshot of it.

Then just click the copy button and paste the image into your message as an attachment. No mucking about with Imgur or uploading it to an external site.
 
The easiest way to take small (read: non-fullscreen) screenshots in Windows is to click Start > type "snippingtool" into the run box > draw a box around what you want to snip and it'll take a screenshot of it.

Then just click the copy button and paste the image into your message as an attachment. No mucking about with Imgur or uploading it to an external site.

Alternately, simple screenshots for applefags:

-On a Mac, holding down Cmd+Shift+3 will take a snapshot of your entire screen. Cmd+Shift+4 will allow you to select the portion you want to cap. There is also a screencap tool under "Utilities" in your app folder.

-On devices such as iPhones and iPads, holding the 'power' and 'menu' buttons simultaneously will result in a snapshot being directly saved to your camera roll.

Easy as pie.
 
Last edited:
The easiest way to take small (read: non-fullscreen) screenshots in Windows is to click Start > type "snippingtool" into the run box > draw a box around what you want to snip and it'll take a screenshot of it.

Then just click the copy button and paste the image into your message as an attachment. No mucking about with Imgur or uploading it to an external site.
I have a Firefox add-on called Nimbus Screenshot that does the same thing, click "selected area", box in the area that you want to cap and copy it to your clipboard, but it's even easier because it's accessed through a simple button on the toolbar. I'm absolutely braindead when it comes to computers and will bashfully admit that this was the only way I knew how to take screenshots for a while.
 
For anyone interacting with a cow from Deviant Art and you want to archive there who gallery use this tool called RipMe.
Install the full JDK on your system.
Go to the gallery in question, make sure you have selected all in the left hand column of the DA page.
Copy the URL from the address bar.
Paste it into the URL field in RipMe.
Click Rip.

You can change the location of the download folder in settings however if you don't it will create folder in your default downloads folder called rips you can find the archive there.
 
For Firefox (I know it works on Nightly) press SHIFT+F2 and a console comes up.
The command "screenshot" screenshots the current section of the page.
--fullpage screenshots the entire page.
--delay (x) delays the screenshot.
--clipboard saves the image to the clipboard instead of a file.
After those, you can type in a custom filename (the default filename is the date you took the photo, put into the downloads folder).
If you wanted to take the entire page and put it onto your clipboard, you would do "screenshot --fullpage --clipboard"
To save the page to a custom filename, do "screenshot --fullpage Test"
Here's an example of this page taken with this method:
brHKBUs.jpg

Edit Note: This doesn't work on all websites (example, 4chan with 4chanx gives an error)

Since a recent release, you can also use the tag --imgur and it'll upload it there and give you the URL.
 
If you guys are too lazy to copy-paste links to archive.is/tweetsave/whatever, use these bookmarklets instead:
archive.is:
Code:
javascript:void(open('https://archive.is/?run=1&url='+encodeURIComponent(document.location)))
tweetsave:
Code:
javascript:void(open('https://tweetsave.com/?tweet='+encodeURIComponent(document.location)))
go1dfish (for reddit stuff):
Code:
javascript:if (window.location.hostname.match(/reddit/)) {window.location = 'https://r.go1dfish.me' + window.location.pathname + window.location.search;}

save the shit in the code tag as bookmark and click them when you open a page you'd like to archive
 
If you guys are too lazy to copy-paste links to archive.is/tweetsave/whatever, use these bookmarklets instead:
archive.is:
Code:
javascript:void(open('https://archive.is/?run=1&url='+encodeURIComponent(document.location)))
tweetsave:
Code:
javascript:void(open('https://tweetsave.com/?tweet='+encodeURIComponent(document.location)))
go1dfish (for reddit stuff):
Code:
javascript:if (window.location.hostname.match(/reddit/)) {window.location = 'https://r.go1dfish.me' + window.location.pathname + window.location.search;}

save the shit in the code tag as bookmark and click them when you open a page you'd like to archive
How do you do this in chrome
 
  • Dislike
Reactions: BoredAtHome
Recently, a few different threads have had problems with YouTube based lolcows mass-deleting their videos, so I though I'd help you guys out and recommend a program called "Video Downloader Ultimate"
https://videodownloaderultimate.com/en/
It's a program that pops up in your browser when you turn it on, and if it scans pick up a video in one of your browser tabs, it will be copied to the program itself so you can download it. So, let's say you wanted to download several videos all at once, all you'd have to do is open all of them up in your browser. The full version costs money, but the free version is basically just as good.

As for reuploading them, you could open up your own YouTube archive if you really wanted, but if you're lazy like I am, you can use this little website called mixtape.moe
https://mixtape.moe/
It's sort of like Imgur but instead of just images you can upload a variety of file types, including MP4s and MP3s. The cool part about this website is, you don't need an account to upload stuff, and anybody can view and download said stuff openly as long as you give them the link. So basically, you can download videos, then upload them to mixtape.moe and drop the link here so anybody can view and download them.

A little wordy, but I thought this would be at least a little helpful for you guys.
 
Back