Archival Tools - How to archive anything.

Next Highlight
Just informing you that
i had to see bowser dick to get this.png

this line of code allow archive.is to bypass the safe mode filter of tumblr. So now you can freely archive nsfw tumblr blog.
 
  • Informative
Reactions: Isaac Clarke, Coach Kreeton Of All That, NoFeline and 1 other person
Is there any way to download twitch streams after they're done or do I need to dub out a copy in OBS or something?
 
Googleweblight is a service from google with the purpose of aiding your browsing experience... by simplifying everything to the most basic form possible

basic stuff.png

http://googleweblight.com/?lite_url=https://kiwifarms.net/threads/archival-tools.6561/page-3

No pop-up, no fancy css, image were resized to be smaller. It looks ugly, but it serve its intended purpose of making site load faster by reducing everything. Several site like wikia just lose its functionality. So why am I posting this here?

Well it's capability to simplifying everything helps in archiving with archive.is. For example, archiving patreon comments:
adv yandere.png
This is the normal archive
intermediate yandere.png

https://archive.is/4HyiD
This is the GW archive
basic yandere.png

https://archive.is/U0R72

Note that while the GW archive is messy, it retains some comment from the patreon site. The normal archive did not
 
  • Informative
  • Winner
Reactions: anameisaname, Isaac Clarke, walnut and 3 others
If you want to archive videos, streamable.com may be useful. I'm not sure if it's a reliable site but I seem to have no problem with it so far. You don't need an account to upload and the site uses a cookie to keep track of what you upload from your specific browser.
 
  • Informative
Reactions: Isaac Clarke
Just on the off chance this info could be useful, I found this webm online showing how you can use inspect element on a video player to download a video without a program. I know it shows you how to download videos from some sketchy streaming site, but I figured it might be able to be applied to unusual video players/ video players without a downloader online.
 

Attachments

  • Informative
Reactions: highdegger, AnOminous and Wärring Ornac
I'm hesistant to use such a service since its longevity hasn't been proven. In comparison, archive.org will almost certainly last for many years, though they may takedown copyrighted content. Archive.is is used frequently and seems like it will last, though not with the certainty of the Internet Archive.
 
If you want to archive an entire account on a site with infinite scroll, such as ask.fm, or Twitter, forget archive.is, it's shit.

Instead, infinite scroll down to the oldest tweet/question (preferably using 'tweets with replies', rather than 'tweets'), then 'save complete web page' (in chrome) or probably other browser.

seems to work fine.

note that for 'tweets with replies' you need to be logged in. So you'll want to use a completely anonymous twitter account for this purpose, or you'll need to remove all traces of doxxing yourself.
 
unclefucker said:
Instead, infinite scroll down to the oldest tweet/question (preferably using 'tweets with replies', rather than 'tweets'), then 'save complete web page' (in chrome) or probably other browser.

seems to work fine.
Click to expand...
One problem with doing it this way is that the libraries required to correctly display the site might not be available forever. I think you can scroll down to the beginning as you say, then use "full page caputre" which printscreens the whole page with the simple click of a button. I use a Chrome-extension named "Full Screen Capture" which exports to png IIRC.

You can of course also download the HTML as you suggest and download what you can of the CSS and JS required to display the site but I'm not sure how comprehensible the result will be.
 
dannyfrickenp said:
Just on the off chance this info could be useful, I found this webm online showing how you can use inspect element on a video player to download a video without a program. I know it shows you how to download videos from some sketchy streaming site, but I figured it might be able to be applied to unusual video players/ video players without a downloader online.
Click to expand...

If you want to download videos use youtube-dl

https://rg3.github.io/youtube-dl/download.html

On a Mac you just type

Code: 
brew install youtube-dl

On Windows just run the exe file

On Linux you can install with

Code: 
sudo pip install --upgrade youtube_dl

or with

Code: 
sudo curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl
sudo chmod a+rx /usr/local/bin/youtube-dl

Or maybe your distro's package manager has it.

It's a really nice tool - you can do

Code: 
youtube-dl -F <youtube_url>

to see a list of formats ant then

Code: 
youtube-dl -f <format> <youtube_url>

to download. Generally I just use

Code: 
youtube-dl -f mp4 <youtube_url>

I.e. format=mp4 for video+audio and

Code: 
youtube-dl -f mp4 <youtube_url>

I.e. format=m4a for audio only and just get the default quality. mp4 is H.264 video and AAC audio, m4a is just AAC audio and both are pretty widely supported, aka 'the GPU in your old ass phone and computer can probably decode them with low CPU usage'.
 
  • Informative
Reactions: Cow Poly, A_Single_Egg and You Promised
Wärring Ornac said:
Ah, I get it now

Speaking of archive.today, I advised anyone who failed to archived a page with it to use via.hypothes.is
Example:
https://mastodon.xyz/@donleykogn
View attachment 402340

Now if you use via.hypothes.is : https://via.hypothes.is/https://mastodon.xyz/@donleykogn
View attachment 402341

EDIT: So apparently if you try to archive deviantart with archive.today, this is what you got:
View attachment 403248

View attachment 403256


I've tried this with several blog, but the result is still the same. But it turns out that using via.hypothes.is is also a solution to this problem:
View attachment 403257
View attachment 403258
Click to expand...
I'd like to add that with Mastodon instances you will want to use this. Just about all mastodon instances tend to block archival tools, but archiving it through a proxy (like the hypothes.is trick does) to archive it works. You'll need to look in the source code to see spoilered/content warning posts though.
 
I remember from a long time ago one neat thing Internet Explorer supported was saving a webpage as MHTML, which is like saving the complete website but all in one file. The same effect can be achieved by zipping the folder with all the page resources for a little less convenience.
 
I found a tool that might be useful for saving ongoing live streams it's called streamlink and it works on the Linux command line and in windows power shell (If you have python installed)

Link to the GitHub project: https://github.com/streamlink/streamlink

To download a live stream from the start while it is in progress you use this command:
Code: 
streamlink <URL to twitch stream or YouTube stream>  best --hls-live-restart -o out.mp4



 
  • Like
  • Informative
Reactions: awoo, You Promised and Ask Jeeves
Nobody4353 said:
I found a tool that might be useful for saving ongoing live streams it's called streamlink and it works on the Linux command line and in windows power shell (If you have python installed)

Link to the GitHub project: https://github.com/streamlink/streamlink

To download a live stream from the start while it is in progress you use this command:
Code: 
streamlink <URL to twitch stream or YouTube stream>  best --hls-live-restart -o out.mp4
Click to expand...

Looks like there's a homebrew version too for MacOS.

brew install streamlink

Also looks like it has a lot of other options, including crunchyroll (!).
 
Last edited:
  • Informative
Reactions: anameisaname, Nobody4353 and You Promised
You must log in or register to reply here.
Back