Archival Tools - How to archive anything.

Does anyone know how to download/archive an "Instagram Live" video? Most IG downloader apps only work if the user has saved the video to their "story", but the one I am after was only a livestream and it expires about 20 hours from now. Worst case scenario I'll use a screen recorder app but that will wreck the quality especially as most of the content I want to record is not so much the video as things happening in the on-screen comments, which I need to be readable as a consequence.

Any pointers? Windows or Android. Running out of time here!
This works for stories, so it may work for a saved live.

Go here and enter the IG name in the field. All of their "stories" will show up. Click the save button below it, then save to your computer.

Update if it works for a live or not.

edit: this only works for public accounts, not private. ALSO, visit their profile through there. It will show highlights, saved vids, and saved stories, then follow above instructions.
 
  • Informative
Reactions: Serial Grapist
No luck, that one can only do saved stories. I've used a screen recorder and the quality is acceptable. Would be good to have a reliable method to get these IG Lives though that doesn't require you to spend hours with your phone recording itself, I'm tracking a live clusterfuck to document for a future OP and most of the drama is on Instagram, where it's very easy to DFE (and likely, in this case, because it involves batshit cultists). Thanks anyway dude.
 
Does anyone know how to download/archive an "Instagram Live" video? Most IG downloader apps only work if the user has saved the video to their "story", but the one I am after was only a livestream and it expires about 20 hours from now. Worst case scenario I'll use a screen recorder app but that will wreck the quality especially as most of the content I want to record is not so much the video as things happening in the on-screen comments, which I need to be readable as a consequence.

Any pointers? Windows or Android. Running out of time here!
Haven't tried this (as I don't have an Instagram account), but it works in general for most means of HTML5 video playback, on Twitter, FB, new sites etc. You need youtube-dl.

Go to the video page in Chrome/Brave/whatever. I assume this video doesn't autoplay. Open the developer tools with F12. Open the network tab. Now, return to the page, start playing the video. On the Network tab of the Developer Tools, you should see one or more m3u8 files being retrieved. The first one is likely the 'master' reference for the video, so right click, copy the URL, then run youtube-dl with that as the parameter:
Code:
youtube-dl https://example-cdn.com/blah/video.m3u8
Alternatively it might be a .ts file. Downloading a few things and playing them is the only way to be sure.
 
Here's a handy helper function in Bash for getting an mp4 file at a specified resolution

Put this in ~/.bash_profile (or whatever your Unix distro's bash startup file) is and then do a source ~/.bash_profile do load it into bash's memory.

Code:
function ytdl
{
    youtube-dl -f "bestvideo[height<=${1}][ext=mp4]+bestaudio[ext=m4a]/best[height<=${1}][ext=mp4]" "${2}" "-o%(title)s-%(id)s-%(height)sp.%(ext)s"
}

This takes two parameters,

$1 is the resolution
$2 is the URL

What that rather impressively ugly command line means is 'get the best video only stream, subject to the constraints that the format is mp4 and the height is less than or equal to $1, get the best audio-only stream subject to the constraint that the format is m4a and then combine them to an mp4 or, alternatively get the best video/audio stream subject to the constraint that the format is mp4 and the resolution is less than or equal to $1 from the URL $2. Name the output something like "Video Title-Youtube ID-360p.mp4"

Now you will say 'Why not grab the highest resolution video"? Well because the Farms will often refuse that with the error 'file too large'. I figure uploading a 360p or even 240p version of a video is better than no upload at all. Also if you've got a video of someone talking even 1080p might be overkill. I normally get the 360p version, or, if that is too large, the 240p one.

Here's what it looks like working. I picked this as an example because it's only a couple of minutes long

Code:
$ ytdl 240 https://www.youtube.com/watch?v=6GqweJZ66ns
[youtube] 6GqweJZ66ns: Downloading webpage
[youtube] 6GqweJZ66ns: Downloading video info webpage
[download] Destination: Teddy Hart Back In Jail-6GqweJZ66ns-240p.f133.mp4
[download] 100% of 699.12KiB in 00:00
[download] Destination: Teddy Hart Back In Jail-6GqweJZ66ns-NAp.f140.m4a
[download] 100% of 1.94MiB in 00:00
[ffmpeg] Merging formats into "Teddy Hart Back In Jail-6GqweJZ66ns-240p.mp4"
Deleting original file Teddy Hart Back In Jail-6GqweJZ66ns-240p.f133.mp4 (pass -k to keep)
Deleting original file Teddy Hart Back In Jail-6GqweJZ66ns-NAp.f140.m4a (pass -k to keep)

It shouldn't be hard to get this working as a Windows batch file. There the parameters are referenced %1, %2 etch instead of $(1), $(2). That means you might need to escape any % signs such as the ones in the output filename specifier.

Or if that doesn't work if you look at what formats youtube tends to use it looks like this

Code:
youtube-dl -F https://www.youtube.com/watch?v=6GqweJZ66ns | grep mp4
140          m4a        audio only tiny  130k , m4a_dash container, mp4a.40.2@128k (44100Hz), 1.94MiB
160          mp4        256x144    144p   44k , avc1.4d400c, 30fps, video only, 417.17KiB
133          mp4        426x240    240p   73k , avc1.4d4015, 30fps, video only, 699.12KiB
134          mp4        640x360    360p  175k , avc1.4d401e, 30fps, video only, 1.78MiB
135          mp4        854x480    480p  353k , avc1.4d401f, 30fps, video only, 3.41MiB
136          mp4        1280x720   720p  694k , avc1.4d401f, 30fps, video only, 6.50MiB
18           mp4        640x360    360p  364k , avc1.42001E, mp4a.40.2@ 96k (44100Hz), 5.46MiB
22           mp4        1280x720   720p  562k , avc1.64001F, mp4a.40.2@192k (44100Hz) (best)

You can specify muxed video and audio directly as video_fmt+audio_fmt and it will automagically mux them for you

So 360p is 134+140, 240p is 133+140 in this example. It seems kinda stable too.

Another option, of course, is to grab the best quality video and then transcode it. E.g. for 240p

Code:
ffmpeg -i "BigVideo.mp4" -vf "scale=320:240" -c:v libx264 -preset medium -crf 23 -c:a copy "BigVideo-240p.mp4"

This is surprisingly fast on a modern machine, about 60x real-time on an 8th gen, quad-core i5 U series chip which is a mid-range mobile chip by modern standards. So you can encode a minute of video per second.

Or if the thing you're archiving is just talking heads you can grab just an audio stream directly and convert it to mp3

Code:
youtube-dl --extract-audio --audio-format mp3 -f m4a https://www.youtube.com/watch?v=6GqweJZ66ns

The idea here is that you grab the m4a audio and convert to mp3 which The Farms media player seems occasionally support.

Last but not least, get a mega.nz account. There you can upload humongous files for free and when it runs out of space, just get another account.
 
Last edited:
Well because the Farms will often refuse that with the error 'file too large'.
If it's important you can message null and he'll archive it for you
idk about right now tho because seems like he's got his hands full at the moment
 
The idea here is that you grab the m4a audio and convert to mp3 which The Farms media player seems occasionally support.
If all you have is audio I don't believe the uploader will take things with .m4a or .mp3 as the extension.

.m4a's at least will probably work if you rename them to .mp4, as .m4as use the MP4 container format anyway. I have also uploaded MP3's in an MP4 container, though you might not even have to do that, renaming them to .mp4 might be enough.
 
Come to think of it, you can update that ytdl function comme ça

Bash:
function ytdl
{
    if [[ $1 == "mp3" ]];
    then
        youtube-dl --extract-audio --audio-format mp3 -f m4a $2
    else
        youtube-dl -f "bestvideo[height<=${1}][ext=mp4]+bestaudio[ext=m4a]/best[height<=${1}][ext=mp4]" "${2}" "-o%(title)s-%(id)s-%(height)sp.%(ext)s"
    fi
}

That way you can do

Bash:
ytdl mp3 url

to get audio or

Bash:
ytdl 360 url

to get the 360p video or

Bash:
ytdl 240 url

to get the 240p video
 
Some reason, archive.li won't capture twitter images in posts properly. I'll have to find another way.
 
What's the most effective way to archive/download an entire instagram gallery? As with most sites with some kind of login requirement, a lot of the standard archive methods don't work very well. Is there an all-in-one program or plugin(I tend to use brave) that can just grab it all easily?
 
What's the most effective way to archive/download an entire instagram gallery? As with most sites with some kind of login requirement, a lot of the standard archive methods don't work very well. Is there an all-in-one program or plugin(I tend to use brave) that can just grab it all easily?
JDownloader pulls the photos down fine.
 
This is the right place for this. I'm gonna give a quick tutorial on how to archive entire YouTube Channels.

1. Be using GNU/Linux
2. Install youtube-dl
3. Create the following 4 files:

yt.conf
Bash:
-o "%(uploader)s/%(title)s-%(id)s.%(ext)s"
                                    ###         The -o flag tells youtube-dl where to store the files
--no-overwrites                     ###         Do not overwrite files
--continue                          ###         Force resume of partially downloaded files.
--ignore-errors                     ###         Continue on download errors, for example to skip unavailable videos in a playlist
--verbose                           ###         Print various debugging information
--download-archive archive.txt      ###         Keep a text archive of the videos you downloaded
--username=YourUsername@You.com     ###         Your YouTube Username. /// Feel free to remove this line if you aren't archiving age-gated content
--password=YourPAssword             ###         Your YouTube Password. /// Remove this line if you removed the username line         
--batch-file channels.txt           ###         Telling youtube-dl to download the URLs contained in the file 'channels.txt'
--format 'bestvideo+bestaudio'      ###         Telling youtube-dl to download the best possible video and audio quality
--write-description                 ###         Write video description to a .description file
--write-info-json                   ###         Write video metadata to a .info.json file
--write-annotations                 ###         Write video annotations to a .annotations.xml file
--write-thumbnail                   ###         Save the thumbnail
--all-subs                          ###         Download all the available subtitles of the video
--merge-output-format mkv           ###         Take the 'bestvideo+bestaudio' files you downloaded and mux them into an mkv file.


archive.txt
This file should be left blank

channels.txt
list all channels or videos you want to archive.

yt.sh
youtube-dl --config-location yt.conf

After this is all done, run the script with following command:
bash yt.sh

You can also set up a cron job to archive stuff automatically every however often you want.
 
Last edited:
I'm more in favor of understanding what the flags do before using someone's script.
I think bestvideo+bestaudio is by default now. -w might be covered by -a (not sure about this one). --username and --password are not needed unless they are private videos and I generally wouldn't tell youtube account info unless it's absolutely necessary. -i is generally good for scripts but you want to make sure to check errors. Annotations and description are covered by info-json (I checked). I think all-subs is unnecessary; thumbnails are up to you.
 
I'm more in favor of understanding what the flags do before using someone's script.
I think bestvideo+bestaudio is by default now. -w might be covered by -a (not sure about this one). --username and --password are not needed unless they are private videos and I generally wouldn't tell youtube account info unless it's absolutely necessary. -i is generally good for scripts but you want to make sure to check errors. Annotations and description are covered by info-json (I checked). I think all-subs is unnecessary; thumbnails are up to you.

Admittedly, that is an old script and I haven't bothered reading any youtube-dl documentation in a while. I put the username and password for videos that are age-gated.


When you archive 100s of channels like I do, you run into errors like this a lot.
Refetching age-gated info webpage
ERROR: Content Warning
This video may be inappropriate for some users.
Sign in to confirm your age
I recommend you make another account just for archiving.

>I'm more in favor of understanding what the flags do before using someone's script.
Feel free to redo it but with comments on each line.


EDIT: I added comments. Now you know you're not downloading a super virus with youtube-dl
 
Last edited:
Here's another idea. Record the live stream direction, code stolen shamelessly from here

https://archive.vn/wip/pE2JN

Consider the following function (I took out the -t 00:02:00 because I want to grab the whole thing, not just 2 minutes)

Bash:
function yt_record
{
    ffmpeg -i $(youtube-dl -f 91 -g  ${1}) \
    -i $(youtube-dl -f 95 -g ${1} ) \
    -c:v copy -c:a aac -strict experimental output.mp4
}

I tested it with this which was live at the time of writing

https://www.youtube.com/watch?v=d4ZV2zCq848

Code:
$ youtube-dl -F https://www.youtube.com/watch?v=d4ZV2zCq848
[youtube] d4ZV2zCq848: Downloading webpage
[youtube] d4ZV2zCq848: Downloading m3u8 information
[youtube] d4ZV2zCq848: Downloading MPD manifest
[info] Available formats for d4ZV2zCq848:
format code  extension  resolution note
91           mp4        256x144    HLS  269k , avc1.4d400c, 30.0fps, mp4a.40.5@ 48k
92           mp4        426x240    HLS  507k , avc1.4d4015, 30.0fps, mp4a.40.5@ 48k
93           mp4        640x360    HLS  962k , avc1.4d401e, 30.0fps, mp4a.40.2@128k
94           mp4        854x480    HLS 1282k , avc1.4d401f, 30.0fps, mp4a.40.2@128k
95           mp4        1280x720   HLS 2447k , avc1.4d401f, 30.0fps, mp4a.40.2@256k (best)

You use it like this

yt_record https://www.youtube.com/watch?v=d4ZV2zCq848

It's a bit limited because it assumes that format codes 91 and 95 exist, that 91 is the audio and 95 is the video. It should probably do something like ytdl does and try to figure out the best ones at runtime. Also, it looks like it is reencoding the AAC audio and not copying it, and it is not clear if this is necessary. Lastly, for this stream it looks like both 91 and 95 are audio and video muxed together. It's not clear if this will always be the case and the superuser post seems to assume 91 is audio and 95 is video, reencodes the audio, and then muxes it to the video. It complains about http keepalives not working. I think this is because of the host changing in the .m3u8 file, or maybe it's because the audio and video streams are on different hosts and ffmpeg gets confused.

Bash:
[https @ 0x7ff33887ce00] Cannot reuse HTTP connection for different host: r5---sn-5hne6ns6.googlevideo.com:-1 != r5---sn-aigl6ned.googlevideo.com:-1

[hls @ 0x7ff33a000000] keepalive request failed for '[redacted]' when opening url, retrying with new connection: Invalid argument

Despite all that it works. output.mp4 was created. I killed the process with ctrl+C, which is obviously not ideal. Presumably, the last part of the video will be corrupted by that. Then again H.264 and AAC are designed to be streamable and, hopefully, your decoder won't crash when it hits the end of the rails. VLC just stops playing.

I've got no idea if the process will just stop when the stream ends.

It needs a bit more work, but it seems promising.



Another interesting thing I noticed with PPP's stream is that even after the channel was down Youtube would still play the stream if you kept the player window open. I think this implies the .m3u8 playlists and the TS files stay up. This means if you got the URLs to them with -g you'd still be able to capture the stream for some time after a channel takedown.

@Guntleman hopefully some of this information may prove useful the Autistic Crusade.

Here's an improved version

Bash:
function yt_record
{
    echo     ffmpeg -i $(youtube-dl -g ${1} ) -c copy out.mp4
    ffmpeg -i $(youtube-dl -g ${1} ) -c copy out.mp4
}

Did a test here

https://www.youtube.com/watch?v=YBdBBU-kSgQ

See out.mp4

Here's version 3

Right now this stream is live

https://www.youtube.com/watch?v=B8a6Dp5bitY

Let's look at what formats are available

Bash:
$ youtube-dl -F https://www.youtube.com/watch?v=B8a6Dp5bitY
[youtube] B8a6Dp5bitY: Downloading webpage
[youtube] B8a6Dp5bitY: Downloading m3u8 information
[youtube] B8a6Dp5bitY: Downloading MPD manifest
[info] Available formats for B8a6Dp5bitY:
format code  extension  resolution note
91           mp4        256x144    HLS  197k , avc1.42c00b, 30.0fps, mp4a.40.5@ 48k
92           mp4        426x240    HLS  338k , avc1.4d4015, 30.0fps, mp4a.40.5@ 48k
93           mp4        640x360    HLS  829k , avc1.4d401e, 30.0fps, mp4a.40.2@128k
94           mp4        854x480    HLS 1380k , avc1.4d401f, 30.0fps, mp4a.40.2@128k
95           mp4        1280x720   HLS 2593k , avc1.4d401f, 30.0fps, mp4a.40.2@256k
96           mp4        1920x1080  HLS 4715k , avc1.640028, 30.0fps, mp4a.40.2@256k (best)

Now consider the following function

Bash:
function yt_record
{
    ffmpeg -i $(youtube-dl -f "bestvideo[height<=${1}][ext=mp4]+bestaudio[ext=m4a]/best[height<=${1}][ext=mp4]" -g ${2} ) -c copy out.mp4
}

This works like ytdl - you pass the max resolution as the first parameter and the URL as the second.

E.g. if you do

yt_record 720 https://www.youtube.com/watch?v=B8a6Dp5bitY

You will get you a 720p capture. I will not upload this foulness here, but it works. You should probably see what formats are available first and then pick one. Because the comparison is <= if you pick a format which isn't available you'll get the next lowest one. E.g.

yt_record 700p https://www.youtube.com/watch?v=B8a6Dp5bitY

Would get you the 480p in this example. To give you some idea of file sizes consider this stream


Code:
$ ytdl -F https://www.youtube.com/watch?v=YBdBBU-kSgQ
140          m4a        audio only tiny  135k , m4a_dash container, mp4a.40.2@128k (44100Hz), 84.04MiB
160          mp4        256x144    144p  160k , avc1.4d400c, 30fps, video only, 44.08MiB
133          mp4        426x240    240p  298k , avc1.4d4015, 30fps, video only, 102.63MiB
134          mp4        640x360    360p  690k , avc1.4d401e, 30fps, video only, 199.30MiB
135          mp4        854x480    480p 1203k , avc1.4d401f, 30fps, video only, 353.82MiB
136          mp4        1280x720   720p 2373k , avc1.4d401f, 30fps, video only, 712.49MiB
137          mp4        1920x1080  1080p 4849k , avc1.640028, 30fps, video only, 1.35GiB
18           mp4        640x360    360p  427k , avc1.42001E, 30fps, mp4a.40.2@ 96k (44100Hz), 277.58MiB
22           mp4        1280x720   720p 1225k , avc1.64001F, 30fps, mp4a.40.2@192k (44100Hz) (best)

I.e. a one and a half hour stream is 199MB at 360p, 712MB at 720p and 1.35GB at 1080p. If you want to shrink a video to reduce the size, do something like this

ffmpeg -i in.mp4 -vf "scale=320:240" -c:v libx264 -preset medium -crf 23 -c:a copy output.mp4

This will reencode to 320x240 but you can easily change that. It seems like this produces 240p versions of 360p videos which are about half the size, which is the same ratio Youtube gives. Then again it's likely that Youtube is using ffmpeg or a fork of it in production.
 

Attachments

  • out.mp4
    14.9 MB
Last edited:
I'll throw in this script here now that Streamlink is finally straightforward to install on Windows, only taking a few clicks.

Youtube-dl is great but Streamlink is necessary whenever there's a risk of the stream getting flagged before it's even over, or if there's a risk of VOD getting deleted by the streamer soon after. In those cases youtube-dl doesn't cut it.
  1. Download and install the latest version of Streamlink
  2. Make sure you have the Visual C++ 2010 Redistributable installed if it isn't already (otherwise Streamlink may not save streams correctly)
  3. Save the script attached here in the folder where you want to save streams
  4. Change the script's extension to .bat so you can run it (forum blocks upload with .bat extension)
  5. Make sure the script is looking in the right place for Streamlink (C:\Program Files (x86)\Streamlink by default)
  6. Run
You can feed the script a URL for a stream that hasn't started yet, and it will wait and check once a minute until it starts. Unfortunately Youtube recently changed the way it handles rewinding, so as of now Streamlink can only capture from the point where it started recording. Better than nothing though.

[Edit 2020-06-22: Fixed the script stupidly ignoring the given requested filename]

Code:
@echo OFF
title %~n0
:: Drop this script where you want to save streams to, and make sure
:: the STREAMLINK_DIR variable is set to the location of your Streamlink
:: install.
::
:: Windows installer:
::     https://github.com/streamlink/streamlink/releases/latest
:: Option documentation:
::     https://streamlink.github.io/cli.html
::
:: Note: Youtube broke rewinding using HLS in Apr 2020, so options for
::       that won't work.

set STREAMLINK_DIR=C:\Program Files (x86)\Streamlink

:: CHECK_DELAY: Seconds between checks to see if stream has begun
set CHECK_DELAY=60
:: RECONN_TIMEOUT: Max seconds waiting for a stalled stream to restart
set RECONN_TIMEOUT=300

set OUT_DIR=%cd%
cd %STREAMLINK_DIR%
:PROMPT
set /p URL="URL: "
set FN=out.ts
set /p FN="OUTPUT [out.ts]: "
set QUAL=480p
set /p QUAL="QUALITY [480p]: "
streamlink.exe "%URL%" %QUAL% --retry-streams %CHECK_DELAY% --hls-timeout %RECONN_TIMEOUT% --force -o "%OUT_DIR%\%FN%"
echo.
goto :PROMPT
 

Attachments

Last edited:
For some reason, archive.md has been giving me the 403 forbidden error
Screenshot_2020-05-24 403 Forbidden.png
 
  • Dumb
Reactions: BingBong
I think it's on your end. I can use the site just fine. No errors at all.
I get a accept the risk warning before, I try it. I don't know what it's problem is.

I actually found a thread for this, lol I'm retarded

Sorry about that, wish I could delete this now
 
  • DRINK!
Reactions: BingBong
Back