Archival Tools - How to archive anything.

  • 🐕 I am attempting to get the site runnning as fast as possible. If you are experiencing slow page load times, please report it.
Not strictly an archival tool itself, but a way to reduce the file size of PNG images (ie screenshots). This script will reduce the image in-place to 256 colours or fewer, allowing it to be saved as a PNG-8, which takes up a lot less space than a standard PNG-24. It also runs it through a PNG optimiser to squeeze out any unnecessary bytes.

I tend to run all Twitter screenshots through this, though if they've got embedded photos you'll want to make sure you post the photo JPEG as well (as it'll be posterised in the screenshot PNG). Requires imagemagick and oxipng.
Bash:
magick mogrify -monitor -colors 256 -define 'png:format=png8' "$1" \
    && oxipng --opt max --strip safe "$1"
 
Is there like some tutorial on how to use it, assuming I'm not a blind bat who missed it in the files?
I don't think so, but if you're familiar with the shell, grab and extract the latest CLI zip and (on Unix-likes) run DiscordChatExporter.Cli.dll with dotnet.

I don't have it installed currently, but from my shell history:
Code:
dotnet DiscordChatExporter.Cli.dll exportall --help

Just a PSA for snscrape users: Twitter seems to be trying to block access to their frontend API. At the moment you'll want to clone the snscrape repo and install the master branch with pip.
 
Couple of convenience things for iPhone phonefags (I imagine these will also work on iPads, maybe also recent versions of macOS).

Both are these are Shortcuts so you’ll need to add them there. I’m not sharing the file exported from Shortcuts as I’m phoneposting(!) and don’t want to risk doxing myself.

Archive Page​

Press the share button when you’re looking at the web page you want to archive. Choose the “Archive Page” shortcut, which will open a search of archive.today for the URL, where you can see previous captures or, if there aren’t any, trigger a new one.
B06AF6BC-9785-486A-B5F6-D3A966037DB5.jpeg1CCDF03C-A23D-440E-8241-8EAFE04873F5.jpeg
Things to note: if there exists a capture of the page already but you want to re-capture it as it is now, it’s fiddly and you effectively have to do it manually (because the capture link doesn’t shown on the archive search page when there are existing captures).

Also, if you are viewing a tweet on a Nitter instance, it will pick out the /username/status/tweet_id part of the URL and archive the original from Twitter.com. If other (non-twitter) sites have this pattern it’ll catch them too, so let me know if you use this and run into problems — I can add an additional check.

Double Link​

When you‘re on an archive.today (or .ph or whatever) page this will add BBCode links to both the original page and the archived version to your clipboard. For example: (twitter.com, archive.ph)
16D96D3A-2AAD-4311-A51E-A09B305790D0.jpeg39333D00-0833-4FCB-860D-C3053AD105EA.jpeg
Here’s the JavaScript that gets run on the page (the screenshot only shows a portion of it):
JavaScript:
let source = document.querySelector("input[name=q]").value
let archive = document.querySelector("link[rel=canonical]").href

function domain(url) {
  const re = /^https?:\/\/([^\/]+)/i
  let res = re.exec(url)
  return res[1]
}

function bb(source, archive) {
  let s = domain(source)
  let a = domain(archive)
  return `([URL="${source}"]${s}[/URL], [URL="${archive}"]${a}[/URL])`
}
let result = bb(source, archive)
// Call completion to finish
completion(result);

Let me know if you try these and have any problems.
 
  • Like
Reactions: Aspiring Artist
Since I see a lot of retards who do not bother to properly archive Steam links, I might as well write a guide on how to properly do so. Given how easy it is to access this data too, this an also double up as an infosec guide for Steam.

DISCLAIMER:
Given the nature of Steam on archive.vn, such as .vn occasionally catching content warnings on games with no way to accept nor able to check "View More Info" on people's profiles, archive.org might be a better bet for archiving certain Steam content. When archiving with archive.vn, with ID64s, it redirects you to the same page with their custom URL, effectively archiving both the ID64 and custom URL with a single archive, and also when searching from their custom URL, it says that the archive is redirected from the ID64. If you do this with archive.org, I believe it does the same thing, but gives a 302 message and redirects you to their custom URL, but not the other way around, unlike archive.vn.
tl;dr: use archive.vn to archive for convenience, use archive.org to archive pages with content that .vn cannot access/mitigate, which happens more often than not.



Always, ALWAYS, grab the ID64 of the lolcow/group, since they can change it at any time. Because other people are retarded with archiving Steam profiles, you might have use both ID64 and custom URL in order to actually get the full picture of someone's transformation to lolcowdom on Steam, and due to Wayback's archives of Steam being more complete than archive.vn's, archive.org is usually better for accessing older archives of people's Steam profiles, especially for 2000s/early-mid 2010s profiles, as archive.vn was not as prominent or outright nonexistent at those time, and for Steam groups, since next to no one uses id64s because it's not as easy to get as profiles, there might not even be any id64 archives of groups.

PROFILES:
Profiles are braindead easy; if it's not already their main link (you can tell if their profile link contains /profile/{NUMBERS} as opposed to /id/{CUSTOM_URL}) you can go to Steamrep, put their URL into the search, and right below their name, there will be their ID64. Below that, there's a few more links. Most of them can be disregarded, but the steam3ID can be used as explained later.
1673896917159.png
GROUPS:
Groups are a bit more complicated, but still easy; you get the link to the group, add /memberslistxml/?xml=1 at the end, and you will be redirected to XML source of the page. At the very top, in <groupID64>, you can get the group ID for the group. And to access the new permanent link, you replace the groups/{CUSTOM_URL} with gid/{GROUPID64}.
https://steamcommunity.com/groups/Korps/ Basic group link
1673897191007.png
https://steamcommunity.com/groups/Korps/memberslistxml/?xml=1 XML link with groupID64
1673897258732.png
https://steamcommunity.com/gid/103582791429609910 GroupID link, which redirects to the original group.

USER CONTENT:
Now for certain aspects of a person's profile, which is extremely handy if they have a private profile. These are all added after the main profile URL (https://steamcommunity.com/profiles/{ID64}).

-/screenshots, shows the person's screenshots tab
-/images, shows the person's artwork tab
-/videos, shows the person's video tab, and videos on there can be traced to the person's linked Youtube account

-/myworkshopfiles, shows the person's workshop tab, alongside how many folowers they have; it does not show who their followers are
-/myworkshopfiles/?section=merchandise, shows the person's merchandise tab
-/myworkshopfiles/?section=collections, shows the person's collections tab
-/myworkshopfiles/?section=collections, shows the person's guide tab
Adding &p={PAGE NUMBER} at the end of any tab with /myworkshopfiles to go to a specific page of the person's tab

-/posthistory to check their forum posts; /?p={PAGE NUMBER} to go to a certain page of forum posts. NOTE: this only works for posts made on game forums/announcements, it does not work for posts made in group forums.
https://steamcommunity.com/profiles/76561197994787161/posthistory Example link
1673898637308.png

-/friends to check their friends; NOTE: only works if their profile is both public and friends are public, but it does work on community banned profiles if they do have the two prerequisites.

-/allcomments to check their comments in batches of fifty instead of 6 at a time; only works if profile is public, and also works on community banned profiles; /?ctp={PAGE NUMBER} to go to a specific page.

Groups:

-/announcements to get a group's announcements; /?p={PAGE NUMBER} for a specific page

-/discussions to get a group's forum; "/{SUBFORUM NUMBER}" for a specific subforum; zero indexed, /fp={PAGE NUMBER} for a specific page

-/events for group events

-/members for group members; /?p={PAGE NUMBER} for a specific page

-/comments for the comments in batches of fifty; no known way to archive a specific page, as clicking on a different page does not change the url.

-/curation for the group's curator page.

If you want to check a person's post history in a certain group, you can search with /discussions/search/?author={STEAM3ID} at the end of the group's link; &p={PAGE NUMBER} for a specific page.


For archiving a specific page/post on a thread, you go to the thread page, and put in ?ctp={PAGE NUMBER} at the end of the page. This works both on the Steam Community forums and group forums. Do keep in mind that this will archive based on the 15 posts per page setting. To archive a specific post, on the post, on the bottom right, there is the post number in the thread, right click and click "Copy Link Address" for the link. Just like the previous post, the url created assumes you're using the 15 posts per page setting. There is no option for this with the OP, but it does not matter since it appears at the top of every page of the thread.
https://steamcommunity.com/discussions/forum/30/864973577904147767/?ctp=18615#c3758850346186677295 Example link: page 18615, post 279211.
1673898518420.png


STEAMREP DATA:
Under "Historical Data" (right to "SteamRep Profile Overview"), you can check the previous names/profile pictures/friends of the person's profile, which a new listing is added to the person every time their SteamRep is checked. The formatting for each one goes as : {DATE OF ARCHIVE} | {STEAM NAME} | {AMOUNT OF FRIENDS}/{AMOUNT OF STEAMREP BANNED FRIENDS}. You can check the specific data of each profile, but you are required to be signed in with your Steam profile to be able to check. As far as I know, the website does not tell others if you decide to check someone's specific historical data listing.
1673898021198.png
WARNING: There is an option to disable historical data, where some users might opt-in for that, and you will be prompted with this message; you can still check for the historical data for the past month.
1673898140896.png

Steamrep Forums can also be used to find old aliases of a user that otherwise might not be on any other Steam related databases, if the user has posted on the forums (most likely to report someone for a scam/appeal a SteamRep ban). The username on the forums can be set to something different from their Steam name, and cannot be changed at all, period, and right below their SteamRep forum username is their ID64.
1673898175593.png

STEAMID.UK DATA:
Steamid.uk has fallen from grace over the past several years since most of the features have been paywalled, but you are still able to get some data from there that Steamrep cannot offer. Some useful features that aren't paywalled are being able to see the person's ENTIRE name history, although usually the last several letters are censored), the ability to see all of their profile pictures they used, and seeing ALL of their previous custom URLS (later two require you to be logged into Steam). You can also search a profile picture for a higher resolution and get a list of people who are currently using the picture, but this is limited to 10 times per day. There were some other options, like seeing every historical friend they had in a single list and a few other things that I forgot about, but that's all behind a Patreon paywall unfortunately.

Another example with Keffals profile https://steamid.uk/profile/76561197994787161
1673898268106.png
1673898284431.png


Now go out there and archive some lolcows with style!
 
Last edited:
I know I am probably asking a really trivial question, but does anyone know how to archive past the 18+ content warning on Reddit? I am trying to do a thread on /r/ChrisChanSonichu and the retardation that occurs on that sub.
 
Is there something to use to download off anchor.fm or spotify that isnt a browser extension?
 
Is there something to use to download off anchor.fm or spotify that isnt a browser extension?
Not exactly what you're looking for, but it might help:
https://www.listennotes.com/
I used this once before when I wanted to download a lolcow-related podcast episode. The episode was on anchor.fm, and I searched around until I also found it on ListenNotes, which thankfully allows direct audio downloading. I'm not sure if whatever you're looking for will also be there, though. Hopefully!🤞
 
  • Like
Reactions: GenociderSyo
Is there something to use to download off anchor.fm or spotify that isnt a browser extension?
Not particularly convenient (and this doesn’t work for Spotify obviously) but in these cases I tend to crack open the browser dev tools, start the audio playing, and the network tab will point you at the file.
 
Not sure what thread is better to ask this in so I will here. What is a good general account search tool? It having an archival function built in would be good but just something that targets well known social media and forums and searches specifically their userbases based on a name input. I know these tools exist but I'm not sure I've seen them named. There's plenty of crappy ones that pop to the top of search results but either they only run on one os, haven't been updated in years, have little documentation of all of the above. For example https://github.com/xHak9x/finduser
 
  • Like
Reactions: notafederalagent
Not sure what thread is better to ask this in so I will here. What is a good general account search tool? It having an archival function built in would be good but just something that targets well known social media and forums and searches specifically their userbases based on a name input. I know these tools exist but I'm not sure I've seen them named. There's plenty of crappy ones that pop to the top of search results but either they only run on one os, haven't been updated in years, have little documentation of all of the above. For example https://github.com/xHak9x/finduser
This is a good directory of general research tools. It doesn't have any actual archival features itself but you can archive what you find.

Usernames specifically are at the top of the list. The UI looks like this:
Screenshot 2023-02-13 211112.png
 
So I’ve successfully archived a Discord channel. How do I post it? I’m afraid that the raw address would contain my personal info.

EDIT: Excuse me for all these questions I keep piling up, but I can't find how to archive YouTube comments. Using the permalink in archive.md results in a "Not Found (yet?)" message. Same goes for the raw video link.
 
Last edited:
So I’ve successfully archived a Discord channel. How do I post it? I’m afraid that the raw address would contain my personal info.

EDIT: Excuse me for all these questions I keep piling up, but I can't find how to archive YouTube comments. Using the permalink in archive.md results in a "Not Found (yet?)" message. Same goes for the raw video link.
I am working to try to do that same thing for a (probably) different community. Hit me up with a DM, and we can put our heads together. Your tool would help me with my project, and I can probably help you anonymise the data.
 
Did protonmail start requiring email/sms verification or is that only being tripped since I'm on TOR? Is there a decent alternative that will be accepted on most sites?
 
Back