I wish I had used Elasticsearch from the beginning, to be honest. Postgres is slow, and people running massive queries that take 15 minutes is not a great idea.
Alright I'll concede I was wrong about that
I think I finally realized what you were talking about, and it's the mutual guilds, not the messages. Scraping every single member in a server sucks because Discord wanted to avoid yet another Dis.cool. There are two main methods of getting every single member in a server:
- Scraping the sidebar: This only scrapes online members and takes a very long time (~30 minutes for 15K members) due to ratelimits on Discord's side.
- Bruteforcing the search box: This takes even longer than the sidebar, mostly due to Discord adding absolutely massive ratelimits on it. This also returns offline members, but that's outweighed by the time it takes.
This was also what I was experimenting with in the two days the site was offline.
That's why I chose fetching directly from Discord every single time someone looks up a user, due to it being more reliable, which is what also made it easy for Discord to find the accounts I was using. My only remaining theory is that Discord added a Cloudflare WAF rule to log every single request with an ID they look up, which then basically leads them straight to the accounts.
I was able to find another way to do the same, but since it's meant for something else other than checking someone's profile, only their member JSON is returned (no pronouns, no connected accounts), which is why that currently doesn't work.
Some other stuff that broke was the cool graphs on server pages due to Discord making ratelimits on fetching servers absolutely gigantically high. It also affects normal users (you can't leave more than 3 servers per 5 minutes because it ratelimits you).
Edit: Money isn't the main thing about Spy.pet; it's more about the scraping itself. All of the infrastructure is still paid for from my own pocket, and all the money Spy.pet has ever made is still in a Trezor under the plants in my garden, remaining completely unused. I've also adjusted it so that the messages are backed up every 24 hours instead of every week.
is the crypto you do send through cryptomus anonymous, or should I use a burner wallet address?
I'm pretty sure Cryptomus uses an external service for wallets. If you're really concerned, use XMR.