After talking with someone who knows more than I do about hardware, we've deduced the issue is that:
We store 1.5 TiB of static content on the site. This data is accessed all the time. I tried to save money by buying a big HDD, but this is becoming a problem. As we store so much and it's accessed so often, the one harddrive is bottlenecking the site. As a test, I disabled the Cloudflare's CDN, which automatically caches and serves a lot of static content for us. Doing this instantly threw the disk into 100% utilization and crashed the site. As it stands, we already over-utilize what resources I have and only because I've optimized CF to aggressively cache is the site not already dead. The problem will get worse as the site's archive and the utilization of the archive continues to grow.
4+4x3.4GHz is probably just not good enough for a site this size anymore, at least for the front end. I've been fine tuning things since Chch trying to tard cum the server for everything it's worth and now we're at a limit. I have an
entire server dedicated to the backend that is on better hardware with more redundancy, but now I'm going to need to move the front end onto its own hardware. Currently, the KF frontend cohabitates a smaller blade with some other smaller sites.
Also, CrunkLord420 runs the Matrix/Riot and Pleroma instances on Server 4, and that server itself is running out of disk space.
Consider donating via crypto if that's an option for you.
I'm probably going to do a merch run to finance this expansion soon. Will likely try selling stickers since it was highly requested last time. Keep your eyes out for that.
