What happened?
On Thursday, 29th of August at around 9 pm UTC, someone managed to exploit a vulnerability in the site's code which allowed them to trick the site software into deleting images. We realised what was happening within an hour of the incident and shut down the server to prevent any further damage. Unfortunately by the time we had shut down the server, they had already deleted all of the images on /ot/. We believe they wrote a script to automate this behaviour which is why they were able to do it so quickly.
The only thing that was affected was the images in /ot/. No user data has been compromised, and there has been no impact to user privacy.
Who did this?
We have not been able to find anyone claiming responsibility for the attack online (but please let us know if you have!), although we believe it may have been someone without any history with the site. We received an email from someone trying to ransom money in exchange for details about a vulnerability, but it's unclear whether they were the attacker or an opportunist trying to take advantage of the situation. They have made no further contact.
Why did the site have to be taken down?
We had to bring the site down in order to protect it from further damage, i.e. other boards being wiped. Additionally, shutting down the server immediately increased the chances that we could recover the deleted files if we needed to.
Why are the images still missing?
Unfortunately, all of the image backups going back to 2019 were corrupt and very little could be recovered from them. We made the mistake of assuming that the backup script that had been used all that time had been functioning correctly, and evidently, it had not.
As a result, we’ve had to resort to other methods to try and recover the images. This is an ongoing process, and we are going to keep an updated count stickied in /ot/ to be transparent about how that is going. At the time of writing this, we’ve recovered just over 10% of the images and about 47% of thumbnails.
Some good news is that we have been able to recover a large number of the images through forensic data recovery. Unfortunately, the original image names/locations have been lost and as a result, we have no easy way to tie them back to the posts they belong to. We have a script that is processing every image and trying to link them to their posts based on the metadata we’ve stored about the image, but it is a very slow process.
Additionally, we’ve written a script that is attempting to recover the images from the Wayback Machine via its API. Unfortunately, that also runs quite slowly due to rate limits imposed by the Internet Archive.
Why did it take so long to fix?
There were several reasons:
- We were not willing to bring the site back online until we were sure that it was safe to do so
- We could not turn the server on at all until we had created a complete copy of the disk and transferred it to one of the admin’s local machines for data recovery. Given that the server disk is multiple TB in size, it took 5 days to make and then transfer the image
- Our hosting provider was utterly worthless throughout all of this, which delayed us starting the process of transferring the disk image. They first refused to assist us in getting a local data recovery specialist to deal with it (which would have saved us the 5 day transfer) and then took a very long time to set up the rescue image we needed to make the transfer itself
- Only one of the admins had the experience necessary to work on this issue, and she had to fit this around her day job (unfortunately she does not live in her mother’s basement like null)
What have we done to stop this from happening again?
First of all, we’ve purchased a robust cloud-based backup system that is independent of our hosting, which we have already tested to our satisfaction. No matter what goes wrong in the future, the site will be safe and restoring lost data will be much easier/faster. We didn’t do this previously as it’s quite expensive, but given the trouble we’ve had with this incident, we have decided to eat the cost.
We have also combed through the site code and patched anything that could potentially be exploited. We are being intentionally vague on the details as we don’t want to give away information to people who may use it against other sites running the same board software. We have also implemented additional security features that should prevent anything similar from happening in the future, even if we haven’t plugged every hole.
Tl;dr - The site was attacked, and all the images on /ot/ were deleted. Our backups were not viable, so we are slowly working to restore the images through other means. We have improved the security of the site in response to the issue, and invested in a good backup solution to better protect the site in future