Postmortem September 17th outage and rollback

[PeggieBigCock]

We’re so back, thank fuck I need the dopamine

 

[Real Fakeman]

More likely than you think if they're rated for the same writes, come from the same batch, are installed at the same time, and used in a raid.

If four SSDs fail at the exact same time I would suspect that it's a problem with the firmware, controller, backplane or software.

 

[Dog-O-Tron 5000v5.0]

An entire days worth of autism. Gone. Like tears in the rain.

It's entirely possible the greatest post I ever made in 30 years of posting online was made yesterday.

Dammed if I can remember what I posted though, but trust me, it was pretty good.

 

[Dynamite Ninja]

The Kiwi Farms went down at approximately 5:30am EU. The database locked up unexpectedly and the site became unavailable. @CrunkLord420 correctly identified it was a disk issue, so I called in the two guys who help me manage our storage array ("raid"). They diagnosed that all four enterprise raid NVMe harddrives had failed simultaneously, which wiped out the entire database and all forum software. I verified we had a fresh copy of the backup and a recent-ish backup of the actual software.

I had two options moving forward:
1) Pull out remote hands on a US Saturday night and have them determine if the drives can be salvaged and overnight parts to fix whatever has actually broken, or
2) Completely reinstall the entire forum on a different raid until we can do #1 and then move it later.

I decided to go with #2 as the faster but more laborious option, which is why we've had data rollback.

This is was very close to a total nightmare scenario where the server would be completely destroyed. I've reinstalled the Kiwi Farms so much at this point that I know the procedure very well, and it only took about 7 hours, where most of that was just waiting on the database to import over 120,000,000 post stickers.

Anyways, sorry, we're back up. Let me know if there's any weirdness. I'll get chat up in a bit.

It's shit like this that makes you the GOAT (Greatest Of All Time) of the Kiwi Farms.

 

[Oxyjen]

I am guessing there is some sort of adapter on the frontloader's backplate which combines them to a single mobo PCIe slot or something and that shit out.

It’s going to be that or a power rail. There’s no way 4 would die simultaneously short of hammer attack or lightning.

Well done and thanks.

 

[Ethan Ralph Is 5'1"]

NO

TOTAL UPDOOT EXISTENCE

 

[Haloperidol]

I got to use ‘niggercattle’ for the first time and now it’s gone, I’m just here to restore balance.

Niggercattle.

 

[Kerry]

Appreciate it.

 

[Gatdam Animal Person]

I heard that Liz Fong Jongs works at WD.
*insert 41% joke here*

 

[Working]

I can post, I CAN SNEEED!!

 

[Oxyjen]

Hard drives are specifically magnetic platter drives. NVMe are pretty much always SSD drives.

NVM means non-volatile memory, which is another name for flash memory, so yes.

 

[AN/ALR56]

@Null thank you for getting stressed and working in a weekend and fighting a half blind chink tranny and the whole entire internet to keep us laughing at retards, I hope you get tons of alcohol, pizza and pussy to make it up to you.

 

[pursuitsnail]

Oh wow. It's really unusual for all four drives to shit themselves all at once. I hope your hands on the scene can figure out if that was anything other than crazy bad luck.

 

[Dojima's Dragon]

We’re so back, thank fuck I need the dopamine

Huh bayo i thought you get your dopamine from killing demons

 

[lurkerplsdontcancelme]

@Null What RAID configuration were the NVME drives in? Please don't say it was RAID-0.

 

[Doseferatu]

Thanks for putting in the extra labor to get the site up and running!



Time to make more cancerous posts that will only prolong additional maintenance in the future.

 

[That_Guy]

"Down for a bit, Restored from backup, Lost a few hours of posts" is pretty damn good for losing the whole disk array. That actually went pretty well. Yay for working backups!

 

[Dude Snakes]

It's good to be back good luck avoiding more issues in the future.

 

[Blue Miaplacidus]

Making a useless post but thanks @Null appreciate you dealing with trannies, discord faggots, journoscum, alogs, and hardware problems just so we can laugh at retards while saying nigger.

 

[PeggieBigCock]

Huh bayo i thought you get your dopamine from killing demons

I’m going to kill Satan himself