Postmortem September 17th outage and rollback

  • 🏰 The Fediverse is up. If you know, you know.
  • Want to keep track of this thread?
    Accounts can bookmark posts, watch threads for updates, and jump back to where you stopped reading.
    Create account
Null said:
More likely than you think if they're rated for the same writes, come from the same batch, are installed at the same time, and used in a raid.
Click to expand...
If four SSDs fail at the exact same time I would suspect that it's a problem with the firmware, controller, backplane or software.
 
Null said:
The Kiwi Farms went down at approximately 5:30am EU. The database locked up unexpectedly and the site became unavailable. @CrunkLord420 correctly identified it was a disk issue, so I called in the two guys who help me manage our storage array ("raid"). They diagnosed that all four enterprise raid NVMe harddrives had failed simultaneously, which wiped out the entire database and all forum software. I verified we had a fresh copy of the backup and a recent-ish backup of the actual software.

I had two options moving forward:
1) Pull out remote hands on a US Saturday night and have them determine if the drives can be salvaged and overnight parts to fix whatever has actually broken, or
2) Completely reinstall the entire forum on a different raid until we can do #1 and then move it later.

I decided to go with #2 as the faster but more laborious option, which is why we've had data rollback.

This is was very close to a total nightmare scenario where the server would be completely destroyed. I've reinstalled the Kiwi Farms so much at this point that I know the procedure very well, and it only took about 7 hours, where most of that was just waiting on the database to import over 120,000,000 post stickers.

Anyways, sorry, we're back up. Let me know if there's any weirdness. I'll get chat up in a bit.
Click to expand...

It's shit like this that makes you the GOAT (Greatest Of All Time) of the Kiwi Farms.
 
Null said:
I am guessing there is some sort of adapter on the frontloader's backplate which combines them to a single mobo PCIe slot or something and that shit out.
Click to expand...
It’s going to be that or a power rail. There’s no way 4 would die simultaneously short of hammer attack or lightning.

Well done and thanks.
 
I got to use ‘niggercattle’ for the first time and now it’s gone, I’m just here to restore balance.

Niggercattle.
 
@Null thank you for getting stressed and working in a weekend and fighting a half blind chink tranny and the whole entire internet to keep us laughing at retards, I hope you get tons of alcohol, pizza and pussy to make it up to you.
 
"Down for a bit, Restored from backup, Lost a few hours of posts" is pretty damn good for losing the whole disk array. That actually went pretty well. Yay for working backups!
 
Making a useless post but thanks @Null appreciate you dealing with trannies, discord faggots, journoscum, alogs, and hardware problems just so we can laugh at retards while saying nigger.
 
You must log in or register to reply here.
Back
Top Bottom