Postmortem September 17th outage and rollback

[Null]

The Kiwi Farms went down at approximately 5:30am EU. The database locked up unexpectedly and the site became unavailable. @CrunkLord420 correctly identified it was a disk issue, so I called in the two guys who help me manage our storage array ("raid"). They diagnosed that all four enterprise raid NVMe harddrives had failed simultaneously, which wiped out the entire database and all forum software. I verified we had a fresh copy of the backup and a recent-ish backup of the actual software.

I had two options moving forward:
1) Pull out remote hands on a US Saturday night and have them determine if the drives can be salvaged and overnight parts to fix whatever has actually broken, or
2) Completely reinstall the entire forum on a different raid until we can do #1 and then move it later.

I decided to go with #2 as the faster but more laborious option, which is why we've had data rollback.

This is was very close to a total nightmare scenario where the server would be completely destroyed. I've reinstalled the Kiwi Farms so much at this point that I know the procedure very well, and it only took about 7 hours, where most of that was just waiting on the database to import over 120,000,000 post stickers.

Anyways, sorry, we're back up. Let me know if there's any weirdness. I'll get chat up in a bit.

 

[The Hero of Kvatch]

I thought trannies were sending mortar fire at Kiwi Farms HQ again.

 

[Peace Or Love]

Adding a couple more stickers to make the next migration slower

 

[TheGuntinator]

How recent was the backup? Sounds like we nearly experienced Total Kiwi Death.

Also LMAO at the fact that trannies can't do as much damage to KF as a hard drive error can.

 

[AbyssGazer]

Every time KF is rebooted a tranny ACKS.

 

[9gfuwegw9j9]

NVMe hard drives?

 

[Null]

How recent was the backup? Sounds like we nearly experienced Total Kiwi Death.

I maintain twice daily backups which I retain for about a month. The attachments are continuously backed up every hour.

 

[Soycast Guy]

I thought it was another Troon attack. Good to see we’re back up though.

 

[Lurker]

thank you noooool

 

[The Hero of Kvatch]

How recent was the backup? Sounds like we nearly experienced Total Kiwi Death.

I checked the Cyraxx thread and 1-2 pages are gone. Yikes. Better than the whole forum being gone.

 

[Peanut Butter in Peril]

All four at once? My inner schizo wants to believe it was deliberate but it's probably not.

 

[whatever I feel like]

Hum, four enterprise harddrives failed simultaneously. Surely thats just a coincidence and not something more serious?

 

[Null]

NVMe hard drives?

I don't know the terminology for drives, I don't even like touching them. They're WD M.2 1.6TB enterprise drives in a frontloading form factor which should not shit out unless I am seriously abusing them with writes continuously.

 

[Lidl Drip]

Phew.

 

[ChromaQuack]

Total Sticker Death.

 

[Ethan Ralph Is 5'1"]

Back status: So

 

[geocities]

Thank you Null.

 

[Null]

Hum, four enterprise harddrives failed simultaneously. Surely thats just a coincidence and not something more serious?

I am guessing there is some sort of adapter on the frontloader's backplate which combines them to a single mobo PCIe slot or something and that shit out.

 

[Sugriva]

Have you ever thought it possible that someone might try to physically sabotage Kiwi Farms?

 

[Dull Pencil]

How likely is it that 4 disks can break simultaneously?