Postmortem September 17th outage and rollback

[MunsterCheeseJr]

What is your RAID configuration? Raid 5? Raid 5 + Spare? Raid 6 + Spare is looking quite attractive.

Simple Raid 5 configurations thrash around like a mother fucker once a single drive goes down. Not unusual for an additional drive to bite it during that time period.

 

[Internet Brain Rot]

I maintain twice daily backups which I retain for about a month. The attachments are continuously backed up every hour.

Nice. What software do you use for backups? Does it work well? I've been considering Duplicati for my NAS, but I'm curious as to what you use for such a heavy workload.

 

[Banana Hammock]

9/17, never forget

 

[Friend of Dorothy Parker]

Just - thank you for your work, patience, and fortitude, @Null. The technicals of all the various potential failure spots are beyond my knowledge, but I appreciate and thank you for everything you do.

 

[Null]

Nice. What software do you use for backups? Does it work well? I've been considering Duplicati for my NAS, but I'm curious as to what you use for such a heavy workload.

It's just mysqldump -> lzop -> s3-style service (local s3 + cloud). I use a FIDO2 key for all my cloud shit. The script for the archival is just a bash script I wrote years ago.

 

[TheClappening]

Four NVMEs simultaneously conking out seems bonkers to me. Then again I fix boomer laptops, not data centers and enterprise grade gear.

 

[UERISIMILITUDO]

I've just participated in the time-honoured tradition of making a backup of important data upon this news of data-loss. I invite each and every one of you to join me.

 

[TerrifyingPterodactyl]

I don't know which one of you posted something so terrible to make the drives all 41% themselves at the same time but I'm glad I never have to see it. They took the hit so we don't have to

 

[BarberFerdinand]

where most of that was just waiting on the database to import over 120,000,000 post stickers.

Nuke stickers null, you know you want to.

 

[GenderCop]

Since it was right after null posted Dylan Mulvaney strutting on the red carpet at New York Fashion Week on the front page, I figured Dylan's public relations team had taken us down.

Bud light be damned, he still has million$ in pricey endorsements and shit.

 

[Sneeds]

The telegram during the shutdown was interesting. The user that kept posting the Norm McDonald clip got mad at you

And

 

[Foghotten]

nevar 3get

 

[Kalsarikännit]

Nuke stickers null, you know you want to.

But why the heck would I keep using the site if I can't watch number go up??????

 

[Local Wordmason]

Reaction farmers on suicide watch

 

[Slav Power]

So uhh

Are you working on it?

 

[SIGSEGV]

So uhh
View attachment 5343260
View attachment 5343263Are you working on it?

Nothing of value has been lost.

 

[Glowie]

All four at once? My inner schizo wants to believe it was deliberate but it's probably not.

Tranny AIDS killed drives.

 

[ImagineTheSmell]

Sounds like this is all @secret watcher 's fault

 

[Troonos]

Joshua, you work so hard and sacrifice so much for us. We love and appreciate you.

 

[Inafune Fanfiction]

WE'RE SO BACK!
View attachment 5342908

Ebola better not be back.