Postmortem September 17th outage and rollback

[AnOminous]

So I replace the Seagate for a WD and it dies within days (4 times this has happened), or it plain out refuses to work at all (one occurrence). So I tuck my tail between my legs and go back to Seagate, which all work well beyond their supposed lifespan.

It seems like both WD and Seagate go through epochs of being utter ass. Currently have a WD SSD, hope we're not in one of them.

 

[Polish Businessman]

If it's an SQL dump recovery, disable INDEX and FOREIGN KEY constraints on the sticker table columns before you COPY table data. Afterwards you index and add foreign key constraints. This will be much faster.

I'm afraid someone already thought about it, beacuse this is exactly how pg_dump works by default. First copy, then create indexes and foreign keys.

 

[DavidS877]

I'm afraid someone already thought about it, beacuse this is exactly how pg_dump works by default. First copy, then create indexes and foreign keys.

And if Mysql was using pg_dump that would be a useful fact. Most seems to say that Mysql does index creation as part of "CREATE TABLE" in the dump file. But you'd have to look at the exact files being loaded and see what's going on. Ideally with a reduced dataset and a test DB to do some timing.
mysqldump has "--disable-keys" which may or may not be in use in the current backups being done.

Obviously the real solution is active/standby real time WAN replication. Which I'm sure could be setup for only some money and time and I'm sure Null has plenty of both.

 

[Seething Troon Collector]

Here's a dumb question, there's obviously people on KF that are in the tech industry and have the knowledge to help keep the forum alive. Would it be prudent to create a place for people to come together to work on keeping the site alive so Jersh doesn't have to shoulder the burden himself?

 

[Flaming Dumpster]

Here's a dumb question, there's obviously people on KF that are in the tech industry and have the knowledge to help keep the forum alive. Would it be prudent to create a place for people to come together to work on keeping the site alive so Jersh doesn't have to shoulder the burden himself?

Supreme tech autists can never agree on anything when shoved into a room. The problem with tech is there's like 100 different ways to achieve the same end result and filtering through the solutions which are just dumb or incompatible is a tonne of effort. There's also people who want to shove their pet project into the mix for their own agendas (e.g. the Urbit people) even though it's a square peg.

Also managing that many people, building trust, it all takes time and is a job in of itself. It's why 20 developers working on a project is slower than 2 developers.

I think if the autists are bored, they should work on building Kiwi-adjacent services, like high quality archiving tools and maybe a fediverse instance worth a damn.

 

[Polish Businessman]

And if Mysql was using pg_dump that would be a useful fact. Most seems to say that Mysql does index creation as part of "CREATE TABLE" in the dump file. But you'd have to look at the exact files being loaded and see what's going on. Ideally with a reduced dataset and a test DB to do some timing.
mysqldump has "--disable-keys" which may or may not be in use in the current backups being done.

Obviously the real solution is active/standby real time WAN replication. Which I'm sure could be setup for only some money and time and I'm sure Null has plenty of both.

I looked around a bit and found this alternative to mysqldump:

https://github.com/mydumper/mydumper

It has an option "innodb-optimize-keys" which does exactly the thing. Might be worth checking out.

 

[IT Dude]

Every WD drive I had died on me, tho with enough time to move everything.

Meanwhile every seagate drive I had completely shat the bed without warning including a fucking backup seagate HDD that died while copying stuff being recovered from another dead seagate drive.

Meanwhile I got 15-year old HGST drives that plain refuses to die, but HGST sold out to WD and doesn't exists anymore.

No idea which SSD brand is the most reliable right now.

My 10 year old 4TB HGSTs in an old home server are still kicking strong.

In my experience dealing with datacenter stuff, Seagate has been fine since the 3TB HDD days. Everything 4 or 6TB and larger are great and last as long as you'd expect a HDD to last.

Any HDD made in the last 6 years is fine. They're at maximum maturity now. Just go with the best GB per $ option. WD is usually more expensive. Not worth it. But I'm not really keen on Toshiba drives. They seem to be similar priced to Seagate but have less performance (cache size and sustained Read/Writes)

My recommended SSD brands:
M.2
- Team Group - Best value and usually one of the highest endurance consumer drives. Their gen4 stuff is great
- Sabrent - Usually the largest capacity and fastest drive options. All quality stuff
- Kingston - Their higher end stuff is great

WD, Samsung, and Seagate are bad value for the performance. Samsung has been fucking up their firmware lately too

SATA
- Team Group - Usually best value. The AX2 and EX2 are best value SATA drives but about 1 in 10 can have minor issues that ZFS complains about
- Samsung - Best value for large capacity. Rock solid SATA drives
- Intel - Best SATA drives for endurance. No longer made I think

Realistically just go with the cheapest SATA drive. Any will do fine.

U.2
- Samsung
- Intel
- Micron
- WD

Here's a dumb question, there's obviously people on KF that are in the tech industry and have the knowledge to help keep the forum alive. Would it be prudent to create a place for people to come together to work on keeping the site alive so Jersh doesn't have to shoulder the burden himself?

He basically has the expertise on hand to keep the site up, even if I am volunteering only when needed. The only need would be maybe money and hosting capacity options. His new beefy server is enough to handle KF for a very long time alone. I wouldn't even expect this thing to have further storage issues. It has capacity to expand it further on size or more redundancy. If it were cost effective, having server redundancy would be better. But that ticks up colo costs and there's money required to get more hardware. Adding more cooks in doesn't help, unless they know more advanced setups.

You sound like you’re from Reddit. I don’t like Reddit niggers.

I haven't used Reddit since you were in elementary school. Sorry that I'm not schizophrenic.

 

[doodoocaca]

This is was very close to a total nightmare scenario where the server would be completely destroyed. I've reinstalled the Kiwi Farms so much at this point that I know the procedure very well, and it only took about 7 hours, where most of that was just waiting on the database to import over 120,000,000 post stickers.

Are sticker stored in some unindexed way lol

 

[Pepsi Man]

Are we ever gonna see the return of .net? Or is that dead and gone for good?

 

[Flaming Dumpster]

Are we ever gonna see the return of .net? Or is that dead and gone for good?

Null said on stream he's aiming to have it back for Halloween.

 

[NPC_]

tfw this happened right as my cow was having one of their best moments in a while, so now one of my posts about it in their thread got nuked and the others' attachments won't show up

 

[seri0us]

Wrong. The sticker King @AnOminous Lives.

View attachment 5344604
Long live the King.

There's a sticker feederboard?

 

[Harm]

Found an archive of a Patreon post by Chris Gesualdi from May 22nd, 2018 which appears consistent with his current story.

View attachment 5342048
Source - Archive

Credit where it's due, this appears to be the story he was telling within a couple weeks of the incident.

EDIT: Removed an unrelated Patreon post I thought was following up on this one.

https://youtu.be/5nUdIhuwtAA?si=8gEsgcN26h5qrHbI

Vito thinks isom was intended to be some kind of conservative anti-woke vehicle that Eric got a few million dollars promoting and then put out a product filled with right-wing messagimg.
From what I know about it it was a project put out by a libertarian that was tired of the marvel crap and wanted to put a product that was free of the racebending and gay shit.
Vito who is a libtard who has a hate boner for the antiwoke got triggered by the three million-dollar campaign and tried to weasel his way into getting super killer a boost by demanding that Eric signal boost the comics of people with a "different political view" and Eric told him to suck his ass hole

Can someone point out the conservative or Christian messages in isom

crossposting from /cow/
View attachment 5342447
the domain squatting is pretty interesting

View attachment 5342452View attachment 5342451View attachment 5342450
https://archive.ph/IlXem


View attachment 5342448
ttps://archive.ph/9d2BH

mynerdyhome.com - leads to vito's youtube channel (archive)
yellowflashguy.com - leads to wikipedia article on grooming (archive)
ryankinel.com - leads to vito's youtube channel (archive)

the ryankinel.com domain used to redirect to this page
View attachment 5342458

View attachment 5342460
archive

View attachment 5342466
archive
joecristalli.com is now down (archive)

Posting this here as it's likely related to the outage, but the attachments in these three posts from the Vito thread seem to no longer be functioning. Attempting to open the mp4 attached to the second post is giving this error:



I'm not quite computer literate enough to know whether it's safe to publicly repost the "document tree" as it appears in my error message or if it's compromise something. The images in the other post lead to this page:

 

[cybertoaster]

My 10 year old 4TB HGSTs in an old home server are still kicking strong.

Good, you're not gonna find any new, all you see for sale are actually taken from enterprise servers and are all near EOL anyway.

In my experience dealing with datacenter stuff, Seagate has been fine since the 3TB HDD days. Everything 4 or 6TB and larger are great and last as long as you'd expect a HDD to last.

You sure? because if a 6TB crashes on me I'm going postal, gonna be fedposting in seagate HQ, not minecraft.

My recommended SSD brands:
M.2
- Team Group - Best value and usually one of the highest endurance consumer drives. Their gen4 stuff is great
- Sabrent - Usually the largest capacity and fastest drive options. All quality stuff
- Kingston - Their higher end stuff is great

What about Crucial?

Zero experience with TG but are those really that good? because the price its suspiciously low, $150 for an NVME 4TB drive.

 

[IT Dude]

What about Crucial?

Zero experience with TG but are those really that good? because the price its suspiciously low, $150 for an NVME 4TB drive.

Crucial's stuff is not good value usually. The P1 and P3 are usable but slow compared to other options. They've come a long way from their old SATA SSD days though where their firmware would lock up Windows.

And Team's M.2 NVMes are very solid. I use them in desktops and servers (both OS boot drives and in multi-raid configs) so I have a lot of experience with them. It's their cheap SATA SSDs that sometimes have issues but they're so cheap, I just swap out the bad ones later. Never had a Team NVMe have an issue. And so far, their speed is as advertised and their endurance is holding up.

My favorite drive they have out right now is The Team Cardea A440 Pro https://pcpartpicker.com/product/Cd...2-2280-nvme-solid-state-drive-tm8fpr002t0c129
$95-$100 usually for 2TB Gen4, 7.4/7.0 GB/s Read/Write, 1mil IOPS, and 1.4 PB Drive endurance rating

Either way, never expect your drive to not fail at some point. People paid a premium for a premium brand like Samsung 980 NVMes and those things died out of the blue very early.

 

[Alligator Alcatraz]

Every WD drive I had died on me, tho with enough time to move everything.

Meanwhile every seagate drive I had completely shat the bed without warning including a fucking backup seagate HDD that died while copying stuff being recovered from another dead seagate drive.

Meanwhile I got 15-year old HGST drives that plain refuses to die, but HGST sold out to WD and doesn't exists anymore.

No idea which SSD brand is the most reliable right now.

Real shit I also have a 720GB HGST drive that still functions for zero reason. I also have a few Western Digitals still functioning, and all my Seagates died within their first year. Hilarious how HDDs are so consistent. I think for the most part, though, the safest bets on SSDs are Samsung, Kyoxia (Toshiba), and Crucial (Micron). I have never had a Crucial product fail, and I attribute that to them being US-based.

 

[cybertoaster]

Crucial's stuff is not good value usually.

Well fuck, just bought one. Actually not just, like over a month ago and can't return it, bought a bunch of parts but didn't have time to put it all together because of work.

Also my motherboard which was supposed to have 2 gen4 NVME M.2 sockets only had one, for some reason the fuckers forgot to add the 2nd socket, the pinout its there. Now I have to RMA that shit because its too late for returns.

in multi-raid configs

I have no idea if you can do RAID6 with these M.2 drives but at least with HDDs if one fails you can get your data out most of the time, I've done it with tons of drives. If shit gets real you can change the board or even open it up and unstuck the arm which is what happened most of the time, or build a clean box with some MDF, plexi, a vacuum cleaner and a cheapo HEPA to do things like remove the platters to another drive.

But every time an SSD fails and I see somebody working on it it looks like fucking heart surgery. All microsoldering and expensive tools, and of course they will charge you out the ass for it because its a ton of work.

People paid a premium for a premium brand like Samsung 980 NVMes and those things died out of the blue very early.

Well shit didn't know that, got a SATA 970 on a backup machine and its been working fine for about 6-7 years.

 

[Homicide]

They diagnosed that all four enterprise raid NVMe harddrives had failed simultaneously, which wiped out the entire database and all forum software.

I can't decide if this is normal or ludicrous. Probably both.

 

[RMQualtrough]

How recent was the backup? Sounds like we nearly experienced Total Kiwi Death.

Also LMAO at the fact that trannies can't do as much damage to KF as a hard drive error can.

They can't even decide what sex they are, how are they going to achieve anything requiring any semblance of competence? Have you seen the DISASTER tier attempts to pass by troons? They're incompetent...

 

[Autistic Joe]

Real shit I also have a 720GB HGST drive that still functions for zero reason. I also have a few Western Digitals still functioning, and all my Seagates died within their first year. Hilarious how HDDs are so consistent. I think for the most part, though, the safest bets on SSDs are Samsung, Kyoxia (Toshiba), and Crucial (Micron). I have never had a Crucial product fail, and I attribute that to them being US-based.

For HDD I've used plenty of WD in the past and Seagate currently, and been happy with both. Haven't had a drive die on me since the 90's - knock on wood!

No joke it might be because I'm a cheap SOB and always liked bringing drives home from work when they decommission someone's desktop. They're never near end-of-life, usually just 2-3 years old, but still considered outdated. So I'd be getting something already broken-in but nowhere near broken-down. I kinda miss that, now everyone has laptops at work so there's a lot less ability to scavenge old parts.

Currently using a SanDisk SSD on my primary computer, still going great after 6+ years. 94% life remaining, lol this thing might outlive me!