<2022-09-18T09:14:37.000Z> Ravioli_Pillow: Hey, I saw your most recent poast and I wanted to see if there was anything you need help with. I work in computer security and have handled my fair share of incidents. I'm sure you've got a bunch of people reaching out, but if you want some more assistance I'm available.
<2022-09-18T09:25:20.000Z> josh: I'm just reviewing what has happened. The attack had access to my account.
<2022-09-18T09:25:32.000Z> josh: Immediately after I logged in to the ACP he deleted every forum on the site. Before then, he was hard deleting threads.
<2022-09-18T09:26:03.000Z> josh: I am pretty sure me logging in gave him access to my security token somehow. I assume he had some way of intercepting web traffic and its headers.
<2022-09-18T09:30:55.000Z> Ravioli_Pillow: If he had access to the proxy then that's very possible. I'm assuming that kind of makes the access logs you have useless since he could make it look like he was coming from anywhere behind the proxy by munging the X-Forwarded-For header. Do you have any database access exposed through the web interface, any way he could have exfilled peoples password hashes?
<2022-09-18T09:33:04.000Z> josh: No. There were hundreds of login attempts a second though.
<2022-09-18T09:33:17.000Z> josh: but hundreds really shouldn't be fast enough to crack any passwords, I think that's actually totally unrelated.
<2022-09-18T09:33:25.000Z> josh: my account has 2fa to even sign in
<2022-09-18T09:33:32.000Z> josh: he must have directly gotten my auth token either from headers or from the redis db
<2022-09-18T09:33:48.000Z> josh: but if he had had access to redis he'd have had my auth token before I signed in
<2022-09-18T09:37:42.000Z> Ravioli_Pillow: It could have been a diversion or just unrelated since it sounds like that's been happening for weeks now in the background anyways. What is your Redis database set-up like? You have authentication turned on, right? I'm assuming it's not externally accessible?
<2022-09-18T09:38:04.000Z> josh: Yes, I have redis authentication turned on but I had issues with redis before
<2022-09-18T09:38:25.000Z> josh: our first hack was the exact same thing but the guy used it as an opportunity to actually sign in to everybody's accounts currently signed in and scrape their email addresses
<2022-09-18T09:38:46.000Z> josh: this guy only signed into my account and had to wait until seconds after I 2fa'd into ACP to do any admin damage.
<2022-09-18T09:46:08.000Z> Ravioli_Pillow: All things considered it sounds like you got lucky. The corporate suit in me says you should think about forcing password resets once you bring the forum back up "in an abundance of caution". I'd also reccomend restoring a back up from before the vsys compromise, especially if you're not sure that his actions are limited to your account to make sure he hasn't done anything like logged in as other users and used invite keys, changed their passwords, emails, etc.
<2022-09-18T09:47:35.000Z> Ravioli_Pillow: Also depending on where the server's hosted and all the fun legal jurisdiction stuff, the clock might be ticking for GDPR notification requirements but that's definitely out of my area of expertise.
<2022-09-18T09:56:49.000Z> josh: yeah i'm not bringing it up until I redo everything
<2022-09-18T10:02:38.000Z> Ravioli_Pillow: I've got to head off for a bit, but I'm also on XMPP at ravioli_pillow@jabber.otr.im and if you need like emergency help, my number is +1 216 230-4529. Cheers and good luck, man.
<2022-09-18T10:03:21.000Z> josh: I'll add you, thanks for reaching out
<2022-09-27T02:02:21.000Z> Ravioli_Pillow: Hey, I saw your most recent post about progress towards getting KF back up and running and I wanted to offer some friendly advice in regards to hosting your own nameservers.First off, I'd recommend hardening PowerDNS by changing a few settings - the biggest being utilizing setuid and setgid (doc.powerdns.com/authoritative/security.html#securing-the-process).You might also want to switch version-string to anonymous so that the specific version of PowerDNS you're running isn't exposed to everyone who runs a `dig ch txt version.bind @ns1.kiwifarms.net` (doc.powerdns.com/authoritative/settings.html#version-string), and set edns-cookie-secret to enable DNS Cookies (kb.isc.org/docs/aa-01387, doc.powerdns.com/authoritative/settings.html#edns-cookie-secret). It looks like you've already restricted or disabled AXFR and IXFR requests, which is good too.I don't know what your infrastructure looks like right now, but for a site like KF I'd recommend getting a few more name servers in geographically dispersed locations and on separate ASNs for more resiliency. Anycast is also worth thinking about, but that can be it's own massive ballache.It's also probably a good idea (if you're not already) to have a dedicated hidden master nameserver and have all of the public nameservers slave off of that one, it provides a lot of positives.Finally, I'd suggest looking in to enabling DNSSEC on your zones. It can feel overwhelming if you've never set up zone signing before, but once you ave it up and running and rotating keys automatically, it's mostly set-and-forget (doc.powerdns.com/authoritative/dnssec/index.html)Good luck!
<2022-09-27T11:42:38.000Z> josh: the dns server has completely shit itself because the fucking galera instead just died randomly
<2022-09-27T11:42:47.000Z> josh: I hate, hate, hate, hate banging my fucking head against a fucking wall
<2022-09-27T11:42:51.000Z> josh: I google my fucking issues and there's zero results
<2022-09-27T11:42:53.000Z> josh: a bunch of shit in chinese
<2022-09-27T11:42:59.000Z> josh: almost all of it related to fucking docker or other instancing software
<2022-09-27T11:43:07.000Z> josh: because the fucking dumb ass piece of shit cannot connect TO ITS FUCKING SELF
<2022-09-27T11:43:08.000Z> josh: ON LOOPBACK
<2022-09-27T11:43:46.000Z> josh: even when I completely flush my ip tables there's nothing, it cannot connect to ITSELF
<2022-09-27T11:43:50.000Z> josh: I turn off galera and it boots just fine
<2022-09-27T11:43:56.000Z> josh: I turn it on and it cannot connect to its fucking self
<2022-09-27T11:44:09.000Z> josh: and this is not the thing I want to be sinking hours into
<2022-09-27T11:44:16.000Z> josh: setting up the FUCKING DNS 
<2022-09-27T19:19:47.000Z> Ravioli_Pillow: Man, I've never uesd Galera for anything so I don't think I can help you there. I've only fucked around with PowerDNS a little, my go-to for nameservers is BIND. Not as performant as PowerDNS or Knot or yadifa but it's the defacto standard and finding answers for stupid shit you run into is easy because fucking everybody runs it.
<2022-09-27T19:27:32.000Z> Ravioli_Pillow: I don't know what kind of use-case you're trying to solve for by using galera clustered MySQL-backed PowerDNS, but you don't have like thousands of records, you don't have hundreds of zones.. It sounds like overengineering and overkill. You can get a master and two slaves in BIND running in like an hour and if you set it up right it will stay chugging along for years. Need to handle more traffic? Just add another slave, add the TSIG key, let it grab the zones it needs from the hidden master, and you're done.