When the single most important business application is stuck in a reboot loop
04:35
And nobody knows how to fix it
04:35
the most senior resource doesn't generally get to rest :P
M
05:17
Michichael
But it's all fixed now
05:17
so
05:17
*dies*
KZ
05:22
Kes Zerda
yayfixed
M
05:30
Michichael
I mean not really
05:30
I have no idea what caused it
05:30
But we rebuilt on a new server
KZ
05:30
Kes Zerda
ahh, so monday, you get to pick at the bones of the old server to try to find what difference caused it
M
05:32
Michichael
Nope
05:32
Spent 30 hours doing that. I have no idea.
05:32
There's no log, no indicator, no error
05:32
Boots, gets to login sometimes. Then just resets like it lost power
05:33
Nothing in windows logs, nothing in vmware logs. If i set last known good, i can get into the os and everything is fine, but it cannot reboot cleanly
05:34
Full shutdown, new hyper? Same problem. Convert from esx5 to 5.5? Same problem. Veeam restore from 12/8? Same problem. Attach disk to new vm profile and boot? Same problem.
05:35
Literally no indication of what the issue is. We're past 5.5 3a so shouldn't be the timeclock issue
05:36
If i had time I'd set up 6.5 and update since we're so behind anyway
KZ
05:37
Kes Zerda
doesn't sound like a vmware-side problem. out of left field thought is possibly weird disk corruption -- something got mangled in disk. other is get dumps of shit like the registry, try to poke through fuckery that way
M
05:37
Michichael
My money's on it being a data store issue and the vm fencing for some reason when it's booting
05:38
But yeah. No data at all gets written to the disk on failed boots
05:38
Checked thoroughly
KZ
05:38
Kes Zerda
okay, nada getting written sounds super weird
05:39
because at the very least, half-written stuff should be dumped that gets rolled back on reboot
M
05:39
Michichael
Yup
05:40
Literally nothing. But if i roll back, or make non os file changes, they're persistent across the failed boots
05:40
And the rolled back boots that successfully permit login log fine
yeah, I'd wanna spin up a new vm and attach that disk as a secondary to try poking and prodding
M
05:42
Michichael
The VMware logs show the failures simply as the world initializing, loading the disks, initializing, etc, then guest is ready, bout 30s later, guest is not ready, heartbeat is red, world is empty, soft reset detected
05:42
Yep, it's attached as a secondary disk to the new host and perfectly accessible
05:43
The odd part is that it persisted across both new vmx AND veeam disk restores to new vmdk
05:43
Which implies OS layer
KZ
05:43
Kes Zerda
yah
M
05:44
Michichael
But if it was that catastrophic that data can get written the partially written boot logs should work or it should be unable to make it to the point where i can start entering credentials
05:44
The credential manager can not load without the event subsystems
KZ
05:44
Kes Zerda
boot in debugging mode and hook up a virtual serial port?
I am gonna maybe not be here tomorrow. Actually funny. Today is my dad b-day.
11:11
Honestly. @Hurileno I would think about it. It isn't far until end of high School. And I also would safe more money. My uncles and I decided whatever college I go to. I am gonna have a head start. After highschool I am moving to my college.
The waters were coming; the Incestuous Lord prayed, sang, danced; then the waters came, but the Incestuous Lord had not yet reached perfection.
The Incestuous Lord swam: with his wife he swam; in the waters they danced, prayed and sang. They were overtaken with religious fervor; at the end of two months they gained strength.
13:06
i love creation myths of other cultures. they're all WAY more interesting and cool than "in the beginning god created the heaven and the earth".
C
13:07
Carci
Shinto's creation myths are wild. >.>
13:07
Norse creation myths are crazy, and metal as hell. :p
