My computer hadn't finished logging in this morning when I got a frantic call from the network team lead.
"Hey, can you help? This site is reporting that they can't get any addresses from DHCP."
Huh, that's interesting, I can't even ping the BMC interface on the host where that DHCP server should be running. Did anything change?
"Oh yeah we swapped out the WAN link for that site."
...
How about you double-check the thing you changed before coming to me about DHCP issues.
*sigh*
Whew, the problematic VM came back up without a fight after rebooting the problematic host. I was very afraid it had gotten corrupted and wouldn't play nice.
Still not quite sure exactly happened (thinking storage hiccup during backup execution, but still digging), but at least the fire has stopped burning.
I'm going on break.
@XSGeek Indeed a Windows machine, but had to reboot the ESXi host in order to get it back.
It was fun going through VMware's checklist of how to deal with a completely unresponsive VM/host. Lots of commands to try, then "if it still doesn't work, reboot the host."
My other "how is it only Wednesday" adventure this morning involves an "CRITICAL" VM which stopped responding, on a host which disconnected from vCenter and refuses to reconnect.
SSH to the host try to restart hostd/vpxa; can't find the process.
`ps | grep vpxa` and kill the listed PID; no such process.
kill the hung VM; no such process.
Need to bounce the host. 10 other (prod) VMs on it are fine, but won't migrate while the host is disconnected.
Coordinating downtime now...
#CoSoTech