Dunno if anybody has noticed but I’ve been a bit scarce lately. Haven’t added many comics, replied to much email or been on IM. Lets just say some serious shit has gone down.
And by shit, I mean our machine room. Yes, in the space of 8 days, the machine room where I work lost power…..TWICE. And for 2 totally different reasons.
If I was still a perl monkey, this wouldn’t have bothered me too much. But now that I’m a sysadmin again; lets just say me and
fsck are pretty good friends right now.
Insane story #1:
Last Sunday at about 10pm, one of the 2 very large cooling towers on the top of our building sprunk a major leak. Now, our machine room is on the 6th floor (2nd from the top), and as in most machine rooms the walls are hollow, and so is the crawl-space under the floor. So you can imagine what happened when the water got down to this level, the crawl space under our servers filled up like a swimming pool.
Down there is where the power is. Power cables don’t like water much, so they all shorted out and every machine in the room went dead. The first to notice were the help desk guys working night shift, as you’d expect their phones started going nuts. They went to the machine room and saw condensation steaming down the window on the door. This is not something one normally expects to see!!
So, all the tiles were lifted, the water pumped out, and the area dryed with these very large fans. Water had managed to make it down 6 levels, and into the elevator shafts, so you could say there a lot of it. At about 7am Monday morning we got the power back on and started bringing up the machines again. Unfortunately some of them came up before the SAN and this caused a lot of extra shit that I just didn’t need on a Monday morning. When it was over, we though “Heh, OK that was a shitty day. But next week we’ll just look back on it and smile”….
Insane story #2:
Yesterday I’m in a meeting and I have some guy on a conference call. During the meeting he says “Damn, my corporate desktop just crashed, oh well”. When I get out of the meeting I go back to my area and one of the DBAs tells me that I might want to go into the machine room. So I go in, and what do I find:
Complete and utter silence.
This is bad, because servers make NOISE when they are on. Also, there was a burnt transistor smell….not to mention a lot of VERY worried faces.
To add to the insanity, it wasn’t a water leak this time. One of the UPSs exploded and this took out the other 2. One of the NT guys who was in the room at the time said sparks flew out the top of this thing like a volcano. He just stood there frozen waiting to get electrocuted!!
So, we get the sparkies in and in a little under 2 hours power is back again. We start turning on servers and once again, they need a lot of help to come up clean. We got the SAN up first this time but I still didn’t get home until midnight.
The fallout from these outages is unreal. Having working in this game for 10 years and only having experienced this happen once before, seeing it twice in one fortnight is just crazy. Makes things interesting though.
And now, I sleep.