One For The Record Books

My colleague, the Network Administrator for our branch of the company, has spent the last week banging his head over a server in our Malaysia branch.

It's a nicely set up machine with a RAID array for obvious purposes.

The person, not pictured to the left there, "in charge" of simply watching the server to see if any of the drives are failing so that he can be instructed to hotswap the drive.  This person shal henceforth be referred to as "The Malaysian".

Our young Malaysian go-getter noticed 2 blinking amber lights.  Decided that wasn't enough to notify anyone.  But when the third amber light appeared, he picked up the phone, snapped on his artificial opposable thumbs and dialed into our offices here to tell my poor colleague that there "might be a problem".

Now...in a RAID 5 environment, with one amber light, one would be motivated to action..yeah, there's still some redundancy, but hey...let's not waste time...maybe there's some important stuff on these drives.  Perhaps, one might think, man-hours upon man-hours have been used up to populate those drives with stuff called "hard work".  But hey, maybe he was busy...and trusted his skills to deal with it before anything got out of control.

Moving on...two amber lights blinking and it's go time.  Redundancy no longer exists and there is zero room for error.  If, God forbid, another drive were to fail at this point it would be what's called a "data disaster".

At this point, the netadmin still hasn't been notified, emailed, called, no flares were shot off, no smoke signals, no Bat-Signal to our netadmin when the second amber light was blinking.

Obviously, a third drive failed.

This lead to the technical term those in the business refer to as "Holy Fucking Mother-fucking Data Loss" (HFMFDL).  At that point...there is so much shit splattered all over the fan it's already started to dry up and fall off.

The option then, is to slap the person on the wrist and instruct our netadmin to "build another server and ship it out to them, mmmmk?"

Skip ahead a week, the server arrives in Malaysia.  Our go-getter out there says, "Hmm, I gotta email our netadmin".  (I know, right?)  So he does...and he asks the netadmin for the root password to the server because he can't logon.  After our netadmin wiped the coffee off his monitor from laughing at and swiftly denying this request, he got another email saying that the reason our Malaysian needs the root password is because he can't log into the server.

Uh-oh, did the netadmin fuck this server up?  Did he build a piece of crap and send it out there?  Did he screw up the install?

No.

The Malaysian forgot to plug the server in.