You can tell from the responses who has a real DR plan and who are just winging it. The DR-backed commenters can switch to site B with nary a worry. Everyone else is trying to justify keeping the server room running while something is burning. To them a misplaced backhoe is a bigger problem than you know a server burning.

I love the sanity check part. But really, you're keeping someone on hand to drag your ass out after you pass out while you sniff up toxic fumes.

You ask the fire marshal what was burning, about an hour or so after you've Big Red Button'ed the server room.

This story is giving the Japanese engineer in me apoplexy.

IR/thermal imaging cameras are SO USEFUL. I had a fire (bathroom fan caught fire due to being 45y old, knocked it down and extinguished it myself, but was worried about extension in the ceiling/duct).

Oakland FD came out and used their IR camera to check the heat from the ceilings nearby. Hilariously they found a hot water pipe (running between bathroom and kitchen) and almost axed the ceiling open (turning $1500 in damage into $3k+), but their captain was smart and figured it out from another angle.

Really tempted to hack an EOS 5Dm3 into an IR camera next. Not so much for fires as night vision, but it would be useful for fires too. I'm not sure how useful an IR camera is at detecting heat, since things which aren't yet on fire are not quite so infrared, though.

I usually use a Fluke IR temp meter when cooking and to find hot wires/etc. in the datacenter, though.

Maybe we need a diy thermal sensor that plugs into an android or iphone device?

Oh wow, it exists:

http://www.instructables.com/id/Thermal-Imaging-Phone-Camera...

http://www.robhopeless.com/search/label/Thermal%20Imaging

http://www.kickstarter.com/projects/andyrawson/ir-blue-therm...

http://rh-workshop-llc.myshopify.com/

Costs only $150 to make?

Open-source: https://github.com/RHWorkshop/

I couldn't answer the question there because of reputation thing. (just signed up)

Here is a little bit different approach;

If you are in the room and smelled the burn, that means something is already happened and you are dealing with its result and possible side effects, and that gives you possibly enough time before shutting down everything or getting out of the room. Your chance of not being harmed by this situation is high at least for 5-10 minutes.

In this case, having a termal check would not help you a lot since burned hardware is most probably not functioning anymore and might be colder than the regular servers. The other option would be that it is still working but not causing any fire yet so heat is not much different than the usual.

Now, smell is your only evidence,

I am hoping you guys have air conditioner in the server room. Put it on the max level so that the smell will not be so strong everywhere. This can help you identify where the smell is coming from. Before you check the smell, get out of the server room, breath as much as fresh air you can, so that your realisation will be sharper when you get back to server room. Having your colleague with makes the process faster.

This would be my first reaction to these kind of situations.

It is of course costly to turn off whole system but don't forget that it is not important than you!

I'm not sure if everyone responding really read this all that carefully. There was absolutely no mention of smoke in this question. There was a "smell". If you drop an entire datacenter, you are easily looking at $100K+ in damages just to reset the room. So, getting a buddy and taking some time to look for the problem until it is found or until you actually are seeing smoke or other specific danger seems like a pretty reasonable course of action.

For some more context, this was the OP's previous question in Stackoverflow.

http://serverfault.com/questions/420877/ive-inherited-a-rats...

Doesn't that change the entire question!

It is pretty simple. You call the fire department.

They have a TIC (thermal imaging camera) that can detect heat/overheating sources pretty quickly. Plus, it's kind of nice to have them on hand in case a smell progresses to a fire.

It's been a while since I've been in IT, but don't failing UPS batteries put off fumes that destroy your lungs?

A friend of a friend was a hero and shutdown his datacenter cleanly/recovered some hard drives during a situation like this. He got severe lung damage (not from fire).

Lead acid batteries will have a rotten smell after shorting out - more so in wet ones, but SLA will smell the same.

If you wait to the point that an SLA smells, it has probably expanded and caused internal damage to your UPS/server rack, albeit minor if you can manage to get it out without dismantling anything.

Just an idea: the server room should be segmented into smaller areas with isolated power circuits. Using the sniff test, if you are truly concerned that you are about to have a fire, then it's only responsible to start an orderly powerdown to prevent equipment loss, and more importantly, prevent injuries to people.

If you start shutting down areas of the datacentre that appear to be closest to the smoke, then you will have a better chance of locating the issue in the fastest possible timeframe, with minimum disruption.

On top of this, if you then have critical infrastructure that you must keep running, then you keep your failover servers in different areas and failover to that equipment.

I'm not a server or datacentre guy in any way, but doesn't this seem sensible?

'hmmm... probably UPS battery venting' click 'yup'

When UPS batteries vent it has a distinctive odor. It's very pungent and sulfuric, but it doesn't smell like a fire or melting silicon. Any experienced operations guy has smelled it before.

Additionally in most fire suppression systems the Big Red Button is the abort button. A well designed room will dump itself when it detects smoke after a short evacuation alarm. It's precisely designed to keep people from screwing around with a real fire. They must make the active decision to stop fire suppression rather than start fire suppression.

HVAC. Has no noticeable smoke (it'd probably be outside the building anyway) but pumps a burning smell into the room when the motors start dying or aren't oiled right.

Don't hit the big red button just because you smell burning.

I've had this happen to me once. No alerts and all boxes were up, but there was a smell in the room. I went machine by machine and UPS by UPS and nothing was wrong or burning(1).

Next day we find out the breaker panel next door had a short that blew out several breakers. Smell was vented into the server room.

So, not always your room, could be something else just as or more dangerous.

1) shut down all machines, unplug all UPSes, open every case

What kind of server room is this which is not equipped with smoke detectors?

I spent a couple of summers interning at IBM and one of the things they taught you in orientation was the sound of "imminent halon dump" (the alarm that said Halon was about to be used in the machine room). The instructions were, hold your breath and make for an exit immediately. Failure to do so would lead to asphyxiation.

Lead based batteries often have a usable life of 3-5 years. Chances are, the others of that vintage are already dry and have already failed. Then they will rupture, often with smoke as their series connected brothers try to push electrons.

If it's that important, buy a TIC (thermal imaging camera). They can be had for under $10,000 and will show you actual hotspots. Walk through, sweeping every item.

Temperature indicator stickers.

Thanks for this story. I am ordering a fire extinguisher for our server room now.

It helps to prioritize what you look for. 9 times out of 10 it's power related.

Get in an electronics "expert" they know what component smells like what :)

Infra red camera!

Can of deodorant.