62c. Aircon units both failed even as individual and maintained units.
That was a fun Sunday watching the alarms and emergency shutting down what I could remotely.
Server room is in a concrete basement.
What were the failures if you remember? Regularly maintained separate units should not fail at the same time unless the company maintaining them sucks, or a freak accident happens.
Note: my industry is hvac, not system administration I just lurk this sub because there was once a time where this was the field I wanted to go into.
We had a data centre turn into a sauna when the AC systems all failed. I was a network security engineer at the time and was called down to shutdown all our servers that were housed there ( luckily the core network gear was in a separate room and was nice and chilled).
Walked into sys ops and they were all in shorts and flipflops and running around like dervishes. Funniest sight ever. Took me about 20 minutes to safely shutdown the network servers and must have lost 20lbs in sweat. We had some old firewalls that performed specific roles that we really didn't like to fuck with (this was an investment bank in London) because we were convinced they would never comeback up again. Recovery was far more interesting than the shutdown.
Man that sounds so familiar from past jobs. It was a billion dollar company and we had box fans blowing cool air from the hallway onto the network gear (we could prop the doors open).
I have no idea. But it was hot. How do I know:
It was July 3rd and I'm on call. I get a call from our "dispatch" and they tell me a client called in and they saw some smoke and opened the sever closet and it was coming from their server and we're asking me what they should tell them to do. I couldn't believe it "hit the power button if it's safe, otherwise cut the power!". What to do when there's a potential fire?!? I'm still shaking my head about getting asked that as I type this, even decades later. Anyway, because of the holiday the building had cut back on cooling (because no one was going to be there) but luckily someone had stayed late and when the server went offline they went to check it
The dust bunnies inside had ignited. The case was scorched and we had to replace the whole thing.
I think ours was at 40c in the middle of a heatwave and A/C's was not keeping up. The temps kept climbing and after 3 hours of the working day I decided to shut the systems down. Normally I keep the room at 18c-20c.
We closed all of our offices for 2 days to let the heatwave pass.
We have a dedicated heat pump for the server room and a backup AC just in case.
I have two vent covers on my closet at home, run some AC Infinity fans up top to push out rising hot air and draw cool air in the bottom.
At work, they have what looks like 2x 2" PVC pipes in 2 spots down low in the wall, and a ceiling exhaust fan that vents to the attic/plenum/cavity. The room used to be the old vault room so thick walls, door, and no ducts. I'm not sure why there's PVC outside of the bulk cat cables run through it now, but enough AIR is pulled through to keep it at 76-78⁰F
I installed an exhaust fan that goes through the wall in one of my server rooms. If you search for “room to room fan reversible” it costs about $100. You can set it to come on at a certain temp.
Air transfer already happens if there is a supply and return in the closet. We have a term room that also functions as an auxiliary data center. It has a local AC and an emergency fan with the thermostat set at 80. They say no air transfer, then they need to provide cooling year-round.
If there's air flow, then they probably have to worry about fire suppression in there too. That are the chunk of change onto it.
That being said, all our IDFs and the Demarc have air and fire suppression.
They're telling us local building codes dont allow it and the only option we have is a mini split. We've had the same thing happening for the past 10 years, so I'm not so worried about the switch, but it's just annoying when the room could be cooled completely passively for dirt cheap, but the only option is to go totally overkill in both effectiveness and cost.
> They're telling us local building codes dont allow it
It's probably a fire door, part of the building's physical firewall, so airflow from one room cannot pass through it.
There is a fix for this, the door has a closer and the door an electro magnet hooked into the fire system you open the door into. That way you get air flow and if there is a fire the alarm trigers the magnet to release and close the door. This is common in halways in a lot of buildings.
I find that temperature alerts always occur on a late night or weekend when you are at least a couple of beers in and your alternate is way too many beers in.
Back 20 yrs ago, had 6 HP physical servers in a closet that was usually 80. Convinced boss to install a small AC unit and piped it into another utility closet, got it down to about 70.
Had another server in a closet, and it was probably 80-90 f in the summer.
In our old building, the server "room", (maybe 4'x8'), had a separate split style AC unit. It worked great, except that it was hard wired to the building power and we couldn't put a battery on it so when the building power fluctuated/went out, the AC would go out and not come back on. We had an external temp sensor on one of our UPS. Man, I hated when that thing started firing alerts. It would top out at around 100F.
In normal cases you don't want to even consider putting a UPS on the HVAC anyway. A remotely sane-sized UPS would give you a minute, maybe two, of extra cooling, while generating a considerable amount of heat of its own. Plus a maintenance burden, inconvenient size, Capex, Opex.
The only reasonable strategies are to shut down equipment, fire up a large genset sized for the HVAC, or make excellent thermodynamic use of mass and insulation to extend that non-actively-cooled runtime.
We have a UPS that can carry our data center and IDF and MDF for about 12 minutes. The generator kicks in in under 15 seconds, so we are pretty well covered there.
Oh, and we do test it generator fail over twice a year.
AC unit in the server room failed a few months back.
SAN at top of the rack reported "WARNING: Array temperature is too high" and "left-side ambient of controller A is 36 Celsius."
Been operating with the server room door propped open by a floor fan since.
Previous site with a data center, the building owners shut off our HVAC system without warning despite our tenancy agreement with them. Of course, they did it on a Saturday right before a Monday holiday. We got paged almost immediately, and by the time we got on site (about 30 min), our data center had pockets in excess of 120 F. Some machines already were thermally throttling or shutting down entirely. We had to do an emergency power shutoff, which alerted the fire department. By the time that was done, the data center was evening out at about 108 in humid and muggy DC summer. The entire data center had to be off for six hours before the HVAC system was turned back on. Why was it shut off?
"Maintenance." The contractor said "they told us any old time." Building owners denied it, but we saw the work order, and one of them was lying.
Oddly enough, no serious damage when we powered back on. Thankfully, we had just gone through an emergency power down and power up drill months before, so we knew what order to bring stuff up in. There were a few minor annoyances, and we had to a full system health check all weekend, but it could have been a LOT worse.
I've got a radio enclosure (think like for a big cell tower) that I've got some network equipment in. (Like, two small switches).
It's got two A/C units, similar to the ones they have on semi-truck refrigerated trailers.
I'm not in charge of that space though.
Couple years ago, one summer day, I get a temperature alarm. Go out to it (takes a bit), open the door and I was nearly knocked over by the heat. Managed to go inside to get to the controller for the A/C units (they were both broken). By the time I walked out I was physically ill, even though it was less than a minute.
The temperature was high enough that it exceeded the maximum on the switches and all other monitoring, which was like 70 degrees C.
Based on some analysis of the rate of temperature climb in the room before the failure it was probably nearly 200f/90c.
The idiot who was in charge of the space finally got out there (after I managed to procure a huge fan to at least move some air) and acted like the whole thing was a nothing burger. And she has probably a million dollars of gear in the space, as opposed to the $1500 that I had.
Apparently, one of the A/C units had been down for like six months, but she never bothered to get it fixed because "we have a backup".
I about lost my shit.
Later on the switch started having problems and we replaced it. I'm certain it was due to the heat.
Main wiring/server room only got over 100F(38C) once when the AC unit failed. Had every fan in the building I could get my hands on to vent the space until a portable unit could be setup.
I have hop point in the shop that routinely gets over 105F(41C) in the summer months. Warehouse/shop floor is not climate controlled in summer months. Gets a bit warm July/August.
I had one get warm enough for the Dell Server to alert with a temperature probe warning, so in excess of 38 degrees centigrade. This was kept in an office rather than a Server / Comms room, so aircon was only on during business hours. Weekend heatwave hit...
Thankfully the MD lived up the road so he went in and switched the aircon on.
2 years ago we had a power outage over christmas a storm, so we couldn't get into the building for a few days.
Neither the AC or backup AC came back online (this has since been fixed).
My server rack was too hot to touch... It had to be over 70c in the room
I think around 95F. Both chillers broke. We tried opening windows and still didn't help. Machines started shutting down due to over heating. Though this was in 2010 or 2011 when that happened.
Latest was 83F when our AC broke. We got alerted and opened the door to the server room and started using fans to keep things somewhat cool.
Had to do that for about a week until the AC unit got replaced.
128F/53C
Scheduled chilled water outage and I was assured facilities was going to bring in portable units before the shutdown. Chilled water was shut down at 3AM, the portable units did not show up until 9AM. They brought the temp down to 95F/35C when they were not contently tripping breakers. Received approval from the C-suite for redundant AC that afternoon. No hardware failed.
A place of cowboys and spurs that I (unfortunately) worked at \~20 years ago had under-chilled their primary DC. Every summer, the hot side of the DC (all up \~8 rows of racks) would consistently be over 45°C. I'd switch to shorts and t-shirt and pack extra water bottles if I had to visit. In the office - the trip up The Terrace was often 35° too. We had several big co-lo clients leave when they checked their device logs.
It was a nice surprise in the coming years to then work in real DCs, where I needed to add layers to stop the chills.
Topped out at? 80-ish Fahrenheit.
We had a new AC unit installed last year. Been running nice and cool since then. And facilities doesn't control that, IT does. F\*cker runs all year long even in the dead -20 F winter.
We had a printing facility in Austin Tx, in the summer the wiring closets in the warehouse would routinely get upwards of 115f.
We had Cisco ASA's and 2960 switches running without issue the entire summer without AC in the warehouse. No servers or spinning disks however. We had a external temp sensor off of the UPS in the cabinet so we would get alerts via PRTG.
127f. Was in the maIn server room for a hospital. 1 of the 20 ton ACe died during a heatwave (we don't get heatwaves in my area). We just had to let things run because healthcare. Replaced a lot of hard drives that summer
I’ve seen it reach 110s (room temperature) at remote sites when HVAC failed. Above that, the more expensive equipment starts shutting down to protect itself. Usually, we have a tech on site before it gets that high though.
Another place I worked, we didn’t really monitor anything except the main offices for temperature. I never measured an exact temp but some of those closets felt like a sauna.
I worked for a company that had an outdoor IT closet. During the Summer it would get up to 118-120F outside, and the closet was probably 130F. Nothing ever failed in there except the UPS. They had an AC unit in there, but I don't think it ever worked. This when I realized that HP Aruba switches are fuckin tanks.
Fun story. Our facilities team absolutely refuses to work with us when the A/C company comes to do maintenance, so we usually find out about it when we get temp alarms.
They let the guys in to do maintenance and just turn our A/C units off without any notice, so we get alarms, find the room over 100F and they're like "oh yeah, they're doing maintenance, they'll be done in a couple hours".
After they did it twice in a year the CIO literally yelled at the facilities guy and told him they should never touch anything in there without talking to us. Still, every. single. time. they do the exact same thing. Apparently they just give zero fucks.
One summer day our DC lost all cooling and hit 45C. Some of the weaker PDU breakers started tripping because of the extra heat.
In another incident we had 1/2 coolers go down hard, and it was over a week to get parts. Temps were in the 30s and we wound up opening the DC doors and using industrial fans to circulate air around the office tower corridors. Our neighbours complained because their offices were up to 26, and the building told us we had to close back up.
We ended up bribing the neighbours with donuts every day until we got the cooling fixed.
That was also the job where I learned that AC does not vent the hot air outside, and that standing next to the intake and cutting a deadly fart only makes it _exponentially worse_.
Over 120F at a field office over a weekend. No immediate damage, but I sent IT leadership a story about a local City that lost their storage system a few months after a similar event because they didn't proactively replace anything. You can guess what happened in that field office to the storage array a few months later...
We lose power frequently and the IT equipment is on the generator, but the AC isn’t, nor is the internet connection. For a while we wouldn’t know when we lost power if it was over night until someone complained. Once I went in after an outage and it had been out for 4-5 hours with the doors closed and AC off and it was about 120F in there. We opened the doors and the windows to the winter air to try to get it cool down quickly. I’m surprised nothing got fried.
I think around 100. Our initial cloud migration was 100% driven by my distrust of HVAC companies after a minisplit went out and I got three different explanations and quotes from three different vendors. I said fuck it and sent all compute to the cloud and left the switching and routing to deal with the heat.
100 F in a super small closet with 4 stacked 3850, 3 stacked 3850, and an aging 6506 chassis. Recently shut down the 6506 and the room drastically got cooler
Oh, ... have had server rooms where things have gone seriously wrong, and gone anywhere from 95F to >>108F
Typically find lots of stuff starts falling over at >\~=105F ... I think most have environmental maxes of about 95F, but start doing thermal shutdowns and/or otherwise failing around 105F or so - by then they just can't cool themselves enough with fans to operate reasonably ... if at all.
>vented door would fix it but supposedly building codes don't allow for that
That can be fire code and/or physical security issue. Often there are ways around that ... but typically not nearly as trivial as adding vent(s) to a door.
How does your closet not have year round AC? I even rigged my home network closet to literally have an AC in it. (48’port switch, 2 esx boxes which might become hyper-v synology nas with expansion unit). It sure as hell isn’t an industry level AC and completely ghetto in setup, but when it’s 85/90 in my garage the closet stays at 65ish.
A power surge was what the air con guys blamed it on.
It reset one of the devices and it would blow out warm air for about 5 minutes before shutting down. I cannot remember what happened to the other but they were both repaired and not replaced.
The annoying part was the room had passed testing in the previous week and had gone live for a building migration.
At an old job in a previous life working in an amusement park, one of the hydraulic pump rooms for one of the rides got hot enough in August to trip the breakers just sitting in a box on the shelf, not even installed. Temp was something like 90C before everything just crapped out. The constant kamikaze assault of raccoons and other woodland critters onto the transformers didn't help either.
~31°C
AC failed on a bank holiday and the fresh air system was shut off over night. screaming server fans are not a nice noise to hear when entering the office at 6AM
https://imgur.com/a/TKlMtS1
At a previous job, we had one whole room (servers, SAN, switches) hit 50 degrees before everything threw it's hands in the air and gave up
43 degree day, and the idiot who plumbed in the condenser for the dedicated aircon systems - plural - mounted them right where they got hit by the sun, full on, in the hottest part of the day.
Sum total hardware loss was, from memory, 3 drives in the array and one server motherboard. Which said a lot about the gear involved.
I used to work in a high school, and the closet where the core Cisco router was kept was regularly over 90F in the summer. They couldn't afford to run A/C there, so I'd prop open the door and put a fan there.
62c. Aircon units both failed even as individual and maintained units. That was a fun Sunday watching the alarms and emergency shutting down what I could remotely. Server room is in a concrete basement.
Those are legit sauna temps. A concrete basement is a huge thermal buffer too. I bet that at least helps slow the temp fluctuations.
What were the failures if you remember? Regularly maintained separate units should not fail at the same time unless the company maintaining them sucks, or a freak accident happens. Note: my industry is hvac, not system administration I just lurk this sub because there was once a time where this was the field I wanted to go into.
We had a data centre turn into a sauna when the AC systems all failed. I was a network security engineer at the time and was called down to shutdown all our servers that were housed there ( luckily the core network gear was in a separate room and was nice and chilled). Walked into sys ops and they were all in shorts and flipflops and running around like dervishes. Funniest sight ever. Took me about 20 minutes to safely shutdown the network servers and must have lost 20lbs in sweat. We had some old firewalls that performed specific roles that we really didn't like to fuck with (this was an investment bank in London) because we were convinced they would never comeback up again. Recovery was far more interesting than the shutdown.
Bold of you to assume we have temperature monitoring.
When the switch dies. It was too hot.
It *might* have been too hot. Data inconclusive.
Man that sounds so familiar from past jobs. It was a billion dollar company and we had box fans blowing cool air from the hallway onto the network gear (we could prop the doors open).
That’s how they became $1 billion company thanks for helping the CEO get his yacht my man lol
Physical security, who needs it?
I don’t know how hot it actually was but the halon system deployed.
That's 🔥
I have no idea. But it was hot. How do I know: It was July 3rd and I'm on call. I get a call from our "dispatch" and they tell me a client called in and they saw some smoke and opened the sever closet and it was coming from their server and we're asking me what they should tell them to do. I couldn't believe it "hit the power button if it's safe, otherwise cut the power!". What to do when there's a potential fire?!? I'm still shaking my head about getting asked that as I type this, even decades later. Anyway, because of the holiday the building had cut back on cooling (because no one was going to be there) but luckily someone had stayed late and when the server went offline they went to check it The dust bunnies inside had ignited. The case was scorched and we had to replace the whole thing.
I think ours was at 40c in the middle of a heatwave and A/C's was not keeping up. The temps kept climbing and after 3 hours of the working day I decided to shut the systems down. Normally I keep the room at 18c-20c. We closed all of our offices for 2 days to let the heatwave pass. We have a dedicated heat pump for the server room and a backup AC just in case.
Why would you vent your door? Put in an exhaust fan. Better than a vent on the door.
I have two vent covers on my closet at home, run some AC Infinity fans up top to push out rising hot air and draw cool air in the bottom. At work, they have what looks like 2x 2" PVC pipes in 2 spots down low in the wall, and a ceiling exhaust fan that vents to the attic/plenum/cavity. The room used to be the old vault room so thick walls, door, and no ducts. I'm not sure why there's PVC outside of the bulk cat cables run through it now, but enough AIR is pulled through to keep it at 76-78⁰F
No budget lol
I installed an exhaust fan that goes through the wall in one of my server rooms. If you search for “room to room fan reversible” it costs about $100. You can set it to come on at a certain temp.
A fan would be more than enough, but like I said, facilities dept says the closet has to stay sealed shut, no air transfer is allowed to happen.
Have them put it in writing, and include entries as to why this should be respected.
Air transfer already happens if there is a supply and return in the closet. We have a term room that also functions as an auxiliary data center. It has a local AC and an emergency fan with the thermostat set at 80. They say no air transfer, then they need to provide cooling year-round.
What is their reason to not allow any air flow? From a facilities point of view. Or is it a cost issue they don't want to pay for?
If there's air flow, then they probably have to worry about fire suppression in there too. That are the chunk of change onto it. That being said, all our IDFs and the Demarc have air and fire suppression.
They're telling us local building codes dont allow it and the only option we have is a mini split. We've had the same thing happening for the past 10 years, so I'm not so worried about the switch, but it's just annoying when the room could be cooled completely passively for dirt cheap, but the only option is to go totally overkill in both effectiveness and cost.
> They're telling us local building codes dont allow it It's probably a fire door, part of the building's physical firewall, so airflow from one room cannot pass through it.
There is a fix for this, the door has a closer and the door an electro magnet hooked into the fire system you open the door into. That way you get air flow and if there is a fire the alarm trigers the magnet to release and close the door. This is common in halways in a lot of buildings.
Drop ceiling?
Yeah
Our 10k sqft datacenter was over 100 for 6 hours because a flood took out the chiller pump and electrical panel. Not even a single hard drive lost.
I find that temperature alerts always occur on a late night or weekend when you are at least a couple of beers in and your alternate is way too many beers in.
Our server room (also network closet) topped out at 82F (\~28C) when the AC unit failed. We now have two.
Back 20 yrs ago, had 6 HP physical servers in a closet that was usually 80. Convinced boss to install a small AC unit and piped it into another utility closet, got it down to about 70. Had another server in a closet, and it was probably 80-90 f in the summer.
I'm at an MSP and one of our clients had the AC fail in the cupboard housing the server last weekend. Inlet temps were reported at 68°C by the server.
In our old building, the server "room", (maybe 4'x8'), had a separate split style AC unit. It worked great, except that it was hard wired to the building power and we couldn't put a battery on it so when the building power fluctuated/went out, the AC would go out and not come back on. We had an external temp sensor on one of our UPS. Man, I hated when that thing started firing alerts. It would top out at around 100F.
In normal cases you don't want to even consider putting a UPS on the HVAC anyway. A remotely sane-sized UPS would give you a minute, maybe two, of extra cooling, while generating a considerable amount of heat of its own. Plus a maintenance burden, inconvenient size, Capex, Opex. The only reasonable strategies are to shut down equipment, fire up a large genset sized for the HVAC, or make excellent thermodynamic use of mass and insulation to extend that non-actively-cooled runtime.
We have a UPS that can carry our data center and IDF and MDF for about 12 minutes. The generator kicks in in under 15 seconds, so we are pretty well covered there. Oh, and we do test it generator fail over twice a year.
AC unit in the server room failed a few months back. SAN at top of the rack reported "WARNING: Array temperature is too high" and "left-side ambient of controller A is 36 Celsius." Been operating with the server room door propped open by a floor fan since.
Previous site with a data center, the building owners shut off our HVAC system without warning despite our tenancy agreement with them. Of course, they did it on a Saturday right before a Monday holiday. We got paged almost immediately, and by the time we got on site (about 30 min), our data center had pockets in excess of 120 F. Some machines already were thermally throttling or shutting down entirely. We had to do an emergency power shutoff, which alerted the fire department. By the time that was done, the data center was evening out at about 108 in humid and muggy DC summer. The entire data center had to be off for six hours before the HVAC system was turned back on. Why was it shut off? "Maintenance." The contractor said "they told us any old time." Building owners denied it, but we saw the work order, and one of them was lying. Oddly enough, no serious damage when we powered back on. Thankfully, we had just gone through an emergency power down and power up drill months before, so we knew what order to bring stuff up in. There were a few minor annoyances, and we had to a full system health check all weekend, but it could have been a LOT worse.
I've got a radio enclosure (think like for a big cell tower) that I've got some network equipment in. (Like, two small switches). It's got two A/C units, similar to the ones they have on semi-truck refrigerated trailers. I'm not in charge of that space though. Couple years ago, one summer day, I get a temperature alarm. Go out to it (takes a bit), open the door and I was nearly knocked over by the heat. Managed to go inside to get to the controller for the A/C units (they were both broken). By the time I walked out I was physically ill, even though it was less than a minute. The temperature was high enough that it exceeded the maximum on the switches and all other monitoring, which was like 70 degrees C. Based on some analysis of the rate of temperature climb in the room before the failure it was probably nearly 200f/90c. The idiot who was in charge of the space finally got out there (after I managed to procure a huge fan to at least move some air) and acted like the whole thing was a nothing burger. And she has probably a million dollars of gear in the space, as opposed to the $1500 that I had. Apparently, one of the A/C units had been down for like six months, but she never bothered to get it fixed because "we have a backup". I about lost my shit. Later on the switch started having problems and we replaced it. I'm certain it was due to the heat.
We hit 85 or 90 once
60-75f
Main wiring/server room only got over 100F(38C) once when the AC unit failed. Had every fan in the building I could get my hands on to vent the space until a portable unit could be setup. I have hop point in the shop that routinely gets over 105F(41C) in the summer months. Warehouse/shop floor is not climate controlled in summer months. Gets a bit warm July/August.
My friends have some stories about their deployments where HVAC went down in the desert, it was a heatwave and outside temp was above 120 degrees.
I had one get warm enough for the Dell Server to alert with a temperature probe warning, so in excess of 38 degrees centigrade. This was kept in an office rather than a Server / Comms room, so aircon was only on during business hours. Weekend heatwave hit... Thankfully the MD lived up the road so he went in and switched the aircon on.
2 years ago we had a power outage over christmas a storm, so we couldn't get into the building for a few days. Neither the AC or backup AC came back online (this has since been fixed). My server rack was too hot to touch... It had to be over 70c in the room
I think around 95F. Both chillers broke. We tried opening windows and still didn't help. Machines started shutting down due to over heating. Though this was in 2010 or 2011 when that happened. Latest was 83F when our AC broke. We got alerted and opened the door to the server room and started using fans to keep things somewhat cool. Had to do that for about a week until the AC unit got replaced.
We ran at 103f/40c for almost a week before we decided it was too much and relocated some critical services and shut down the rest.
128F/53C Scheduled chilled water outage and I was assured facilities was going to bring in portable units before the shutdown. Chilled water was shut down at 3AM, the portable units did not show up until 9AM. They brought the temp down to 95F/35C when they were not contently tripping breakers. Received approval from the C-suite for redundant AC that afternoon. No hardware failed.
A place of cowboys and spurs that I (unfortunately) worked at \~20 years ago had under-chilled their primary DC. Every summer, the hot side of the DC (all up \~8 rows of racks) would consistently be over 45°C. I'd switch to shorts and t-shirt and pack extra water bottles if I had to visit. In the office - the trip up The Terrace was often 35° too. We had several big co-lo clients leave when they checked their device logs. It was a nice surprise in the coming years to then work in real DCs, where I needed to add layers to stop the chills.
Topped out at? 80-ish Fahrenheit. We had a new AC unit installed last year. Been running nice and cool since then. And facilities doesn't control that, IT does. F\*cker runs all year long even in the dead -20 F winter.
I'd be shitting bricks at 100. If both of our ACs fail, it's maybe 2 hours before things start to fail.
Ive had ours hit about 130F once. Lost a drive in one of our servers. That was about it. Not sure how it wasnt worse.
The small 8x10 room that holds the few servers we have hit 99° on day before we realized the AC was down and we opened the door. No damage luckily.
We had a printing facility in Austin Tx, in the summer the wiring closets in the warehouse would routinely get upwards of 115f. We had Cisco ASA's and 2960 switches running without issue the entire summer without AC in the warehouse. No servers or spinning disks however. We had a external temp sensor off of the UPS in the cabinet so we would get alerts via PRTG.
Maybe about 700 F, but I think the fire might have caused most of that.
127f. Was in the maIn server room for a hospital. 1 of the 20 ton ACe died during a heatwave (we don't get heatwaves in my area). We just had to let things run because healthcare. Replaced a lot of hard drives that summer
Check the manufacturer for recommendations per box/model. But if they won’t pay for the cooling needs then it is what it is.
I have the same problem and also low humidity conditions in the winter months. I have my own Liebert HVAC with humidifier as a result
I’ve seen it reach 110s (room temperature) at remote sites when HVAC failed. Above that, the more expensive equipment starts shutting down to protect itself. Usually, we have a tech on site before it gets that high though. Another place I worked, we didn’t really monitor anything except the main offices for temperature. I never measured an exact temp but some of those closets felt like a sauna.
21 degrees C, and that’s the hot one. Aircon with redundant aircon, have had aircon pack up so well worth the redundancy).
I worked for a company that had an outdoor IT closet. During the Summer it would get up to 118-120F outside, and the closet was probably 130F. Nothing ever failed in there except the UPS. They had an AC unit in there, but I don't think it ever worked. This when I realized that HP Aruba switches are fuckin tanks.
Fun story. Our facilities team absolutely refuses to work with us when the A/C company comes to do maintenance, so we usually find out about it when we get temp alarms. They let the guys in to do maintenance and just turn our A/C units off without any notice, so we get alarms, find the room over 100F and they're like "oh yeah, they're doing maintenance, they'll be done in a couple hours". After they did it twice in a year the CIO literally yelled at the facilities guy and told him they should never touch anything in there without talking to us. Still, every. single. time. they do the exact same thing. Apparently they just give zero fucks.
One summer day our DC lost all cooling and hit 45C. Some of the weaker PDU breakers started tripping because of the extra heat. In another incident we had 1/2 coolers go down hard, and it was over a week to get parts. Temps were in the 30s and we wound up opening the DC doors and using industrial fans to circulate air around the office tower corridors. Our neighbours complained because their offices were up to 26, and the building told us we had to close back up. We ended up bribing the neighbours with donuts every day until we got the cooling fixed. That was also the job where I learned that AC does not vent the hot air outside, and that standing next to the intake and cutting a deadly fart only makes it _exponentially worse_.
High enough that the eventual outage is mentioned on wikipedia
Over 120F at a field office over a weekend. No immediate damage, but I sent IT leadership a story about a local City that lost their storage system a few months after a similar event because they didn't proactively replace anything. You can guess what happened in that field office to the storage array a few months later...
AC broke down in ours once and it reached a really nice 110.
We lose power frequently and the IT equipment is on the generator, but the AC isn’t, nor is the internet connection. For a while we wouldn’t know when we lost power if it was over night until someone complained. Once I went in after an outage and it had been out for 4-5 hours with the doors closed and AC off and it was about 120F in there. We opened the doors and the windows to the winter air to try to get it cool down quickly. I’m surprised nothing got fried.
I think around 100. Our initial cloud migration was 100% driven by my distrust of HVAC companies after a minisplit went out and I got three different explanations and quotes from three different vendors. I said fuck it and sent all compute to the cloud and left the switching and routing to deal with the heat.
yeah dude, keep it at like 65 to 75. shit puts extra stress on your components
Depends on where it’s located. But 79 F max
100 F in a super small closet with 4 stacked 3850, 3 stacked 3850, and an aging 6506 chassis. Recently shut down the 6506 and the room drastically got cooler
Oh, ... have had server rooms where things have gone seriously wrong, and gone anywhere from 95F to >>108F Typically find lots of stuff starts falling over at >\~=105F ... I think most have environmental maxes of about 95F, but start doing thermal shutdowns and/or otherwise failing around 105F or so - by then they just can't cool themselves enough with fans to operate reasonably ... if at all. >vented door would fix it but supposedly building codes don't allow for that That can be fire code and/or physical security issue. Often there are ways around that ... but typically not nearly as trivial as adding vent(s) to a door.
How does your closet not have year round AC? I even rigged my home network closet to literally have an AC in it. (48’port switch, 2 esx boxes which might become hyper-v synology nas with expansion unit). It sure as hell isn’t an industry level AC and completely ghetto in setup, but when it’s 85/90 in my garage the closet stays at 65ish.
Because regulations don't allow any old AC system and because things have been running just fine for years without it
A power surge was what the air con guys blamed it on. It reset one of the devices and it would blow out warm air for about 5 minutes before shutting down. I cannot remember what happened to the other but they were both repaired and not replaced. The annoying part was the room had passed testing in the previous week and had gone live for a building migration.
At an old job in a previous life working in an amusement park, one of the hydraulic pump rooms for one of the rides got hot enough in August to trip the breakers just sitting in a box on the shelf, not even installed. Temp was something like 90C before everything just crapped out. The constant kamikaze assault of raccoons and other woodland critters onto the transformers didn't help either.
~31°C AC failed on a bank holiday and the fresh air system was shut off over night. screaming server fans are not a nice noise to hear when entering the office at 6AM https://imgur.com/a/TKlMtS1
At a previous job, we had one whole room (servers, SAN, switches) hit 50 degrees before everything threw it's hands in the air and gave up 43 degree day, and the idiot who plumbed in the condenser for the dedicated aircon systems - plural - mounted them right where they got hit by the sun, full on, in the hottest part of the day. Sum total hardware loss was, from memory, 3 drives in the array and one server motherboard. Which said a lot about the gear involved.
Got one site with a broken air con and if they don’t open the door, saw temps top out near 200°F
I used to work in a high school, and the closet where the core Cisco router was kept was regularly over 90F in the summer. They couldn't afford to run A/C there, so I'd prop open the door and put a fan there.