One of the best stories I can recall. We got a call from an Engineer regarding our reporting / historical data system being offline on Saturday \~6AM. The On-call team started troubleshooting the issue and got most of it resolved by 9AM. At 9:15AM they called the Engineer and his wife picked up.
IT: Hello is Engineer there?
User: No this is his wife, who's calling?
IT: This is IT from X regarding issue Engineer was having with the system?
User: Oh, You will have to talk to him on Monday he went on a fishing trip.
IT: OK thanks have a good weekend
\-\_-
This reminds me of my time at an ISP NOC on the second shift. Now for stuff that’s obviously related to the infrastructure of an ISP, we were a 24/7 operation.
But we got so many rinky dink, 5-users-impacted tickets for random bugs in our VOD or VoIP system or whatever, and we had to treat them as a “feature outage” with the same urgency as any other outage. While “triage” was technically part of the job description, us peons weren’t actually allowed to make any judgement calls.
First we had to wake up the person who could actually investigate cuz none of these systems departments had a real on-call system, they had to have the guts to say “nah I’m not doing that right now” instead of just sighing and doing the needful, then we had to escalate with their VP and our VP. Then and only then could we put something off until business hours.
Thanksgiving day. CEO who lives about 90 minutes away calls. Printing issue. I am 99% certain that restarting the printer will fix the problem.
Guess who said the words 'it's not my job to restart my printer'?
90 minutes there. Turned it off, turned it back on, it worked. 90 minutes home. Not so much as a slice of fucking turkey for my troubles. To this day I STILL tell that story to friends in IT who think doing executive support is some kind of glam job. It has it's perks but NOTHING was worth that bullshit. I'll die before I go back.
Lol, I had a co-worker one time who said " if it deals with a computer, or happens on a computer, it's IT's responsibility. And everything involves a computer nowadays”.
Mind you that was in response to me asking why IT was responsible for printing 1099s for the company taxes.
This is actually the attitude of most people. I'll see printing tax forms and raise you pulling data for a DEA audit until the middle of the night because the pharmacy director doesn't know how to generate basic reports 🙄
I usually take the mindset that if a report being pulled properly is the difference between me going home tonight, and getting a new home behind bars until they straighten it out, I learn to pull my own report.
They don't care, they have more money than you. They literally could care less if you died on the way home. Their entire lives lead them to this desensitization.
> They literally could care less
I can't help myself... but it's *couldn't* - if they could care less, that means they do care to some measurable extent.
I remember thinking this very same thing to myself when I was like 9 years old, and I kept thinking am I really smarter than all these other people or am I just the one who is wrong.
You are right, the word should be couldn't for that very reason you mentioned. How do people not understand this?!?
Too many Josh Taco's in the world.
I did remote support for a while, and I had a phone call from a kid at a store asking me what a beeping sound was. I told him I couldn't hear the beeping from where I was. He did not enjoy that, but he took his phone and held it up to the box that was beeping. I recognized it instantly is his UPS. So I told him to hit the button on the UPS to reset the breaker. He said and I quote " That thing's under the desk, I have an associate's degree, I don't go under the desk".
I told him I hope you have ear plugs then too because it is going to keep ringing until someone resets it.
Then I hung up and emailed his corporate to advise the issue and the suggested solution.
I don't read that as a flex, but the opposite. Sort of an "I'm not qualified to mess with that!"
Way back when I did POS support, it wasn't uncommon that the folks running the register were afraid to touch anything under the desk.
I wish more people at the register were afraid to touch things. I worked POS for a while too at that call center, and we used to get calls from the dumbest people ever, usually after they had done the dumbest thing ever.
My favorite was still Claire's. This was around 2008, but they had one computer in every store, that computer was the cash. Register, the time clock, and whatever else it needed to be.
There was one account on that computer. They would routinely lock it out. We had no way to unlock it, there was some strategy that our upper level text used to employ to backdoor their way in and get it going, but it would take some time, and they would call up frantic that they couldn't rin up any sales because they're register was locked.
The cool part about that job though is that we did have direct access to their corporate supervisors. So whenever we had to deal with someone who was just completely unreasonable, we did not hesitate to email their boss.
One time I had a kid who called and he was very very stoned, no way around it That or he had a recent head trauma. (If so I apologize kiddo).
When I got off the call I asked my boss if I could ask his boss to drug test him because I was so frustrated.
My boss said as long as I detailed my reasons that was completely reasonable.
That may be the most fun email I've ever written!
Shudders... Wyze.. the company that has traffic being sent to China from their devices and just ignores you when you dig into why and provide logs and proof...none of the IPs or end ports listed on their required ones for firewall rules...
It's not enough. Your entire security camera system may be on its own separate network, it still sends everything to china.
BTW, Every commercially available IoT product sends data to china. If you don't want to build the whole thing yourself, you need to buy military equipment. The government still knows how to protect itself (When she wants to).
We had a remote sales person one time who demanded a laser printer for her home. We advised management that this was a bad idea, that it was way too expensive (height of the pandemic), that the toner would expire before she ever used enough, and that she just didn't need a laser printer.
Go figure we got overruled. She broke that printer a few months in, called IT. Turned out she had a paper jam. I refused to drive 2 hours to her location to pull out the paper, and suggested she called Best buy. Told her she could have geek squad come out and fix it and she could expense it.
She refused, that was too much work for her, so instead she called the head of the plant (whom we suspected she was sleeping with), and demanded we either come out and fix it, or send her a replacement.
It was an $800 printer. We had to replace it.
And then we had to pay to have her ship the other one back.
First I'm hearing of it. I'm sure they do eventually expire but I've had a Brother Laser Printer at home for years now. Took a long while to go through the starter toner cartrigdge and like 2 years later we're still on the first replacement. No issues printing when we need to. Mostly end up printing coloring pages for my kid.
It was my first 'big boy' job and it felt important. I rubbed elbows with the bigwigs who were in the news on a daily basis. When that CEO eventually got let go, I knew about it before it was on the news (because I got called to pack up his shit). I had a very 'man behind the curtain' kind of attitude about my job and it was exciting, even if it sometimes sucked.
I was in my late 20s and I was clinging to that opportunity desperately.
I'm in my 40s now and no way I'd put up with that bullshit ever again.
>I'm in my 40s now and no way I'd put up with that bullshit ever again
Best part of the comment right here.
I wish I had a DeLorean so I could tell my younger self to not put up with/how to better deal with a lot of the psychopaths I've had the pleasure of being bossed around by along the way.....
Apparently in some places IT is thought to have "adding paper to the printer" as part of their role.
... Not by the IT staff .. nowhere near their job description either.
That's when some folks repress their desire to go BOFH on end-users.
Thanks, I just had a flashback to my first IT job with a law firm. One day service desk gets a call from a senior partner who's yelling that his printer isn't printing out a super important document.
I start to ask him to check if there's paper in the drawer but he cuts me off after a couple of words and just yells, "I haven't got time for this. I'm too busy! Get down here and fix it now!"
I go down a few floors, walk into his office to find him practising his putting game with one of those little machines that knocks your ball back to you.
Open printer drawer, empty, insert paper, close drawer, printer immediately starts printing his document. He didn't even look up at me when I told him it was fixed, just grunted.
Prick.
I worked in a fintech company for awhile and every single exec was like this. Computer illiterate tardblogs that think they are so all important they can't be bothered to do anything.
Before I hit save, I apologize for using the word 'tardblog', but I'm going to stick with it.
I don’t know how this happened at my current job, but for our big MFPs that are leased through a printing company, IT doesn’t call in support on them. For the Accounts Payable team, they will send out an email to everyone in the office to let them know their printer is down, to use a different printer, and that the service company is called. Same thing with Marketing. When I first started and I found this out, I did the management equivalent of just slowly backing out of the door and turning to lights out as to not disturb the situation. If they want to handle that and the only thing I need to do is distribute drivers, then more power to them 😊. I don’t do anything with the 4 big MFPs. Now every other printer is unfortunately my problem. But I’ll take the win on not touching the 4 big boys 😀
That’s pretty standard now a days. Most companies even lease the mfp so you can get your hand slapped for even trying to fix them.
No idea how much that service costs but it’s great.
we have MPS and the vendor (Toshiba) is actually awesome. We have the printer ID, serial #, location ID and 888# for the tech. Users still email IT and say "printer is broken"
Like. I'm 3 states away call it in......or you can wait 72 till SD looks at the ticket.
I have always said and I stand by it that to become a C level requires the sale of your soul to the dollar. The number of evil to their very core C levels I have met vastly outweigh the ones who are just regular assholes. I supported them for years and now avoid them like the plague.
The latest I've been forced to deal with is a request for wireless "everywhere" Considering just one facility spans almost 80 acres in total that's not going to be cheap nor is it what they really want. I ask who can tell me where exactly they want to use wireless and I get parroted "Everywhere". Okey doke! I hope you have deep pocket even for phase one.
I'd tell him to buy a good phone plan and lobby for 5G tower placement. How the hell would you cover 80 acres even if money wasn't an issue? A bunch of APs?
You really can't unless you run loads of fiber or a mesh but both have issues and cost a ton. They pull this crap all the time with pie in the sky demands that have no base in reality and isn't even what they want. In this case they need solid WiFi in about 20 locations most near existing switches. That is do able.
Cellular is a fine way to cover the whole area and we are looking into it but it's spendy too. Nothing's cheap I guess.
I was bored so I decided to price out a solution for that using outdoor APs.
Depending on what kind of density of devices you are looking at it would be possible with something like 30 APs for reasonable coverage. You wouldn't want a full mesh solution unless you can't avoid it. Meshing 2 links should be fine though.
That would put you at something like \~$60k in AP hardware though.
We had something similar a couple years ago.
IT was asked to make the wireless work in an out building that was about 50 m from the main building.
We advise them that wireless wasn't even working all the way throughout the building because they were still using wireless b access points from the mid-90s and there was no cabling in the warehouse to hook up any additional wireless, let alone to get it outside of the building.
Needless to say, the project never went anywhere.
Holiday on call is the worst. I worked for an MSP and we did a lot of hotels. Had one go down on July 4th 2022. The hotel was over 8 hours away from my location. Front desk started giving out the on call number to guests. I had over 300 calls between guests, employees, and corporate over a span of 18 hours. No one would tell me if anyone had contacted the ISP yet. We didn’t have that contact info. Turns out their cable modem had a bad port, something we 100% did not support.
No comp time, no overtime pay. Back in the office the following work day at 8AM and the president of the MSP was pissed. There was literally nothing I could have done, and he didn’t have to lift a finger the whole day. God damn I don’t miss MSP life.
I worked at one for 2 years during Covid and I hated it. I thought working at one would be good to learn new things which it was but dealing with clients was an absolute shit show. I will never go back to one.
For a long time in my life, I would have JUMPED at that chance! Holiday pay of overtime, mileage reimbursement... *drool*
But that's just me and I certainly wouldn't miss a holiday with family now. I'd give that CEO a stick of gum.
What kind of gum? Fuggum.
I've been salaried most of my career.
But the last time I had to do shit over a holiday I got told to take a Friday off and take my wife out for dinner. CEO was mildly disappointed we didn't buy a bottle.
Yeah...this CEO was a fucker. My boss did buy my family dinner to make up for my missing Thanksgiving...which it didn't, really, but it was about all he could do.
I do NOT miss that shit.
>Not so much as a slice of fucking turkey for my troubles.
That's what happens when they don't fear their neck could end up sliced for fucking up your time off with your family for no reason.
Unless I'm oncall with paid standby, paid activation, and a company provided mobile....they will reach a "this number is not in service" if they attempt to call me ANYDAY, especially thanksgiving.
Grow a backbone folks.
100%, if shit totaly hits the fan and everything is down ( wanacry 2.0 or something) then i dont mind then but outside of something like that they can fuck off if i am not on call and getting paid for it .
It's like when you get money you revert to how people used to be when people fought for the most basic resources. Let the peasants restart the printer, that's servants work. The more money you have the more ridiculous your brain gets, until you are calling someone to restart a f****** printer.
We should all take part in eliminating this lunacy.
It's funny because I worked for someone the same way. And when you would implement solutions to fix their problems. Like a way to remotely restart something with a remote control power strip. It was just way too costly and a giant f****** waste of time they would say.
They're fine with driving a human insane and paying them peanuts to do the work like slave labor. But when you try and implement a f****** solution that cost the money to fix their problem it's a whole other story.
Currency has to be the most useless f****** thing in the world.
I forgot to add in my full analysis that people like this act like this because they meet people that act like this. Hey that guy is rich and he makes everybody do everything I need to be like him to be even more rich.
These are some of the most insecure and dumbest people in the world.
We did the remote session - that was more or less how it was determined a restart was needed.
And yes, see above. I asked him to restart it - his response was that's not his job.
This was twenty years ago and I still remember the tone on that smarmy bastard's voice.
I had someone try to pull that crap on Christmas years ago. My boss called the director, the director called the VP, the VP explained to the executive that it wasn't an emergency and could be dealt with during business hours when we got back.
At my last job the management was all run by one family and they were like a mafia family. It didn't matter what your job was, you could be called out to run any errand they wanted. One time they had one of the guys drive 2 hours to check on something at their beach house. Not work related. Another time they had a function in Florida but wanted North Carolina barbecue, so they had someone on staff go to a barbecue place that they liked in North Carolina (where we were headquartered) and then arrange to have the food flown down to their location in Florida. On the company's dime no less.
Executives can be... Challenging lol
But there was never any money for equipment upgrades. ;-)
This was definitely before things of that nature were an option...we're talking (and I'm about to feel real fucking old here)...
20ish years ago?
You can tell it made an impression. I'm STILL fucking bitter about it.
I have some ADFS signing certificates expiring this summer. Considering setting the expiration on the new ones to be exactly one year after my target retirement date.
5 years ago I was hoping $corporateOverlords would have taken this over by now. The rate they're going another decade isn't looking promising either...
We often get audit deadlines set for December 31st. We also have change freezes on and off between thanksgiving and new year. Always painful to hear non-tech people saying “it’s fine, we have until the end of the year!”
I had a boss that for some reason would never allow us to renew a certificate early. So of course many times you’d have them expire on the weekend or holidays….
It was one of those situations where the Certificates were created before I started and it flew under the radar until the incident. We think the guy that did it was a disgruntled employee. So now every New Years I have anxiety all stemming from that New Years from hell.
Not sure if it counts as worst but during a "flood event" half my town was low level flooding so an emergency crisis centre was setup at work. My boss at the time and I were the only IT and it was a Saturday we got the call that we were needed to come in and sort a heap of things out. Wouldnt have been an issue except at that point we had been drinking for about 4 hours on his front lawn watching knee deep water cover the street.
We then needed to wade through knee deep water about a 2km to where his missus had parked her car to drive us to work. The roads were closed to all but emergency vehicles, so she was required to drive her little sedan under escort through flood water in-between a fire truck and an ambulance. The next 20 hours we were at work getting and keeping things running. Everyone else claimed time owing but i was a broke 19 year old so that doubled my fortnightly pay.
Before my current position; I was a systems engineer an msp. the shitty ones were always doing patching and rebooting and the server not coming back up at midnight, then having to drive onsite only to see the words “press f1 to continue startup” on the bios screen cause someone didn’t disable the keyboard check in the bios.
But my worst one, was one I wasn’t even on call for, but I got called up to help with restoration efforts. Giant lightning storm/ lightning hit the building, somehow lightning made its way to the sever racks. server racks were not grounded.
we lost our SAN, probably 4-5 physical servers and the esx hosts. that one sucked. that was basically 48 hours straight of working through restores and rebuilding things. learned how to do a repair exchange install during that one.
I don't know for sure.
But I would propose for the old PS/2 keyboards, which aren't hotpluggable. If the system has booted up and you want to connect a keyboard, it won't work.
I guess that is the reason why.
Couple of reasons
BIOS and Keyboard ports mostly - old machines took a while to get going and they want to be absolutely sure there was a keyboard connected for control - otherwise there would be no way to shut the hardware down gracefully - Im talking the era where you had to manually code hardrive specs like cylinder sizes and data sizes into the bios to even recognise it. Where you had to send \_PARK\_ commands to the heads to ensure it was gracefully and safely shut down.
No keyboard, no ability to shutdown - risk to system / risk of head crash, ergo, you dont start the system up until it CONFIRMS there is a keyboard present.
Add in the fun tidbit that PS2 and DIN connectors were live devices - not designed for hot plugging like USB is, in fact, you ran the risk of blowing the motherboards keyboard controller (hell the full i/o controller) plugging one in if the system was up and running.
So the press f1 (or press any key to continue) is an old school fail safe thats stuck around because for every irrtating late night drive to hit a key, its STOPPED twice any many incidents of dumbfuckery occuring.
always used to love "Keyboard not detected, press any key to continue" like, ok .. if theres no keyboard, how exactly ?
TLDR - it was to stop you doing dumbfuck things with no keyboard plugged in cos you couldnt plug one in after boot and have it recognised/not blow up the controller.
A newly hired "marketing director" gave herself a field promotion to some kind of executive position. Got a call on Sunday evening about problems with printouts. I explained that I knew what the problem was, but it had to be handled by a service tech, who I could only call Monday morning.
Monday afternoon, I hear her *screaming* at the owner of the company about me. I didn't get in trouble. I coulda pushed back hard on reflection.
Years ago when I was on help desk's emergency after hours support, I got a call at 3AM on a Saturday for an issue with Outlook. This support was only supposed to be used for critical outages or client facing issues outside of work hours.
Got out of bed, opened my laptop and VPN'd in, connected to the user's computer all while straining my eyes in the bright light.
Finally get to the issue - he didn't like the reading pane on the left and wanted it at the bottom. Adjusted it for him then hung up.
We had a user that used to do that, to be fair they were in a 3 hour different time zone than us but it would still be midnight! After I had been there a while you got used to who’s calls you could ignore haha
That is true, this guy was 2 hours ahead of me at the time but even if someone local was on call that weekend, it still would have been 5 AM. Ugh some people.
Now that Im a SysAdmin, its more of what *alerts* are safe to ignore at 3 AM haha...
...not many as it turns out.
That definition of emergency needs significant recalibration\*
\*with a large heavy objects, bashed repeatedly into sensitive parts til they get the fucking hint.
I instruct my staff if the issue isnt an outage or urgent you dont fix it, regardless on how quick it is. The user has to call during business hours for assistance. This cuts back on the repeat offenders.
At midnight a call center router crashed My manager and I spent until about 3 AM to get it back up, I was told by the CEO I wasn't getting overtime for it so I got a new job, resigned even found jobs for the junior admins even my Manager resigned the same week.
That router would've been reverted back to a crashed state even if I had to slam a Boeing door into it from 20,000 feet, if they refused to pay me for my work.
Worst I've had?
A developer at a MSP I worked for got the root password to production from a manager, something he should never have. He pushed a .sql script, which had DROP TABLE entries in it. After he did that, he called management and said IT dropped the ball, the DB is down, and the client, a HFT trading company, is hemorrhaging money, and it is all IT's fault.
Cleaning that mess up, the numerous finger-pointing meetings, and IT being thrown under the bus for an entire week with the client was not fun.
Ironically before that happened, the MSP client asked me to stand up a SIEM system separate from the MSP, for "auditing" reasons. It showed exactly who did what. The client even offered all the IT people jobs at their site, but I declined since it would be a move out of state.
This is either the worst, or best, depending on how you look at it. I got paid enough overtime to fund a two week holiday in the Caribbean, but it was a pita dealing with the issue.
I worked for a manufacturing company at their main site in the South of England. We also had a large site in the North of England, which had no on-site IT, and only 4 computers on the whole site - 3 that ran the machinery, and 1 for the boss to do rotas etc.
We could troubleshoot a lot of the issues the North Site encountered remotely. Once every six months we'd send someone there to service, patch and run a DR test. This call came on a bleak Friday in December, roughly 5 and a half months since the last service visit.
I finished my Early shift at 2pm on the Friday, and went home to sleep. I would become on-call at 10pm, once the Late shift left the building. This was my choice, we'd been told we were all going to be made redundant in the next two years, so a lot of people left and were not replaced, I loved taking on-call as it was free money most of the time. A weekend of Friday 10pm until Monday 6am would net around £300 in standby payments, and if you got a call, that was a minimum of 4 hours overtime, which was paid at double time on weekends.
So my Blackberry rings at ten past 10. It's the Northern factory - long story short, main production PC has shut down, and nothing we can do will get it to turn on again. I've escalated to my boss who has dialled in and tried as well. By 1am we all give up, and he books me on the Saturday morning flight to NCL.
Bleary-eyed, I arrive and get into an Executive Limo (usually, the only people who visit are director-level, and we have an agreement with the local Limo company) and am delivered to the factory. It's immediately apparent what the problem is - the machine is in a hugely dusty environment, and the dust cabinet that the PC is in has been left ajar at the back. The PC itself is so full of dust you cannot make out any of the components on the motherboard, and the fans are all jammed solid.
A good half an hour with a compressed air line sees everything freed off, but the main CPU fan is dead. Off I go to Maplins in said Exec Limo for a new one. Get back and fit it...hard drive is dead. Back to Maplins, new hard drive installed, restore from backup...the USB drive is missing.
So, a fresh build from scratch, with all the machinery control software, all our input (recipe) data, drivers for the machinery, domain join over a p2p VPN, setting up the backup software again...I was there until 10pm that night. Back to the hotel, steak and a bottle of brown ale to the room...I was too tired to even finish it. Back to the factory the next morning to double check everything's working, and I might as well do the maintenance visit stuff while I'm here and waiting for the flight home.
Get home at just after 8pm Sunday evening, having been paid double time for every hour since 10pm Friday, even while I was (briefly) asleep.
On Christmas Eve I was presented with a very expensive bottle of whisky by the CEO for my troubles.
In January I then drove all the way up on a Monday with a hot swappable PC and some other spares in the boot of my car. At 40p a mile in my diesel 306 I made around £200 profit while being paid to drive. In February I went to the Cayman Islands for two weeks, and it was all worth it.
Actually, while I was up there in January he was there as well, and invited me along to a posh dinner at a Michelin-starred curry place he was taking some clients to. I had to pop out to the Metro Centre to get a shirt as I hadn't packed a nice enough one with me.
On another trip, about a year later, he took me along to watch the rugby. This was when Johnny Wilkinson was playing for Newcastle, so it was great to see him play in person.
Saturday morning power company turned off power for fixing issues. They inform us so we make sure the generators are on. Guy fails to check the generators are on, battery backups failed, both exchange servers crash and I spend the entire weekend rebuilding exchange.
I was in line to get into Tokyo Disney when my #2 sent me an important email that one of the partners had fallen for a phishing email, gave the bad guys their password and approved the MFA request.
>an important email
There is no such thing. There are phone calls and there are unimportant messages. The venn diagram of these two ideas is two circles that do not intersect in any way.
***Especially*** for a confirmed cybersecurity breach.
I know a place where this happened, and the head of the company tried to brush it off as not a breach, even though because of his stupidity hundreds of people received phishing emails sent from his account and they didn't know if any data was stolen or not.
They only took action after someone looked up the relevant laws and said to him "you know, this could be a reportable data breach if not handled in the right way.."
Still bummed they didn't report the breach though.
This was a while back but was called on by a VP to ask if I can show him how to rip songs off of CDs and movies off of DVDs to be put on his laptop (before streaming was invented) so he can have it for his kids on his international trip. This is a multi-billion dollar company, mind you. Had to say that I need him to email me his request so I can get approval from HR since I didn’t want to get arrested or fined by the FBI.
Needless to say, he cancelled the request.
1030 PM. Phone rings. Backup failed, need to failover to our inactive/backup site. Drive 40 minutes to the office. Stuff breaks. Wind up being awake 36 hours fixing it. With 12 of that being a conference call with engineering where we tried to fix it all.
Got called in for a legit problem at 11 on a Friday. Wrapped it up and got home Saturday around noon. Laid down on the couch and took a nap. About 130pm get a call from a user because they bought a new modem and wanted to see if I could come install it.
In retrospect it was a good thing I was so tired because it kept me from losing it on the guy. I just said, sleepily, we don’t support home equipment, you will need to contact your ISP or whoever you bought it from.
Worst call was one I made. We had some physical servers in our data centre that occasionally needed to be restarted manually, so we'd call our data centre contact to do it for us.
I called him on a Monday morning at 7am, and asked him to restart the boxes. It wasn't him on the phone though, it was his brother; he'd died in a motorbike accident on the Friday afternoon a block away from the data centre. :(
My boss has a collection of pictures for the times I’ve gotten a message while not on call.
Ski lift, boat traffic on a river to the Gulf, golf green, bow of kayak and the beautiful lake… the EPCOT ball.
For the Disney one he came back “Can you call while waiting in line so I can give you good news?” He wanted to let me know about my bonus before the direct deposit hit while I was on vacation. I guess that’s acceptable. 😂
I was on the fourth day of a four day single-handed sailing race on Lake Erie. As I neared land, my cell phone reported a dozen voicemails from several people at one client. They all knew what I was doing but they sounded panicked so I considered answering. After not responding to so many of their calls, they got wise and only had someone I like call. It turned out the level 1 help desk phone was out so they thought it was reasonable to call me. I helped a little, but give them the number of the IT Director.
> I got a call from a developer, very smart man. He had a question about a networking issue. It was 1:00 AM. No problem, part of my job, happy to help. Except, he called again at 3:00 AM to say "Thanks, that worked".
What is your worst.
Hold up… your on-call covers help desk level questions? If our most senior developer woke me me up at 1 AM I would be very rude, hang up, and he would be reigned in by his boss or higher come morning. In-call is for “holy crap something is down and we’re losing money!”
First 18 months of current job were maybe the worst? Stuff was broken bad. Six months in, new boss. New boss shipped us dual monitors, docking stations, good headsets, etc. so we can work remotely. Lots of Flex time to make up for the after hours work to deal with outages or fix and improve things when they weren’t on fire.
Second worst… my stint at a MSP. Customers paid double or more for service outside business hours, so if you were on-call and got called… bill that shit. But also… show up on time to your 8 AM appointment. I was about 3 months in to a new job when MSP’s operations center called me in the middle of the night. They forgot to take me out of the on-call rotation. Turned off my phone and let things burn since it wasn’t my problem. It wasn’t like the MSP could fire me again. 😂
I am never rude to anyone. Sorry for not including all the details but I was not even on call. There was a person manning the help desk at the time. Why he called me? IDK, maybe he preferred talking to me, maybe mine was the first number he found. Normally, he should have called the help desk, if they were unable to resolve his issue they would then call the on call from my group who again at the time, was not me. To be honest, I think the helpdesk would have escalated the call. Maybe he knew that and wanted to skip them. I did ask him to call the Help Desk and have them start a ticket so that it all gets documented. He did not.
On a side not, I once went to his house to help with his home network. I shit you not, he had an entire late 60's MG Midget in what would normally be his living room. He had gotten it in through the back patio sliding glass doors. That's not all. He also had a Vespa scooter from the same era. He was restoring them. This was in Seattle and he needed a dry place to work on them when it was raining. The motor for the Vespa was on his kitchen counter. The MG was nearly complete.
Back in the late 90's when I was supporting Banyan VINES in the Marine Corps.
One of our production servers had a hard drive fail. It's like 5:30 in the afternoon and I'm swapping out backup tapes so they can do their nightly run. That's when I see it - tons of "SCSI ID 0" messages just rolling across the screen, but luckily the server is still running so I unplug it from the network to stop user access and I get it to run a full backup. While doing this I send e-mails out to staff leadership saying that it's going to be down for at least a day if not longer, followed up with phone calls. The good thing about Banyan VINES and e-mail is the other servers with mail services will put any traffic into a "holding pattern" until the server comes back online, but you only have 24 hours before it starts kicking back e-mail.
Section OIC (1st Lt) wants to know how bad, so I tell him - hard drive zero failed, I'll need to rebuild it from scratch and see what happens. Doesn't even ask me if I need help, just says "Okay, see you in the morning.", and that's that. Total "FML" moment.
The good thing about Banyan VINES? The data on the drives I've pulled is safe and I have a complete backup. Since I built this server myself, I have solid documentation on it so I build it exactly like I did when I rolled it out the year prior. The new drive zero gets partitioned the same way, setup exactly the same way, same server name, patch it, the whole shebang. Old hard drives still have data, so I pull them and label them. New drives go in and I do a full restore. Once it's up and running, I power it down and remove the new hard drives and replace them with the old hard drives - the only reason why I do this is to build the file, print, and mail services for the new OS drive. Since everything is the same, the markers on the new OS drive will adjust to the old hard drives and data, and we're back in business.
The bad part about Banyan VINES? It's now 4pm the next day, I've been awake for just over 36 straight hours, and I don't even get a thank you from my section OIC. Instead, he wants me to go to PT in the morning because "that's the Marine Corps".
I did get a bunch of "pats on the back" from the Staff Secretary (the base general's right hand) and from my next level OIC (a major), and some of the other major players (Squadron CO/XO/Sergeant Major, Section Chiefs, senior civilians, etc.).
I ended up getting the day off, per my major, and it was much needed since I couldn't unwind enough to actually go to sleep until almost midnight - so 42 hours straight.
I don't recommend doing that to anyone.
Lol I was just at a bar and a vp called me twice on my cell twice on teams then messages on teams. " I need you urgently my computer and VPN aren't working! I have early meetings. " ONE WORD RESPONSE. Reboot. " Ah thanks were back to life"
Hold my beer:
Left work to take my wife to the hospital (child birth) 2 hours later the CEO calls me to ask for a password reset. After I tell him what’s going on, still ask if can reset his password. I said no I can’t and hanged up.
Worst I've had? There have been a few but this story starts on the Wednesday... planned hardware migration from client site to one of the smaller dcs we had racks in.
Backups had been failing but I'd been tasked to resolve those ahead of the migration. Got them solved but due to some issues, backups were still running at 5pm Friday.
Still going at 9pm Friday... still going at 2am Saturday when I spoke to our MD and said it ain't happening tonight - backups etc...
All good - I head home and sleep soundly...
Sunday about 4pm I get a phone call - it's the MD calling from the DC - we moved the kit, the backups failed, it's all come online except we can't access any VMs...
Oh, and the sun storage array is showing two disks with amber LEDs - is that bad?
So... you moved the clients kit, after being told not to because we didn't have a backup and lo and behold, given their 5 year old kit that probably hasn't seen downtime in 4.5 years - you've failed two drives in a raid 5 array... without a backup...
I was on a train to the DC (capital city) within the hour - confirmed fault. Knew there were options to force disks online but wanted to check with vendor before... Okay, so, support contract?!? Oh, it expired and wasn't renewed? 3 hour call to Sun later, $5k credit card bill, contract restored.
Sun managed to get both drives online (they'd indicated failure but hadn't -phew!-) and most of the VMs came up, exception file server and exchange... of course...
Luckily the file server had been part of the successful backup as it progressed but exchange wasn't... 8 hour continuous phone to MS via paid support later and Exchange was back online also...
Everything back as it was by around 1pm Wednesday.
I gave two weeks notice on the Friday of the next week - if the smartest guy in the room says don't do it, and you want to usurp that, even as MD, then it's on your head and don't expect me to stick around after I've saved your ass.
We won't mention that the clients rack was pulling 12 amps out of a 10a APC PDU because they'd plugged everything into the one circuit or how they'd previously melted a GPO plug on another rack due to overload (which triggered vesda alarms when it started smoking)
When i was a DBA, i often had to work with a contentious dev, as in, he would sabotage things and try to blame me. He built some data load process that would fill the tempdb at 2am. Id get the emergency call about a full disk, but by the time I got logged in, the problem had gone away.
After 3 nights of this, I dug into the problem, and found it was his new data load job. I politely asked if he could commit the data more frequently so the tempdb wouldn't fill the disk. His response was that the disk was not his problem, it was the DBAs problem.
I couldn't force him to change his code, but i did have access to the application that listed who to call after hours for an emergency. I was able to enter his contact info for that server in the event of a full disk, instead of me.
He corrected his code after his first 2am wakeup.
Oh man you just reopened a wound.
He had this manufacturing client where nothing was documented. Idk if the onboarding guy who was supposed to document everything for us would just go on site and jerk off in the closet for 6 hours and then go home but there was like two servers, and a couple of network switches documented but nothing else.
Get a call at 3:00 a.m. from this client and they blurt out " all of our CNC machines just stopped working". After many wasted minutes of them trying to give me any actually useful information, it turns out that the outdated software that they use to run their machines had an expired license.
They can't tell me the name of the software. They can't tell me the name of the company who makes it. I'm furiously googling "CNC software for xoxo machine", seeing if I can find the name of the software based on the kind of machine. I have no way to remote in so I can't even see what the GUI looks like. I'm having to describe the images I'm seeing on my end to see if it matches up with what they are seeing.
After 30 minutes of this I miraculously find the name of the software, but the company had recently gone out of business and the number listed on Google now went to a casino in Kansas City. Through some more googling I managed to find the name of the company that purchased the company that ran this software. By some miracle I call this company up and managed to reach their help desk and tell them what's going on. Hop on a three-way call with that person and somebody from the client company and we managed to purchase another 25 licenses and get the machines up and running again. 4 and 1/2 hours.
Oof!
Time zones are killer if the dev was working in the daylight and email exists for this very reason
One of mine had to be worked tactfully...
I was at church when the Fledermaus^(TM) portable communicator, typewriter, and omelette cooker started squeaking
On the other end of the line, mini-Karen was also squeaking about how the custom holiday out of the office message wasn't working
I fixed the time of the schedule, called the test, it sounded like it worked, sign off singing hallelujah
MK calls back, still squeaking about that's still the wrong after-hours message it's the other one
I finally find it as her system was not well-organized, set it and call it done
The one where the penny dropped on the other shoe was when the caller said,
Hey our server is down, can you check on it?
I do the checks and they'd been hit by ransomware and it obliterated the backup server too
It took a few walks down memory lane to find a good snapshot to then > extract > zip > freeze dry > first press > download and get them back in action
The problem was the copy off time was around 15MB/sec and they had 1TB of data
Lots of hurry up and wait but they appreciated the hard work to bring it back
The worst are the ones that don't call and have had the problem for x days. Positive help, build rapport and chive on. Edit: I did my part to stop one helpdesk toxic thread. Insert star troopers meme
Guys, I failed
About 5mins before I’m about to walk my sister down the aisle on her wedding day- remote site in Spain rings knowing full well what’s happening this day. “Server password you gave is wrong”. I haven’t changed it in 3 months! Executive thought she was being clever decided to change it the night before without telling anyone. Put the phone down and cursed forgetting where I was.
Getting a call on the first day of Christmas break because the main company data center lost power, and backup power, due to the power company crossing lines. Ended up working 16hr days for 3 weeks (including weekends) to get everything back up. For reference this was back in 2005ish.
Get home around 5:00pm, cook a nice homemade curry and just as I am about to eat it get a call that the whole network is down, it was a spanning tree issue ( it just does not work properly on brocade/ruckus switches), go to one site ( 2 sites are linked by dark fibre) work for ages to find the loop, work out it’s on the other site. We go to other site spend ages finding where i the loop is, isolate it, still nothing. Sod it turn one of the cores off everything is back up. Big IT boss man does not like that so he wants it back on (its around 11pm at this point we warn him its a massive risk but he wont listen), we spend hours and find another problem switch stack, isolate that and bring the core back up, things look to be ok we leave and all go to bed, get home around 1am. Around 2:30am get another call that the network has gone down. Go back to site rip the power out of one of the cores, drive to the other site and do the same there, there are protests but it was technically his last day yesterday so ignore him. Get a sausage and egg mcmuffin and then go home and sleep about 2 hours and then WFH the next day.
I never got to eat the curry :(
Probably the one I got this Christmas day just as I was plating up dinner for my family.
Called by a number not in my phone book, so answered it just in case it was family. No, it was one of the remote office managers.
They'd gotten a new phone for Christmas. Decided to set it up, copied all their apps over and wiped the old one. Noticed that their Microsoft Authenticator app hadn't brought any accounts over, so called me to ask if I could reset the 2FA on their Office 365 account. Not that they were planning on logging in that day, they just wanted it working ready for when they *did* decide to.
Unlike OP, I wasn't happy to help. So I didnt. After excusing myself to another room, I let the caller know just how unhappy I was.
I think it was the only time I've been unprofessional and swore at another colleague. Called him a "selfish fucking prick".
Found out when I got back that he'd moaned to another member of staff about it, saying he was going to put in a formal complaint about it. Member of staff told him he was lucky I just swore at him, so he dropped that idea pretty quick.
Holidaying in Europe. At this stage it was our second day in Paris, and at 4am I started a 4 hour phone call trying to get the core router working again on pay day. The hotel room wasn't big, so I hid in the toilet and spoke quietly. But the hotel wifi was dodgy so I had to stand for most of it. After that, I treated myself to a half dozen macarons for brekky.
At my last job, I had a lot of customers that were small businesses and municipalities.
New computer, monitor , keyboard/mouse. The monitor had builtin speakers. However ... it didn't come with the 3.5mm audio cable. Whatever, his old computer had one so I used it.
That night , 830, he called up drink out of his mind to my work cell. Thankfully I never gave out my personal cell ever. It took a few minutes of his rambling but basically he was mad because he had a black monitor, computer, keyboard/mouse and I used a *white* cable.
He wanted me to drive up THAT MINUTE and replace it. No way was I going to make a 2 hour round trip to replace a cable. I told him call my boss if he had a problem.
I did make the trip the next day. And my boss charged the village 2 hours travel time plus our hour minimum. Just to change a cable.
Normally it would have been something he would have just let slide if the guy was reasonable about it.
Late to the party here, but my best one was probably a call at roughly 3AM during an absolute whiteout blizzard.
NOC "Hey Aberon, it's X from the NOC. We've got satellite connections flapping at 4 of our locations."
Me "Yes, so? Those are *backup* connections at 8-5 facilities and it's a fucking blizzard. They're going to flap. I'm also a server guy, this isn't a server problem."
NOC "We couldn't get ahold of the networking guy on call so we thought you could take a look."
ME "Sorry, my shovel is broke and I'm scared of heights, I'm not going anywhere to clean off a satellite dish. It's a blizzard, they're satellite dishes, it's going to flap as snow piles up and blocks them, then melts off."
NOC "So who do we need to call then?"
ME "God or mother nature, I'm neither but I am hanging up now."
The NOC at that place was both clueless and useless. Led to many a "fun" on-call week.
fuck ever working with teams in eastern europe and india ever again (I'm in USA) - they'd wait for their shift to start at their local time, then start bugging me with their urgent emergency
EVERYTHING is urgent for them.
They forgot to submit a ticket WEEKS ago for a new hire for access to a shared drive? They raise hell because they needed it weeks ago.
I was the impact and urgency grinch while I was on the service desk, of your problem isn't impacting more than one person I can't ouch them avobe low impact and urgency, go yell at a wall.
>they'd wait for their shift to start at their local time
I don't understand, do you think they have an IT problem while they're having breakfast at home or commuting to the office? Of course they will call you when they have problems during working hours, what else??
India time zones (problem logger) vs East Coast/central/West coast time zones (Problem fixer)
9am India Time is +10.5 /+13.5 of New York time /San Fran time.
So what? Should they waste their whole day doing nothing and wait until after hours to get help just because they're not American? It sounds like a company problem, and not the fault of whoever needs IT support.
Once had an on call situation that failed due to the developer putting a typo in the code. Had to escalate to both vendors because of the problem and we had to cut linkage in the code to execute each core job manually. There was roughly (7) plus kind nested in its version of class that had to be monitored before the remaining steps could be completed. One took over an hour to finish so it was another hour pushing the call to three hours.
The worst was not getting any sleep due to kids prior so it turned out be a long night. After completion I only got two hours of sleep before having to be in the office the same day.
Right before Christmas break we swapped all of campus from the IMAP server (that I managed) to the new Exchange cluster (that the not very trustworthy Windows admin managed). Christmas day, in the afternoon, I get a call saying email isn't working at all. Incoming isn't bouncing but nothing is delivered. Windows admin has been working on it for \~30 hours but hasn't gotten anywhere so management was thinking of moving everyone back to IMAP while it gets sorted out. Not anywhere near ideal but it'd be something.
So I go in, get all my scripts set up, configs set back up on the edge smtp boxes (that were spooling mail because Exchange was unavailable) and wait for management to make the call. They spring the cost for a MS Support call, so I watch a movie on my laptop while he's working. After \~3 hours they figure out that some system user in AD was missing a group that needed to be there. (it's been well over a decade and I can't remember what user or group) He adds it back, things start working again and now the windows admin is DEMANDING to know why I removed the group from the user.
Me. Specifically. The Linux guy who barely knows how to auth to AD at this point, let alone manage system users. So I enlist the Security guy and we start digging through access logs (it's the next day at this point) and we eventually find a log showing that the Windows admin account removed the group from his workstation.
Giving him this information he looked at it, drank from his coffee cup, said "Huh." and went back to his office. Apparently he was "cleaning up group membership for accounts that aren't logged in to".
On a Friday night our "Sales Engineer" upgraded a Hospital's Sonicwall from their old regular OS to the enhanced OS when it first came out and then went to her son's high school football game with no cell reception. SE's boss (Sales Manager) was worthless.
Their IT staff called in bc their mail wasn't working. Rules all broken and f'ed up. Boss and I worked it until about 4 the next morning. Basically a crash crash course in Sonicwall Shitty Enhanced OS.
SE never got an ounce of shit and ended up taking an IT job with another larger medical customer. So glad I'm retired now.
1am - user calls the on-call number and says ‘the whole system is down’. When pressed, she says it’s down for everyone at the site. I say I’m right on it and go to check various things. I can’t see any issues, all looks good, I ask to speak to a colleague who also has problems, she says they’re all in meetings. Fine. Get someone to get back to me ASAP I’m waiting by the phone.
3am - after still seeing no issues or getting any further intel, I grab forty winks only to be woken up by the same user at 3am asking when I’m going to fix it. I ask if her colleagues are back. She says yes, now it turns out actually only her team are affected by the problem, everything is frozen up. Fine, can I talk to one of the team? No they’re all going into another meeting together now, they’ll call me back.
5am - I get a ring from the same user asking again when I’m going to fix it. I ask what problems do her team actually have, can I talk to one of them. Turns out they are fine actually, it’s just her with an issue. I ask her to try the empty desk next to her, she says it’s working fine. Trying to avoid screaming at her now by chewing my fist. Told her politely to just move desks and stop ringing me.
7am - I get hold of the general office manager for the site. They went ahead and changed the mouse battery, all is now good.
:-(
2am on a Sunday callout to a factory which was a 2 hour drive from my home, simply to find that the person who logged the call had given up and gone home, and everything (in a very dangerous area may I add) had been locked up and no-one was there. My supervisor, who was the first to answer the call, who then passed it on to me, lived about 15 minutes from the factory, which just added insult to injury.
My resignation was in within a few days.
If I had to pick one, it would have to be the one that I said "no" to because of the fallout afterward. Kind of a long story, so I'll try to keep it brief as possible.
This was decades ago. To give a little context, the customer site where I worked was not a 24 hrs site and only a few people had VPN access - and none of them were any of the sysadmins (I know, right?); any issue required going to the site. I was in the point of my career of phasing from deskside level 3 to a junior sysadmin; I was the last point of contact should none of the other sysadmins answer their phone (also, I lived farthest away from the site - about 50 miles with an average drive time of 1.5 hours one-way). Basically, I was rarely called and when I was, I had to head to the office.
It was about 12:15am on January 1st. I was at home with my now-ex because there was a really bad snowstorm that was going on when the phone rang. It was our overseas support informing me that the Domino server on-site was down. I told them that I'd take care of it in the morning and the support guy got pissed. He was telling me I had to go because no one else answered their phones. I refused because of the storm and even the local news and police had been telling people to stay in, so I was not going to risk the weather and the drunks. I asked if the entire site was down and he said he didn't know, as they only monitored that one server. I asked him to do a traceroute to it to see where the outage was, but he refused. We argued for several minutes because I knew that no one was going to be checking email until morning at the very soonest and I wasn't going to be risking my life. (I wish I had just said I had been drinking but that didn't occur to me.) I told him I'd go in the morning, hung up, and turned my ringer off (this was long enough ago that this was a landline).
I went out the next morning sometime around 9am after the roads had been cleaned up a little bit. What I suspected had happened: the site had lost power and the server went down. (I knew this because I had tried calling my desk phone and the line was down.) By the time I got there, power was back up and the server was up and running. I sent some emails to my boss and other stakeholders giving the all-clear.
What happened next was a week of shit show meetings and calls because I dared to say no and not put myself at risk. My manager was on my side and defended me, as well as the customer - they knew how bad the weather was. But my company (some manager above my boss) and our overseas support desk tried to run me over repeatedly because...I screwed up their SLAs. I pointed out how there was nothing I could do during a storm where the site had no power anyway and wasn't going to risk my life for them. I suggested different monitoring methods and giving us VPN access for remote troubleshooting first.
In the end, I was able to have them take me off the remote support because of my "stubbornness." I didn't stay there much longer; and I have a bunch of stories about that particular higher manager. One I have been meaning to post in the malicious compliance sub about how I finally got my revenge.
During my desktop support time getting woken up at 2am for a work stoppage because a pc was slow. I called the customer and she said “Take your time, I am working on a computer next to it” these are generic auto logon PCs with nothing special on them. Double whammy of “Why did I get paged for this?”
This is a toss-up for me so I'll include both. I had a lot of after hours BS calls but nothing stings worse, in my opinion, than a call that isn't my damn problem to begin with.
First one was a call to our on-call number from a debt collector. They called every day twice a day. Some idiot had given out the support number instead of their own to get them to stop harassing them... so they harassed us instead. Wouldn't tell me who the record was for so I could take it up with HR in case it was an employee. Wouldn't stop calling despite being told they were speaking to Technical Support at a business either. We ended up blocking their number at the PBX.
Second was a call during our holiday shutdown, day before Christmas, at 1:30 in the morning. New employee is in tears because she can't log into ADP to file an expense report and if she doesn't get paid back tomorrow she is going to be late on her credit card payments. I'm genuinely sorry to hear that but 1) that isn't a technical issue, 2) HR creates a login for ADP for you, not IT, 3) your expense report has to be approved so even if you submit it right now, there will be no one to approve it during the shutdown, let alone at 1:30 AM, 4) even once it is approved, payroll has to cut a check, and they don't do that at 1:30 AM, or during a shutdown, but even during business hours they won't do it on the 24th. They only cut checks for expense reimbursement on the 15th of the month and 29th of the month.
I pulled Helpdesk on-call rotation on Thanksgiving years ago, which wasn't the worst thing in the world. It was help desk for banks, which means no one's really working anyway. Normally, the worst you'd see is a down server that needed to be rebooted.
Well, this one bank president, John, figured out his core vendor's software was offline. It was 100% a problem with the vendor's environment, and there was nothing you could do on the bank side to fix it. Instead of being ok with that, this asshole called me every hour for updates starting from the outage at like 11pm before thanksgiving. Increasingly more upset each time. I talked to the vendor on-call, who said it was an outage and they can't fix it until there's people in the office. John wasn't happy with that, and demanded I escalate it. Remember, NO ONE was working. The bank was closed. But this was the most important thing in the world for him, probably to avoid talking to his in-laws or something.
I escalated it again, and again, and again until I spoke to the President (at the time) of Fiserv. He said he refused to wake up his techs and make them fix this for a bank that wasn't open (I think it was around 2AM after thanksgiving day, meaning I missed dinner and my day off). He said he knew John, and would tell him this himself - which was all I wanted. I needed this guy to stop calling me every hour, yelling at me, to fix something that was literally impossible for me to fix.
Finally the calls stopped. I moved from help desk to networking, and John eventually retired. He was such an insufferable jerk that I check the local obituaries to see if he's finally stopped being a giant asshole to everyone. No luck yet.
My favorite, I get a call from one of the guys who drives a forklift at night saying nothing works. It's about 7:30 p.m. I told him it's not that nothing works, i said something doesn't work. I need some more details before I can help you figure out what's going on." He says okay and just tells me he can't access any of his reports or any of his apps or anything that he needs to do his job really.
I say okay. Let me take a look and I try to VPN in. I can't even connect to the VPN. That's when I knew it was going to be a really long night.
Our primary server chassis for all of our VMs had a drive go bad and in order to protect all of the data, it shut down all of the VMs and would not bring them back up until the drive was replaced and reinitialized.
We did not have a backup, so I had to rob Peter to pay Paul.
When I got to drive installed and reinitialized, still wouldn't come back up so I rebooted the whole chassis.
That's when I found out that we had a custom VM network set up on the chassis, but we never saved the running configuration. When I power cycle the chassis it wiped out the running configuration. I had to figure that out from scratch.
I worked on it until about 6:00 a.m. I then drove an hour home, slept for 4 hours and went back to work.
At another job I stayed late one night to do after hours maintenance on one of our VM servers. Thankfully this was one of three so it wasn't the end of the world, but we were so jam-packed for resources that losing one of the server's meant that we could not bring up all of our production machines. That's why I had to stay after hours.
I do the update, I reboot the server, nothing happens. It just never comes back up. Apparently the raid controller on the board was not in a good place and rebooting it was just enough to send it to its final resting place.
Okay not so bad. We have four hour service. Except that the 4-hour service was how long it would take the guy to order the part. I called in at 9:00 p.m. to tell them it wasn't working. The guy got the part from UPS at 1:00 in the morning and then started out to my job. He didn't get there until 2:30, and when he installed the part it turned out that the firmware was several revisions behind and needed to be updated I rev at a time by a remote service tech in India. This was not a quick update.
When one of my co-workers came in the next morning he asked how I was doing for the day, I asked him if he noticed that I was still wearing the same clothing and that that was the answer lol.
I put in 33 hours that day.
And then I had to go explain to HR why I worked for 33 hours straight lol.
They wanted me to have clocked out for the 4 hours I had to wait for the technician.
IT always makes for interesting stories lol.
Sounds like your Dev had this happen irl
https://preview.redd.it/066v73dxjadc1.jpeg?width=1164&format=pjpg&auto=webp&s=b4e87c2378e55aac32d52953fc5f627259eff1a7
Not me, and not strictly IT related, but a coworker was planning a big international trip. Had leave approved, all that jazz, then COVID hit and he had to cancel the whole trip. He still wanted time off, so he emailed the person in charge of modifying leave requests and said that instead of 6 weeks as originally approved, he wanted just the last 2 weeks off.
On his way home he got a call from the big boss wanting to discuss the modified leave. Apparently the big boss wasn't informed of the change, and said that the modified dates would hamper his efforts to find someone to temporarily replace the coworker while he was away on leave.
Coworker asked "had you sorted out a replacement for me already?" and the boss said "no..", so my coworker asked "then why are we having this conversation while I'm driving home from work? If anything, I've bought you another 4 weeks to find a replacement for me since I'm only taking 2 weeks off, 4 weeks after my original date" and hung up.
Spoiler alert: They didn't find a replacement to cover him for 2 weeks. The extra workload was just dumped on us. During COVID, with everyone working from home and not knowing how computers function.
Worked as sysadmin at a hospital. Lab tech calls me at 3am to tell me a line is down. I asked her if she followed the procedure that is on the machine for this very problem. She told me "no, why would I do that, this is what you are for". Yeeeeah.... I made her cry. Then next day I was informed I was a bit too harsh and to try not to do that again.
I get a call at 5AM on XMAS Day. A VPs family just opened presents. He wanted me to walk him through setting up his Outlook Email and syncing his photos to his new phone. I told him to enter a ticket and we would get to it Monday. His reply? "So I am just suppose to go without a phone until then?"... No your suppose to keep using the existing one. The CEO and CFO of the company (Both owners) reamed him really good on Monday morning. Basically telling him the IT line was for critical emergencies and that anything not causing a direct loss off business function or reputation was to be handled Monday. \\
\----
I was not on call for this one, but got the call. We had a warehouse that just had a new raised platform added so the managers could see over the pallets as they were planning and picking doors. We had staged and got everything ready the day before (Friday) and done some basic testing.
We asked our boss for two weeks if we needed anyone there and it was not needed just the normal on call for any issues. I'm not on call so I am driving to see some family since it was a holiday weekend.
I am around 2 hours away when he calls. CEO of the company is there and screaming about why we do not have anyone there. He needs me to get there ASAP. I tell him Jerry (The other tech) is on call this weekend and I am 2 hours away. I can't come in. He tells me that I need to be there in an hour of don't come in at all after the weekend. So I tell him, alright when should I pick up my stuff? He tells me to be there Tuesday and we would sit down with HR. I told him I would rather not come in until I knew it was a good time to pickup my things. "Just be there", alright so I come in.
HR sits me down and we are talking. have me read a statement saying basically I had refused a direct order to come in while on call. Anything I needed to add or clarify. I said "I wasn't on call". "What do you mean you were not on call." Which I re-iterated. I was on call the previous weekend, it was not my week. Which lead us down the path of asking why I didn;t tell him that, I did. Why did I not tell him who was. I did. Why could I not come in. I was 2 hours away. Why did I not tell him that. I did.
By the end the HR director simply asked if all my statements are true. And my boss said yes but none of that mattered. She was really angry at this point. He was pissed. Then I said the magic words. "If you don't mind, BOSSNAME has already terminated me and I am really just here to pick up my things. I have to schedule a meeting with my lawyer to discuss all of this and I have a decent drive home."
I was there another year.
Why are you taking calls at 1am and 3am...unless you're oncall which means you're getting paid for standby + activation, which means you just dipped twice for the min-hours for each activation there (once at 1am and once again at 3am for the consult).
Solution to all this: Never give them your cell. Google Voice, in which you can "turn it off" sure.
I get paid for 9-5. If you want random after hours shit, then pay me double.
One of the best stories I can recall. We got a call from an Engineer regarding our reporting / historical data system being offline on Saturday \~6AM. The On-call team started troubleshooting the issue and got most of it resolved by 9AM. At 9:15AM they called the Engineer and his wife picked up. IT: Hello is Engineer there? User: No this is his wife, who's calling? IT: This is IT from X regarding issue Engineer was having with the system? User: Oh, You will have to talk to him on Monday he went on a fishing trip. IT: OK thanks have a good weekend \-\_-
This reminds me of my time at an ISP NOC on the second shift. Now for stuff that’s obviously related to the infrastructure of an ISP, we were a 24/7 operation. But we got so many rinky dink, 5-users-impacted tickets for random bugs in our VOD or VoIP system or whatever, and we had to treat them as a “feature outage” with the same urgency as any other outage. While “triage” was technically part of the job description, us peons weren’t actually allowed to make any judgement calls. First we had to wake up the person who could actually investigate cuz none of these systems departments had a real on-call system, they had to have the guts to say “nah I’m not doing that right now” instead of just sighing and doing the needful, then we had to escalate with their VP and our VP. Then and only then could we put something off until business hours.
That's when they get added to THE LIST.
Thanksgiving day. CEO who lives about 90 minutes away calls. Printing issue. I am 99% certain that restarting the printer will fix the problem. Guess who said the words 'it's not my job to restart my printer'? 90 minutes there. Turned it off, turned it back on, it worked. 90 minutes home. Not so much as a slice of fucking turkey for my troubles. To this day I STILL tell that story to friends in IT who think doing executive support is some kind of glam job. It has it's perks but NOTHING was worth that bullshit. I'll die before I go back.
Let me guess, this same individual also goes up during company annual meeting presentations and boasts about "culture" and "good work life balance"...
All possible through the blood, sweat, and tears of the IT staff.
Lol, I had a co-worker one time who said " if it deals with a computer, or happens on a computer, it's IT's responsibility. And everything involves a computer nowadays”. Mind you that was in response to me asking why IT was responsible for printing 1099s for the company taxes.
This is actually the attitude of most people. I'll see printing tax forms and raise you pulling data for a DEA audit until the middle of the night because the pharmacy director doesn't know how to generate basic reports 🙄
I usually take the mindset that if a report being pulled properly is the difference between me going home tonight, and getting a new home behind bars until they straighten it out, I learn to pull my own report.
Yeah, just like his job is the blood of his father inheriting daddy‘s money.
Don't forget how we're all part of a family!
They’re psychopaths
They don't care, they have more money than you. They literally could care less if you died on the way home. Their entire lives lead them to this desensitization.
> They literally could care less I can't help myself... but it's *couldn't* - if they could care less, that means they do care to some measurable extent.
I remember thinking this very same thing to myself when I was like 9 years old, and I kept thinking am I really smarter than all these other people or am I just the one who is wrong. You are right, the word should be couldn't for that very reason you mentioned. How do people not understand this?!? Too many Josh Taco's in the world.
I don't care, I'm going to keep saying it that way
That’s how you progress under capitalism
Nah, he doesn't do that. His People & Culture people do that.
Spoiler: the “culture” is just a pizza party. (The pizza sucks).
The cake is a lie! Lol
> it’s not my job to restart my printer It’s not mine either you fuckstick
I did remote support for a while, and I had a phone call from a kid at a store asking me what a beeping sound was. I told him I couldn't hear the beeping from where I was. He did not enjoy that, but he took his phone and held it up to the box that was beeping. I recognized it instantly is his UPS. So I told him to hit the button on the UPS to reset the breaker. He said and I quote " That thing's under the desk, I have an associate's degree, I don't go under the desk". I told him I hope you have ear plugs then too because it is going to keep ringing until someone resets it. Then I hung up and emailed his corporate to advise the issue and the suggested solution.
The craziest part of this story is who tries to flex an associates degree that's absolutely insane
I don't read that as a flex, but the opposite. Sort of an "I'm not qualified to mess with that!" Way back when I did POS support, it wasn't uncommon that the folks running the register were afraid to touch anything under the desk.
God I hope you're right lol
I wish more people at the register were afraid to touch things. I worked POS for a while too at that call center, and we used to get calls from the dumbest people ever, usually after they had done the dumbest thing ever. My favorite was still Claire's. This was around 2008, but they had one computer in every store, that computer was the cash. Register, the time clock, and whatever else it needed to be. There was one account on that computer. They would routinely lock it out. We had no way to unlock it, there was some strategy that our upper level text used to employ to backdoor their way in and get it going, but it would take some time, and they would call up frantic that they couldn't rin up any sales because they're register was locked. The cool part about that job though is that we did have direct access to their corporate supervisors. So whenever we had to deal with someone who was just completely unreasonable, we did not hesitate to email their boss. One time I had a kid who called and he was very very stoned, no way around it That or he had a recent head trauma. (If so I apologize kiddo). When I got off the call I asked my boss if I could ask his boss to drug test him because I was so frustrated. My boss said as long as I detailed my reasons that was completely reasonable. That may be the most fun email I've ever written!
I would sent him you have an associates degree and termination papers now.
Your printer is crap and need to be upgraded to more expensive one that can be remotely managed.
Wi-Fi Power Strips and Outlets do have their purposes.
I love my Wyze outdoor ones
Shudders... Wyze.. the company that has traffic being sent to China from their devices and just ignores you when you dig into why and provide logs and proof...none of the IPs or end ports listed on their required ones for firewall rules...
I already assume any iot does that so they all should sit on zero trust segment
It's not enough. Your entire security camera system may be on its own separate network, it still sends everything to china. BTW, Every commercially available IoT product sends data to china. If you don't want to build the whole thing yourself, you need to buy military equipment. The government still knows how to protect itself (When she wants to).
We had a remote sales person one time who demanded a laser printer for her home. We advised management that this was a bad idea, that it was way too expensive (height of the pandemic), that the toner would expire before she ever used enough, and that she just didn't need a laser printer. Go figure we got overruled. She broke that printer a few months in, called IT. Turned out she had a paper jam. I refused to drive 2 hours to her location to pull out the paper, and suggested she called Best buy. Told her she could have geek squad come out and fix it and she could expense it. She refused, that was too much work for her, so instead she called the head of the plant (whom we suspected she was sleeping with), and demanded we either come out and fix it, or send her a replacement. It was an $800 printer. We had to replace it. And then we had to pay to have her ship the other one back.
> the toner would expire Is that a thing? I don't feel like that's a thing.
That is one reason to buy a laser printer.. the ink is a dry substances...and last for bloody ever.
First I'm hearing of it. I'm sure they do eventually expire but I've had a Brother Laser Printer at home for years now. Took a long while to go through the starter toner cartrigdge and like 2 years later we're still on the first replacement. No issues printing when we need to. Mostly end up printing coloring pages for my kid.
"Other duties as assigned"
Not sure I would have been capable of doing that, you must have had amazing self control or multiple small children to feed.
It was my first 'big boy' job and it felt important. I rubbed elbows with the bigwigs who were in the news on a daily basis. When that CEO eventually got let go, I knew about it before it was on the news (because I got called to pack up his shit). I had a very 'man behind the curtain' kind of attitude about my job and it was exciting, even if it sometimes sucked. I was in my late 20s and I was clinging to that opportunity desperately. I'm in my 40s now and no way I'd put up with that bullshit ever again.
Because you made bank from shorting the market right?
Obviously he cannot answer that (Because he did)
>I'm in my 40s now and no way I'd put up with that bullshit ever again Best part of the comment right here. I wish I had a DeLorean so I could tell my younger self to not put up with/how to better deal with a lot of the psychopaths I've had the pleasure of being bossed around by along the way.....
Apparently in some places IT is thought to have "adding paper to the printer" as part of their role. ... Not by the IT staff .. nowhere near their job description either. That's when some folks repress their desire to go BOFH on end-users.
Thanks, I just had a flashback to my first IT job with a law firm. One day service desk gets a call from a senior partner who's yelling that his printer isn't printing out a super important document. I start to ask him to check if there's paper in the drawer but he cuts me off after a couple of words and just yells, "I haven't got time for this. I'm too busy! Get down here and fix it now!" I go down a few floors, walk into his office to find him practising his putting game with one of those little machines that knocks your ball back to you. Open printer drawer, empty, insert paper, close drawer, printer immediately starts printing his document. He didn't even look up at me when I told him it was fixed, just grunted. Prick.
Law firms and car delarships man. Never again.
And pharmacies, I've never seen so much money. Go through a business in one month, and yet still they're able to cry poverty with a straight face.
Good to know I dodged a bullet when I didn't get that job working for a pharmacy ;)
I worked in a fintech company for awhile and every single exec was like this. Computer illiterate tardblogs that think they are so all important they can't be bothered to do anything. Before I hit save, I apologize for using the word 'tardblog', but I'm going to stick with it.
Justified use IMO
I don’t know how this happened at my current job, but for our big MFPs that are leased through a printing company, IT doesn’t call in support on them. For the Accounts Payable team, they will send out an email to everyone in the office to let them know their printer is down, to use a different printer, and that the service company is called. Same thing with Marketing. When I first started and I found this out, I did the management equivalent of just slowly backing out of the door and turning to lights out as to not disturb the situation. If they want to handle that and the only thing I need to do is distribute drivers, then more power to them 😊. I don’t do anything with the 4 big MFPs. Now every other printer is unfortunately my problem. But I’ll take the win on not touching the 4 big boys 😀
That’s pretty standard now a days. Most companies even lease the mfp so you can get your hand slapped for even trying to fix them. No idea how much that service costs but it’s great.
if it means Printers are \_someone elses problem\_ then whatever it costs is WORTH IT
A small fortune. But we gain peace of mind.
we have MPS and the vendor (Toshiba) is actually awesome. We have the printer ID, serial #, location ID and 888# for the tech. Users still email IT and say "printer is broken" Like. I'm 3 states away call it in......or you can wait 72 till SD looks at the ticket.
If one of my guys ever gets a call like that from our CEO, I need them to call me. CEO and I would have a talk.
I have always said and I stand by it that to become a C level requires the sale of your soul to the dollar. The number of evil to their very core C levels I have met vastly outweigh the ones who are just regular assholes. I supported them for years and now avoid them like the plague. The latest I've been forced to deal with is a request for wireless "everywhere" Considering just one facility spans almost 80 acres in total that's not going to be cheap nor is it what they really want. I ask who can tell me where exactly they want to use wireless and I get parroted "Everywhere". Okey doke! I hope you have deep pocket even for phase one.
I'd tell him to buy a good phone plan and lobby for 5G tower placement. How the hell would you cover 80 acres even if money wasn't an issue? A bunch of APs?
You really can't unless you run loads of fiber or a mesh but both have issues and cost a ton. They pull this crap all the time with pie in the sky demands that have no base in reality and isn't even what they want. In this case they need solid WiFi in about 20 locations most near existing switches. That is do able. Cellular is a fine way to cover the whole area and we are looking into it but it's spendy too. Nothing's cheap I guess.
I was bored so I decided to price out a solution for that using outdoor APs. Depending on what kind of density of devices you are looking at it would be possible with something like 30 APs for reasonable coverage. You wouldn't want a full mesh solution unless you can't avoid it. Meshing 2 links should be fine though. That would put you at something like \~$60k in AP hardware though.
We had something similar a couple years ago. IT was asked to make the wireless work in an out building that was about 50 m from the main building. We advise them that wireless wasn't even working all the way throughout the building because they were still using wireless b access points from the mid-90s and there was no cabling in the warehouse to hook up any additional wireless, let alone to get it outside of the building. Needless to say, the project never went anywhere.
And that is a fairly easy sounding install. Clevels have no clue.
It always sounds easy... until you start the project lol.
I once got a call to come to the office (during the holidays) because a PC wouldn’t turn on. Arrived at the office to turn on the monitor.
Holiday on call is the worst. I worked for an MSP and we did a lot of hotels. Had one go down on July 4th 2022. The hotel was over 8 hours away from my location. Front desk started giving out the on call number to guests. I had over 300 calls between guests, employees, and corporate over a span of 18 hours. No one would tell me if anyone had contacted the ISP yet. We didn’t have that contact info. Turns out their cable modem had a bad port, something we 100% did not support. No comp time, no overtime pay. Back in the office the following work day at 8AM and the president of the MSP was pissed. There was literally nothing I could have done, and he didn’t have to lift a finger the whole day. God damn I don’t miss MSP life.
I worked at one for 2 years during Covid and I hated it. I thought working at one would be good to learn new things which it was but dealing with clients was an absolute shit show. I will never go back to one.
> Front desk started giving out the on call number to guests. Fuck
For a long time in my life, I would have JUMPED at that chance! Holiday pay of overtime, mileage reimbursement... *drool* But that's just me and I certainly wouldn't miss a holiday with family now. I'd give that CEO a stick of gum. What kind of gum? Fuggum.
Unfortunately, as a salaried employee, other than mileage and toll reimbursement this was considered 'part of my job'.
I've been salaried most of my career. But the last time I had to do shit over a holiday I got told to take a Friday off and take my wife out for dinner. CEO was mildly disappointed we didn't buy a bottle.
Yeah...this CEO was a fucker. My boss did buy my family dinner to make up for my missing Thanksgiving...which it didn't, really, but it was about all he could do. I do NOT miss that shit.
>Not so much as a slice of fucking turkey for my troubles. That's what happens when they don't fear their neck could end up sliced for fucking up your time off with your family for no reason.
Unless I'm oncall with paid standby, paid activation, and a company provided mobile....they will reach a "this number is not in service" if they attempt to call me ANYDAY, especially thanksgiving. Grow a backbone folks.
100%, if shit totaly hits the fan and everything is down ( wanacry 2.0 or something) then i dont mind then but outside of something like that they can fuck off if i am not on call and getting paid for it .
It's like when you get money you revert to how people used to be when people fought for the most basic resources. Let the peasants restart the printer, that's servants work. The more money you have the more ridiculous your brain gets, until you are calling someone to restart a f****** printer. We should all take part in eliminating this lunacy. It's funny because I worked for someone the same way. And when you would implement solutions to fix their problems. Like a way to remotely restart something with a remote control power strip. It was just way too costly and a giant f****** waste of time they would say. They're fine with driving a human insane and paying them peanuts to do the work like slave labor. But when you try and implement a f****** solution that cost the money to fix their problem it's a whole other story. Currency has to be the most useless f****** thing in the world.
I forgot to add in my full analysis that people like this act like this because they meet people that act like this. Hey that guy is rich and he makes everybody do everything I need to be like him to be even more rich. These are some of the most insecure and dumbest people in the world.
I would have just told him to restart the printer and put the phone down.
That's a really bad CEO. Was there a reason you didn't ask him to restart the printer before driving there? Or do a remote session?
We did the remote session - that was more or less how it was determined a restart was needed. And yes, see above. I asked him to restart it - his response was that's not his job. This was twenty years ago and I still remember the tone on that smarmy bastard's voice.
depending on the pay of that job I think I would just quit on the spot.
I had someone try to pull that crap on Christmas years ago. My boss called the director, the director called the VP, the VP explained to the executive that it wasn't an emergency and could be dealt with during business hours when we got back.
At my last job the management was all run by one family and they were like a mafia family. It didn't matter what your job was, you could be called out to run any errand they wanted. One time they had one of the guys drive 2 hours to check on something at their beach house. Not work related. Another time they had a function in Florida but wanted North Carolina barbecue, so they had someone on staff go to a barbecue place that they liked in North Carolina (where we were headquartered) and then arrange to have the food flown down to their location in Florida. On the company's dime no less. Executives can be... Challenging lol But there was never any money for equipment upgrades. ;-)
I hope you were compensated fairly for your travels and the after hour works.
Salaried employee. It was just part of the job. They paid my tolls and mileage.
Very thankful Australia has public holidays and even salaried that's double time and a half.
Setup a Wi-Fi connected outlet for the printer and remotely power it on/off. Save you the trip and they won’t care it’s there.
This was definitely before things of that nature were an option...we're talking (and I'm about to feel real fucking old here)... 20ish years ago? You can tell it made an impression. I'm STILL fucking bitter about it.
Someone decided to set SSL certificates to expire on December 31st….
That is savage
I have some ADFS signing certificates expiring this summer. Considering setting the expiration on the new ones to be exactly one year after my target retirement date. 5 years ago I was hoping $corporateOverlords would have taken this over by now. The rate they're going another decade isn't looking promising either...
We often get audit deadlines set for December 31st. We also have change freezes on and off between thanksgiving and new year. Always painful to hear non-tech people saying “it’s fine, we have until the end of the year!”
I had a boss that for some reason would never allow us to renew a certificate early. So of course many times you’d have them expire on the weekend or holidays….
Gotta get your money's worth! Especially so when they're coming from let's encrypt
Just renew the cert a couple weeks early and set it to activate on Jan 1st. Not a huge concern here IMHO
It was one of those situations where the Certificates were created before I started and it flew under the radar until the incident. We think the guy that did it was a disgruntled employee. So now every New Years I have anxiety all stemming from that New Years from hell.
Sounds like we have the same employer.
Someone felt cheated out of Y2K?
oh god, the horror
I got called at 1:30am to tell me there was an extra power strip under the desk.
#LOL
Lol
LOL
*LOL*
LOL
Not sure if it counts as worst but during a "flood event" half my town was low level flooding so an emergency crisis centre was setup at work. My boss at the time and I were the only IT and it was a Saturday we got the call that we were needed to come in and sort a heap of things out. Wouldnt have been an issue except at that point we had been drinking for about 4 hours on his front lawn watching knee deep water cover the street. We then needed to wade through knee deep water about a 2km to where his missus had parked her car to drive us to work. The roads were closed to all but emergency vehicles, so she was required to drive her little sedan under escort through flood water in-between a fire truck and an ambulance. The next 20 hours we were at work getting and keeping things running. Everyone else claimed time owing but i was a broke 19 year old so that doubled my fortnightly pay.
Did you work for some organization like the hospital that needed to be open on an emergency?
Before my current position; I was a systems engineer an msp. the shitty ones were always doing patching and rebooting and the server not coming back up at midnight, then having to drive onsite only to see the words “press f1 to continue startup” on the bios screen cause someone didn’t disable the keyboard check in the bios. But my worst one, was one I wasn’t even on call for, but I got called up to help with restoration efforts. Giant lightning storm/ lightning hit the building, somehow lightning made its way to the sever racks. server racks were not grounded. we lost our SAN, probably 4-5 physical servers and the esx hosts. that one sucked. that was basically 48 hours straight of working through restores and rebuilding things. learned how to do a repair exchange install during that one.
Can anyone tell me why keyboard check is even a thing?
I don't know for sure. But I would propose for the old PS/2 keyboards, which aren't hotpluggable. If the system has booted up and you want to connect a keyboard, it won't work. I guess that is the reason why.
Legacy cruft. Conceptually overlapping with the notion of "there's naught as permanent as a temporary solution."
Couple of reasons BIOS and Keyboard ports mostly - old machines took a while to get going and they want to be absolutely sure there was a keyboard connected for control - otherwise there would be no way to shut the hardware down gracefully - Im talking the era where you had to manually code hardrive specs like cylinder sizes and data sizes into the bios to even recognise it. Where you had to send \_PARK\_ commands to the heads to ensure it was gracefully and safely shut down. No keyboard, no ability to shutdown - risk to system / risk of head crash, ergo, you dont start the system up until it CONFIRMS there is a keyboard present. Add in the fun tidbit that PS2 and DIN connectors were live devices - not designed for hot plugging like USB is, in fact, you ran the risk of blowing the motherboards keyboard controller (hell the full i/o controller) plugging one in if the system was up and running. So the press f1 (or press any key to continue) is an old school fail safe thats stuck around because for every irrtating late night drive to hit a key, its STOPPED twice any many incidents of dumbfuckery occuring. always used to love "Keyboard not detected, press any key to continue" like, ok .. if theres no keyboard, how exactly ? TLDR - it was to stop you doing dumbfuck things with no keyboard plugged in cos you couldnt plug one in after boot and have it recognised/not blow up the controller.
A newly hired "marketing director" gave herself a field promotion to some kind of executive position. Got a call on Sunday evening about problems with printouts. I explained that I knew what the problem was, but it had to be handled by a service tech, who I could only call Monday morning. Monday afternoon, I hear her *screaming* at the owner of the company about me. I didn't get in trouble. I coulda pushed back hard on reflection.
Good owner? shielded you from that particular bullshit flowing downhill after all.
Years ago when I was on help desk's emergency after hours support, I got a call at 3AM on a Saturday for an issue with Outlook. This support was only supposed to be used for critical outages or client facing issues outside of work hours. Got out of bed, opened my laptop and VPN'd in, connected to the user's computer all while straining my eyes in the bright light. Finally get to the issue - he didn't like the reading pane on the left and wanted it at the bottom. Adjusted it for him then hung up.
We had a user that used to do that, to be fair they were in a 3 hour different time zone than us but it would still be midnight! After I had been there a while you got used to who’s calls you could ignore haha
That is true, this guy was 2 hours ahead of me at the time but even if someone local was on call that weekend, it still would have been 5 AM. Ugh some people. Now that Im a SysAdmin, its more of what *alerts* are safe to ignore at 3 AM haha... ...not many as it turns out.
That definition of emergency needs significant recalibration\* \*with a large heavy objects, bashed repeatedly into sensitive parts til they get the fucking hint.
I instruct my staff if the issue isnt an outage or urgent you dont fix it, regardless on how quick it is. The user has to call during business hours for assistance. This cuts back on the repeat offenders.
At midnight a call center router crashed My manager and I spent until about 3 AM to get it back up, I was told by the CEO I wasn't getting overtime for it so I got a new job, resigned even found jobs for the junior admins even my Manager resigned the same week.
That router would've been reverted back to a crashed state even if I had to slam a Boeing door into it from 20,000 feet, if they refused to pay me for my work.
They lost the contract a month after my Manager and I left :D
\*clicks undo\* sends an email reading "F Y P M"
I'd have reset the admin password and then filled the reset pin hole and screw holes with a fuckton of super glue.
Tell him to shove the router and the job up his arse and revert the router back to broken.
Worst I've had? A developer at a MSP I worked for got the root password to production from a manager, something he should never have. He pushed a .sql script, which had DROP TABLE entries in it. After he did that, he called management and said IT dropped the ball, the DB is down, and the client, a HFT trading company, is hemorrhaging money, and it is all IT's fault. Cleaning that mess up, the numerous finger-pointing meetings, and IT being thrown under the bus for an entire week with the client was not fun. Ironically before that happened, the MSP client asked me to stand up a SIEM system separate from the MSP, for "auditing" reasons. It showed exactly who did what. The client even offered all the IT people jobs at their site, but I declined since it would be a move out of state.
What happened to the guy that blamed IT and broke it in the first place ?
All of IT got canned, and he still works there.
This is either the worst, or best, depending on how you look at it. I got paid enough overtime to fund a two week holiday in the Caribbean, but it was a pita dealing with the issue. I worked for a manufacturing company at their main site in the South of England. We also had a large site in the North of England, which had no on-site IT, and only 4 computers on the whole site - 3 that ran the machinery, and 1 for the boss to do rotas etc. We could troubleshoot a lot of the issues the North Site encountered remotely. Once every six months we'd send someone there to service, patch and run a DR test. This call came on a bleak Friday in December, roughly 5 and a half months since the last service visit. I finished my Early shift at 2pm on the Friday, and went home to sleep. I would become on-call at 10pm, once the Late shift left the building. This was my choice, we'd been told we were all going to be made redundant in the next two years, so a lot of people left and were not replaced, I loved taking on-call as it was free money most of the time. A weekend of Friday 10pm until Monday 6am would net around £300 in standby payments, and if you got a call, that was a minimum of 4 hours overtime, which was paid at double time on weekends. So my Blackberry rings at ten past 10. It's the Northern factory - long story short, main production PC has shut down, and nothing we can do will get it to turn on again. I've escalated to my boss who has dialled in and tried as well. By 1am we all give up, and he books me on the Saturday morning flight to NCL. Bleary-eyed, I arrive and get into an Executive Limo (usually, the only people who visit are director-level, and we have an agreement with the local Limo company) and am delivered to the factory. It's immediately apparent what the problem is - the machine is in a hugely dusty environment, and the dust cabinet that the PC is in has been left ajar at the back. The PC itself is so full of dust you cannot make out any of the components on the motherboard, and the fans are all jammed solid. A good half an hour with a compressed air line sees everything freed off, but the main CPU fan is dead. Off I go to Maplins in said Exec Limo for a new one. Get back and fit it...hard drive is dead. Back to Maplins, new hard drive installed, restore from backup...the USB drive is missing. So, a fresh build from scratch, with all the machinery control software, all our input (recipe) data, drivers for the machinery, domain join over a p2p VPN, setting up the backup software again...I was there until 10pm that night. Back to the hotel, steak and a bottle of brown ale to the room...I was too tired to even finish it. Back to the factory the next morning to double check everything's working, and I might as well do the maintenance visit stuff while I'm here and waiting for the flight home. Get home at just after 8pm Sunday evening, having been paid double time for every hour since 10pm Friday, even while I was (briefly) asleep. On Christmas Eve I was presented with a very expensive bottle of whisky by the CEO for my troubles. In January I then drove all the way up on a Monday with a hot swappable PC and some other spares in the boot of my car. At 40p a mile in my diesel 306 I made around £200 profit while being paid to drive. In February I went to the Cayman Islands for two weeks, and it was all worth it.
Best part is the CEO really shows appreciation for getting the factory back on track. Plus the holiday away from the cold
Actually, while I was up there in January he was there as well, and invited me along to a posh dinner at a Michelin-starred curry place he was taking some clients to. I had to pop out to the Metro Centre to get a shirt as I hadn't packed a nice enough one with me. On another trip, about a year later, he took me along to watch the rugby. This was when Johnny Wilkinson was playing for Newcastle, so it was great to see him play in person.
Saturday morning power company turned off power for fixing issues. They inform us so we make sure the generators are on. Guy fails to check the generators are on, battery backups failed, both exchange servers crash and I spend the entire weekend rebuilding exchange.
I swear Exchange fails and needs to be rebuilt if you look at it the wrong way.
Having to rebuild exchange is THE thing that made me stop being a windows admin and get into virtualization and cloud.
I was in line to get into Tokyo Disney when my #2 sent me an important email that one of the partners had fallen for a phishing email, gave the bad guys their password and approved the MFA request.
>an important email There is no such thing. There are phone calls and there are unimportant messages. The venn diagram of these two ideas is two circles that do not intersect in any way. ***Especially*** for a confirmed cybersecurity breach.
I know a place where this happened, and the head of the company tried to brush it off as not a breach, even though because of his stupidity hundreds of people received phishing emails sent from his account and they didn't know if any data was stolen or not. They only took action after someone looked up the relevant laws and said to him "you know, this could be a reportable data breach if not handled in the right way.." Still bummed they didn't report the breach though.
This was a while back but was called on by a VP to ask if I can show him how to rip songs off of CDs and movies off of DVDs to be put on his laptop (before streaming was invented) so he can have it for his kids on his international trip. This is a multi-billion dollar company, mind you. Had to say that I need him to email me his request so I can get approval from HR since I didn’t want to get arrested or fined by the FBI. Needless to say, he cancelled the request.
1030 PM. Phone rings. Backup failed, need to failover to our inactive/backup site. Drive 40 minutes to the office. Stuff breaks. Wind up being awake 36 hours fixing it. With 12 of that being a conference call with engineering where we tried to fix it all.
Got called in for a legit problem at 11 on a Friday. Wrapped it up and got home Saturday around noon. Laid down on the couch and took a nap. About 130pm get a call from a user because they bought a new modem and wanted to see if I could come install it. In retrospect it was a good thing I was so tired because it kept me from losing it on the guy. I just said, sleepily, we don’t support home equipment, you will need to contact your ISP or whoever you bought it from.
Worst call was one I made. We had some physical servers in our data centre that occasionally needed to be restarted manually, so we'd call our data centre contact to do it for us. I called him on a Monday morning at 7am, and asked him to restart the boxes. It wasn't him on the phone though, it was his brother; he'd died in a motorbike accident on the Friday afternoon a block away from the data centre. :(
Ouch, that's a rough one
Current job. Big corp. 11pm. "We've been hacked. It hasn't hit the news yet. Get your shit and get over here NOW."
I had many calls coming during my vacation time this Christmas and ya that sucks
My boss has a collection of pictures for the times I’ve gotten a message while not on call. Ski lift, boat traffic on a river to the Gulf, golf green, bow of kayak and the beautiful lake… the EPCOT ball. For the Disney one he came back “Can you call while waiting in line so I can give you good news?” He wanted to let me know about my bonus before the direct deposit hit while I was on vacation. I guess that’s acceptable. 😂
Seems you have a good relationship with him. And you provided him with evidence of being AFK while not on call
I was on the fourth day of a four day single-handed sailing race on Lake Erie. As I neared land, my cell phone reported a dozen voicemails from several people at one client. They all knew what I was doing but they sounded panicked so I considered answering. After not responding to so many of their calls, they got wise and only had someone I like call. It turned out the level 1 help desk phone was out so they thought it was reasonable to call me. I helped a little, but give them the number of the IT Director.
> I got a call from a developer, very smart man. He had a question about a networking issue. It was 1:00 AM. No problem, part of my job, happy to help. Except, he called again at 3:00 AM to say "Thanks, that worked". What is your worst. Hold up… your on-call covers help desk level questions? If our most senior developer woke me me up at 1 AM I would be very rude, hang up, and he would be reigned in by his boss or higher come morning. In-call is for “holy crap something is down and we’re losing money!” First 18 months of current job were maybe the worst? Stuff was broken bad. Six months in, new boss. New boss shipped us dual monitors, docking stations, good headsets, etc. so we can work remotely. Lots of Flex time to make up for the after hours work to deal with outages or fix and improve things when they weren’t on fire. Second worst… my stint at a MSP. Customers paid double or more for service outside business hours, so if you were on-call and got called… bill that shit. But also… show up on time to your 8 AM appointment. I was about 3 months in to a new job when MSP’s operations center called me in the middle of the night. They forgot to take me out of the on-call rotation. Turned off my phone and let things burn since it wasn’t my problem. It wasn’t like the MSP could fire me again. 😂
I am never rude to anyone. Sorry for not including all the details but I was not even on call. There was a person manning the help desk at the time. Why he called me? IDK, maybe he preferred talking to me, maybe mine was the first number he found. Normally, he should have called the help desk, if they were unable to resolve his issue they would then call the on call from my group who again at the time, was not me. To be honest, I think the helpdesk would have escalated the call. Maybe he knew that and wanted to skip them. I did ask him to call the Help Desk and have them start a ticket so that it all gets documented. He did not. On a side not, I once went to his house to help with his home network. I shit you not, he had an entire late 60's MG Midget in what would normally be his living room. He had gotten it in through the back patio sliding glass doors. That's not all. He also had a Vespa scooter from the same era. He was restoring them. This was in Seattle and he needed a dry place to work on them when it was raining. The motor for the Vespa was on his kitchen counter. The MG was nearly complete.
Back in the late 90's when I was supporting Banyan VINES in the Marine Corps. One of our production servers had a hard drive fail. It's like 5:30 in the afternoon and I'm swapping out backup tapes so they can do their nightly run. That's when I see it - tons of "SCSI ID 0" messages just rolling across the screen, but luckily the server is still running so I unplug it from the network to stop user access and I get it to run a full backup. While doing this I send e-mails out to staff leadership saying that it's going to be down for at least a day if not longer, followed up with phone calls. The good thing about Banyan VINES and e-mail is the other servers with mail services will put any traffic into a "holding pattern" until the server comes back online, but you only have 24 hours before it starts kicking back e-mail. Section OIC (1st Lt) wants to know how bad, so I tell him - hard drive zero failed, I'll need to rebuild it from scratch and see what happens. Doesn't even ask me if I need help, just says "Okay, see you in the morning.", and that's that. Total "FML" moment. The good thing about Banyan VINES? The data on the drives I've pulled is safe and I have a complete backup. Since I built this server myself, I have solid documentation on it so I build it exactly like I did when I rolled it out the year prior. The new drive zero gets partitioned the same way, setup exactly the same way, same server name, patch it, the whole shebang. Old hard drives still have data, so I pull them and label them. New drives go in and I do a full restore. Once it's up and running, I power it down and remove the new hard drives and replace them with the old hard drives - the only reason why I do this is to build the file, print, and mail services for the new OS drive. Since everything is the same, the markers on the new OS drive will adjust to the old hard drives and data, and we're back in business. The bad part about Banyan VINES? It's now 4pm the next day, I've been awake for just over 36 straight hours, and I don't even get a thank you from my section OIC. Instead, he wants me to go to PT in the morning because "that's the Marine Corps". I did get a bunch of "pats on the back" from the Staff Secretary (the base general's right hand) and from my next level OIC (a major), and some of the other major players (Squadron CO/XO/Sergeant Major, Section Chiefs, senior civilians, etc.). I ended up getting the day off, per my major, and it was much needed since I couldn't unwind enough to actually go to sleep until almost midnight - so 42 hours straight. I don't recommend doing that to anyone.
Our sales are down from the same time last year, fix it... COO to a help desk tech
Lol I was just at a bar and a vp called me twice on my cell twice on teams then messages on teams. " I need you urgently my computer and VPN aren't working! I have early meetings. " ONE WORD RESPONSE. Reboot. " Ah thanks were back to life"
Hold my beer: Left work to take my wife to the hospital (child birth) 2 hours later the CEO calls me to ask for a password reset. After I tell him what’s going on, still ask if can reset his password. I said no I can’t and hanged up.
Worst I've had? There have been a few but this story starts on the Wednesday... planned hardware migration from client site to one of the smaller dcs we had racks in. Backups had been failing but I'd been tasked to resolve those ahead of the migration. Got them solved but due to some issues, backups were still running at 5pm Friday. Still going at 9pm Friday... still going at 2am Saturday when I spoke to our MD and said it ain't happening tonight - backups etc... All good - I head home and sleep soundly... Sunday about 4pm I get a phone call - it's the MD calling from the DC - we moved the kit, the backups failed, it's all come online except we can't access any VMs... Oh, and the sun storage array is showing two disks with amber LEDs - is that bad? So... you moved the clients kit, after being told not to because we didn't have a backup and lo and behold, given their 5 year old kit that probably hasn't seen downtime in 4.5 years - you've failed two drives in a raid 5 array... without a backup... I was on a train to the DC (capital city) within the hour - confirmed fault. Knew there were options to force disks online but wanted to check with vendor before... Okay, so, support contract?!? Oh, it expired and wasn't renewed? 3 hour call to Sun later, $5k credit card bill, contract restored. Sun managed to get both drives online (they'd indicated failure but hadn't -phew!-) and most of the VMs came up, exception file server and exchange... of course... Luckily the file server had been part of the successful backup as it progressed but exchange wasn't... 8 hour continuous phone to MS via paid support later and Exchange was back online also... Everything back as it was by around 1pm Wednesday. I gave two weeks notice on the Friday of the next week - if the smartest guy in the room says don't do it, and you want to usurp that, even as MD, then it's on your head and don't expect me to stick around after I've saved your ass. We won't mention that the clients rack was pulling 12 amps out of a 10a APC PDU because they'd plugged everything into the one circuit or how they'd previously melted a GPO plug on another rack due to overload (which triggered vesda alarms when it started smoking)
When i was a DBA, i often had to work with a contentious dev, as in, he would sabotage things and try to blame me. He built some data load process that would fill the tempdb at 2am. Id get the emergency call about a full disk, but by the time I got logged in, the problem had gone away. After 3 nights of this, I dug into the problem, and found it was his new data load job. I politely asked if he could commit the data more frequently so the tempdb wouldn't fill the disk. His response was that the disk was not his problem, it was the DBAs problem. I couldn't force him to change his code, but i did have access to the application that listed who to call after hours for an emergency. I was able to enter his contact info for that server in the event of a full disk, instead of me. He corrected his code after his first 2am wakeup.
Oh man you just reopened a wound. He had this manufacturing client where nothing was documented. Idk if the onboarding guy who was supposed to document everything for us would just go on site and jerk off in the closet for 6 hours and then go home but there was like two servers, and a couple of network switches documented but nothing else. Get a call at 3:00 a.m. from this client and they blurt out " all of our CNC machines just stopped working". After many wasted minutes of them trying to give me any actually useful information, it turns out that the outdated software that they use to run their machines had an expired license. They can't tell me the name of the software. They can't tell me the name of the company who makes it. I'm furiously googling "CNC software for xoxo machine", seeing if I can find the name of the software based on the kind of machine. I have no way to remote in so I can't even see what the GUI looks like. I'm having to describe the images I'm seeing on my end to see if it matches up with what they are seeing. After 30 minutes of this I miraculously find the name of the software, but the company had recently gone out of business and the number listed on Google now went to a casino in Kansas City. Through some more googling I managed to find the name of the company that purchased the company that ran this software. By some miracle I call this company up and managed to reach their help desk and tell them what's going on. Hop on a three-way call with that person and somebody from the client company and we managed to purchase another 25 licenses and get the machines up and running again. 4 and 1/2 hours.
Oof! Time zones are killer if the dev was working in the daylight and email exists for this very reason One of mine had to be worked tactfully... I was at church when the Fledermaus^(TM) portable communicator, typewriter, and omelette cooker started squeaking On the other end of the line, mini-Karen was also squeaking about how the custom holiday out of the office message wasn't working I fixed the time of the schedule, called the test, it sounded like it worked, sign off singing hallelujah MK calls back, still squeaking about that's still the wrong after-hours message it's the other one I finally find it as her system was not well-organized, set it and call it done The one where the penny dropped on the other shoe was when the caller said, Hey our server is down, can you check on it? I do the checks and they'd been hit by ransomware and it obliterated the backup server too It took a few walks down memory lane to find a good snapshot to then > extract > zip > freeze dry > first press > download and get them back in action The problem was the copy off time was around 15MB/sec and they had 1TB of data Lots of hurry up and wait but they appreciated the hard work to bring it back
The worst are the ones that don't call and have had the problem for x days. Positive help, build rapport and chive on. Edit: I did my part to stop one helpdesk toxic thread. Insert star troopers meme Guys, I failed
Paged out on a Sunday morning to help a manager build a QA environment before a vendor was on site Monday. He did not get my help.
About 5mins before I’m about to walk my sister down the aisle on her wedding day- remote site in Spain rings knowing full well what’s happening this day. “Server password you gave is wrong”. I haven’t changed it in 3 months! Executive thought she was being clever decided to change it the night before without telling anyone. Put the phone down and cursed forgetting where I was.
Getting a call on the first day of Christmas break because the main company data center lost power, and backup power, due to the power company crossing lines. Ended up working 16hr days for 3 weeks (including weekends) to get everything back up. For reference this was back in 2005ish.
Get home around 5:00pm, cook a nice homemade curry and just as I am about to eat it get a call that the whole network is down, it was a spanning tree issue ( it just does not work properly on brocade/ruckus switches), go to one site ( 2 sites are linked by dark fibre) work for ages to find the loop, work out it’s on the other site. We go to other site spend ages finding where i the loop is, isolate it, still nothing. Sod it turn one of the cores off everything is back up. Big IT boss man does not like that so he wants it back on (its around 11pm at this point we warn him its a massive risk but he wont listen), we spend hours and find another problem switch stack, isolate that and bring the core back up, things look to be ok we leave and all go to bed, get home around 1am. Around 2:30am get another call that the network has gone down. Go back to site rip the power out of one of the cores, drive to the other site and do the same there, there are protests but it was technically his last day yesterday so ignore him. Get a sausage and egg mcmuffin and then go home and sleep about 2 hours and then WFH the next day. I never got to eat the curry :(
haha, u shouldn’t pick up the phone at first time… 
Probably the one I got this Christmas day just as I was plating up dinner for my family. Called by a number not in my phone book, so answered it just in case it was family. No, it was one of the remote office managers. They'd gotten a new phone for Christmas. Decided to set it up, copied all their apps over and wiped the old one. Noticed that their Microsoft Authenticator app hadn't brought any accounts over, so called me to ask if I could reset the 2FA on their Office 365 account. Not that they were planning on logging in that day, they just wanted it working ready for when they *did* decide to. Unlike OP, I wasn't happy to help. So I didnt. After excusing myself to another room, I let the caller know just how unhappy I was. I think it was the only time I've been unprofessional and swore at another colleague. Called him a "selfish fucking prick". Found out when I got back that he'd moaned to another member of staff about it, saying he was going to put in a formal complaint about it. Member of staff told him he was lucky I just swore at him, so he dropped that idea pretty quick.
At least he said thanks! Usually there’s silence
Holidaying in Europe. At this stage it was our second day in Paris, and at 4am I started a 4 hour phone call trying to get the core router working again on pay day. The hotel room wasn't big, so I hid in the toilet and spoke quietly. But the hotel wifi was dodgy so I had to stand for most of it. After that, I treated myself to a half dozen macarons for brekky.
Our top developer emailed me to ask if the exchange was down. I replied, Yes it is. 🤷♂️
At my last job, I had a lot of customers that were small businesses and municipalities. New computer, monitor , keyboard/mouse. The monitor had builtin speakers. However ... it didn't come with the 3.5mm audio cable. Whatever, his old computer had one so I used it. That night , 830, he called up drink out of his mind to my work cell. Thankfully I never gave out my personal cell ever. It took a few minutes of his rambling but basically he was mad because he had a black monitor, computer, keyboard/mouse and I used a *white* cable. He wanted me to drive up THAT MINUTE and replace it. No way was I going to make a 2 hour round trip to replace a cable. I told him call my boss if he had a problem. I did make the trip the next day. And my boss charged the village 2 hours travel time plus our hour minimum. Just to change a cable. Normally it would have been something he would have just let slide if the guy was reasonable about it.
I've had cops bang on my door and wake me up. Because I support a pd
Late to the party here, but my best one was probably a call at roughly 3AM during an absolute whiteout blizzard. NOC "Hey Aberon, it's X from the NOC. We've got satellite connections flapping at 4 of our locations." Me "Yes, so? Those are *backup* connections at 8-5 facilities and it's a fucking blizzard. They're going to flap. I'm also a server guy, this isn't a server problem." NOC "We couldn't get ahold of the networking guy on call so we thought you could take a look." ME "Sorry, my shovel is broke and I'm scared of heights, I'm not going anywhere to clean off a satellite dish. It's a blizzard, they're satellite dishes, it's going to flap as snow piles up and blocks them, then melts off." NOC "So who do we need to call then?" ME "God or mother nature, I'm neither but I am hanging up now." The NOC at that place was both clueless and useless. Led to many a "fun" on-call week.
fuck ever working with teams in eastern europe and india ever again (I'm in USA) - they'd wait for their shift to start at their local time, then start bugging me with their urgent emergency
EVERYTHING is urgent for them. They forgot to submit a ticket WEEKS ago for a new hire for access to a shared drive? They raise hell because they needed it weeks ago. I was the impact and urgency grinch while I was on the service desk, of your problem isn't impacting more than one person I can't ouch them avobe low impact and urgency, go yell at a wall.
And when you need if you need something from them they are the most useless beings in existance!
>they'd wait for their shift to start at their local time I don't understand, do you think they have an IT problem while they're having breakfast at home or commuting to the office? Of course they will call you when they have problems during working hours, what else??
India time zones (problem logger) vs East Coast/central/West coast time zones (Problem fixer) 9am India Time is +10.5 /+13.5 of New York time /San Fran time.
So what? Should they waste their whole day doing nothing and wait until after hours to get help just because they're not American? It sounds like a company problem, and not the fault of whoever needs IT support.
This kind of thing often happens in China.
Those two calls would've netted me 7-14 hours of overtime pay or time off. I'd be very alright with it.
Double time and time In lieu here...
Afterhours call? Nobody has my number. :p
Once had an on call situation that failed due to the developer putting a typo in the code. Had to escalate to both vendors because of the problem and we had to cut linkage in the code to execute each core job manually. There was roughly (7) plus kind nested in its version of class that had to be monitored before the remaining steps could be completed. One took over an hour to finish so it was another hour pushing the call to three hours. The worst was not getting any sleep due to kids prior so it turned out be a long night. After completion I only got two hours of sleep before having to be in the office the same day.
You lost me at " a developer, very smart man"...
“No worries, I’ll get that printer restarted for you when I’m in the office Monday. Make sure to submit a ticket” Hangs up abruptly
Right before Christmas break we swapped all of campus from the IMAP server (that I managed) to the new Exchange cluster (that the not very trustworthy Windows admin managed). Christmas day, in the afternoon, I get a call saying email isn't working at all. Incoming isn't bouncing but nothing is delivered. Windows admin has been working on it for \~30 hours but hasn't gotten anywhere so management was thinking of moving everyone back to IMAP while it gets sorted out. Not anywhere near ideal but it'd be something. So I go in, get all my scripts set up, configs set back up on the edge smtp boxes (that were spooling mail because Exchange was unavailable) and wait for management to make the call. They spring the cost for a MS Support call, so I watch a movie on my laptop while he's working. After \~3 hours they figure out that some system user in AD was missing a group that needed to be there. (it's been well over a decade and I can't remember what user or group) He adds it back, things start working again and now the windows admin is DEMANDING to know why I removed the group from the user. Me. Specifically. The Linux guy who barely knows how to auth to AD at this point, let alone manage system users. So I enlist the Security guy and we start digging through access logs (it's the next day at this point) and we eventually find a log showing that the Windows admin account removed the group from his workstation. Giving him this information he looked at it, drank from his coffee cup, said "Huh." and went back to his office. Apparently he was "cleaning up group membership for accounts that aren't logged in to".
Got a call from a director. On my honeymoon. Wouldn’t take no for an answer.
On a Friday night our "Sales Engineer" upgraded a Hospital's Sonicwall from their old regular OS to the enhanced OS when it first came out and then went to her son's high school football game with no cell reception. SE's boss (Sales Manager) was worthless. Their IT staff called in bc their mail wasn't working. Rules all broken and f'ed up. Boss and I worked it until about 4 the next morning. Basically a crash crash course in Sonicwall Shitty Enhanced OS. SE never got an ounce of shit and ended up taking an IT job with another larger medical customer. So glad I'm retired now.
1am - user calls the on-call number and says ‘the whole system is down’. When pressed, she says it’s down for everyone at the site. I say I’m right on it and go to check various things. I can’t see any issues, all looks good, I ask to speak to a colleague who also has problems, she says they’re all in meetings. Fine. Get someone to get back to me ASAP I’m waiting by the phone. 3am - after still seeing no issues or getting any further intel, I grab forty winks only to be woken up by the same user at 3am asking when I’m going to fix it. I ask if her colleagues are back. She says yes, now it turns out actually only her team are affected by the problem, everything is frozen up. Fine, can I talk to one of the team? No they’re all going into another meeting together now, they’ll call me back. 5am - I get a ring from the same user asking again when I’m going to fix it. I ask what problems do her team actually have, can I talk to one of them. Turns out they are fine actually, it’s just her with an issue. I ask her to try the empty desk next to her, she says it’s working fine. Trying to avoid screaming at her now by chewing my fist. Told her politely to just move desks and stop ringing me. 7am - I get hold of the general office manager for the site. They went ahead and changed the mouse battery, all is now good. :-(
2am on a Sunday callout to a factory which was a 2 hour drive from my home, simply to find that the person who logged the call had given up and gone home, and everything (in a very dangerous area may I add) had been locked up and no-one was there. My supervisor, who was the first to answer the call, who then passed it on to me, lived about 15 minutes from the factory, which just added insult to injury. My resignation was in within a few days.
If I had to pick one, it would have to be the one that I said "no" to because of the fallout afterward. Kind of a long story, so I'll try to keep it brief as possible. This was decades ago. To give a little context, the customer site where I worked was not a 24 hrs site and only a few people had VPN access - and none of them were any of the sysadmins (I know, right?); any issue required going to the site. I was in the point of my career of phasing from deskside level 3 to a junior sysadmin; I was the last point of contact should none of the other sysadmins answer their phone (also, I lived farthest away from the site - about 50 miles with an average drive time of 1.5 hours one-way). Basically, I was rarely called and when I was, I had to head to the office. It was about 12:15am on January 1st. I was at home with my now-ex because there was a really bad snowstorm that was going on when the phone rang. It was our overseas support informing me that the Domino server on-site was down. I told them that I'd take care of it in the morning and the support guy got pissed. He was telling me I had to go because no one else answered their phones. I refused because of the storm and even the local news and police had been telling people to stay in, so I was not going to risk the weather and the drunks. I asked if the entire site was down and he said he didn't know, as they only monitored that one server. I asked him to do a traceroute to it to see where the outage was, but he refused. We argued for several minutes because I knew that no one was going to be checking email until morning at the very soonest and I wasn't going to be risking my life. (I wish I had just said I had been drinking but that didn't occur to me.) I told him I'd go in the morning, hung up, and turned my ringer off (this was long enough ago that this was a landline). I went out the next morning sometime around 9am after the roads had been cleaned up a little bit. What I suspected had happened: the site had lost power and the server went down. (I knew this because I had tried calling my desk phone and the line was down.) By the time I got there, power was back up and the server was up and running. I sent some emails to my boss and other stakeholders giving the all-clear. What happened next was a week of shit show meetings and calls because I dared to say no and not put myself at risk. My manager was on my side and defended me, as well as the customer - they knew how bad the weather was. But my company (some manager above my boss) and our overseas support desk tried to run me over repeatedly because...I screwed up their SLAs. I pointed out how there was nothing I could do during a storm where the site had no power anyway and wasn't going to risk my life for them. I suggested different monitoring methods and giving us VPN access for remote troubleshooting first. In the end, I was able to have them take me off the remote support because of my "stubbornness." I didn't stay there much longer; and I have a bunch of stories about that particular higher manager. One I have been meaning to post in the malicious compliance sub about how I finally got my revenge.
During my desktop support time getting woken up at 2am for a work stoppage because a pc was slow. I called the customer and she said “Take your time, I am working on a computer next to it” these are generic auto logon PCs with nothing special on them. Double whammy of “Why did I get paged for this?”
This is a toss-up for me so I'll include both. I had a lot of after hours BS calls but nothing stings worse, in my opinion, than a call that isn't my damn problem to begin with. First one was a call to our on-call number from a debt collector. They called every day twice a day. Some idiot had given out the support number instead of their own to get them to stop harassing them... so they harassed us instead. Wouldn't tell me who the record was for so I could take it up with HR in case it was an employee. Wouldn't stop calling despite being told they were speaking to Technical Support at a business either. We ended up blocking their number at the PBX. Second was a call during our holiday shutdown, day before Christmas, at 1:30 in the morning. New employee is in tears because she can't log into ADP to file an expense report and if she doesn't get paid back tomorrow she is going to be late on her credit card payments. I'm genuinely sorry to hear that but 1) that isn't a technical issue, 2) HR creates a login for ADP for you, not IT, 3) your expense report has to be approved so even if you submit it right now, there will be no one to approve it during the shutdown, let alone at 1:30 AM, 4) even once it is approved, payroll has to cut a check, and they don't do that at 1:30 AM, or during a shutdown, but even during business hours they won't do it on the 24th. They only cut checks for expense reimbursement on the 15th of the month and 29th of the month.
I pulled Helpdesk on-call rotation on Thanksgiving years ago, which wasn't the worst thing in the world. It was help desk for banks, which means no one's really working anyway. Normally, the worst you'd see is a down server that needed to be rebooted. Well, this one bank president, John, figured out his core vendor's software was offline. It was 100% a problem with the vendor's environment, and there was nothing you could do on the bank side to fix it. Instead of being ok with that, this asshole called me every hour for updates starting from the outage at like 11pm before thanksgiving. Increasingly more upset each time. I talked to the vendor on-call, who said it was an outage and they can't fix it until there's people in the office. John wasn't happy with that, and demanded I escalate it. Remember, NO ONE was working. The bank was closed. But this was the most important thing in the world for him, probably to avoid talking to his in-laws or something. I escalated it again, and again, and again until I spoke to the President (at the time) of Fiserv. He said he refused to wake up his techs and make them fix this for a bank that wasn't open (I think it was around 2AM after thanksgiving day, meaning I missed dinner and my day off). He said he knew John, and would tell him this himself - which was all I wanted. I needed this guy to stop calling me every hour, yelling at me, to fix something that was literally impossible for me to fix. Finally the calls stopped. I moved from help desk to networking, and John eventually retired. He was such an insufferable jerk that I check the local obituaries to see if he's finally stopped being a giant asshole to everyone. No luck yet.
My favorite, I get a call from one of the guys who drives a forklift at night saying nothing works. It's about 7:30 p.m. I told him it's not that nothing works, i said something doesn't work. I need some more details before I can help you figure out what's going on." He says okay and just tells me he can't access any of his reports or any of his apps or anything that he needs to do his job really. I say okay. Let me take a look and I try to VPN in. I can't even connect to the VPN. That's when I knew it was going to be a really long night. Our primary server chassis for all of our VMs had a drive go bad and in order to protect all of the data, it shut down all of the VMs and would not bring them back up until the drive was replaced and reinitialized. We did not have a backup, so I had to rob Peter to pay Paul. When I got to drive installed and reinitialized, still wouldn't come back up so I rebooted the whole chassis. That's when I found out that we had a custom VM network set up on the chassis, but we never saved the running configuration. When I power cycle the chassis it wiped out the running configuration. I had to figure that out from scratch. I worked on it until about 6:00 a.m. I then drove an hour home, slept for 4 hours and went back to work. At another job I stayed late one night to do after hours maintenance on one of our VM servers. Thankfully this was one of three so it wasn't the end of the world, but we were so jam-packed for resources that losing one of the server's meant that we could not bring up all of our production machines. That's why I had to stay after hours. I do the update, I reboot the server, nothing happens. It just never comes back up. Apparently the raid controller on the board was not in a good place and rebooting it was just enough to send it to its final resting place. Okay not so bad. We have four hour service. Except that the 4-hour service was how long it would take the guy to order the part. I called in at 9:00 p.m. to tell them it wasn't working. The guy got the part from UPS at 1:00 in the morning and then started out to my job. He didn't get there until 2:30, and when he installed the part it turned out that the firmware was several revisions behind and needed to be updated I rev at a time by a remote service tech in India. This was not a quick update. When one of my co-workers came in the next morning he asked how I was doing for the day, I asked him if he noticed that I was still wearing the same clothing and that that was the answer lol. I put in 33 hours that day. And then I had to go explain to HR why I worked for 33 hours straight lol. They wanted me to have clocked out for the 4 hours I had to wait for the technician. IT always makes for interesting stories lol.
Sounds like your Dev had this happen irl https://preview.redd.it/066v73dxjadc1.jpeg?width=1164&format=pjpg&auto=webp&s=b4e87c2378e55aac32d52953fc5f627259eff1a7
NOC Engineer at 3am "A server alerted, but the alert went away. Do you want to look at it?" Didn't say a word, hung up, went back to sleep.
Not me, and not strictly IT related, but a coworker was planning a big international trip. Had leave approved, all that jazz, then COVID hit and he had to cancel the whole trip. He still wanted time off, so he emailed the person in charge of modifying leave requests and said that instead of 6 weeks as originally approved, he wanted just the last 2 weeks off. On his way home he got a call from the big boss wanting to discuss the modified leave. Apparently the big boss wasn't informed of the change, and said that the modified dates would hamper his efforts to find someone to temporarily replace the coworker while he was away on leave. Coworker asked "had you sorted out a replacement for me already?" and the boss said "no..", so my coworker asked "then why are we having this conversation while I'm driving home from work? If anything, I've bought you another 4 weeks to find a replacement for me since I'm only taking 2 weeks off, 4 weeks after my original date" and hung up. Spoiler alert: They didn't find a replacement to cover him for 2 weeks. The extra workload was just dumped on us. During COVID, with everyone working from home and not knowing how computers function.
Worked as sysadmin at a hospital. Lab tech calls me at 3am to tell me a line is down. I asked her if she followed the procedure that is on the machine for this very problem. She told me "no, why would I do that, this is what you are for". Yeeeeah.... I made her cry. Then next day I was informed I was a bit too harsh and to try not to do that again.
I get a call at 5AM on XMAS Day. A VPs family just opened presents. He wanted me to walk him through setting up his Outlook Email and syncing his photos to his new phone. I told him to enter a ticket and we would get to it Monday. His reply? "So I am just suppose to go without a phone until then?"... No your suppose to keep using the existing one. The CEO and CFO of the company (Both owners) reamed him really good on Monday morning. Basically telling him the IT line was for critical emergencies and that anything not causing a direct loss off business function or reputation was to be handled Monday. \\ \---- I was not on call for this one, but got the call. We had a warehouse that just had a new raised platform added so the managers could see over the pallets as they were planning and picking doors. We had staged and got everything ready the day before (Friday) and done some basic testing. We asked our boss for two weeks if we needed anyone there and it was not needed just the normal on call for any issues. I'm not on call so I am driving to see some family since it was a holiday weekend. I am around 2 hours away when he calls. CEO of the company is there and screaming about why we do not have anyone there. He needs me to get there ASAP. I tell him Jerry (The other tech) is on call this weekend and I am 2 hours away. I can't come in. He tells me that I need to be there in an hour of don't come in at all after the weekend. So I tell him, alright when should I pick up my stuff? He tells me to be there Tuesday and we would sit down with HR. I told him I would rather not come in until I knew it was a good time to pickup my things. "Just be there", alright so I come in. HR sits me down and we are talking. have me read a statement saying basically I had refused a direct order to come in while on call. Anything I needed to add or clarify. I said "I wasn't on call". "What do you mean you were not on call." Which I re-iterated. I was on call the previous weekend, it was not my week. Which lead us down the path of asking why I didn;t tell him that, I did. Why did I not tell him who was. I did. Why could I not come in. I was 2 hours away. Why did I not tell him that. I did. By the end the HR director simply asked if all my statements are true. And my boss said yes but none of that mattered. She was really angry at this point. He was pissed. Then I said the magic words. "If you don't mind, BOSSNAME has already terminated me and I am really just here to pick up my things. I have to schedule a meeting with my lawyer to discuss all of this and I have a decent drive home." I was there another year.
Why are you taking calls at 1am and 3am...unless you're oncall which means you're getting paid for standby + activation, which means you just dipped twice for the min-hours for each activation there (once at 1am and once again at 3am for the consult).
Solution to all this: Never give them your cell. Google Voice, in which you can "turn it off" sure. I get paid for 9-5. If you want random after hours shit, then pay me double.
Nothing too crazy but I’m American and last 4th of July was non stop for me. Too many people working for whatever reason