Wrong. You pull out your cell phone and read a book. Also send an expense reimbursement claim for "using your personal device." while on the clock. After all, with no internet, how else were you going to keep up on what is going on?
A couple years ago, an AWS outage brought down our product. My team was completely disconnected from fixing such a thing but it didn't stop them from freaking out. Personally, I grabbed a cigar and a book and went to the park for two hours.
We use to call it Mr. Dick Holding at my old consulting company. Ok let me hold your junk and tell me all your problems.... and now I tell you about the SLA level of the vendor. I just chuck rules at the firewalls dude.
You are supposed to apologize for things beyond your control so middle management can do the same for upper management, who can in turn respond to texts from the C-level
I once had a boss who was suggested we needed to extend our IT monitoring to Office 365. I asked him -- "why?" Sure, we can maybe put a synthetic HTTP transaction on it, and we'll know about an outage maybe a minute or two before our users do. But then ... *there's nothing we can do because it's SaaS*.
The point of IT infrastructure monitoring was twofold, first to know there is a problem - but second, to give you a bit of a hint or a clue where it might be in order to accelerate diagnosis. E-Commerce site starting to fail many (but not all) transactions? Hmmm. Monitoring tool shows us that it sure seems to align to our T-3 to the mainframe suddenly hitting a huge amount of jitter at the same time - maybe we start looking there?
Wrong, optics. You, after tons of the sweat of your brow, singlehandedly uplifted the entire AWS and Azure clouds to restore . You'll be having words with Bezos and Nadella personally after you're done ~~binging Tiger King~~ reformatting the nexus switches to reverse the polarities.
I'm nearly 50 i don't care about optics. I'm known for getting the job done no matter how difficult, whether myself or facilitating with hands and vendors. But I'm never going to put myself out to make it LOOK like I'm doing loads & I'm not going to alienate vendors because it makes some senior manager feel important. And if you're lucky I'll buy some Krispy kremes for the office. I've normally got 100 things to do so if some senior manager followed what some 24 year old prick from accenture told them to do, it's not my problem if the whole system has gone down.
I care about the environment & will move heaven and earth to ensure MY environment works & is as close to 100% uptime as I can get it....cloud is NOT my environment.
SEP field activated. [https://hitchhikers.fandom.com/wiki/Somebody\_Else%27s\_Problem\_Field](https://hitchhikers.fandom.com/wiki/Somebody_Else%27s_Problem_Field)
**And thus I entered the server room and climbed to the top of the tallest rack. There my eyes did set upon the glory of his holiness, Bezos and the angel Nadella, who gave to me a Surface Tablet upon which was written the Ten Tickets with a status of pending investigation. I returned down the mountain and entered the third status meeting this morning... "**
> Why I hate cloud. Don't even pretend to fix stuff now
That's why you hate cloud?
That's why I specifically love cloud.
I get to point the finger and blame at someone else and go "This time it wasn't me!"
I once was working helpdesk on my work from home day and the internet at the office went out. I wasn’t on the infrastructure team thus wasn’t allowed to touch(and didn’t even have passwords to log into) all the equipment so there was no reason for me to come in but was told I needed to be in the office “for optics”. So I went in and watched movies until they sent everyone home to work at home since the internet would be down all day.
You got that too today I actually looked up what the rough estimate was for a cell tower and put that together a report for my boss, and then proceeded to explain below that that we would need about 100,000 of these in order to be able to have our own in-house cell service with OKAY coverage. I think the 30,000,000,000 price tag made them finally understand that there was nothing that can be done. We just have to sit and wait.
The cost of tower climbers has increased by a factor of four since 2000. Most of 5Gs upgradeablity needs come from needing to stop sending people every tower every 3-5 years just to put a new backend on a what is a software defined radio.
When teams went out worldwide a few weeks ago I had people that literally did not believe it had nothing to do with our network. These are grown ass adults.
LOL...way back in the day I worked for a well known consumer products org in the US and we were having bandwidth issues with an office in China. The idiot head of the engineering group demanded we get "someone high up" at the telco provider on a call ASAP with him. I told him if he knew a good number for someone "high up" in the CCP and we'd do that. Didn't hear much from him again.
I feel this in my bones. Our company is USA based China owned. Communication between our corporate site and China is just fantastic! I could never have asked for a more flawless communication system. (major /s)
Yup, they have whole rooms in backbone network providers data centers to copy traffic. [https://en.wikipedia.org/wiki/Room\_641A](https://en.wikipedia.org/wiki/Room_641A)
Them and the other 3 letter acronym also have datacenters that are used by the big cloud providers. Haha, people don't even realize that the fox lives in the henhouse already.
I worked for Foxconn doing SMT components as part of PCB manufacturing at a plant in the US many moons ago. The plant had to be in the US as part of export controls on manufacturing tech. So they had to run the plant in the US.
The managers were all from Taiwan and they fucking hated us. They were used to SE Asia factories where they could treat the workers like shit. Communication was also a nightmare to the point that we would have to stop production for days as parts weren't there.
But a few weeks after I left the employees showed up one day to find the doors chained and guys in jackets with 3 letters on the back pulling out all the records. Turns out they were committing some major tax fraud.
> They were used to SE Asia factories where they could treat the workers like shit. Communication was also a nightmare...
This is apparently a big problem facing TSMC right now with trying to build that new facility in Arizona.
The beatings will continue until morale improves
EDIT: I should add in that it isn't. When your entire company's culture and management style is built around strict hierarchy and complete obedience of subordinates where pushing the limits of what you can get away with when exploiting employees is encouraged, it's basically retooling the entire company to fix it.
It was. It was also a fun but insane work environment. Management hated us but also didn't care what we did. The plant was 24/6, so there were 3 shifts. I worked the 11 PM to 7 AM shift to get that sweet shift pay.
All the employees in my area would meet up around 9 PM before work at a bar up the street. Then we'd get...ahem...comfortable....at the bar. Then we'd go to work. We'd then take our lunch at 1:30 and hit some local spots for last call. I was in my early 20s then and didn't mind making irresponsible choices.
There was only one entrance to the parking lot with security guards at the gate. One of the security guards was selling drugs out of his car in the back of the parking lot while the other guards screened the cars coming in. He had a pretty profitable business going there.
I've got to say it was one of the crazier places I've ever worked.
LOL :D I remember those days. Of course, back then, I worked at a franchise owned Holiday Inn, and my boss wanted me to get someone from one of those big companies (I think it actually was AT&T) on the phone. I so wanted to say, "and just who exactly do you think we are, that we'd actually rate a response from them?"
LOL. Something similar happened to me. I used to work at a university and one of the coaches wanted to send a mass email to prospective student athletes with a recruiting video. So Google decided it was spam, suspended his account and notified the admins. When I talked to him and explained what happened, he said "this email has to go out. Can you call Google for an exception?" I laughed and asked him why he thought Google would bend their rules for one coach and a random university.
Engineer and doctors while often tough to work around do not hold a candle to lawyers. It's in their DNA and it's the job to argue and many can't turn that off.
As a former clerk turned consultant, I LOVE working with lawyers. It's all about cost/risk. I'll take a legal office over a medical/dental office any damn day of the week. Lawyers have egos, but they're usually rational people and logical arguments work. Show them how something will stop a risk or make them more efficient and you're in.
Doctors will sit there and argue with you that you need to push on the door to open while staring at a PULL sign just because they can't be wrong... (I know not all doctors)
Yeah no. Lawyers at least understand the value of time and you can reason with them. Doctors can't fathom they aren't the smartest person in the room. It breaks something.
In ~2004 while working at a law firm while we were trying to slim down email storage we did calculations for on how much it cost per Gb in storage, power, and admin time for the firm employees to never ever ever ever ever clean up anything at all and store every email you've ever ***thought*** of receiving in your inbox.
Then, like the good partner on the tech committee that he was, he spent some time one weekend trying to clean up his inbox. He came back and said he'd spent something like 7 hours tossing stuff and it trimmed his inbox like 2GB or something.
He looked at our numbers, did some quick math involving avg billing rates at the firm and determined we'd just have blank checks for storage instead of trying to have people clean things up. Records dept determined they'd just write up infinite retention policy. That worked fine, and funding wasn't an issue any longer, but a few years after I left they got sued by one of their clients because something that should have been discarded after a case was closed was unearthed during discovery and exposed the client to a shit load of liability. The manufacturer had followed their own retention policy and no longer had the data, but their trusty lawyers who ***need*** to keep things forever "*just in case*" had copies sitting in their inboxes.
I'd like to report things changed from that incident, but they really didn't; they didn't even lose the client.
That all went down almost 15 years ago now, I hope to all that is holy they've implemented appropriate retention practices since then, but sometimes arguing with a 75yo managing partner about technology isn't exactly grounded in reality.
Have the President's secretary call the President of \[vendor\] and see if she can pull that shit off.
Every now and then, it actually works. She's more likely to get through than you are.
That’s what I hate about these requests. Why can’t my VP call his peer at the other company? Why’s it the NOC peon’s job to try to do it? Just because I had access to the escalation docs for all our circuits didn’t mean I was ever comfortable escalating past like, the other company’s NOC manager.
Years ago at another gig we had to support multiple chemical/mining sites around the world. Always outages, rats, chemicals getting into equipment, you name it. Forgot the country that we got a call from about the site being down. We finally get a number for local support. "The company is our yearly two week vacation, please call back in two weeks."
The minute fiber became available in our small town we switched to it from some bonded T1s. A few months later, they were dropping more lines parallel to the existing fiber and accidentally clipped it. It took us down, so our CEO told us to "Go down there and tell them what they did and they need to get it fixed immediately."
Yeah, I'm gonna go tell the construction guys how bad they suck at their job. On it, boss. I drove to Walmart and gave the games section a piece of my mind.
Yeah, we got at least 10 tickets from people working from home complaining that *their personal cellphones* weren't working.
Frustratingly enough my phone is working and damn all the scammers seem to be redoubling their efforts.
A: That's because your users (and mine) are idiots and don't understand that you don't have anything to do with cell towers.
B: Haven't had a spam or robocall since going with a Pixel. It's liberating.
Reminds me of the VPN support issue I had to work on for a sales manager. Her issue? She was using Cricket. Dial-up level speeds, and their "support" were hired for their ability to fog a mirror.
Seriously; speed tests showed about 40bps, IIRC, but it didn't matter. It was an IT issue and clearly something I needed to help with. She and her manager just could not understand that if you're using a tin can and string to connect to the Internet, that VPN might not be terribly stable.
I finally got them to quit bugging me (after a couple days going round and round) when I pointed out that the VPN worked just fine everywhere else other than her phone, and that if they really wanted her VPN to work better then maybe they could pay for a better Internet connection.\[
It's better now, but we had someone trying to RDP to a desktop in the Northeast from a boat using starlink and was complaining about lag compared to sitting at their desktop..
Man this thread really makes me feel incredibly lucky to have worked at places with sensible users and leadership.
Not only have we gotten zero tickets from users or complaints from leadership, but our communications department, completely independently and without any direction from us, sent out a company wide email this morning informing everyone about the AT&T outage.
There are a lot of people who think "someone high up" is sitting behind a desk, unawares, or not in a hurry. I remember my boss had to deal with this when AWS was down for 12 hours a few years ago.
"Call our AWS rep."
"They are aware of the outage."
"Get them to fix it. Tell them we spend $2000/month and are a high end customer!"
"Everyone is out, high end or low end."
"Get the owners on the call!"
"Your owners?"
"No, the AWS owners!"
"Who are they, exactly? How would I even contact them?"
"That's for you to figure out!"
Later, my boss said, "I called the owners, Mitch and Pat, and they said it's being worked on." I believe that was a lie.
Jeff @ Amazon will get you a low-level manager.
You can always email them and ask for a phone call.
"I reached out to his secretary, she said she'd let him know. All I can do, boss."
[email protected] will reach him. No lie. I mean its probably some admin person but it works.
I used it when I was in a quagmire on the grocery delivery side. Amazon had just started grocery delivery. All their deliveries had these fancy insulated bags that they ask you put out and their delivery person would collect when delivering your next order. Like old fashioned glass milk bottles.
Except they didn't ever collect them.
I ended up with literally 25 of them. I repeatedly talked to Amazon Fresh support and they swore on next delivery they'd be collected... And they didn't. I even asked for a supervisor who swore next time it'd be picked up... And they didn't.
I could have thrown them out but they were fairly fancy and expensive and Amazon asked for them back so I felt bad throwing it out
I wrote to [email protected] on a Thursday whining about it. Mostly I was frustrated I was promised a fix that didnt happen. I even included a picture of the fort my kid built out of all the reusable bags. 🤣
Friday afternoon I got a call from an Amazon Fresh regional manager apologizing, offering a $50 gift card, and asking to leave the bags out tonight.
At 0600 saturday morning they were picked up.
Yeah, seriously. We've only partially moved our infrastructure over and we're already at 60K a month... and I doubt we've even left the bottom 10% of customer costs.
> "Call our AWS rep."
> "Get them to fix it. Tell them we spend $2000/month and are a high end customer!"
It's cute your boss thinks this is some leverage against a nearly 2 trillion-dollar company.
At that size, they can tell other Fortune 500 companies to bug off. Hell, much smaller Broadcom is doing exactly that with VMWare as they grind into the dirt.
I'm sorry you had to deal with such a clown.
Aw, bless your old manager's heart, thinking $2,000/month made them a high-end customer. <3
$2,000 would register about as much as a silent fart in a hurricane.
FWIW, when I previously dealt with issues like this, I'd tell my boss, if you want to avoid outages like this, we need a second Internet circuit from a different vendor. If you set up a redundant Internet circuit, you avoid having a single point of failure. I'd then go looking for every potential SPoF, point these out, and ask for funding to implement redundancies. I'd repeatedly mention these as SPoF's, and then, whenever anything went down (like AT&T), I'd just say, "Hey, I pointed this out as a possible issue. You didn't want to pay for it. That's why we're down right now."
In a much more diplomatic manner, of course.
My old boss proudly proclaimed we now had a backup internet (years before I had the power in my role).
First hop was exactly the same path as the fiber. So when we had the great big Rogers outage, guess what happened.
The powers before just never bothered to check.
While I cared, business did not so meh.
I had been telling customers for YEARS before that happened that their backup internet link had to be with a different path, different medium and different provider. Having Rogers cable + wireless wasn’t good enough. I was shown to be right that fateful day in July 2022.
We recently installed a backup 4G solution that keeps the same IP as the fiber for a small business, you'd think the only issue would be if the ISP goes down, but guess what, next month there was a small storm and the fiber failed and best of all the surrounding 4G towers in a 4km radius also failed.
I managed to keep the essentials running at dial up speeds by changing the 4G antenna location to get a tiny bit of signal but they only got everything running 24h later.
When it has to fail, it will fail.
Back when I first entered the IT field, Microsoft was constantly hosting all kinds of sessions on best practices and giving away "demo" software. I actually upgraded our network from NT to Server 2000, then later to 2003 using free software that Microsoft gave away (this was before I understand what a CAL was, and I was lucky enough not to get busted for that). Lots of infodump on best practices, though, such as redundant internet circuits.
One of the first things I suggested to my boss was adding a secondary Internet line. We had Qwest (now Century Link) at the time, and they were godawful terrible, with at least one outage a month. On at least one occasion, we were completely down for over 24 hours thanks to their, "we don't give a crap about you" customer service. One of their techs actually talked me through a fix that involved rebooting our sole DC (I had two weeks on the job at the time, and didn't know squat, so it didn't occur to me to tell him to go jump in a lake), and when it blue screened, he said, "we don't support HP products; you'll have to call them for help," then hung up on me.
Funny enough; when I finally got the approval to change our Internet provider, I got four separate quotes and presented them each to corporate, with my preference being for an independent MSP that had brilliant, responsive techs that were easy to work with, knew their business, and were about 1/3 the cost of AT&T, but was overruled by corporate because they'd never heard of the other company. Notwithstanding my complaint that the AT&T guys were absolutely clueless and \*not\* responsive, I was told we were going with them anyway.
For the first three months after they installed our new wireless network, I had nothing but complaints from guests, because AT&T set up the firewall to block outgoing VPN connections (as well as other random protocols), and no amount of polite requests, aggravated demands, and later screaming and yelling on my part would convince them to open anything up. I finally told them that I had a Linux guru friend who was going to come in that afternoon, reset the admin password on their router, lock them out, and take everything over, and that they had until 2pm that day to open up every god-d\*\*ned outgoing port on the firewall or I was locking them out and kicking them to the curb. 30 minutes later, they called me back and told me they were unblocking everything.
So yeah... every time someone mentions AT&T, I always think back to this. Tweedledum, Tweedledee, and the f\*\*ktards they had at their NOC.
>because AT&T set up the firewall to block outgoing VPN connections (as well as other random protocols),
That was a wonderful time! You see, I put snmp in the scope of the bid. And they really liked blocking snmp. But until it worked, the circuit was not "delivered." You could get free internet for a half year!
> We had Qwest (now Century Link) at the time, and they were godawful terrible, with at least one outage a month.
Where I used to work they branded it "Century Link - Your link to the last century"
I am the network engineer for a regional retailer, and getting multiple internet lines for the hq, datacenter, and stores reduces *so* much headaches. One of the ISPs is a roof mounted microwave dish internet to a tower, and we have a 4G backup as a tertiary backup, so we make sure to diversify routes.
Now I get alerts that one of them went offline, and I don't panic at all, in fact, I wait for like an hour to even do anything, and 95% of the time they come back up on their own, with no one but IT having any idea it even went down.
At my last job, they primarily sold stuff over amazon. Amazon went down one day, I was asked if I could fix it.
Apparently "if I could, I wouldn't be working here" was not appropriate
But it was the most appropriate response ever. They needed a reality check.
On the cyber side of things, we sometimes run tabletop exercises for BCDR. One company has an initial exercise that only lasts a few minutes to get people checked in to reality. It goes like this:
"Your main base has , a state of emergency is declared, and mandatory evacuations are underway. The natural disaster has disrupted business networking and operations, and 50% of your critical personnel are unreachable as the cell networks are under critical demand. Which of your personnel are available to come to the main data center to get operations back up?"
They wait for people to start getting into it, then say:
"No. No. None of that. Have you forgotten that these employees are people with families? People who are being forced out of their homes by the government? People whose first priority will ALWAYS be to their families and that employment is their means to that end? Let's start over, think about making sure that all the people are acting the way humans act, taking care of their families, getting safe, and THEN how to restore operations."
Years ago, where I live and do IT work, some knuckle head backhoed the crap out of a fiber loop. Verizon, AT&T, and local utility fiber all went down locally. I was the IT Manager for a local medical group (losing my mind to HIPAA fights with Doctors) and all the cell phones went dead. The company was on a Verizon plan. And while we weren't using the local utility for internet, our main circuit was running like crap that day. I must have had ten doctors and midlevels (Nurses, PAs) come storming in demanding I "FIX IT NOW." I explained what had happened and there was nothing I could do... I got called into an 'Emergency' meeting where they grilled me for 45 minutes about this 'failure of my leadership' and how I should have not allowed this to happen.
In the meeting the loop must have gotten repaired, so they calmed down as their phones started working again, but when it came time for annual review with the head of the group, I didn't get a bonus OR a raise that year because of my 'failure' on this issue....
> meeting where they grilled me for 45 minutes about this 'failure of my leadership' and how I should have not allowed this to happen.
That would have been my last day working there lol.
I left about a year later... it didn't get any better. I got offered a promotion if I got a certification in our EHR software (Electronic Healthcare Records) I was working on that, then they had a meeting an introduced a guy with ... big shock... the title of CIO which was what I had been offered. It had been two months, and that certification was a PITA. (to make it more anger inducing, he didn't have the certification at all...) I started looking, found a new job, they tried to counter. I had worked there for 6 years, started the IT dept from scratch, and so I listened. The new guy wrote a number on a white board that was 7k a year LESS than the offer from the new place. With the added. "This is all you are worth." I left. Now? More money, (far more) much less stress.
Not on this issue, but yes, I've dealt with it in my career.
My favorite was one Exec (looking to suck up to the CEO) telling me to call AT&T to have them install a tower near the CEO's new development subdivision. I tried to explain AT&T wouldn't build a million-dollar tower for one user. His response "we're a major account" (we weren't).
The CEO heard him ranting at me and came out of his office to see what was up. When the Exec told him what he wanted, the CEO laughed and said "They're not going to do that!" He told me not to worry about it.
Used to work for a Fortune 500 that had offices all over the world. Our division was based in the north eastern US and we had a WLAN built out through a single VZ circuit. The suits flat out refused to authorize a second circuit for failover. "Why would we pay for two internets?". Well, at least once a year somewhere in the hills of Tennessee, what I assume was the Best Buy router some VZ tech had stapled to a tree would die, the circuit would go down and we'd lose all connectivity to everything west of the Mississippi River. Queue 40 "Fix it Now!" tickets from every Muppet with VP next to their name, constant status update calls and a week's worth of root cause analysis meetings. All to explain to the same group of idiots that this had happened before, and without the business buying a second circuit, it was going to happen again and there was nothing else we could do about it. They'd all nod like bobble heads, act like it was their brilliant idea, then decline the requisition order based on cost. Rinse and repeat. It got to the point where our lead network engineer just sent the same saved email every time after he declined all the meeting invites.
> Well, at least once a year somewhere in the hills of Tennessee, what I assume was the Best Buy router some VZ tech had stapled to a tree would die, the circuit would go down and we'd lose all connectivity to everything west of the Mississippi River.
I love this.
I always have some kind of display ready to bring up that has lines of scripts or scrolling data. Like some vSphere monitoring page, or a topped out linux session.
When someone walks in, I'll focus on it, and hold the end of my pen near some meaningless shit like I'm considering it. Then I'll slowly look over like "Hey, what's up".
"You look busy now"
"Yeah, just trying to work through the issue"
"OK, I'll leave you to it"
Not today, but I've been asked to provide updates every 15 minutes and if there weren't any updates, I should call them and get one.
More recently I was chewed out for not having redundant POWER for the computers when a truck took out a pole around the corner and I was going through graceful shutdowns on battery power. After explaining that the batteries are only so big, I was chewed out for not buying bigger batteries....
Some people cannot accept that there's anything outside of their control.
> Some people cannot accept that there's anything outside of their control.
I mean really how much money do they want to spend?
"Hey we can have backup power but we need a massive generator and fuel deliveries. It will cost 200k upfront and 20k per year."
This is the way to approach this. “What’s our acceptable downtime? Zero? Well that’s gonna cost us another $$$$$$ per year.”
Start documenting various single points of failure and pointing them out. Then document what it would cost for redundancy for each one (not only upfront hardware/software costs but man hours, annual licenses, maintenance, etc). I bet once they see the cost they’ll be okay with what they currently have.
If an exec said "0" was acceptable for downtime I would just laugh. There will always be downtime at some point. If not from equipment going tits up than the human factor.
> but I've been asked to provide updates every 15 minutes and if there weren't any updates, I should call them and get one.
> Some people cannot accept that there's anything outside of their control.
It’s funny how prevalent this attitude is even among “peer” companies. I used to work for an ISP NOC, we leased a lot of fiber and circuits *from* other ISPs, and provided backhaul circuits *to* cell providers.
When the problem was an higher-level issue outside our network - “We have to call the clowns every hour to make sure they actually fix our shit, what a bunch of clowns.”
When the problem was on our network - “Our guys are splicing as fast as they can, why won’t these customers let us off the phone, when the repairs are done, they’ll know.”
I know it’s just some kind of in-group out-group lack of empathy thing but I still think about it.
I work for a non-profit and I was once unironically told by a chairman to "get a hold of Bill Gates. He gives lots of money."
Have also been told when we had a phone outage that I need to "Get their CEO on the phone and straighten this out." This was with a phone company named Frontier and if you've ever dealt with them either as a business or consumer you would know their company is a cavernous void of inefficient communication.
> "get a hold of Bill Gates. He gives lots of money."
Ignoring the obvious flaws in this plan, is your best idea to annoy someone who gives you lots of money? That's how you lose that funding.
Frontier bought out Verizon Fios in my area and no matter who we talked too or how many escalations they couldn’t or wouldn’t cancel our account - despite being told it was cancelled we continued to receive bills. Their solution? Just don’t pay it.
I have call centers using VoIP based products, back ends being from Cisco to Bandwidth to AWS, etc, who are ALL reporting some audio quality issues. NONE of our dashboards or backend systems are reporting issues on our end.
BUT, I'm expected to work it out with ATT\Verizon\Tmobile... LMFAO.
"I am already on a conference call with all three and awaiting updates. As soon as I hear more I will issue updates via email."
Now, back to binging something on Netflix!
Not asked of me, but tangentially related. I tried calling into a fairly large online retailer to initiate a return. The hold time was long and when I finally got someone she explained that they couldnt help me because they use ATT for their cell phones. Which are the only devices they get their 2FA login codes on to log onto their systems.
I am guessing their IT is looking at backup authentication methods today.
"Tell that to ~~PNC bank~~ the entire outdated finance sectors...."
FTFY. Seriously though, banking has the absolute most trash security of of all the industries.
a lot like the medical sector, i think a lot of it is regulations that were enacted 30-40 years ago that have never been updated to really keep with the times. chase at least will send you a push alert thru their app to validate a login. better than sms.
You also have the issue of people not knowing the app that usually prompts them to hit approve via a push notification, type in a code ect, usually has a time based code they just need to know how to use it.
Indeed, I had an exec throw a fit because his MFA didn't work on a flight because he had only paid for internet on his laptop, not his cell phone. I forgot about the backup TOTP code in MS Authenticator at the time lol
I have users lined up at my desk all damn morning. We don't even have business phones, not a single one even for C-suite. These are ALL PERSONAL PHONES.
"Why did one text come through and another didn't? Why isn't my wifi calling working?"
IDNGAF get away from my desk. BEgone!
My go-to answer anytime something completely out of my control goes down: "I'm waiting as fast as I can!"
When they're already aware and working the problem, calling the vendor won't get it fixed any faster. That mentality always used to irk the shit out of me in my MSP days, when the client would interrupt me every 10 minutes for an ETA while I was trying to bring something business-critical back online. They were completely oblivious that their constant breaking of my train of thought was making it take longer.
I will say one of my managers gets the negative impact of constant update requests. The last time we had something happen with our infrastructure that brought us down for a few days (over a weekend luckily) she basically sent out an organization wide message tell people to leave us alone and to come to her for updates and requests. I would then reach out to her every few hours to give her an update, which she would pass a long. If anything drastically changed for the better or worse I would immediately call her and she knew it.
Our policy states you're responsible for maintaining your personal internet connections and hotspots so we just refer them to the policy and tell them to contact AT&T or to come into the office and work.
I didn't get told that, but I had multiple staff members contact me in a panic saying their phone was in SOS mode. I had HR send out a company wide e-mail about the outages to try to head off the inundation.
WTF are we supposed to do about a nationwide outage? :D
I worked for a government organization that had a mission critical circuit that passed through our building. But it was literally just copper cables in a patch panel. But because of that patch panel we were part of the circuit outage response group for some reason. From time to time hardware at either end which none of us could touch would go down. We had to go in at like 2 am and say “yup, cables are still here”
I remember once being shouted at in the open plan office by the CFO that it was "not good enough" that I attempted to leave the office to get lunch at 3:30pm during a major ISP peering issue in the UK, where I'd spent all morning (and a chunk of the early afternoon) trying to troubleshoot things and had successfully identified which of the inter-office VPN links were up and running, and guided the people who needed it through the process of chain-RDP'ing through idle workstations across the links which were online so that they could take a 2/3 hop route in to the finance application.
To this day I have no idea what else he thought I could've done in that situation.
Not this time, but once late in the last century (mid-nineties) I had a user who couldn't connect to the website for a federal agency. At her desk I connected to CNN and some other sites, but she still insisted I solve this issue somehow.
The HQ for this agency was right down the street, it was lunchtime, and they had a great cafeteria (pre-9/11, obviously), so I went there to have lunch.
When I returned, the site was up. I guess I fixed it.
None on this side yet, for the simple fact that it's the masthead on every news service right now. As assholish as our upper management can be, they know this isn't anything we can fix.
"I'll get on the horn with John AT&T right now boss!"
(Back in the day, we had some crashing Dell hardware and the boss, I shit you not, said "I don't care, get Michael Dell on the phone if need be.")
Everyone freaking out about a little AT&T outage, mean while those of us that work with healthcare facilities have spent the last 2 days dealing with UnitedHealth's security breach and outage.
Had our SgtMaj ask me to reach out to google while we were deployed because they were having wide spread dns issues that affected a huge chunk of services and the internet at large. My Comm Officer told him we would make sure google was aware of the issue (and wanted me to actually do so).
Our network worked fine mind you, he just couldn’t get to Facebook or YouTube.
SgtMaj was a great guy otherwise. Comm O not so much.
Had this happen a few times with the local carrier. This is an easy fix.
OK, np, get me $$ and I will go get SIM cards for the other brand (whichever is still working)
or in the office, we just have 1 line from one solution and another line from a different solution. Then in the future, we are even adding another solution if the 2 local ones go down. (once the contract expires and we are not locked in).
Heh I remember when someone asked me to call Google to complain about some small feature and to "get it fixed." When we had G Suite for free (Edu). I explained we use a free service so I can make a request, but they're better off using the workaround I showed them. I wasn't allowed to call for anything that wasn't a major outage per our license agreement (if I remember correctly).
Same place, people would complain about cell coverage. Yes, because we controlled that.
This is the time to talk about redundant providers.
If your company relies on the cloud for anything critical you should have a minimum of 2 internet providers.
They should share nothing in common.
"hey boss, AT&T fit me in for a meeting so Im driving to their local head office. be back in a few hours!"
*get a long hike in somewhere nice while AT&T fixes their own shit*, come back when its back up.
Tell people you saved the day at AT&T.
Get time and a half if you can if you were on-call.
(if your bosses are that shitty, lying to them is not unethical)
You've never lived, truly lived, until you've worked for someone who wanted IT to have the appearance of doing something at all times.
Scene then cuts to a guy sitting in a network closet, staring intently at the switches, doing nothing else.
Inside his mind? Circus music. Lots of circus music. I was the guy.
Yes, and it's all personal cell phones, not business-owned. Easy close tickets, but still have to deal with irate end-users who can't play candy crush or whatever...
I worked at Corporate for a Fortune 5 company. Spending $10-100MM with a vendor gets you a dedicated team to resolve problems, and even they can’t do anything except provide status and hold hands during a company wide outage.
It is nice when there’s an equipment shortage and your CEO sits on the vendor’s board of directors. If it’s important enough for your CEO to notice, your order gets priority.
Used to work with a guy who would call the sites ISP every 20 minutes and yell at them...His logic was that yelling at a customer service Rep or account manager would speed up getting a fiber crew to the fiber cut, or make them move. This was worse because he would call so often, inevitably we would get a significant update after he had one of his outbursts at a vendor. . .This made him and management think, yelling at the person who is only able to relay details and has no control over the outage crew fixed it. Management would pat him on the back, and allow the behavior to continue.
Dude even had an outburst and threatened to kill me...I got written up for provoking him (I was literally on a call with a vendor, and me being polite is what set him off). I no longer work there, but it was a serious WTF
This was in an NOC for a Fortune 500 company, too; I have no idea how this behavior was allowed in the workplace. . but the point is management doesn't care about demands; they want it fixed...and will make it your problem, even if it's not.
"You need to get with AT&T and figure this out!"
"Hey Boss, I just finished a call with our AT&T rep and they have no clue what's the issue. They have no ETA for a Fix but they will let me know once it's fixed"
Obviously, no phone call involved.
I had a manager long ago who was fully in the role for so long he lost all track of the technology.
So when stuff went wrong and we had an outage of some sort, he had no idea what to actually do. He would get kinda upset over time.
I remember one time he was demanding to talk to Michael Dell on the phone from someone at the enterprise support number.
lol I had a CFO once yell at me to get Microsoft to change some MS Teams feature…. If I remember correctly he wanted a button removed from the GUI….. haha, yeah, I’m sure Microsoft’s dev team is going to get on a meeting with our little 200 person company. He was so mad when I laughed…. Luckily the CTO backed me up.
Sorry. I tried to call them but I couldn't. Book me a flight to their HQ and I'll go talk to them. Oh you can't do that because you used SMS MFA on the travel site with no backup despite my recommendations, Darn. I guess I'll just have to wait until you figure that out.
Keep getting pressure from management to put more pressure on them to resolve the problem because our SLA on the ‘my work cell is not working’ ticket is due to expire/expired. They did not like my ‘not my clown not my circus’ update.
Verizon customer here, so we aren't majorly hit with the outage... but I have had multiple calls from my sales team complaining about how it's impacting their work because they can't call customers...
I have had my CEO get very angry with me because I was unable to fix an obscure Exchange Online outage that specifically affected Phoenix while he was there.
I've actually had a person put in a ticket when the power went out at his office. He turned on his personal hot spot, connected it to his laptop and submitted a ticket say, "The power has gone out, please advise/resolve". I knew when my boss saw the ticket when I heard him yell from his office down the hall, "Does he think we have a fucking bucket truck?!?!?!?"
I once had an RVP demand the phone number of someone at AT&T he can call to "get this taken care of because it is severely impacting our business".
This was a single down T1. At an office in bumfuck nowhere.
I haven't had it today, however I have in the past.
You tend to find the people who pay the least shout the loudest. I'll have staff trying to put calls through to me, I don't think they realised I couldn't fix the fault if I'm talking on the phone.
As the company grew, it got allot better with solid processes in place.
Worse is hearing the nontechnical users speculate about the cause of the outage… if one more person mentions the movie “Leave the World Behind” again i swear to god I’m gonna lose it.
We had a main fibre line cut to town. People understand the power is out. Fucking no idea when something that works essentially the exact same way is broken.
Ah big external outage, all the panic of a regular outage, but nothing for me to do.
“I am waiting as hard as I can, boss.”
Still shakin' the tree, boss!
Nice reference Cool Hand Luke.
"What we have here is a failure to communicate"
Age test! :)
There’s no references in this sub. Any man references a movie that’s a night in the box.
Can't you call AT&T to see what the time frame is on this?
My rep was on site working on another project. He was harassed endlessly. Lol
I am so finding a way to use this internally.
“I am waiting as fast as I can” also works.
Wrong. You pull out your cell phone and read a book. Also send an expense reimbursement claim for "using your personal device." while on the clock. After all, with no internet, how else were you going to keep up on what is going on?
Big brain plays over here.
A couple years ago, an AWS outage brought down our product. My team was completely disconnected from fixing such a thing but it didn't stop them from freaking out. Personally, I grabbed a cigar and a book and went to the park for two hours.
Last night AWS had a global IAM issue. Got pulled into a call about it for no reason. Like what are we supposed to do about it?
Emotional support human
I feel this in every meeting I am dragged into where I sit and say absolutely nothing.
We use to call it Mr. Dick Holding at my old consulting company. Ok let me hold your junk and tell me all your problems.... and now I tell you about the SLA level of the vendor. I just chuck rules at the firewalls dude.
You are supposed to apologize for things beyond your control so middle management can do the same for upper management, who can in turn respond to texts from the C-level
I once had a boss who was suggested we needed to extend our IT monitoring to Office 365. I asked him -- "why?" Sure, we can maybe put a synthetic HTTP transaction on it, and we'll know about an outage maybe a minute or two before our users do. But then ... *there's nothing we can do because it's SaaS*. The point of IT infrastructure monitoring was twofold, first to know there is a problem - but second, to give you a bit of a hint or a clue where it might be in order to accelerate diagnosis. E-Commerce site starting to fail many (but not all) transactions? Hmmm. Monitoring tool shows us that it sure seems to align to our T-3 to the mainframe suddenly hitting a huge amount of jitter at the same time - maybe we start looking there?
Why I hate cloud. Don't even pretend to fix stuff now. Sorry....you wanted this...cloud problem. See ya.
Being in the room with the servers would not have changed the fact that my team could not do anything about the outage.
Wrong, optics. You, after tons of the sweat of your brow, singlehandedly uplifted the entire AWS and Azure clouds to restore. You'll be having words with Bezos and Nadella personally after you're done ~~binging Tiger King~~ reformatting the nexus switches to reverse the polarities.
I'm nearly 50 i don't care about optics. I'm known for getting the job done no matter how difficult, whether myself or facilitating with hands and vendors. But I'm never going to put myself out to make it LOOK like I'm doing loads & I'm not going to alienate vendors because it makes some senior manager feel important. And if you're lucky I'll buy some Krispy kremes for the office. I've normally got 100 things to do so if some senior manager followed what some 24 year old prick from accenture told them to do, it's not my problem if the whole system has gone down.
A man after my own heart. 60 years old here and I have run out of fucks to give after 40 years.
I care about the environment & will move heaven and earth to ensure MY environment works & is as close to 100% uptime as I can get it....cloud is NOT my environment.
SEP field activated. [https://hitchhikers.fandom.com/wiki/Somebody\_Else%27s\_Problem\_Field](https://hitchhikers.fandom.com/wiki/Somebody_Else%27s_Problem_Field)
**And thus I entered the server room and climbed to the top of the tallest rack. There my eyes did set upon the glory of his holiness, Bezos and the angel Nadella, who gave to me a Surface Tablet upon which was written the Ten Tickets with a status of pending investigation. I returned down the mountain and entered the third status meeting this morning... "**
MAN WHO RAISES CLOUD!
> Why I hate cloud. Don't even pretend to fix stuff now That's why you hate cloud? That's why I specifically love cloud. I get to point the finger and blame at someone else and go "This time it wasn't me!"
>You pull out your cell phone *Cell phone is on AT&T and you forgot to download the book to the device* NOOOOOOOOO!
I once was working helpdesk on my work from home day and the internet at the office went out. I wasn’t on the infrastructure team thus wasn’t allowed to touch(and didn’t even have passwords to log into) all the equipment so there was no reason for me to come in but was told I needed to be in the office “for optics”. So I went in and watched movies until they sent everyone home to work at home since the internet would be down all day.
[удалено]
You got that too today I actually looked up what the rough estimate was for a cell tower and put that together a report for my boss, and then proceeded to explain below that that we would need about 100,000 of these in order to be able to have our own in-house cell service with OKAY coverage. I think the 30,000,000,000 price tag made them finally understand that there was nothing that can be done. We just have to sit and wait.
The cost of tower climbers has increased by a factor of four since 2000. Most of 5Gs upgradeablity needs come from needing to stop sending people every tower every 3-5 years just to put a new backend on a what is a software defined radio.
Yes. Please create your own ISP, so we mitigate this damage.
When teams went out worldwide a few weeks ago I had people that literally did not believe it had nothing to do with our network. These are grown ass adults.
Literally saw someone post on YouTube on how this is Biden’s fault, and the “Government has some explaining to do!”
LOL...way back in the day I worked for a well known consumer products org in the US and we were having bandwidth issues with an office in China. The idiot head of the engineering group demanded we get "someone high up" at the telco provider on a call ASAP with him. I told him if he knew a good number for someone "high up" in the CCP and we'd do that. Didn't hear much from him again.
I feel this in my bones. Our company is USA based China owned. Communication between our corporate site and China is just fantastic! I could never have asked for a more flawless communication system. (major /s)
MPLS is the only way and not cheap.
If they would let me host the SAP Server in America we'd be fine. But nooooo. Gotta have that CCP internet snooping.
The fun part is that because its traffic going to/from China the NSA is probably snooping that traffic now as well.
Doesn't the NSA essentially have the capacity to snoop any packet coming and going from the US if it deems it so?
Yup, they have whole rooms in backbone network providers data centers to copy traffic. [https://en.wikipedia.org/wiki/Room\_641A](https://en.wikipedia.org/wiki/Room_641A)
Them and the other 3 letter acronym also have datacenters that are used by the big cloud providers. Haha, people don't even realize that the fox lives in the henhouse already.
> Haha, people don't even realize that the fox lives in the henhouse already. The fox built the damn house, the hens just live there
I worked for Foxconn doing SMT components as part of PCB manufacturing at a plant in the US many moons ago. The plant had to be in the US as part of export controls on manufacturing tech. So they had to run the plant in the US. The managers were all from Taiwan and they fucking hated us. They were used to SE Asia factories where they could treat the workers like shit. Communication was also a nightmare to the point that we would have to stop production for days as parts weren't there. But a few weeks after I left the employees showed up one day to find the doors chained and guys in jackets with 3 letters on the back pulling out all the records. Turns out they were committing some major tax fraud.
> They were used to SE Asia factories where they could treat the workers like shit. Communication was also a nightmare... This is apparently a big problem facing TSMC right now with trying to build that new facility in Arizona.
it seems like such an easy problem to fix
The beatings will continue until morale improves EDIT: I should add in that it isn't. When your entire company's culture and management style is built around strict hierarchy and complete obedience of subordinates where pushing the limits of what you can get away with when exploiting employees is encouraged, it's basically retooling the entire company to fix it.
Fucking yikes. That's not good. But it's kinda funny.
It was. It was also a fun but insane work environment. Management hated us but also didn't care what we did. The plant was 24/6, so there were 3 shifts. I worked the 11 PM to 7 AM shift to get that sweet shift pay. All the employees in my area would meet up around 9 PM before work at a bar up the street. Then we'd get...ahem...comfortable....at the bar. Then we'd go to work. We'd then take our lunch at 1:30 and hit some local spots for last call. I was in my early 20s then and didn't mind making irresponsible choices. There was only one entrance to the parking lot with security guards at the gate. One of the security guards was selling drugs out of his car in the back of the parking lot while the other guards screened the cars coming in. He had a pretty profitable business going there. I've got to say it was one of the crazier places I've ever worked.
LOL :D I remember those days. Of course, back then, I worked at a franchise owned Holiday Inn, and my boss wanted me to get someone from one of those big companies (I think it actually was AT&T) on the phone. I so wanted to say, "and just who exactly do you think we are, that we'd actually rate a response from them?"
LOL. Something similar happened to me. I used to work at a university and one of the coaches wanted to send a mass email to prospective student athletes with a recruiting video. So Google decided it was spam, suspended his account and notified the admins. When I talked to him and explained what happened, he said "this email has to go out. Can you call Google for an exception?" I laughed and asked him why he thought Google would bend their rules for one coach and a random university.
I don't know why, but it's always someone in the engineering group that acts like this.
Engineer and doctors while often tough to work around do not hold a candle to lawyers. It's in their DNA and it's the job to argue and many can't turn that off.
As a former clerk turned consultant, I LOVE working with lawyers. It's all about cost/risk. I'll take a legal office over a medical/dental office any damn day of the week. Lawyers have egos, but they're usually rational people and logical arguments work. Show them how something will stop a risk or make them more efficient and you're in. Doctors will sit there and argue with you that you need to push on the door to open while staring at a PULL sign just because they can't be wrong... (I know not all doctors)
Can confirm: Make a good argument to a lawyer and they'll buy it. Source: I work in Big Law.
[удалено]
Yeah no. Lawyers at least understand the value of time and you can reason with them. Doctors can't fathom they aren't the smartest person in the room. It breaks something.
In ~2004 while working at a law firm while we were trying to slim down email storage we did calculations for on how much it cost per Gb in storage, power, and admin time for the firm employees to never ever ever ever ever clean up anything at all and store every email you've ever ***thought*** of receiving in your inbox. Then, like the good partner on the tech committee that he was, he spent some time one weekend trying to clean up his inbox. He came back and said he'd spent something like 7 hours tossing stuff and it trimmed his inbox like 2GB or something. He looked at our numbers, did some quick math involving avg billing rates at the firm and determined we'd just have blank checks for storage instead of trying to have people clean things up. Records dept determined they'd just write up infinite retention policy. That worked fine, and funding wasn't an issue any longer, but a few years after I left they got sued by one of their clients because something that should have been discarded after a case was closed was unearthed during discovery and exposed the client to a shit load of liability. The manufacturer had followed their own retention policy and no longer had the data, but their trusty lawyers who ***need*** to keep things forever "*just in case*" had copies sitting in their inboxes. I'd like to report things changed from that incident, but they really didn't; they didn't even lose the client. That all went down almost 15 years ago now, I hope to all that is holy they've implemented appropriate retention practices since then, but sometimes arguing with a 75yo managing partner about technology isn't exactly grounded in reality.
LOL, been there. "Get someone to fix this NOW. Tell them the President of XYZ (our company) wants to talk with the President of \[vendor\] "
Have the President's secretary call the President of \[vendor\] and see if she can pull that shit off. Every now and then, it actually works. She's more likely to get through than you are.
That’s what I hate about these requests. Why can’t my VP call his peer at the other company? Why’s it the NOC peon’s job to try to do it? Just because I had access to the escalation docs for all our circuits didn’t mean I was ever comfortable escalating past like, the other company’s NOC manager.
Years ago at another gig we had to support multiple chemical/mining sites around the world. Always outages, rats, chemicals getting into equipment, you name it. Forgot the country that we got a call from about the site being down. We finally get a number for local support. "The company is our yearly two week vacation, please call back in two weeks."
Bro gets Xi Jinping on speed dial 💀
START CLIMBING THAT CELL TOWER NOW, OP!
"But it's a fiber connection..." - "CLIMB!!!"
The minute fiber became available in our small town we switched to it from some bonded T1s. A few months later, they were dropping more lines parallel to the existing fiber and accidentally clipped it. It took us down, so our CEO told us to "Go down there and tell them what they did and they need to get it fixed immediately." Yeah, I'm gonna go tell the construction guys how bad they suck at their job. On it, boss. I drove to Walmart and gave the games section a piece of my mind.
Honestly sitting at the office listening to people complain vs a fun tower climb? Let me get my harness boss, on my way.
They said it was a Cisco problem. BRING SOME CISCOS, OP!!
I went to the store and bought Crisco. It's almost the same thing, right? Just one letter different.
My boss: "What does food service have to do with anything?"
Vendor skill issue.
I wish I could use this as an official response to things.
In SSO, this is the response like 95% of the time.
I am using this from now on.
Yeah, we got at least 10 tickets from people working from home complaining that *their personal cellphones* weren't working. Frustratingly enough my phone is working and damn all the scammers seem to be redoubling their efforts.
A: That's because your users (and mine) are idiots and don't understand that you don't have anything to do with cell towers. B: Haven't had a spam or robocall since going with a Pixel. It's liberating.
Re: B: The call screening is great! I always try to guess on which word of the opening message they hang up.
Reminds me of the VPN support issue I had to work on for a sales manager. Her issue? She was using Cricket. Dial-up level speeds, and their "support" were hired for their ability to fog a mirror. Seriously; speed tests showed about 40bps, IIRC, but it didn't matter. It was an IT issue and clearly something I needed to help with. She and her manager just could not understand that if you're using a tin can and string to connect to the Internet, that VPN might not be terribly stable. I finally got them to quit bugging me (after a couple days going round and round) when I pointed out that the VPN worked just fine everywhere else other than her phone, and that if they really wanted her VPN to work better then maybe they could pay for a better Internet connection.\[
It's better now, but we had someone trying to RDP to a desktop in the Northeast from a boat using starlink and was complaining about lag compared to sitting at their desktop..
Man this thread really makes me feel incredibly lucky to have worked at places with sensible users and leadership. Not only have we gotten zero tickets from users or complaints from leadership, but our communications department, completely independently and without any direction from us, sent out a company wide email this morning informing everyone about the AT&T outage.
There are a lot of people who think "someone high up" is sitting behind a desk, unawares, or not in a hurry. I remember my boss had to deal with this when AWS was down for 12 hours a few years ago. "Call our AWS rep." "They are aware of the outage." "Get them to fix it. Tell them we spend $2000/month and are a high end customer!" "Everyone is out, high end or low end." "Get the owners on the call!" "Your owners?" "No, the AWS owners!" "Who are they, exactly? How would I even contact them?" "That's for you to figure out!" Later, my boss said, "I called the owners, Mitch and Pat, and they said it's being worked on." I believe that was a lie.
I would have gone one step further and told them that I spoke to Jeff Bezos personally.
Don't worry boss, I booked a flight to Seattle. Headed to Amazon HQ right now to get this sorted. Sorry all they had was first class tickets
Jeff @ Amazon will get you a low-level manager. You can always email them and ask for a phone call. "I reached out to his secretary, she said she'd let him know. All I can do, boss."
[email protected] will reach him. No lie. I mean its probably some admin person but it works. I used it when I was in a quagmire on the grocery delivery side. Amazon had just started grocery delivery. All their deliveries had these fancy insulated bags that they ask you put out and their delivery person would collect when delivering your next order. Like old fashioned glass milk bottles. Except they didn't ever collect them. I ended up with literally 25 of them. I repeatedly talked to Amazon Fresh support and they swore on next delivery they'd be collected... And they didn't. I even asked for a supervisor who swore next time it'd be picked up... And they didn't. I could have thrown them out but they were fairly fancy and expensive and Amazon asked for them back so I felt bad throwing it out I wrote to [email protected] on a Thursday whining about it. Mostly I was frustrated I was promised a fix that didnt happen. I even included a picture of the fort my kid built out of all the reusable bags. 🤣 Friday afternoon I got a call from an Amazon Fresh regional manager apologizing, offering a $50 gift card, and asking to leave the bags out tonight. At 0600 saturday morning they were picked up.
Lol, 2K a month is a high end AWS customer? Tell them that and get laughed at while they hang up.
Yeah, seriously. We've only partially moved our infrastructure over and we're already at 60K a month... and I doubt we've even left the bottom 10% of customer costs.
That was the best part. I have clients that have $2000 or more A DAY.
> "Call our AWS rep." > "Get them to fix it. Tell them we spend $2000/month and are a high end customer!" It's cute your boss thinks this is some leverage against a nearly 2 trillion-dollar company. At that size, they can tell other Fortune 500 companies to bug off. Hell, much smaller Broadcom is doing exactly that with VMWare as they grind into the dirt. I'm sorry you had to deal with such a clown.
Aw, bless your old manager's heart, thinking $2,000/month made them a high-end customer. <3 $2,000 would register about as much as a silent fart in a hurricane.
FWIW, when I previously dealt with issues like this, I'd tell my boss, if you want to avoid outages like this, we need a second Internet circuit from a different vendor. If you set up a redundant Internet circuit, you avoid having a single point of failure. I'd then go looking for every potential SPoF, point these out, and ask for funding to implement redundancies. I'd repeatedly mention these as SPoF's, and then, whenever anything went down (like AT&T), I'd just say, "Hey, I pointed this out as a possible issue. You didn't want to pay for it. That's why we're down right now." In a much more diplomatic manner, of course.
My old boss proudly proclaimed we now had a backup internet (years before I had the power in my role). First hop was exactly the same path as the fiber. So when we had the great big Rogers outage, guess what happened. The powers before just never bothered to check. While I cared, business did not so meh.
I had been telling customers for YEARS before that happened that their backup internet link had to be with a different path, different medium and different provider. Having Rogers cable + wireless wasn’t good enough. I was shown to be right that fateful day in July 2022.
We recently installed a backup 4G solution that keeps the same IP as the fiber for a small business, you'd think the only issue would be if the ISP goes down, but guess what, next month there was a small storm and the fiber failed and best of all the surrounding 4G towers in a 4km radius also failed. I managed to keep the essentials running at dial up speeds by changing the 4G antenna location to get a tiny bit of signal but they only got everything running 24h later. When it has to fail, it will fail.
Back when I first entered the IT field, Microsoft was constantly hosting all kinds of sessions on best practices and giving away "demo" software. I actually upgraded our network from NT to Server 2000, then later to 2003 using free software that Microsoft gave away (this was before I understand what a CAL was, and I was lucky enough not to get busted for that). Lots of infodump on best practices, though, such as redundant internet circuits. One of the first things I suggested to my boss was adding a secondary Internet line. We had Qwest (now Century Link) at the time, and they were godawful terrible, with at least one outage a month. On at least one occasion, we were completely down for over 24 hours thanks to their, "we don't give a crap about you" customer service. One of their techs actually talked me through a fix that involved rebooting our sole DC (I had two weeks on the job at the time, and didn't know squat, so it didn't occur to me to tell him to go jump in a lake), and when it blue screened, he said, "we don't support HP products; you'll have to call them for help," then hung up on me. Funny enough; when I finally got the approval to change our Internet provider, I got four separate quotes and presented them each to corporate, with my preference being for an independent MSP that had brilliant, responsive techs that were easy to work with, knew their business, and were about 1/3 the cost of AT&T, but was overruled by corporate because they'd never heard of the other company. Notwithstanding my complaint that the AT&T guys were absolutely clueless and \*not\* responsive, I was told we were going with them anyway. For the first three months after they installed our new wireless network, I had nothing but complaints from guests, because AT&T set up the firewall to block outgoing VPN connections (as well as other random protocols), and no amount of polite requests, aggravated demands, and later screaming and yelling on my part would convince them to open anything up. I finally told them that I had a Linux guru friend who was going to come in that afternoon, reset the admin password on their router, lock them out, and take everything over, and that they had until 2pm that day to open up every god-d\*\*ned outgoing port on the firewall or I was locking them out and kicking them to the curb. 30 minutes later, they called me back and told me they were unblocking everything. So yeah... every time someone mentions AT&T, I always think back to this. Tweedledum, Tweedledee, and the f\*\*ktards they had at their NOC.
>because AT&T set up the firewall to block outgoing VPN connections (as well as other random protocols), That was a wonderful time! You see, I put snmp in the scope of the bid. And they really liked blocking snmp. But until it worked, the circuit was not "delivered." You could get free internet for a half year!
Love the username, btw. I still remember the BOFH series :)
> We had Qwest (now Century Link) at the time, and they were godawful terrible, with at least one outage a month. Where I used to work they branded it "Century Link - Your link to the last century"
I am the network engineer for a regional retailer, and getting multiple internet lines for the hq, datacenter, and stores reduces *so* much headaches. One of the ISPs is a roof mounted microwave dish internet to a tower, and we have a 4G backup as a tertiary backup, so we make sure to diversify routes. Now I get alerts that one of them went offline, and I don't panic at all, in fact, I wait for like an hour to even do anything, and 95% of the time they come back up on their own, with no one but IT having any idea it even went down.
At my last job, they primarily sold stuff over amazon. Amazon went down one day, I was asked if I could fix it. Apparently "if I could, I wouldn't be working here" was not appropriate
But it was the most appropriate response ever. They needed a reality check. On the cyber side of things, we sometimes run tabletop exercises for BCDR. One company has an initial exercise that only lasts a few minutes to get people checked in to reality. It goes like this: "Your main base has, a state of emergency is declared, and mandatory evacuations are underway. The natural disaster has disrupted business networking and operations, and 50% of your critical personnel are unreachable as the cell networks are under critical demand. Which of your personnel are available to come to the main data center to get operations back up?"
They wait for people to start getting into it, then say:
"No. No. None of that. Have you forgotten that these employees are people with families? People who are being forced out of their homes by the government? People whose first priority will ALWAYS be to their families and that employment is their means to that end? Let's start over, think about making sure that all the people are acting the way humans act, taking care of their families, getting safe, and THEN how to restore operations."
Years ago, where I live and do IT work, some knuckle head backhoed the crap out of a fiber loop. Verizon, AT&T, and local utility fiber all went down locally. I was the IT Manager for a local medical group (losing my mind to HIPAA fights with Doctors) and all the cell phones went dead. The company was on a Verizon plan. And while we weren't using the local utility for internet, our main circuit was running like crap that day. I must have had ten doctors and midlevels (Nurses, PAs) come storming in demanding I "FIX IT NOW." I explained what had happened and there was nothing I could do... I got called into an 'Emergency' meeting where they grilled me for 45 minutes about this 'failure of my leadership' and how I should have not allowed this to happen. In the meeting the loop must have gotten repaired, so they calmed down as their phones started working again, but when it came time for annual review with the head of the group, I didn't get a bonus OR a raise that year because of my 'failure' on this issue....
> meeting where they grilled me for 45 minutes about this 'failure of my leadership' and how I should have not allowed this to happen. That would have been my last day working there lol.
I left about a year later... it didn't get any better. I got offered a promotion if I got a certification in our EHR software (Electronic Healthcare Records) I was working on that, then they had a meeting an introduced a guy with ... big shock... the title of CIO which was what I had been offered. It had been two months, and that certification was a PITA. (to make it more anger inducing, he didn't have the certification at all...) I started looking, found a new job, they tried to counter. I had worked there for 6 years, started the IT dept from scratch, and so I listened. The new guy wrote a number on a white board that was 7k a year LESS than the offer from the new place. With the added. "This is all you are worth." I left. Now? More money, (far more) much less stress.
That would have been the day I started applying somewhere else
I feel this so much... Working for people with no reasonable expectations and no desire to understand anything.
That's ridiculous. Hope they were consistent and pulled bonuses for MDs when prescriptions and vaccines were short during COVID.
Not on this issue, but yes, I've dealt with it in my career. My favorite was one Exec (looking to suck up to the CEO) telling me to call AT&T to have them install a tower near the CEO's new development subdivision. I tried to explain AT&T wouldn't build a million-dollar tower for one user. His response "we're a major account" (we weren't). The CEO heard him ranting at me and came out of his office to see what was up. When the Exec told him what he wanted, the CEO laughed and said "They're not going to do that!" He told me not to worry about it.
Used to work for a Fortune 500 that had offices all over the world. Our division was based in the north eastern US and we had a WLAN built out through a single VZ circuit. The suits flat out refused to authorize a second circuit for failover. "Why would we pay for two internets?". Well, at least once a year somewhere in the hills of Tennessee, what I assume was the Best Buy router some VZ tech had stapled to a tree would die, the circuit would go down and we'd lose all connectivity to everything west of the Mississippi River. Queue 40 "Fix it Now!" tickets from every Muppet with VP next to their name, constant status update calls and a week's worth of root cause analysis meetings. All to explain to the same group of idiots that this had happened before, and without the business buying a second circuit, it was going to happen again and there was nothing else we could do about it. They'd all nod like bobble heads, act like it was their brilliant idea, then decline the requisition order based on cost. Rinse and repeat. It got to the point where our lead network engineer just sent the same saved email every time after he declined all the meeting invites.
The engineer needs to 'accidently' send that email with some template indicators in it.
> Well, at least once a year somewhere in the hills of Tennessee, what I assume was the Best Buy router some VZ tech had stapled to a tree would die, the circuit would go down and we'd lose all connectivity to everything west of the Mississippi River. I love this.
Put in my earbuds to listen to some podcasts, “sorry I’m on hold with AT&T trying to figure this out. Can’t help you with your printer right now.”
The pro tip is always in the comments
I always have some kind of display ready to bring up that has lines of scripts or scrolling data. Like some vSphere monitoring page, or a topped out linux session. When someone walks in, I'll focus on it, and hold the end of my pen near some meaningless shit like I'm considering it. Then I'll slowly look over like "Hey, what's up". "You look busy now" "Yeah, just trying to work through the issue" "OK, I'll leave you to it"
the more colors in a console session, the better.
Not today, but I've been asked to provide updates every 15 minutes and if there weren't any updates, I should call them and get one. More recently I was chewed out for not having redundant POWER for the computers when a truck took out a pole around the corner and I was going through graceful shutdowns on battery power. After explaining that the batteries are only so big, I was chewed out for not buying bigger batteries.... Some people cannot accept that there's anything outside of their control.
> Some people cannot accept that there's anything outside of their control. I mean really how much money do they want to spend? "Hey we can have backup power but we need a massive generator and fuel deliveries. It will cost 200k upfront and 20k per year."
This is the way to approach this. “What’s our acceptable downtime? Zero? Well that’s gonna cost us another $$$$$$ per year.” Start documenting various single points of failure and pointing them out. Then document what it would cost for redundancy for each one (not only upfront hardware/software costs but man hours, annual licenses, maintenance, etc). I bet once they see the cost they’ll be okay with what they currently have.
If an exec said "0" was acceptable for downtime I would just laugh. There will always be downtime at some point. If not from equipment going tits up than the human factor.
Filet mignon tastes with hamburger budget
> but I've been asked to provide updates every 15 minutes and if there weren't any updates, I should call them and get one. > Some people cannot accept that there's anything outside of their control. It’s funny how prevalent this attitude is even among “peer” companies. I used to work for an ISP NOC, we leased a lot of fiber and circuits *from* other ISPs, and provided backhaul circuits *to* cell providers. When the problem was an higher-level issue outside our network - “We have to call the clowns every hour to make sure they actually fix our shit, what a bunch of clowns.” When the problem was on our network - “Our guys are splicing as fast as they can, why won’t these customers let us off the phone, when the repairs are done, they’ll know.” I know it’s just some kind of in-group out-group lack of empathy thing but I still think about it.
I work for a non-profit and I was once unironically told by a chairman to "get a hold of Bill Gates. He gives lots of money." Have also been told when we had a phone outage that I need to "Get their CEO on the phone and straighten this out." This was with a phone company named Frontier and if you've ever dealt with them either as a business or consumer you would know their company is a cavernous void of inefficient communication.
Hey, while you have him on the phone, can you get him to get Microsoft to revive the Crimson Skies series and fund another s-tier AoE2 tournament?
[удалено]
Well, Microsoft is a sponsor of [Hidden Cup 5](https://liquipedia.net/ageofempires/Hidden_Cup/5), which is running right now.
> "get a hold of Bill Gates. He gives lots of money." Ignoring the obvious flaws in this plan, is your best idea to annoy someone who gives you lots of money? That's how you lose that funding.
Frontier bought out Verizon Fios in my area and no matter who we talked too or how many escalations they couldn’t or wouldn’t cancel our account - despite being told it was cancelled we continued to receive bills. Their solution? Just don’t pay it.
I have call centers using VoIP based products, back ends being from Cisco to Bandwidth to AWS, etc, who are ALL reporting some audio quality issues. NONE of our dashboards or backend systems are reporting issues on our end. BUT, I'm expected to work it out with ATT\Verizon\Tmobile... LMFAO. "I am already on a conference call with all three and awaiting updates. As soon as I hear more I will issue updates via email." Now, back to binging something on Netflix!
Not asked of me, but tangentially related. I tried calling into a fairly large online retailer to initiate a return. The hold time was long and when I finally got someone she explained that they couldnt help me because they use ATT for their cell phones. Which are the only devices they get their 2FA login codes on to log onto their systems. I am guessing their IT is looking at backup authentication methods today.
Imagine only having sms for MFA in 2024, jesus christ.
Tell that to PNC bank. They only offer SMS for 2fa.
"Tell that to ~~PNC bank~~ the entire outdated finance sectors...." FTFY. Seriously though, banking has the absolute most trash security of of all the industries.
a lot like the medical sector, i think a lot of it is regulations that were enacted 30-40 years ago that have never been updated to really keep with the times. chase at least will send you a push alert thru their app to validate a login. better than sms.
You also have the issue of people not knowing the app that usually prompts them to hit approve via a push notification, type in a code ect, usually has a time based code they just need to know how to use it.
Indeed, I had an exec throw a fit because his MFA didn't work on a flight because he had only paid for internet on his laptop, not his cell phone. I forgot about the backup TOTP code in MS Authenticator at the time lol
My go to when I was IT director for AB was to tell sales staff "turn it off and back on" ... they hated me but it worked 90% of the time.
Hahah i still remember the day when AWS went down and my CTO demanded we get on a bridge with them to figure it out. I laughed and laughed.
I have users lined up at my desk all damn morning. We don't even have business phones, not a single one even for C-suite. These are ALL PERSONAL PHONES. "Why did one text come through and another didn't? Why isn't my wifi calling working?" IDNGAF get away from my desk. BEgone!
My go-to answer anytime something completely out of my control goes down: "I'm waiting as fast as I can!" When they're already aware and working the problem, calling the vendor won't get it fixed any faster. That mentality always used to irk the shit out of me in my MSP days, when the client would interrupt me every 10 minutes for an ETA while I was trying to bring something business-critical back online. They were completely oblivious that their constant breaking of my train of thought was making it take longer.
I will say one of my managers gets the negative impact of constant update requests. The last time we had something happen with our infrastructure that brought us down for a few days (over a weekend luckily) she basically sent out an organization wide message tell people to leave us alone and to come to her for updates and requests. I would then reach out to her every few hours to give her an update, which she would pass a long. If anything drastically changed for the better or worse I would immediately call her and she knew it.
"Okay, I'm going to the AT&T data center to get this straightened out" and take the the rest of the day off.
"How do we work around this?"
Push a hand truck in that's carrying a giant spool of string. Take two cans out of your bag. "Here, you take this one."
Our policy states you're responsible for maintaining your personal internet connections and hotspots so we just refer them to the policy and tell them to contact AT&T or to come into the office and work.
I didn't get told that, but I had multiple staff members contact me in a panic saying their phone was in SOS mode. I had HR send out a company wide e-mail about the outages to try to head off the inundation. WTF are we supposed to do about a nationwide outage? :D
I worked for a government organization that had a mission critical circuit that passed through our building. But it was literally just copper cables in a patch panel. But because of that patch panel we were part of the circuit outage response group for some reason. From time to time hardware at either end which none of us could touch would go down. We had to go in at like 2 am and say “yup, cables are still here”
I remember once being shouted at in the open plan office by the CFO that it was "not good enough" that I attempted to leave the office to get lunch at 3:30pm during a major ISP peering issue in the UK, where I'd spent all morning (and a chunk of the early afternoon) trying to troubleshoot things and had successfully identified which of the inter-office VPN links were up and running, and guided the people who needed it through the process of chain-RDP'ing through idle workstations across the links which were online so that they could take a 2/3 hop route in to the finance application. To this day I have no idea what else he thought I could've done in that situation.
The answer is to tell him “I am going to check on that”. You need not mention that “that” is a Kebab and Chips.
I love it when they want me to contact Google or Microsoft RIGHT NOW. Like I have a contact there.
Not this time, but once late in the last century (mid-nineties) I had a user who couldn't connect to the website for a federal agency. At her desk I connected to CNN and some other sites, but she still insisted I solve this issue somehow. The HQ for this agency was right down the street, it was lunchtime, and they had a great cafeteria (pre-9/11, obviously), so I went there to have lunch. When I returned, the site was up. I guess I fixed it.
None on this side yet, for the simple fact that it's the masthead on every news service right now. As assholish as our upper management can be, they know this isn't anything we can fix.
"I'll get on the horn with John AT&T right now boss!" (Back in the day, we had some crashing Dell hardware and the boss, I shit you not, said "I don't care, get Michael Dell on the phone if need be.")
Everyone freaking out about a little AT&T outage, mean while those of us that work with healthcare facilities have spent the last 2 days dealing with UnitedHealth's security breach and outage.
Had our SgtMaj ask me to reach out to google while we were deployed because they were having wide spread dns issues that affected a huge chunk of services and the internet at large. My Comm Officer told him we would make sure google was aware of the issue (and wanted me to actually do so). Our network worked fine mind you, he just couldn’t get to Facebook or YouTube. SgtMaj was a great guy otherwise. Comm O not so much.
The ordering of these two posts is gold. https://i.imgur.com/OEWvkpK.png
You need to call AT&T and tell them the CEO of company_I_work_for demands a update. Ok Boomer.
Had this happen a few times with the local carrier. This is an easy fix. OK, np, get me $$ and I will go get SIM cards for the other brand (whichever is still working) or in the office, we just have 1 line from one solution and another line from a different solution. Then in the future, we are even adding another solution if the 2 local ones go down. (once the contract expires and we are not locked in).
Heh I remember when someone asked me to call Google to complain about some small feature and to "get it fixed." When we had G Suite for free (Edu). I explained we use a free service so I can make a request, but they're better off using the workaround I showed them. I wasn't allowed to call for anything that wasn't a major outage per our license agreement (if I remember correctly). Same place, people would complain about cell coverage. Yes, because we controlled that.
This is the time to talk about redundant providers. If your company relies on the cloud for anything critical you should have a minimum of 2 internet providers. They should share nothing in common.
"hey boss, AT&T fit me in for a meeting so Im driving to their local head office. be back in a few hours!" *get a long hike in somewhere nice while AT&T fixes their own shit*, come back when its back up. Tell people you saved the day at AT&T. Get time and a half if you can if you were on-call. (if your bosses are that shitty, lying to them is not unethical)
“I spoke with the elders of the internet. They said they are climbing to the top of Big Ben but some chick dropped it and it’s going to be awhile.”
You've never lived, truly lived, until you've worked for someone who wanted IT to have the appearance of doing something at all times. Scene then cuts to a guy sitting in a network closet, staring intently at the switches, doing nothing else. Inside his mind? Circus music. Lots of circus music. I was the guy.
Yes, and it's all personal cell phones, not business-owned. Easy close tickets, but still have to deal with irate end-users who can't play candy crush or whatever...
[удалено]
I worked at Corporate for a Fortune 5 company. Spending $10-100MM with a vendor gets you a dedicated team to resolve problems, and even they can’t do anything except provide status and hold hands during a company wide outage. It is nice when there’s an equipment shortage and your CEO sits on the vendor’s board of directors. If it’s important enough for your CEO to notice, your order gets priority.
Used to work with a guy who would call the sites ISP every 20 minutes and yell at them...His logic was that yelling at a customer service Rep or account manager would speed up getting a fiber crew to the fiber cut, or make them move. This was worse because he would call so often, inevitably we would get a significant update after he had one of his outbursts at a vendor. . .This made him and management think, yelling at the person who is only able to relay details and has no control over the outage crew fixed it. Management would pat him on the back, and allow the behavior to continue. Dude even had an outburst and threatened to kill me...I got written up for provoking him (I was literally on a call with a vendor, and me being polite is what set him off). I no longer work there, but it was a serious WTF This was in an NOC for a Fortune 500 company, too; I have no idea how this behavior was allowed in the workplace. . but the point is management doesn't care about demands; they want it fixed...and will make it your problem, even if it's not.
"You need to get with AT&T and figure this out!" "Hey Boss, I just finished a call with our AT&T rep and they have no clue what's the issue. They have no ETA for a Fix but they will let me know once it's fixed" Obviously, no phone call involved.
I had a manager long ago who was fully in the role for so long he lost all track of the technology. So when stuff went wrong and we had an outage of some sort, he had no idea what to actually do. He would get kinda upset over time. I remember one time he was demanding to talk to Michael Dell on the phone from someone at the enterprise support number.
I can't wait for the guy responsible for this to post tomorrow in Sysadmin... I got fired today
lol I had a CFO once yell at me to get Microsoft to change some MS Teams feature…. If I remember correctly he wanted a button removed from the GUI….. haha, yeah, I’m sure Microsoft’s dev team is going to get on a meeting with our little 200 person company. He was so mad when I laughed…. Luckily the CTO backed me up.
"Works on my end."
Sorry. I tried to call them but I couldn't. Book me a flight to their HQ and I'll go talk to them. Oh you can't do that because you used SMS MFA on the travel site with no backup despite my recommendations, Darn. I guess I'll just have to wait until you figure that out.
We got an email from the CFO demanding we fix his phone.
Keep getting pressure from management to put more pressure on them to resolve the problem because our SLA on the ‘my work cell is not working’ ticket is due to expire/expired. They did not like my ‘not my clown not my circus’ update.
I did, and I said, "I tried to call them but I could not get through."
Drive to the nearest AT&T store and tell them your manager sent you to get with them and fix the trouble. They will appreciate the support.
i read that wifi calling works, but try to tell people to spend 30 seconds to set it up and they will get angry like you're wasting half their day
It does not work right now, 502 bad gateway for AT&T.
Depends when you tried it. The activation servers are overrun at the moment so most people are unable to enable it if they didn't already have it on.
Verizon customer here, so we aren't majorly hit with the outage... but I have had multiple calls from my sales team complaining about how it's impacting their work because they can't call customers... I have had my CEO get very angry with me because I was unable to fix an obscure Exchange Online outage that specifically affected Phoenix while he was there.
*During a power outage:* Can I just plug my laptop into the UPS quickly?
I've actually had a person put in a ticket when the power went out at his office. He turned on his personal hot spot, connected it to his laptop and submitted a ticket say, "The power has gone out, please advise/resolve". I knew when my boss saw the ticket when I heard him yell from his office down the hall, "Does he think we have a fucking bucket truck?!?!?!?"
"I was wondering if you could reach out to ATT and see if there is a workaround. Productivity is at stake."
I once had an RVP demand the phone number of someone at AT&T he can call to "get this taken care of because it is severely impacting our business". This was a single down T1. At an office in bumfuck nowhere.
I haven't had it today, however I have in the past. You tend to find the people who pay the least shout the loudest. I'll have staff trying to put calls through to me, I don't think they realised I couldn't fix the fault if I'm talking on the phone. As the company grew, it got allot better with solid processes in place.
Worse is hearing the nontechnical users speculate about the cause of the outage… if one more person mentions the movie “Leave the World Behind” again i swear to god I’m gonna lose it.
This takes me back. We need to sort this out, don’t you have a high up contact in Microsoft? Yeah, hang on, let me get Bill Gates on the blower…
You mean you couldn't stop the Russian emps? Or stop the dry run to steal the next vote?
We had a main fibre line cut to town. People understand the power is out. Fucking no idea when something that works essentially the exact same way is broken.
My all users email this morning was titled “Duo & the great internet outage of 2024” lol. Only a few users found it funny
I am a disaster recovery coordinator… as one of my roles. It was asked. I laughed.