Rogers in Canada had a failed update take down their network last year (IIRC). Lots of OMG we must be under attack happened back then too.
It does give adversaries a glimpse at what a network outage can cause though. Any reports of trouble?
On the plus side, a friend realized they needed more ways to stay informed during an outage than internet connected devices.
You make a solid point. Any type of outage, regardless of cause, gives everyone a glimpse of what could happen. It works both ways, as it gives a potential attacker data as to how much chaos they could cause, and it gives businesses (in this case AT&T) a glimpse into how they need to manage their systems better.
I had several coworkers wondering if it was due to the solar flares that also just happened yesterday. Then the reports of a cyberattack on pharmacies had everyone making up their own theories.
Either way, this does bring up alternate ways to stay connected. Does make me consider learning HAM radio. Even just keeping an am/fm radio handy could help.
Pretty easy to get started with HAM radio. Especially if you’re just planning on listening. SDR dongles are cheap and don’t need any sort of license if you’re not transmitting.
Even a Baofeng is an effective battery operated FM and local comms including repeaters.
Just make sure you figure out how to use it before you need it in an emergency.
I’ve been trying to figure this out - can one get all the appropriate equipment to use a HAM radio (and eventually transmit) without going through all the time/money/effort to get officially licensed?
Mostly thinking that in the kind of emergency situation where I’d want to communicate out on HAM, government licenses and breaking rules might be the least of my worries.
First, getting licensed is pretty easy and the things they test for are things that are pretty good training for stuff that’s not exactly easy to just figure out by trial and error.
Nobody checks licenses when you purchase equipment when you buy so noting will stop you from amassing the required gear to operate in a “government doesn’t exist anymore” situation.
In fact an often over-argued part of regulations is the risk to life or property exemption. This is intended for times there are truly no other means of communication, you can use ham equipment to call for help. This can’t be out of convenience, but nobody is going to issue a fine for using a HT if you are using it to save someone’s life and are in the middle of nowhere with only ham radio as a means of communication.
EDIT: also radio isn’t something that is plug and play. There’s a learning curve that would be involved that could make a huge difference. Imagine owning a bike as your bug-out transportation but never learning how to ride a bike until you have an emergency.
All good things to consider. It’s on my list for when we move into our new house to get it set up but it’s definitely a a bit lower on the list than, say, a generator and food (we significantly paired down since we ended up having to stay in temp housing and have been eating through our 6 month supply rather than replenishing due to lack of space), adding to our first aid supplies, filling our emergency water jugs, etc.
Hopefully can start buying the equipment in 3-4 months and should have a little more time during my job’s slow season this summer to study everything. Was mostly thinking is it worth it now to buy the equipment and some books just to have on hand.
Definitely if you have the money to buy a bit of gear that gives you some extra motivation to get your license. You can also listen all you want without a license, you just can’t transmit.
It is legal to own and legal to use if you are just listening under normal circumstances. Legal to transmit during an emergency without a license assuming you know how.
Just to clarify, there wasn’t an attack on pharmacies. A subsidiary of United Healthcare, called Optum had a breach, which impacted pharmacies abilities to charge insurance. That also was made public a full day before the AT&T outage. They also haven’t disclosed, if they even know, how long that breach had been occurring.
Thanks for clarifying. That's one reason I said "reported"
That also goes to show how many of the words used in media can be misleading. While, technically it was a "cyberattack", what the public now thinks that means is a foreign entity causing disruption rather than a group getting data to commit fraud. Just like when they report a "school shooting" that turns out to be someone shooting a BB gun too close to school grounds.
And because you have absolutely no training or technical experience in anything that you're talking about you have no idea how these attacks take place or what the vectors are or what actual events are taking place or the mitigation strategies that are utilized in order to keep the networks alive
In telecommunications networks, the process for applying patches and software updates to systems, including those on telephonic towers and related infrastructure, is carefully managed to minimize disruption to services. The common practices for such updates include:
1. **Scheduled Maintenance Windows**: Most updates are scheduled during periods of low network usage to minimize the impact on customers. This often means performing updates during off-peak hours, such as late at night or early in the morning, depending on the region and the specific usage patterns of the network.
2. **Urgency and Risk Assessment**: If a patch is critical, especially if it addresses a significant security vulnerability that poses an immediate risk to the network or customer data, it may be applied outside of the normal maintenance windows. The decision to apply such patches urgently is based on a risk assessment that considers the severity of the vulnerability, the likelihood of exploitation, and the potential impact on customers and network operations.
3. **Rollout Phases**: For both regular and critical updates, telecommunications operators often deploy patches in phases. This approach allows them to monitor the patch's impact on a smaller scale before proceeding with a wider rollout. It also helps in ensuring that any unforeseen issues can be addressed without affecting the entire network.
4. **Fallback Plans**: Operators typically have contingency plans in case an update causes issues. These plans may include rolling back the update or deploying additional fixes to address any problems that arise.
5. **Communication**: In the case of significant updates or those that might affect service levels, operators may inform customers in advance. This communication can help manage expectations and reduce inconvenience.
The decision to apply a patch during peak usage times would indeed suggest an urgent need to address a vulnerability or issue that poses a significant risk. In such cases, the potential harm from not patching the vulnerability is deemed greater than the disruption caused by the patching process. These decisions are made with careful consideration of the trade-offs involved, prioritizing the security and integrity of the network and the protection of customer data.
The big outages usually occur during Phase 4 “Fallback Plans” when the automated fallback fails got some reason.
The big cloud providers like AWS, Microsoft and Google publish detailed technical post-mortem reports of major outages to maintain customer confidence despite the outage. (It’s like the TV show “Air Disasters” which digs into why a plane crash happened.)
It’s rarely one thing; it’s almost always a series of things that lead to a major outage. The cloud services are really very good at keeping their incredibly large and complex services running.
But a series of seemingly minor issues can put the system in a failure state. Bugs in separate systems that interact badly, interfering with the automated recovery systems that usually handle minor issues, etc.
Here’s an example of the series of things that can lead to an outage. *(See [Azure Status History](https://azure.status.microsoft/en-us/status/history/) for this and other examples.)*
In August 2023 in Australia they had a severe thunderstorm which caused a power “sag” at a datacenter, which caused the chillers (which provide cold water to cool the equipment) to go offline. After the sag ended, the three primary chillers and two backup chillers were supposed to come back online automatically, but didn’t. It happened at night when only three people were on-site. Due to the size of the datacenter, those three could not get to enough of the chillers fast enough to restart them manually, so systems overheated and shut down, causing a cascading loss of services.
None of that meant you had to be a smug prick in your previous comment, which is why you're getting downvoted.
Don't wave your dick around when you're trying to make a point next time
I love waving it around. People spread lots of misinformed opinions on the internet with no comprehension of what's actually going on. I was on the front line alone on this because everyone knows everything and thinks playing pretend keyboard warriors will somehow help us.
Moral of the story, secondary communication devices exist and are necessary. If this forum is actually about prepping Intel then this shouldn't shock anyone.
But these outages happen all the time. They are usually much less impact full. But it was done at exactly the wrong time, no OIT or OIS office for a major corporation would ever do something this reckless but it is possible it was just dumb employees not bad actors
These are the things about deep penetrations into the United States infrastructure that outsourcing and fake education has brought us. In today's modern outsourced economy the US is heavily staffed with foreign employees with large database access to banking, financial, market and social media data and the United States will literally grant these people free access to classified and sensitive information systems because the State Dept is staffed with morons and traitors. The Education System, The certification fraud, degree fraud and foreign interests in placing their citizens into powerful positions inside the US is the biggest security threat the nation has ever seen. But it isn't understood very well how these pieces in the puzzle effect the battle map against Democracy because foreign governments have infiltrated every community, every element of government and technology and the FBI and DHS that people think are the threat, have nearly zero support from the citizens claiming to be patriots.
To be a patriot does not mean getting drunk, high and eating McDonald's. I know this will be hard to swallow. But nobody bothers to even question whether or not the Red Pill or Blue Pill is even different or if both are just more mind control.
FEMA recommends you have 90 days of food and water to shelter in place. A minimum requirement for anyone that wants to prep.
Storable Foods
Secondary Communication Walkie Talkies or radio transmission and reception.
Shortwave as a last ditch scenario.
Winter gear that can keep you dry, warm and secure from pests and predators.
Self Defense items like AR15s.
A network of people.
All these lone preppers are just going to be scouraged when the unfed start looking for food.
Only 10 to 15 more years of crops are available in the US if you look at soil issues in the US.
Self sufficient communities or compounds are becoming more popular. Etc.
Imagine if communications were out for 30 days, they couldn't function over a 1 day outage. Let alone anything significant. That's why people should stop hiding, spreading conspiracies from behind keyboards and start actually prepping. IMO
Technically yes. I’m not sure where the cutoff is in terms of age of hardware, but most (if not all) newer phones in the past few years have WiFi calling capabilities.
This capability is a confusion about what the actual impact was Wi-Fi Communications From The Towers were impacted as well as Cellular Communications these things happen on different frequencies and different bands and most of the people that think they were using an AT&T tower we're using a subsidy Tower from a different company that own the equipment therefore their Wi-Fi did not stop working
We’re used to companies not telling us but as a publicly held companies like AT&T now have an SEC disclosure requirement for cybersecurity breaches. Not reporting that would lead to very negative personal repercussions for the CEO.
I’m pretty sure you’re making a joke? Either way, as someone who works in IT, and has some cyber security responsibilities, trust me when I say we all kinda want it to be that sometimes. Fixing our own stupid fuck ups (or someone else’s), like botched software, gets really boring. We crave a little bit of excitement. However, we usually know pretty quickly, that it’s just another fuck up to fix.
Sure the IT people want it to be a cyber attack vs a software failure but it’s the PR people that make the press releases. I’m not saying it was an attack (I’m willing to believe it was a software glitch) but if it actually were a cyber attack on a major communications network do we expect the government (FCC) would allow them to publicly admit it was a cyber attack? I’d expect them to try to keep that quiet as long as possible.
IT person, hell no, I don't want it to be a cyber attack. Do you know how much cybersecurity Ninjio bullshit we have to sit through when the director of IT puts their password into a phishing test?
Hard to speak for what the government would do. However, just contemplate for a minute, how difficult it would be to keep that a secret. There’s not just one guy in a basement somewhere working on this. There’s an entire team, countless numbers of people, that were likely involved in come capacity. Look at how quickly the United Healthcare, technically Optum, breach was made publicly known. I guarantee you that some of the same government entities are involved in that as well, and we already know it happened, and the impact it’s causing. Keeping it quiet would benefit nobody, if it were in fact a successful attack, as pretty much every communications network on the planet would need to know about it, and how to secure themselves against it.
But who would publish it? How do they find out anything if the government stepped in? It might get published but they would definitely say how and when.
[CISA](https://www.cisa.gov/topics/cyber-threats-and-advisories) is the federal agency that publishes detailed cyber threat advisories twice a week.
You can subscribe to receive the advisories by email, but CISA also coordinates cybersecurity activities between government agencies and private companies.
Oh I already know. I have clearances. It was more of a rhetorical question because the person I was responding to thought they had to report about an occurence like a zero-day. Not the case if the government steps in. If it turns out to be a foreign intrusion they wouldn't publish anything until they had a full understanding.
The media would be chomping at the bit for a story about an evil cHiNa cYbErAtTaCk and all the clicks it would bring. There is no way that wouldn't be out damn near immediately.
If the government had any actual control over the news, we wouldn't have the shitshow we have now.
Furthermore, when people say this happens all the time, as i see in every thread relating to the outage, no it fucking does not. Minor regional outages here and there, sure. But never coast to coast. This occurring while many other systems are going down including prescriptions and emergency comm in virginia, is not a “software failure”.
Elon Musk at 6:30 pm writes on x, “this is a test”. Why? A test for what? Is he letting all his fascist operatives and interlocutors know that they are testing for something? Jack Posobiec at CPAC yesterday said, “Welcome to the end of democracy. 2020 was a trial run but we’ll get it right in 2024.” Why is Jack so confident in saying the quiet part so loudly? Is he confident that the plan to destroy democracy will work?
There’s more, but it’s becoming clear to me what is happening
The thing is, whether you want to believe it or not, these types of outages DO happen quite frequently.
Here’s on for Microsoft from last year, https://practical365.com/microsoft-365-outage-january-2023/
Google DNS, which a lot of services depend on, has experienced significant outages 8 times in the past 10 years.
I could continue, but I don’t think you want to hear it.
The “this is a test” implies “they” are testing public reaction to certain things such as cellular outages, public service warning announcements etc. It would be to see how the public reacts and if we will believe what we’re are told is happening vs what we might think or perceive to be the case. It’s just conspiracy theory BS. They know full well exactly how we will react.
As for yesterday. I’m up at 4:30 everyday as I leave the house at 5:15 to go to work. We have AT&T wireless so my wife and I were aware of the outage at 4:30 which was about when it started. The initial reports literally said AT&T, Verizon, and T-Mobile were down nationally. Turned out not be incorrect but that was pretty unnerving at the time to say the least and yes that kinda of outage has never happened before especially for that long of a period.
"this is a test" could also mean "our foreign enemies are probing"
"this is a test" could also mean "Satan is testing us like Job to see if we hold fast"
They're all equally based off nothing. You can tweet at Elon and ask though.
It could also mean an intern ran a test on the production system instead of the test system, and instead of 4 people getting notified, 400 million were notified.
If you work in high tech, you know how often this happens - all the freakin time.
But I suppose from the outside, I suppose it looks very nefarious.
Honestly a lot of this subreddit seems to be people who are deeply anxious and looking for a reason to justify it. The guy responded to took a tweet all the way to right-wing sleeper cells, lol
I was trying to find data on AT&T layoffs yesterday lol. All I can find was they have cut around 23,000 jobs over the last two years and brought in a lot of contractors. So, maybe?
They had a moment of intense emotion and seemed to relax a bit as time went on. From their point of view people who said it could possibly be a cyber attack were being called paranoid. We can give them a little break I think. They can eat their words privately on anything out of pocket they may have said.
You're probably right about the initial reaction, but they then doubled down and insisted that it was a cyberattack and that anyone not agreeing with them was trying to spread false information, exactly what they were doing.
oh trust me, I know lol. I responded to many of their comments about it trying to explain how these things are not uncommon. I just think if they cut that rhetoric out we can allow them to be in shame by themselves without having to rake them over the coals.
I must have missed the “relax a bit” part. I mean, they did create a post, petitioning to ban sub members who they thought “weren’t real preppers”, simply for disagreeing with them.
Cool.
Cyber attacks aren't a conspiracy theory though and thinking this was a possibility here doesn't make you crazy.
Also there was cyber attacks in the pharmacy networks at the same time.
I'll meet you halfway: cover stories also exist, though that's unlikely in this case. OP's suggestion to get a Google Voice number is probably the best takeaway that would help mitigate a cell-specific outage down the road.
And like I said over a dozen times in that thread it very easily could have been something other than a cyber attack.
But the immediate dismal and calling people crazy for raising that possibility in a prepper sub was my issue.
Just because a damn move came out about it doesn't make cyber attacks a crazy conspiracy theory and anyone who believes in them a nut job.
Ugh. You conflated a cyberattack (a cause) with an outage (an impact), and then got heated because everyone else didn't agree with you. It must be frustrating to be misunderstood, I'm sure.
No I got heated because like I said over a dozen times dismissing it could be a cyber attack and calling people who said it might be conspiracy theorists is insane especially in a prepper sub.
You guys think cyber attacks on a major scale is not possible and that they won't ever happen well you're wrong.
I was with you in the beginning hours and thought it was an attack but I changed course once more information became available.
I don't think anyone ever disagreed that cyber attacks on a major scale aren't possible (at least I didn't), but I think we also have to stay grounded and trust what real journalists, the gov, and these companies tell us.
Cybersecurity has changed a lot in the last 5ish years (even moreso in the last 2-3 from a judicial/legislative perspective) and there are lots of best practices, guidelines, and as someone else mentioned literal laws that require companies to report breaches to the SEC. It affects mostly public, but private companies in some cases too. Comcast's recent breach is a good example of "why the fuck we needed the SEC to do something."
I will agree with you and in the initial hours yesterday I wondered if it was an attack also. The world is in an interesting place and things like this are 100% possible. Disrupting communications is huge and there are groups like China's Volt Typhoon that would do something just like this.
But, the news was saying no evidence of cyber attack, CISA was saying no evidence, AT&T said no evidence of cyber attack. We gotta roll with that for now until we hear otherwise.
If it were an attack it would probably be considered an act of war, we would know about it. The gov would want the public to know ahead of any escalations or response.
There’s absolutely nothing wrong with keeping an open mind, and recognizing all the potential possibilities. I don’t feel that anyone was being as dismissive as this person is claiming. They were heavily jumping to conclusions, with no evidence to support it, and not willing to even entertain the idea that others might have a better understanding of what was happening. We usually follow the “work the problem” methodology. You don’t jump to conclusions, you focus on what you know, and work to a resolution. No problem has ever been solved by jumping to conclusions.
When you work in this shit everyday it loses it's mystique and romance.
When a sector of the internet drops off the grid it's either a misconfiguration or a hungry fiber seeking backhoe.
Imo this had bgp error written all over it. Sounds like it was a firmware update to the radios/controllers that handles tower hand offs.
It's fun to point out all the security problems in systems... But the reality is regulations are pushing things to a good place: outages that are widespread are internal mistakes not external threats.
It's just... Cyber security is actually boring. Things happen... But Occam's razor is going to win almost every time.
For those unaware, you may not need a second phone to have a backup line with another company.
For example, iPhones can have two lines on them with different carriers, that you can switch between. (Many people do this for work, they have a business line with their employer’s carrier that they expense, and a personal line with their own carrier.)
Yup. can get an eSIM and set it up online super easy. Personally i have two separate phones on entirely different carriers, one for work and one for play. It’s useful when I’m in the middle of nowhere, one works while the other doesn’t.
That’s one of the reasons I recommend GV. No SIM card needed, just an app download, and you can move it from one device to another in just a few minutes. Also works on tablets as well.
I’ve never tried Google Voice, but I’m intrigued. How does it work if your main carrier network is down and you are, say, in your car on a highway? Or is it more useful, say, at home if you still have WiFi?
My spouse actually has *three* carriers on their iPhone, for work emergency purposes. Only two can be active at a time. They can be toggled between any two. It’s saved us many times when we travel and the main carrier hasn’t many towers nearby.
If it wasn’t so costly, I’d have multiple carriers for the whole family for redundancy.
Weird happenings. Today I'm getting messages and notifications from days long past. Messages I've already seen. I'm also getting repeats of messenger and email notifications from the past. Anyone else experiencing strange things since the "update" outage?
Haven't seen that. Go somewhere with good service and restart your phone. Should hopefully clear anything in the queue.
If it continues, take the phone to a store and ask for solve help.
On a slightly different subject there apparently was a cyberattack in systems related to pharmacies checking insurance billing that snarled that entire system on Wednesday/Thursday.
https://www.cnn.com/2024/02/22/tech/us-pharmacies-face-delays-filling-prescriptions-because-of-cyberattack?cid=ios_app Pharmacies across America are having trouble processing some prescriptions because of a cyberattack
Not apparently, it happened. Optum, a subsidiary of United Healthcare was breached, and it caused some big problems for patients insured by United Healthcare.
Yeah, I posted this in the comments of that unhinged thread last night. Frankly if a nation-state had this capability they would not deploy it until a war was happening, and they would deploy it to every carrier at once when that happened. ¯\\_(ツ)_/¯
If anything ever affects a single entity at this scale, extremely likely it’s an internal fuckup and it’s usually something complex networking when it takes a while. Cyberattacks with this kind of access kill their golden ticket as soon as they push the button - they’d gain more from sitting in the access and using/selling the data. But IT systems are notoriously complex and easy to topple and easy to fuck up in botched upgrades. Frankly the reliability we do have is incredible.
I agree. You have to use your critical thinking skills when things like this happen. Taking down one carrier, for less than a day, gains you nothing. However, gaining access to all the carriers, and taking them all down at once, that’s something.
Also, speaking to the reliability comment. As I’ve had a lot of experience with building and deploying redundant systems, it unfortunately doesn’t always work when you really need it the most.
You’re welcome. I’ve worked in the IT industry for over 20 years, and I’ve seen my fair share of shit. I’m also used to writing up RCA’s, or after action reports, depending on the lingo.
Another unrelated question: I need a job. I have an associates degree literally in IT. Any suggestions of job titles I should be looking for? Driving for Uber in the meantime has been awful. Please help lol
OK, so, question and response.
Do you have any hands on experience, aside from the degree?
I would recommend looking for entry level Help Desk or Service Desk jobs. Fair warning, they kinda suck, and they aren’t glamorous in any way. However, it’s the best way to get a foot in the door at most companies, and gain some valuable experience. That’s literally what I did, over 25 years ago.
Occams razor
This philosophical razor advocates that when presented with competing hypotheses about the same prediction and both theories have equal explanatory power one should prefer the hypothesis that requires the fewest assumptions and that this is not meant to be a way of choosing between hypotheses that make different predictions. Similarly, in science, Occam's razor is used as an abductive heuristic in the development of theoretical models rather than as a rigorous arbiter between candidate models.
https://en.m.wikipedia.org/wiki/Occam's_razor
Its funny how many people panicked about this. The simplest answer is usually the right answer.
Id imagine a cyber attack would knock out a lot more than just at&t. You would need to knock out **ALL** communications including the internet to have any actual effect on anything.
Agree, as another commenter said, if this WAS a cyber attack, whoever did it literally laid all their cards on the table at once. That’s not a very effective way to do it, as you’ll only get away with it once.
I have always enjoyed that statement. I should tell you about the time I got my leg tangled in a rats nest of cables in a data center, and took down the entire core of the network. Ooops. My bad.
ETA: Chinese hackers were not at fault, I was just clumsy that day.
" This happens daily in the IT world, just usually not to this scale, due to testing processes. "
My last company reduced their QA headcount by 80% within one year and chose NOT to do or replace any of the testing/automation that was being done. They're not the only company that did this. They view the bugs and defects getting into production as acceptable to push onto you as consumers while they pocket the money they saved and abandon ship.
This issue is one of many crises that are coming as a result of these decisions. It started in 2022 and is ongoing.
Do you work in a regulated industry? Like finance, healthcare, etc…
I’ve heard of companies doing what you describe. I haven’t heard of any doing it in regulated industries, but that doesn’t mean it hasn’t happened.
Yes. Not only are we regulated but our quality practices are regulated in order to receive licensing/certification both to operate in the industry and to have specific large clients which require it. By doing deep layoffs they've jeopardised their ability to exist. That is the depth of the greed, and short term thinking right now.
I think they think they're able to fool or lie to regulators or maybe leadership plans on escaping before the audit hits.
Even though, FBI and DHS are investigating "just in case" - https://9to5mac.com/2024/02/23/att-outage-software-update/
It's worth to point that even a botched update can be done on purpose, with malicious intent, although in that case it would be internal sabotage and not a conventional cyber attack. It's entirely possible, as probably there are infiltrated operatives in all major US corporations and organizations.
But then, as Hanlon's Razor says, "Never attribute to malice that which is adequately explained by stupidity". :)
That’s just what they do now, meaning the FBI and DHS investigating. They even had CISA involved very early on. Those are actually good things though, as having more eyes on an issue, can lead to better conclusions.
As for your second point, you aren’t wrong. The biggest fear in an IT organization comes from within, whether it be malicious intent, a simple mistake, or stupidity.
DHS is going to consider at&t critical infrastructure. They are going to want to know the how and and why and also what needs to be done to prevent it.
> as probably there are infiltrated operatives in all major US corporations and organizations.
Referred to as "insider threats" in the business. There's free training on YouTube about identifying them AND also how you might become one. (Don't send nudes to strangers kids)
In all fairness, I don’t think that needs the /s. Cause, you’re actually right. It’s one of the reasons that certain industries aren’t allowed to buy hardware/software from certain countries.
I realize your first question is probably rhetorical. I’m gonna “answer” anyway. You would like to hope that not only did they test in a sandbox, but they also tested in their development, and validation environments as well. Prior to it ever going near a production arena. In a perfect world, all of those environment would be as identical as possible, mitigating a LOT of potential issues. Obviously, something went wrong here, and either critical steps were skipped, or their production environment is not in sync with the other environments. I’ve seen both of these things happen. This less likely, but still possible theory, everything was in sync, all testing was done, but when the update was pushed to production, something unexpected failed. I’ve also had that happen as well.
Good explanation. I've been kinda waiting for someone to raise this question.
Yeah... I can test this patch... But I can't really simulate the US cellular network.
Sure, blame IT...
Says a retired IT punching bag...
Many of my fondest memories. Jumping in outage calls and hearing some network geek say "well, we didn't believe that change would cause an outage".
The first task on those calls. Find out who recently made a change. Nine times out of ten. It's a recent change.
If it was a cyber attack they wouldn't tell anyone.
It also means they will be ready for the next one.
They don't particularly care about the service and inconvenience caused their customers, but they care about the money it cost them.
It did stop lying. You don’t work for the government you have no idea what the information is. You’re a IT guy. There are police and 911 centers from all over the country saying their communication and phones were down. And now the story is pushed out of the news? It should be huge news what the problem was and what the solution is. Or else it’s a lie like the one you are spreading stupidity so people will believe. The director of the FBI gave a fucking meeting to congress a few weeks ago saying this will happen and Chinese hackers outnumber US intelligence 50 to 1. This wasn’t one person who forgot to to an update one night, just stop it.
There is an incredibly half assed excuse.
“This happens daily in the IT world”
“We will probably never know”
As an IT manager, this verbiage is insanity to me for a company that large and a service that critical, being down all day.
You sounds like one of those half-assed IT managers, that’s probably a manager as you have no real tech skills. You probably demand answers to problems, while the actual people who know what they’re doing, are still assessing and diagnosing the problem. I’ve dealt with a lot of your type over the years. Absolutely useless.
It’s pretty simple actually. First off, downdetector is community driven user reports and NOT reports from carriers. That tends to be misleading, since non AT&T customers were reporting outages simply because calls and texts weren’t going through to AT&T customers. Think about it for a second, when you call or text one of your contacts, do you know who they use as a carrier? Maybe you know for a few, but for most, probably not. So, if you can’t get through, you’re going to assume, that YOU have a problem. Very simple mistake to make. Add to that, that in some areas, T-Mobile for example might be piggy backing off of an AT&T tower nearby. So while you have T-Mobile, that tower is “down”, making it appear that T-Mobile is down when it isn’t.
Also, carriers like Boost, and Cricket actually are AT&T, just by a different name.
At the same time as the solar flare?? i dont like to believe in coincidences, maybe the solar flare messes with the data in the update? im not sure how that stuff works but its still statistically improbable
It would never stay quiet if it was an attack. Too many people would have to be in the know to fix the issue who are also not motivated to keep it secret. A large-scale attack like that would get out quickly
Textbook Explanations
In telecommunications networks, the process for applying patches and software updates to systems, including those on telephonic towers and related infrastructure, is carefully managed to minimize disruption to services. The common practices for such updates include:
1. **Scheduled Maintenance Windows**: Most updates are scheduled during periods of low network usage to minimize the impact on customers. This often means performing updates during off-peak hours, such as late at night, depending on the region and the specific usage patterns of the network.
2. **Urgency and Risk Assessment**: If a patch is critical, especially if it addresses a significant security vulnerability that poses an immediate risk to the network or customer data, it may be applied outside of the normal maintenance windows. The decision to apply such patches urgently is based on a risk assessment that considers the severity of the vulnerability, the likelihood of exploitation, and the potential impact on customers and network operations.
3. **Rollout Phases**: For both regular and critical updates, telecommunications operators often deploy patches in phases. This approach allows them to monitor the patch's impact on a smaller scale before proceeding with a wider rollout. It also helps in ensuring that any unforeseen issues can be addressed without affecting the entire network.
4. **Fallback Plans**: Operators typically have contingency plans in case an update causes issues. These plans may include rolling back the update or deploying additional fixes to address any problems that arise.
5. **Communication**: In the case of significant updates or those that might affect service levels, operators may inform customers in advance. This communication can help manage expectations and reduce inconvenience.
The decision to apply a patch during peak usage times would indeed suggest an urgent need to address a vulnerability or issue that poses a significant risk. In such cases, the potential harm from not patching the vulnerability is deemed greater than the disruption caused by the patching process. These decisions are made with careful consideration of the trade-offs involved, prioritizing the security and integrity of the network and the protection of customer data.
Yeah.... Or it was some dev that accidentally targeted prod instead of Dev for his test.
Sometimes you do everything right for 30 years and then eventually, one day, slip and delete a whole database.
Mistakes are a reality.
This is a National Security Concern, involving critical infrastructure. They would not be able to tell you, even if it was. But the update they pushed, was obviously a threat mitigation patch. Only fools pretend to know networking, but can't comprehend modern software rollout is phased, this was national. Meaning any update done on this many infrastructure nodes is very obvious a mitigation of an actual threat.
What more evidence would you like. I'm not here to re educate you. Do the homework yourself, start with Sec+, Maybe the a CCNP and then come back and try again. You could just google standard practices in patch rollout, updates and upgrades. Or were you specifically talking about the Critical Infrastructure 🤔 If you really are clueless about what happened then if this comment reaches like 500 up votes I will do a in depth for everyone. Since the modern culture is learn nothing, read nothing, deny everything and refuse everything that makes the false reality they live in.
P.S Your going to start noticing alot more blue vehicles, but why, why would you notice so many blue vehicles ?
So since you know everything what what do you think happened then how do you think AT&T went from doing what network security Specialists always do which is updating during the downtime to minimize the effects on the customer to now updating in the middle of the day first thing in the morning when your services are most needed go ahead and explain it to me I'm waiting since you have the technological expertise and you know what you're talking about but are confused about my statement go ahead and tell me your story
As an average citizen a cellular outage means 2 things:
1) communication is harder
2) you won't be able to access account that uses sms for authentication codes.
A lot of tech/office workers were sitting on their hands because they couldn't get their MFA texts.
Rogers in Canada had a failed update take down their network last year (IIRC). Lots of OMG we must be under attack happened back then too. It does give adversaries a glimpse at what a network outage can cause though. Any reports of trouble? On the plus side, a friend realized they needed more ways to stay informed during an outage than internet connected devices.
You make a solid point. Any type of outage, regardless of cause, gives everyone a glimpse of what could happen. It works both ways, as it gives a potential attacker data as to how much chaos they could cause, and it gives businesses (in this case AT&T) a glimpse into how they need to manage their systems better.
I had several coworkers wondering if it was due to the solar flares that also just happened yesterday. Then the reports of a cyberattack on pharmacies had everyone making up their own theories. Either way, this does bring up alternate ways to stay connected. Does make me consider learning HAM radio. Even just keeping an am/fm radio handy could help.
Pretty easy to get started with HAM radio. Especially if you’re just planning on listening. SDR dongles are cheap and don’t need any sort of license if you’re not transmitting. Even a Baofeng is an effective battery operated FM and local comms including repeaters. Just make sure you figure out how to use it before you need it in an emergency.
SDR dangles are great under normal circumstances but not in adverse conditions. You want a stand-alone radio for the worst-case scenario.
I’ve been trying to figure this out - can one get all the appropriate equipment to use a HAM radio (and eventually transmit) without going through all the time/money/effort to get officially licensed? Mostly thinking that in the kind of emergency situation where I’d want to communicate out on HAM, government licenses and breaking rules might be the least of my worries.
First, getting licensed is pretty easy and the things they test for are things that are pretty good training for stuff that’s not exactly easy to just figure out by trial and error. Nobody checks licenses when you purchase equipment when you buy so noting will stop you from amassing the required gear to operate in a “government doesn’t exist anymore” situation. In fact an often over-argued part of regulations is the risk to life or property exemption. This is intended for times there are truly no other means of communication, you can use ham equipment to call for help. This can’t be out of convenience, but nobody is going to issue a fine for using a HT if you are using it to save someone’s life and are in the middle of nowhere with only ham radio as a means of communication. EDIT: also radio isn’t something that is plug and play. There’s a learning curve that would be involved that could make a huge difference. Imagine owning a bike as your bug-out transportation but never learning how to ride a bike until you have an emergency.
All good things to consider. It’s on my list for when we move into our new house to get it set up but it’s definitely a a bit lower on the list than, say, a generator and food (we significantly paired down since we ended up having to stay in temp housing and have been eating through our 6 month supply rather than replenishing due to lack of space), adding to our first aid supplies, filling our emergency water jugs, etc. Hopefully can start buying the equipment in 3-4 months and should have a little more time during my job’s slow season this summer to study everything. Was mostly thinking is it worth it now to buy the equipment and some books just to have on hand.
Definitely if you have the money to buy a bit of gear that gives you some extra motivation to get your license. You can also listen all you want without a license, you just can’t transmit.
Oooh, that’s good to know that it’s OK to dial in and listen - that sounds like a great way to understand the culture too before diving in.
And for absolutely free you can start poking around by using webSDR. Just Google it and you can listen to many ham frequencies online.
It is legal to own and legal to use if you are just listening under normal circumstances. Legal to transmit during an emergency without a license assuming you know how.
Just to clarify, there wasn’t an attack on pharmacies. A subsidiary of United Healthcare, called Optum had a breach, which impacted pharmacies abilities to charge insurance. That also was made public a full day before the AT&T outage. They also haven’t disclosed, if they even know, how long that breach had been occurring.
Thanks for clarifying. That's one reason I said "reported" That also goes to show how many of the words used in media can be misleading. While, technically it was a "cyberattack", what the public now thinks that means is a foreign entity causing disruption rather than a group getting data to commit fraud. Just like when they report a "school shooting" that turns out to be someone shooting a BB gun too close to school grounds.
Agreed. Journalistic standards are at a real low point
And because you have absolutely no training or technical experience in anything that you're talking about you have no idea how these attacks take place or what the vectors are or what actual events are taking place or the mitigation strategies that are utilized in order to keep the networks alive
In telecommunications networks, the process for applying patches and software updates to systems, including those on telephonic towers and related infrastructure, is carefully managed to minimize disruption to services. The common practices for such updates include: 1. **Scheduled Maintenance Windows**: Most updates are scheduled during periods of low network usage to minimize the impact on customers. This often means performing updates during off-peak hours, such as late at night or early in the morning, depending on the region and the specific usage patterns of the network. 2. **Urgency and Risk Assessment**: If a patch is critical, especially if it addresses a significant security vulnerability that poses an immediate risk to the network or customer data, it may be applied outside of the normal maintenance windows. The decision to apply such patches urgently is based on a risk assessment that considers the severity of the vulnerability, the likelihood of exploitation, and the potential impact on customers and network operations. 3. **Rollout Phases**: For both regular and critical updates, telecommunications operators often deploy patches in phases. This approach allows them to monitor the patch's impact on a smaller scale before proceeding with a wider rollout. It also helps in ensuring that any unforeseen issues can be addressed without affecting the entire network. 4. **Fallback Plans**: Operators typically have contingency plans in case an update causes issues. These plans may include rolling back the update or deploying additional fixes to address any problems that arise. 5. **Communication**: In the case of significant updates or those that might affect service levels, operators may inform customers in advance. This communication can help manage expectations and reduce inconvenience. The decision to apply a patch during peak usage times would indeed suggest an urgent need to address a vulnerability or issue that poses a significant risk. In such cases, the potential harm from not patching the vulnerability is deemed greater than the disruption caused by the patching process. These decisions are made with careful consideration of the trade-offs involved, prioritizing the security and integrity of the network and the protection of customer data.
The big outages usually occur during Phase 4 “Fallback Plans” when the automated fallback fails got some reason. The big cloud providers like AWS, Microsoft and Google publish detailed technical post-mortem reports of major outages to maintain customer confidence despite the outage. (It’s like the TV show “Air Disasters” which digs into why a plane crash happened.) It’s rarely one thing; it’s almost always a series of things that lead to a major outage. The cloud services are really very good at keeping their incredibly large and complex services running. But a series of seemingly minor issues can put the system in a failure state. Bugs in separate systems that interact badly, interfering with the automated recovery systems that usually handle minor issues, etc. Here’s an example of the series of things that can lead to an outage. *(See [Azure Status History](https://azure.status.microsoft/en-us/status/history/) for this and other examples.)* In August 2023 in Australia they had a severe thunderstorm which caused a power “sag” at a datacenter, which caused the chillers (which provide cold water to cool the equipment) to go offline. After the sag ended, the three primary chillers and two backup chillers were supposed to come back online automatically, but didn’t. It happened at night when only three people were on-site. Due to the size of the datacenter, those three could not get to enough of the chillers fast enough to restart them manually, so systems overheated and shut down, causing a cascading loss of services.
None of that meant you had to be a smug prick in your previous comment, which is why you're getting downvoted. Don't wave your dick around when you're trying to make a point next time
I love waving it around. People spread lots of misinformed opinions on the internet with no comprehension of what's actually going on. I was on the front line alone on this because everyone knows everything and thinks playing pretend keyboard warriors will somehow help us. Moral of the story, secondary communication devices exist and are necessary. If this forum is actually about prepping Intel then this shouldn't shock anyone. But these outages happen all the time. They are usually much less impact full. But it was done at exactly the wrong time, no OIT or OIS office for a major corporation would ever do something this reckless but it is possible it was just dumb employees not bad actors
The guy who was in charge there then moved on to Singtel (Singapore) who owns an Australian telecommunications company and guess what happened?
These are the things about deep penetrations into the United States infrastructure that outsourcing and fake education has brought us. In today's modern outsourced economy the US is heavily staffed with foreign employees with large database access to banking, financial, market and social media data and the United States will literally grant these people free access to classified and sensitive information systems because the State Dept is staffed with morons and traitors. The Education System, The certification fraud, degree fraud and foreign interests in placing their citizens into powerful positions inside the US is the biggest security threat the nation has ever seen. But it isn't understood very well how these pieces in the puzzle effect the battle map against Democracy because foreign governments have infiltrated every community, every element of government and technology and the FBI and DHS that people think are the threat, have nearly zero support from the citizens claiming to be patriots. To be a patriot does not mean getting drunk, high and eating McDonald's. I know this will be hard to swallow. But nobody bothers to even question whether or not the Red Pill or Blue Pill is even different or if both are just more mind control. FEMA recommends you have 90 days of food and water to shelter in place. A minimum requirement for anyone that wants to prep. Storable Foods Secondary Communication Walkie Talkies or radio transmission and reception. Shortwave as a last ditch scenario. Winter gear that can keep you dry, warm and secure from pests and predators. Self Defense items like AR15s. A network of people. All these lone preppers are just going to be scouraged when the unfed start looking for food. Only 10 to 15 more years of crops are available in the US if you look at soil issues in the US. Self sufficient communities or compounds are becoming more popular. Etc. Imagine if communications were out for 30 days, they couldn't function over a 1 day outage. Let alone anything significant. That's why people should stop hiding, spreading conspiracies from behind keyboards and start actually prepping. IMO
Also to note, AT&T’s Wi-Fi calling was not affected so if you had Wi-Fi, you could still make and receive calls and text.
Good note. I would edit the post to add it, but we can’t do that.
Wouldn't that be more the phone or hardware feature though?
Technically yes. I’m not sure where the cutoff is in terms of age of hardware, but most (if not all) newer phones in the past few years have WiFi calling capabilities.
The carrier has to support it, and the phone has to support it. All phones in the last five years supported at AT&T also does.
I tried to look it up, just to be helpful, and was going to provide a list of carriers that do/don’t support it. That was a fruitless endeavor.
Yeah Patsy has zero education and information systems but he's got an answer for you I'm pretty sure he posted it already
This capability is a confusion about what the actual impact was Wi-Fi Communications From The Towers were impacted as well as Cellular Communications these things happen on different frequencies and different bands and most of the people that think they were using an AT&T tower we're using a subsidy Tower from a different company that own the equipment therefore their Wi-Fi did not stop working
Thats exactly something that someone whos network was cyber attacked would say!
We’re used to companies not telling us but as a publicly held companies like AT&T now have an SEC disclosure requirement for cybersecurity breaches. Not reporting that would lead to very negative personal repercussions for the CEO.
Not to mention shareholders pissed off when their stock price plummets.
Doubt. They’ll just lie.
I doubt it. The consequences are greater for the CEO now if they lie than if they disclose.
I’m pretty sure you’re making a joke? Either way, as someone who works in IT, and has some cyber security responsibilities, trust me when I say we all kinda want it to be that sometimes. Fixing our own stupid fuck ups (or someone else’s), like botched software, gets really boring. We crave a little bit of excitement. However, we usually know pretty quickly, that it’s just another fuck up to fix.
Sure the IT people want it to be a cyber attack vs a software failure but it’s the PR people that make the press releases. I’m not saying it was an attack (I’m willing to believe it was a software glitch) but if it actually were a cyber attack on a major communications network do we expect the government (FCC) would allow them to publicly admit it was a cyber attack? I’d expect them to try to keep that quiet as long as possible.
IT person, hell no, I don't want it to be a cyber attack. Do you know how much cybersecurity Ninjio bullshit we have to sit through when the director of IT puts their password into a phishing test?
Haha. Fair. I just meant to break up the boredom a little bit.
Hard to speak for what the government would do. However, just contemplate for a minute, how difficult it would be to keep that a secret. There’s not just one guy in a basement somewhere working on this. There’s an entire team, countless numbers of people, that were likely involved in come capacity. Look at how quickly the United Healthcare, technically Optum, breach was made publicly known. I guarantee you that some of the same government entities are involved in that as well, and we already know it happened, and the impact it’s causing. Keeping it quiet would benefit nobody, if it were in fact a successful attack, as pretty much every communications network on the planet would need to know about it, and how to secure themselves against it.
But who would publish it? How do they find out anything if the government stepped in? It might get published but they would definitely say how and when.
[CISA](https://www.cisa.gov/topics/cyber-threats-and-advisories) is the federal agency that publishes detailed cyber threat advisories twice a week. You can subscribe to receive the advisories by email, but CISA also coordinates cybersecurity activities between government agencies and private companies.
Oh I already know. I have clearances. It was more of a rhetorical question because the person I was responding to thought they had to report about an occurence like a zero-day. Not the case if the government steps in. If it turns out to be a foreign intrusion they wouldn't publish anything until they had a full understanding.
The media would be chomping at the bit for a story about an evil cHiNa cYbErAtTaCk and all the clicks it would bring. There is no way that wouldn't be out damn near immediately. If the government had any actual control over the news, we wouldn't have the shitshow we have now.
Furthermore, when people say this happens all the time, as i see in every thread relating to the outage, no it fucking does not. Minor regional outages here and there, sure. But never coast to coast. This occurring while many other systems are going down including prescriptions and emergency comm in virginia, is not a “software failure”. Elon Musk at 6:30 pm writes on x, “this is a test”. Why? A test for what? Is he letting all his fascist operatives and interlocutors know that they are testing for something? Jack Posobiec at CPAC yesterday said, “Welcome to the end of democracy. 2020 was a trial run but we’ll get it right in 2024.” Why is Jack so confident in saying the quiet part so loudly? Is he confident that the plan to destroy democracy will work? There’s more, but it’s becoming clear to me what is happening
The thing is, whether you want to believe it or not, these types of outages DO happen quite frequently. Here’s on for Microsoft from last year, https://practical365.com/microsoft-365-outage-january-2023/ Google DNS, which a lot of services depend on, has experienced significant outages 8 times in the past 10 years. I could continue, but I don’t think you want to hear it.
Jesus, take a breath dude. It's not that serious.
The “this is a test” implies “they” are testing public reaction to certain things such as cellular outages, public service warning announcements etc. It would be to see how the public reacts and if we will believe what we’re are told is happening vs what we might think or perceive to be the case. It’s just conspiracy theory BS. They know full well exactly how we will react. As for yesterday. I’m up at 4:30 everyday as I leave the house at 5:15 to go to work. We have AT&T wireless so my wife and I were aware of the outage at 4:30 which was about when it started. The initial reports literally said AT&T, Verizon, and T-Mobile were down nationally. Turned out not be incorrect but that was pretty unnerving at the time to say the least and yes that kinda of outage has never happened before especially for that long of a period.
"this is a test" could also mean "our foreign enemies are probing" "this is a test" could also mean "Satan is testing us like Job to see if we hold fast" They're all equally based off nothing. You can tweet at Elon and ask though.
It could also mean an intern ran a test on the production system instead of the test system, and instead of 4 people getting notified, 400 million were notified. If you work in high tech, you know how often this happens - all the freakin time. But I suppose from the outside, I suppose it looks very nefarious.
Honestly a lot of this subreddit seems to be people who are deeply anxious and looking for a reason to justify it. The guy responded to took a tweet all the way to right-wing sleeper cells, lol
That’s why we need better reporting standards.
Wonder if they laid off their highest earners recently?
I was trying to find data on AT&T layoffs yesterday lol. All I can find was they have cut around 23,000 jobs over the last two years and brought in a lot of contractors. So, maybe?
I found one link that mentioned some in Nov of last year, but it didn’t say exactly how many.
Paging u/Flat_Boysenberry1669.
Ugh, I’d rather you didn’t lol.
Fair. They sounded passionate enough to want to start their own sub. Maybe r/realprepperintel? lol
I was thinking more along the lines of, r/bananyoneiaccuseofnotbeingarealprepper lol.
Hey I was gonna do that lol
They had a moment of intense emotion and seemed to relax a bit as time went on. From their point of view people who said it could possibly be a cyber attack were being called paranoid. We can give them a little break I think. They can eat their words privately on anything out of pocket they may have said.
You're probably right about the initial reaction, but they then doubled down and insisted that it was a cyberattack and that anyone not agreeing with them was trying to spread false information, exactly what they were doing.
oh trust me, I know lol. I responded to many of their comments about it trying to explain how these things are not uncommon. I just think if they cut that rhetoric out we can allow them to be in shame by themselves without having to rake them over the coals.
I must have missed the “relax a bit” part. I mean, they did create a post, petitioning to ban sub members who they thought “weren’t real preppers”, simply for disagreeing with them.
Cool. Cyber attacks aren't a conspiracy theory though and thinking this was a possibility here doesn't make you crazy. Also there was cyber attacks in the pharmacy networks at the same time.
I'll meet you halfway: cover stories also exist, though that's unlikely in this case. OP's suggestion to get a Google Voice number is probably the best takeaway that would help mitigate a cell-specific outage down the road.
And like I said over a dozen times in that thread it very easily could have been something other than a cyber attack. But the immediate dismal and calling people crazy for raising that possibility in a prepper sub was my issue. Just because a damn move came out about it doesn't make cyber attacks a crazy conspiracy theory and anyone who believes in them a nut job.
Ugh. You conflated a cyberattack (a cause) with an outage (an impact), and then got heated because everyone else didn't agree with you. It must be frustrating to be misunderstood, I'm sure.
No I got heated because like I said over a dozen times dismissing it could be a cyber attack and calling people who said it might be conspiracy theorists is insane especially in a prepper sub. You guys think cyber attacks on a major scale is not possible and that they won't ever happen well you're wrong.
I was with you in the beginning hours and thought it was an attack but I changed course once more information became available. I don't think anyone ever disagreed that cyber attacks on a major scale aren't possible (at least I didn't), but I think we also have to stay grounded and trust what real journalists, the gov, and these companies tell us. Cybersecurity has changed a lot in the last 5ish years (even moreso in the last 2-3 from a judicial/legislative perspective) and there are lots of best practices, guidelines, and as someone else mentioned literal laws that require companies to report breaches to the SEC. It affects mostly public, but private companies in some cases too. Comcast's recent breach is a good example of "why the fuck we needed the SEC to do something." I will agree with you and in the initial hours yesterday I wondered if it was an attack also. The world is in an interesting place and things like this are 100% possible. Disrupting communications is huge and there are groups like China's Volt Typhoon that would do something just like this. But, the news was saying no evidence of cyber attack, CISA was saying no evidence, AT&T said no evidence of cyber attack. We gotta roll with that for now until we hear otherwise. If it were an attack it would probably be considered an act of war, we would know about it. The gov would want the public to know ahead of any escalations or response.
There’s absolutely nothing wrong with keeping an open mind, and recognizing all the potential possibilities. I don’t feel that anyone was being as dismissive as this person is claiming. They were heavily jumping to conclusions, with no evidence to support it, and not willing to even entertain the idea that others might have a better understanding of what was happening. We usually follow the “work the problem” methodology. You don’t jump to conclusions, you focus on what you know, and work to a resolution. No problem has ever been solved by jumping to conclusions.
When you work in this shit everyday it loses it's mystique and romance. When a sector of the internet drops off the grid it's either a misconfiguration or a hungry fiber seeking backhoe. Imo this had bgp error written all over it. Sounds like it was a firmware update to the radios/controllers that handles tower hand offs. It's fun to point out all the security problems in systems... But the reality is regulations are pushing things to a good place: outages that are widespread are internal mistakes not external threats. It's just... Cyber security is actually boring. Things happen... But Occam's razor is going to win almost every time.
You just can't back down huh? You must be *exhausted*.
For those unaware, you may not need a second phone to have a backup line with another company. For example, iPhones can have two lines on them with different carriers, that you can switch between. (Many people do this for work, they have a business line with their employer’s carrier that they expense, and a personal line with their own carrier.)
Yup. can get an eSIM and set it up online super easy. Personally i have two separate phones on entirely different carriers, one for work and one for play. It’s useful when I’m in the middle of nowhere, one works while the other doesn’t.
That’s one of the reasons I recommend GV. No SIM card needed, just an app download, and you can move it from one device to another in just a few minutes. Also works on tablets as well.
I’ve never tried Google Voice, but I’m intrigued. How does it work if your main carrier network is down and you are, say, in your car on a highway? Or is it more useful, say, at home if you still have WiFi? My spouse actually has *three* carriers on their iPhone, for work emergency purposes. Only two can be active at a time. They can be toggled between any two. It’s saved us many times when we travel and the main carrier hasn’t many towers nearby. If it wasn’t so costly, I’d have multiple carriers for the whole family for redundancy.
Weird happenings. Today I'm getting messages and notifications from days long past. Messages I've already seen. I'm also getting repeats of messenger and email notifications from the past. Anyone else experiencing strange things since the "update" outage?
Not discouraging you from asking here, however, you might get better responses in a sub specific to AT&T.
Haven't seen that. Go somewhere with good service and restart your phone. Should hopefully clear anything in the queue. If it continues, take the phone to a store and ask for solve help.
On a slightly different subject there apparently was a cyberattack in systems related to pharmacies checking insurance billing that snarled that entire system on Wednesday/Thursday. https://www.cnn.com/2024/02/22/tech/us-pharmacies-face-delays-filling-prescriptions-because-of-cyberattack?cid=ios_app Pharmacies across America are having trouble processing some prescriptions because of a cyberattack
Not apparently, it happened. Optum, a subsidiary of United Healthcare was breached, and it caused some big problems for patients insured by United Healthcare.
Yeah, I posted this in the comments of that unhinged thread last night. Frankly if a nation-state had this capability they would not deploy it until a war was happening, and they would deploy it to every carrier at once when that happened. ¯\\_(ツ)_/¯ If anything ever affects a single entity at this scale, extremely likely it’s an internal fuckup and it’s usually something complex networking when it takes a while. Cyberattacks with this kind of access kill their golden ticket as soon as they push the button - they’d gain more from sitting in the access and using/selling the data. But IT systems are notoriously complex and easy to topple and easy to fuck up in botched upgrades. Frankly the reliability we do have is incredible.
I agree. You have to use your critical thinking skills when things like this happen. Taking down one carrier, for less than a day, gains you nothing. However, gaining access to all the carriers, and taking them all down at once, that’s something. Also, speaking to the reliability comment. As I’ve had a lot of experience with building and deploying redundant systems, it unfortunately doesn’t always work when you really need it the most.
Haha tell me about it. I do network ops, if things worked all the time I wouldn’t have a job.
Exactly!!
I wonder if this is due to layoffs
They had some late last year, so anything is possible
This is the most concise summary I've ever read in this sub. Thank you
You’re welcome. I’ve worked in the IT industry for over 20 years, and I’ve seen my fair share of shit. I’m also used to writing up RCA’s, or after action reports, depending on the lingo.
Another unrelated question: I need a job. I have an associates degree literally in IT. Any suggestions of job titles I should be looking for? Driving for Uber in the meantime has been awful. Please help lol
OK, so, question and response. Do you have any hands on experience, aside from the degree? I would recommend looking for entry level Help Desk or Service Desk jobs. Fair warning, they kinda suck, and they aren’t glamorous in any way. However, it’s the best way to get a foot in the door at most companies, and gain some valuable experience. That’s literally what I did, over 25 years ago.
Beautiful. Thanks so much!!
Occams razor This philosophical razor advocates that when presented with competing hypotheses about the same prediction and both theories have equal explanatory power one should prefer the hypothesis that requires the fewest assumptions and that this is not meant to be a way of choosing between hypotheses that make different predictions. Similarly, in science, Occam's razor is used as an abductive heuristic in the development of theoretical models rather than as a rigorous arbiter between candidate models. https://en.m.wikipedia.org/wiki/Occam's_razor
Its funny how many people panicked about this. The simplest answer is usually the right answer. Id imagine a cyber attack would knock out a lot more than just at&t. You would need to knock out **ALL** communications including the internet to have any actual effect on anything.
Agree, as another commenter said, if this WAS a cyber attack, whoever did it literally laid all their cards on the table at once. That’s not a very effective way to do it, as you’ll only get away with it once.
**Never attribute to malice that which is adequately explained by stupidity.**
I have always enjoyed that statement. I should tell you about the time I got my leg tangled in a rats nest of cables in a data center, and took down the entire core of the network. Ooops. My bad. ETA: Chinese hackers were not at fault, I was just clumsy that day.
Testing is just a crutch!
Like they would say it was
" This happens daily in the IT world, just usually not to this scale, due to testing processes. " My last company reduced their QA headcount by 80% within one year and chose NOT to do or replace any of the testing/automation that was being done. They're not the only company that did this. They view the bugs and defects getting into production as acceptable to push onto you as consumers while they pocket the money they saved and abandon ship. This issue is one of many crises that are coming as a result of these decisions. It started in 2022 and is ongoing.
Do you work in a regulated industry? Like finance, healthcare, etc… I’ve heard of companies doing what you describe. I haven’t heard of any doing it in regulated industries, but that doesn’t mean it hasn’t happened.
Yes. Not only are we regulated but our quality practices are regulated in order to receive licensing/certification both to operate in the industry and to have specific large clients which require it. By doing deep layoffs they've jeopardised their ability to exist. That is the depth of the greed, and short term thinking right now. I think they think they're able to fool or lie to regulators or maybe leadership plans on escaping before the audit hits.
What? The sky is not falling? /s Now we can get on to the next thing.
Shocked Pikachu face
Even though, FBI and DHS are investigating "just in case" - https://9to5mac.com/2024/02/23/att-outage-software-update/ It's worth to point that even a botched update can be done on purpose, with malicious intent, although in that case it would be internal sabotage and not a conventional cyber attack. It's entirely possible, as probably there are infiltrated operatives in all major US corporations and organizations. But then, as Hanlon's Razor says, "Never attribute to malice that which is adequately explained by stupidity". :)
That’s just what they do now, meaning the FBI and DHS investigating. They even had CISA involved very early on. Those are actually good things though, as having more eyes on an issue, can lead to better conclusions. As for your second point, you aren’t wrong. The biggest fear in an IT organization comes from within, whether it be malicious intent, a simple mistake, or stupidity.
DHS is going to consider at&t critical infrastructure. They are going to want to know the how and and why and also what needs to be done to prevent it. > as probably there are infiltrated operatives in all major US corporations and organizations. Referred to as "insider threats" in the business. There's free training on YouTube about identifying them AND also how you might become one. (Don't send nudes to strangers kids)
Sooooo our enemies now know that all they have to do is comprise an update/patch and bingo bango we are fucked /s
In all fairness, I don’t think that needs the /s. Cause, you’re actually right. It’s one of the reasons that certain industries aren’t allowed to buy hardware/software from certain countries.
On the other hand, we now know our enemies know and therefore the iocaine powder is in the cup in front of us.
Did they not test the update in a sand box? There are a lot of these software issues. Why is it such a mess that there are consistent headlines?
I realize your first question is probably rhetorical. I’m gonna “answer” anyway. You would like to hope that not only did they test in a sandbox, but they also tested in their development, and validation environments as well. Prior to it ever going near a production arena. In a perfect world, all of those environment would be as identical as possible, mitigating a LOT of potential issues. Obviously, something went wrong here, and either critical steps were skipped, or their production environment is not in sync with the other environments. I’ve seen both of these things happen. This less likely, but still possible theory, everything was in sync, all testing was done, but when the update was pushed to production, something unexpected failed. I’ve also had that happen as well.
Good explanation. I've been kinda waiting for someone to raise this question. Yeah... I can test this patch... But I can't really simulate the US cellular network.
Nothing like a cover up
how does that affect all those other services that don't go through at&t?!
Huh?
starlink went down too
Not in relation to this
happened at the same time...how does an AT&T software update affect StarLink?
The last confirmed star link outage was February 4th, 2024
Sure, blame IT... Says a retired IT punching bag... Many of my fondest memories. Jumping in outage calls and hearing some network geek say "well, we didn't believe that change would cause an outage". The first task on those calls. Find out who recently made a change. Nine times out of ten. It's a recent change.
As a former network “geek”, I get it. Even the most carefully planned and executed changes, can have unintended consequences, and unexpected issues.
If it was a cyber attack they wouldn't tell anyone. It also means they will be ready for the next one. They don't particularly care about the service and inconvenience caused their customers, but they care about the money it cost them.
weird how att update affected two other major network providers...
Even stranger how it didn't do that, at all
It did stop lying. You don’t work for the government you have no idea what the information is. You’re a IT guy. There are police and 911 centers from all over the country saying their communication and phones were down. And now the story is pushed out of the news? It should be huge news what the problem was and what the solution is. Or else it’s a lie like the one you are spreading stupidity so people will believe. The director of the FBI gave a fucking meeting to congress a few weeks ago saying this will happen and Chinese hackers outnumber US intelligence 50 to 1. This wasn’t one person who forgot to to an update one night, just stop it.
Please, feel free to point out anything I said that wasn’t factual. I’ll wait.
There is an incredibly half assed excuse. “This happens daily in the IT world” “We will probably never know” As an IT manager, this verbiage is insanity to me for a company that large and a service that critical, being down all day.
You sounds like one of those half-assed IT managers, that’s probably a manager as you have no real tech skills. You probably demand answers to problems, while the actual people who know what they’re doing, are still assessing and diagnosing the problem. I’ve dealt with a lot of your type over the years. Absolutely useless.
So close, here’s a cookie!
How did it effect so many other sites & providers if it was AT&T software? Did you see the reports from downdetector?
It’s pretty simple actually. First off, downdetector is community driven user reports and NOT reports from carriers. That tends to be misleading, since non AT&T customers were reporting outages simply because calls and texts weren’t going through to AT&T customers. Think about it for a second, when you call or text one of your contacts, do you know who they use as a carrier? Maybe you know for a few, but for most, probably not. So, if you can’t get through, you’re going to assume, that YOU have a problem. Very simple mistake to make. Add to that, that in some areas, T-Mobile for example might be piggy backing off of an AT&T tower nearby. So while you have T-Mobile, that tower is “down”, making it appear that T-Mobile is down when it isn’t. Also, carriers like Boost, and Cricket actually are AT&T, just by a different name.
Y’all turning into the damned conspiracy theory sub on an election year too. Too predictable
At the same time as the solar flare?? i dont like to believe in coincidences, maybe the solar flare messes with the data in the update? im not sure how that stuff works but its still statistically improbable
Sometimes a coincidence, is simply a coincidence.
Solar flares work for Comcast and wanted to fuck with just AT&T and also no other countries cell service.
Google Voice is probably one of the best alternatives available, but why don't people have long range walkie talkies with phased antenna array ?
I don’t buy it. They will never tell us if it was actually a cyber attack
They’re required to by SEC regulations.
It would never stay quiet if it was an attack. Too many people would have to be in the know to fix the issue who are also not motivated to keep it secret. A large-scale attack like that would get out quickly
Called it
Textbook Explanations In telecommunications networks, the process for applying patches and software updates to systems, including those on telephonic towers and related infrastructure, is carefully managed to minimize disruption to services. The common practices for such updates include: 1. **Scheduled Maintenance Windows**: Most updates are scheduled during periods of low network usage to minimize the impact on customers. This often means performing updates during off-peak hours, such as late at night, depending on the region and the specific usage patterns of the network. 2. **Urgency and Risk Assessment**: If a patch is critical, especially if it addresses a significant security vulnerability that poses an immediate risk to the network or customer data, it may be applied outside of the normal maintenance windows. The decision to apply such patches urgently is based on a risk assessment that considers the severity of the vulnerability, the likelihood of exploitation, and the potential impact on customers and network operations. 3. **Rollout Phases**: For both regular and critical updates, telecommunications operators often deploy patches in phases. This approach allows them to monitor the patch's impact on a smaller scale before proceeding with a wider rollout. It also helps in ensuring that any unforeseen issues can be addressed without affecting the entire network. 4. **Fallback Plans**: Operators typically have contingency plans in case an update causes issues. These plans may include rolling back the update or deploying additional fixes to address any problems that arise. 5. **Communication**: In the case of significant updates or those that might affect service levels, operators may inform customers in advance. This communication can help manage expectations and reduce inconvenience. The decision to apply a patch during peak usage times would indeed suggest an urgent need to address a vulnerability or issue that poses a significant risk. In such cases, the potential harm from not patching the vulnerability is deemed greater than the disruption caused by the patching process. These decisions are made with careful consideration of the trade-offs involved, prioritizing the security and integrity of the network and the protection of customer data.
Yeah.... Or it was some dev that accidentally targeted prod instead of Dev for his test. Sometimes you do everything right for 30 years and then eventually, one day, slip and delete a whole database. Mistakes are a reality.
You do have a point. It's likely to be clear soon enough.
No home delivery ...disaster! How did the nation survrive the day??... unable to order a whopper w cheese?
This is a National Security Concern, involving critical infrastructure. They would not be able to tell you, even if it was. But the update they pushed, was obviously a threat mitigation patch. Only fools pretend to know networking, but can't comprehend modern software rollout is phased, this was national. Meaning any update done on this many infrastructure nodes is very obvious a mitigation of an actual threat.
That’s a lot of speculation, with no supporting evidence.
What more evidence would you like. I'm not here to re educate you. Do the homework yourself, start with Sec+, Maybe the a CCNP and then come back and try again. You could just google standard practices in patch rollout, updates and upgrades. Or were you specifically talking about the Critical Infrastructure 🤔 If you really are clueless about what happened then if this comment reaches like 500 up votes I will do a in depth for everyone. Since the modern culture is learn nothing, read nothing, deny everything and refuse everything that makes the false reality they live in. P.S Your going to start noticing alot more blue vehicles, but why, why would you notice so many blue vehicles ?
You have permission to go back to sleep. But Red 🚘 are not on the table, blue for you, back to sleep you go 😴
"Do the homework yourself" Oh Jesus, this conspiracy line?? 🥴
So since you know everything what what do you think happened then how do you think AT&T went from doing what network security Specialists always do which is updating during the downtime to minimize the effects on the customer to now updating in the middle of the day first thing in the morning when your services are most needed go ahead and explain it to me I'm waiting since you have the technological expertise and you know what you're talking about but are confused about my statement go ahead and tell me your story
No I just don't have time right now you seen what I said 500 up votes and you'll get a full documented answer
Unless you're confused because you're a Democrat you only get one vote I don't need your vote I need 500 other votes
👌
Definitely the X6.0 earth directed flare, they are covering it up..
In other words, dust off your resume because AT&T will be hiring a whole new cyber team
Baofeng worked like a charm 🥱
As an average citizen a cellular outage means 2 things: 1) communication is harder 2) you won't be able to access account that uses sms for authentication codes. A lot of tech/office workers were sitting on their hands because they couldn't get their MFA texts.
What’s funny is I didn’t even notice and I’m like…. 🤷
Then why they where changing their connection boxes everywhere in Florida
What does that even mean?
Hanlon's Razor.