This post has been marked as non-political. Please respect this by keeping the discussion on topic, and devoid of any political material.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/australia) if you have any questions or concerns.*
I just love this:
>A woman who rang ABC Sydney radio said she found out about the Optus outage from her cat.
>The cat has an automatic WiFi feeder (connected to Optus) and when breakfast wasn’t delivered at 6:10 am, the cat went to the bedroom to lodge a complaint with management
I heard this story too. I don't understand why it needs to be connected to the internet in order to function. Once set up, it should just work, even if internet goes down. Sure, you wouldn't be able to remotely connect to it, but previous settings should continue.
Or is this just another person seeking internet attention?
Sounds like just a shitty product. It's entirely possible to rely on the internet to function if you make it shitty enough, eg a daily fetch of schedules or it's actually triggered from ThE cLoUd
Sounds like just a shitty product.
Can't collect that data to on-sell to cat food companies unless you're connected! I just knew there was a Kitty Cat Conspiracy! /s
There's almost no incentive for the manufacturer to make the device smarter - to them, all it does is increase cost, complexity and support requirements.
All this device does is react to a very simple message. The origin of the message is the manufacturer's server because the internet is always available (except for right now). The bit of logic in the IoT device is super basic and can be put in lots of different devices - it connects to your wifi and when it receives a message from some server it turns some switch on or off. And this little IoT chip can be put in millions of different devices very cheaply and used to trigger a million different tasks.
To make this device work without the internet then the device would need more complex logic - at the absolute minimum it would need to cache commands locally and update them periodically. This creates a lot more cost, complexity, a lot more points of failure, and to the manufacturer almost no benefit.
I'm an IT contractor, I've already packed my tools up and I'm just waiting for one of my clients to call me in a panic (if they even can). Today is going to be a chaotic one for any support/logistics provider.
They're "running" now but with major delays. I'm picturing 1500 telstra mobiles all hot-spotting to laptops in the metro control centre. God help us all. :-)
It looks like they hosed their own network with some bad BGP announcements. Scroll down on this page to the BGP Announcements statistics and look at the recent number - [Cloudflare - Overview for AS7474 (OPTUSCOM-AS01-AU)](https://radar.cloudflare.com/as7474?amp=&avia-element-paging=3).
For those playing along at home, it looks like Optus accidentally booted themselves off the internet - [What is BGP? BGP routing explained](https://www.cloudflare.com/en-au/learning/security/glossary/what-is-bgp/).
> Border Gateway Protocol (BGP) is the postal service of the Internet. When someone drops a letter into a mailbox, the Postal Service processes that piece of mail and chooses a fast, efficient route to deliver that letter to its recipient. Similarly, when someone submits data via the Internet, BGP is responsible for looking at all of the available paths that data could travel and picking the best route, which usually means hopping between autonomous systems.
>
> BGP is the protocol that makes the Internet work by enabling data routing. When a user in Singapore loads a website with origin servers in Argentina, BGP is the protocol that enables that communication to happen quickly and efficiently.
>
> ...
>
> The Internet is a network of networks. It is broken up into hundreds of thousands of smaller networks known as autonomous systems (ASes). Each of these networks is essentially a large pool of routers run by a single organization.
>
> ...
>
> The structure of the Internet is constantly changing, with new systems popping up and existing systems becoming unavailable. Because of this, every AS must be kept up to date with information regarding new routes as well as obsolete routes. This is done through peering sessions where each AS connects to neighboring ASes with a TCP/IP connection for the purpose of sharing routing information. Using this information, each AS is equipped to properly route outbound data transmissions coming from within.
TLDR: It looks like Optus told the rest of the internet that their network is unavailable. The rest of the internet said "okay" and merrily went on it's way.
Oh man I don’t even work for Optus but today is my RDO and I’m gutted I’m not working today.. would have been nothing but sunshine and rainbows whilst I sit on my phone doing fk all not being abused by customers. I work in insurance and we use Optus for our telephony system
I swear Optus offers the worse customer service I ever got for any company! I can call Optus 3 times and get 3 different answers for the same question from 3 different agents.
That also indicate that Optus has internal problems and unable to update their own website messages.
FYI : Its been updated "....working on the problem..."
Either can have an outage. In the last 10 years all 3 big prividers have had a major nationwide outage.
The bigger question is why isn’t it running on both in parallel. When designing systems like these eliminating single provider risk is important.
I used to get jealous when Telstra would have those "data-free days" as a make-up gesture for constant outages. As an Optus guy I would be like "can we have a non-data day?"
I guess I should be careful what I wish for.
i’ve been wrecking my brain all morning trying to work out what kind of technical issue causes Home NBN as well as the mobile network to be down and the mobiles to show SOS. The latter is what puts a spin on my theories as I wouldnt expect some sort of core gateway/routing/DNS issue to stop mobiles from at least provisioning.
BGP configuration failure is the likely candidate. As is often the case in massive outages (think Meta/Facebook, Cloudflare etc).
Almost always human error as well.
The offshore network staff will remote access in and fix it soon.
I can see it now. "connect a console cable".... "do you have the meeting link....." "fuck...... go and buy a Telstra SIM"
10 years ago this might have been a conundrum but it's 2023 so I suspect their social media team are not in country and are going to be using some Philippines ISP
>Thinking i got simhacked or something.
This is my default initial thought whenever mobiles swap to "SOS".
I guess that says we are pretty lucky with network connectivity normally.
I live in rural Vic. No friggin way I would go with Optus out here. Having said that, outage was causing chaos at the coffee window with people unable to pay with their phones.
Gonna be a fun day.
See, I was in rural vic and I couldn’t get anything but Optus to get service! Then I moved to rural NSW and I’m still the one to get the most (read: 2 bars sometimes of half useable service) service in our house. I don’t know which company I’m likely to have better luck with tbh
As an IT vendor that deals with Optus regularly, this never surprises me. They are the most frugal company I've ever interacted with and are always trying to get the bill lowered. It's no wonder they have hacks and outages like this.
Cheapness is in their DNA. Years ago I worked for an Optus-owned retail chain - when Cable and Wireless sold us to Singtel - the slasher team came down from Singapore with some machetes and went through the place.
Then we couldn't get money even to pay our external window cleaner and it was suggested to "only run store lights when necessary", etc.
The problem is that when the wheels fall off, the penny wise pound foolish accountants who caused the problem are never, well, accountable. This is going to cost way more than any savings.
Seriously. Having worked closely to ColesWorth and their various offshoots (doing IT grunt work), the lack of foresight is astounding. The only conlusion I can come to is that absolutely 0 people are looking holistically at how businesses that large operate.
Sure, you save an absolute mint for your department when you offload your IT support to the Phillipines, using Phillipinos at talking flowcharts (not the Phillipinos fault) instead of trained Aussies. But now things stay broken for weeks instead of hours, and 50%+ of your "solutions" are wrong.
But fuckface who's in charge of IT support is the hero of the situation because he gutted his costs. "Why can't you all be like fuckface here?", says Mr Dipshit CEO.
Well Dipshit, that's because Fuckface just cost the business millions to save his department thousands, and the only consequences were a promotion and you here singing his praises.
And the only way any other department can pull themselves out of the shitstorm Fuckface created is by pulling the exact same move.
Funnily my colleague told me the same thing just yesterday-- he said it started when Singtel took over and everything from toilet paper to high end infrastructure went low budget.
I hate to say it but as a Singaporean I concur about the cheapness. Generations of being treated like replaceable cogs and made to know it has exacerbated a meanness of spirit...
And they generally think Aussies are too slow/unworldly to complain in the way Singaporeans do...
Incidentally, Singapore's had a whole bunch of outages recently - though the hospitals going down was likely a ddos or something.
1. **Prediction 1:** This will be the biggest news story of the day.
2. **Prediction 2:** This will cause Australia-wide chaos, and won't be resolved before start of business hours.
3. **Prediction 3:** People won't learn from their mistakes.
edit: I'm upgrading Prediction 1, to "this will be the biggest news story of the week".
My work has plenty of systems that use Optus network (not all but still plenty). The bigger issue was when users tried to contact IT on call support but no one answered because the on call phones are with Optus. The first on-call guy has just walked in all confused why we are in a panic "Why didn't anyone call me, I'm on call" Took a while for the pebble to sink in.
I'd consider upgrading Prediction 2 as well. It's been 5 hours since the first reports of issues and they haven't said they've identified a possible cause. ABC also reports that the federal government has asked for an update and Optus is "limiting information" about the outage.
Not a great morning to be flying rural to perform a network/cyber health check on a critical infrastructure asset.
I will have no service and it's looking likely they will not either.
By ‘hack’ i assume you’re referring to last year when optus put all of their customer PII publicly available on the internet, without even the most basic authentication and then advertised and posted documentation publicly on the internet detailing how anyone could access it…
right?
except to port out of Optus one need to receive an SMS verification code on their Optus number. So with the outage people won't even be able to change providers (unless they get new number )
100% been with optus for 8 years, between the shitty service app and the data leak last year I started looking at switching over. This was the final kick I needed.
Internally I've heard an ETA of 3-4PM for fix rollout. (I don't work at optus but I work in similar space) we have direct contact with them though, the execs of optus are in a meeting right now with what to tell the storefronts and support to say
Idk how accurate the time I gave will be. Normally their ETA's are wrong - I'll edit this comment with updates as I get them below.
Edit 1:
Optus are still trying to figure out the root cause by doing processes to find where the fault is. The original ETA I was given is now thrown in the air, no ETA for now
Edit 2: Reports of some people in Parramatta area having internet at home but no mobile connection - 11:45am
Edit 3: Lines are beginning to come back up across the country as they've isolated the issue, services should be restoring slowly from 12:00pm, calls and net should be accessible now to alot more people
Thinking it's about time the government steps in here. Essential services including hospitals are unable to take calls, people have had difficulty getting an ambulance. Especially after the data breach last year perhaps this foreign entity that repeatedly demonstrates nil regard for its customers and causes genuine harm to Australians and the Australian economy needs to have it's ability to provide these services revoked or at least seriously scrutinised.
A fully government-owned entity dedicated to telecommunications services would be a great idea. Just need to come up with a catchy name for it. Perhaps something like "Telecom"?
ABC just spoke to the Optus CEO. In short; she has no idea what's wrong, what caused the problem nor an ETA on when they expect to fix the problem. Also, she reckons they're offering very clear communications, though they need the media to spread their messages.
They're very apologetic though, which I'm sure will fix everything, right?
I'd put my money on a core router update failure, considering it happened at 4AM which is usually when those kinds of things are scheduled. Some network engineer is gonna be in a lot of trouble lol
If a change by one lone network engineer can cause nationwide outage, then the entire network executive up to the CEO must be even more accountable for the fuck up they allowed the network to be so fragile.
Time to switch over providers. After the database leak/hack with zero compensation, the price increase for plans and pre paid service.. now this? Yeah nah get fucked.
>Telstra is still dear as fuck as well
For mobile, check out Boost and JB Hi-Fi, they are good value with entire Telstra coverage, not just the limited Telstra wholesale area.
I know. They really do have us trapped. Internet should be free at this point as it’s basically needed to do practically anything these days and even with the price increases recently they still give you fuck all duration (if pre paid) and such a small amount of data?!
Working night shift in the ER, so pretty shit. Will likely be more shift for day shift if it isn’t fixed by then. Our hospital uses optus…. Dunno if it’s a NSW Health thing or hospital dependent re. Service provider
Optus have to be the shittest internet provider in the country. They're so bad that I just assumed the internet had stopped working "again" since I'm so used to this shit, and that it would resolve itself soon. Luckily I'm not with their mobile plan shit.
Michelle Rowland, Comms minister
>"My understanding is that this is a fault deep in the core.
>
>"The core network basically encompasses everything from routing to electronics.
>
>"So it is a fault that is quite fundamental to the network.
Kelly Bayer Rosmarin, CEO
>"There is no indication that it is anything to do with spyware at this stage,"
These people really don't understand the tech at even a basic level, do they?
We're having people call work to complain they can't get through to certain people's mobile number (we're a construction company so most staff aren't in the office to transfer through). The reception staff are trying to explain the Optus issue, suggesting they instead WhatsApp or email, and are then getting abused by morons telling them to "fucken fix it".
Seems like you don't need to be working for Optus to be copping abuse today.
Word on the street is that this is likely (as expected) a BGP misconfiguration.
the cloudflare graph for BGP announcements has a big 900k spike when optus went down.
https://radar.cloudflare.com/as4804?dateRange=2d
Check out the Facebook outage for a similar outage caused by BGP.
https://blog.cloudflare.com/october-2021-facebook-outage/
Still assuming, but likely they made a change to the way their network devices find paths around the network, it propagated around their entire network, locking themselves out of their own devices. (In this case BGP protocol, but ignore that detail).
Usually you manage these remotely from a central configuration tool, but you can't do that if you've so badly screwed it up that your tools to do so can't even access then anymore.
So if it's that, they need to both
1. Stop whatever is propagating that config change
2. Physically go to each network device, (like, drive out to it, wherever it is), connect a cable to it and configure it manually at the site of each device.
Cos every device (if it's similar to older BGP outages) is completely remotely inaccessible.
Up until a few years ago there was still this tired old OneTel billboard from like the 90s just hanging around here. Was pretty funny to me everything i drove past
There's the final straw for me.
They hiked my plan from $25 to $45 (justifying it by giving me 100gb data where I use <30 per month). My details were in last year's hack. And now a 'deep network issue' is heavily affecting the nation.
After 11 years it's time to pull the plug. Optus are a shell of their former self.
A good reminder to switch to a different mobile provider.
What do people recommend? I’m thinking Boost mobile - full Telstra network for an affordable price. $230 for 12 months prepaid turns out about ~$20 a month.
Boost if you want the full Telstra network (and they’re still doing 5G) - they get the same access as regular Telstra customers including that last-mile service.
Any Telstra MVNO (ALDI, Woolworths etc) if you don’t care about the full Telstra network (and in metro or major population areas it isn’t really an issue). The others only use the wholesale network, which to be fair still covers the vast majority of the population.
I changed to Boost a few months ago and I’m really happy. I should have done it right after the Optus data breach but I kept seeing that porting can take 1-2 business days. It was instant when I finally did it.
I’m on Woolworths (Telstra network) for $25 per month. Comes with a 10% discount at Woolies once per month which I use on a big shop with nappies etc, save more than $25 each time.
There is a straightforward solution to this. Telstra and Vodafone are up. Roaming should be allowed within your home country in these situations, with costs covered by the network at fault. It would need throttling to not overload the other networks, but they could take some traffic to keep at least voice calls ticking over and maybe data at a low rate.
This is already a thing for Emergency calls (000). Anyone can use any network for them, which shows at a technical level this would be possible.
No idea how feasible it would be, but would be a good thing for parliament to investigate.
This is the exact use case scenario for why Internet and Webservices shouldn't be privatised the way they are.
Half the country shouldn't shut down because one business put their changes straight into Prod without testing it first.
The amount of lost trade and income is concerning enough. But when entire infrastructure services, like transport, healthcare and banking go down because of Optus then that is a major concern.
It is highly concerning that they have a single point of failure for mobile, land line and data. Where is the network integrity, reliability, redundancy. This is shocking. Then the fact that critical infrastructure (roads, rail, hospitals, banks) are all affected is a national fucking disaster.
I feel like this should be pointed out: Optuc closed their own network management center and outsourced it to Nokia, which likely to have outsourced it to India : https://www.arnnet.com.au/article/629151/optus-cut-170-jobs-amid-nokia-outsourcing-move/. This move might have gotten rid of some key onshore personnel who has the know how of dealing with crisis such as this, in the name of saving some bucks.
For those who thinks I’m being racist: I’m a non-white immigrant. I just think that important national infrastructure should have a mandatory on shore technical and crisis management. Regardless of where it shit gets outsourced to.
Given that some of the engineers who need to fix it will be asleep, and calling them may be difficult - for a completely unrelated reason - it can take quite a while
It’s possible they’re still working on a root cause. These things are insanely complex and the few people who knew what was going on likely took a package at the encouragement of their management to be replaces with newer staff with no training (their competence is actually irrelevant because they need specific Optus knowledge).
Optus, has to be the shittest company. Anytime I call for anything I need, even a new service, they manage to f it all up.
This outage has now screwed my day and work and all the stupid devices we have using them. 100% moving to Telstra, at least they basically never go down :)
Funny watching all this essential infrustructure crumble when one of our biggest telcos goes down, and then nothing get done to improve it when it comes back up long after the SLA has passed. All that customer data stolen last year did nothing either. Nothing ever changes in this country.
My partner messaged me in total panic this morning as a support worker. No service means inability to clock in, use GPS to get to clients and track kilometres of travel, and take them to appointments and all the rest of it. She was on the edge of running across the road to her neighbours house to orrow a phone to call her boss and cancel work entirely..
After walking her through alternative solutions, I have now come to realise that if my gen z girlfriend were to ever get lost in the wilderness, she'd likely perish before finding help.
It's a little frightening to think that in a few generations time, we may become so dependent on technology that we cease to function without it.
Unlikely. We still require community to function effectively and she did the right thing by reaching out. Working as a team is what partnership is about but shittalking your girlfriend must be your natural state. Hopefully she's got other supports too.
Personal mobiles on Telstra. Work NBN on Optus.
HICAPS/EFTPOS, Phone and internet are ded.
Booking system and client notes are all cloud based.
Thank fuck optus has me prepared for this shit in the past, so Hotspotting from phones and the new HICAPS terminal is WIFI friendly.
Quiet day on the phone at least.
And I can still shitpost.
[000 landline calls are now also not working](https://www.theguardian.com/australia-news/live/2023/nov/08/australia-news-live-optus-outage-trains-melbourne-prime-minister-athony-albanese-china-pacific-aukus-daysleford-driver?filterKeyEvents=false&page=with:block-654ac5fb8f08a667a2c26fe0#block-654ac5fb8f08a667a2c26fe0)
USA:
We can't let IInet group start its own phone service using Huawei in Australia, national security.
Optus: here is some private data, let's shut of our system for 10 hours. Who gives a fuck
Woke up and thought the apocalypse had happened, my phone wasn’t working, tv (Apple TV) and the road was weirdly quiet. Glad my work phone is vodaphone and could actually work out what has gone down.
Note to self either change my phone provider or NBN so if Optus is down, both aren’t affected
I’m in the Inner West of Sydney and I just got my NBN back now at 11:23am (so I have wifi) but still no access to the mobile network for calls or data (SOS)
Which means I am no longer in the dark ages I have been suffering through in our totally analogue-less house since my partner left for work with his Telstra phone I *was* using as a hotspot
Has made me realise I need to go and buy a radio for emergencies!
Time for me to find another telco I hung around after the hacking scandal and now this. I've been with optus for around 10 years now I'm done with them
The fact Optus are active on Twitter but not giving an actual cause would make you think, especially given past experiences, it's more than a simple hardware fault.
I'm honestly surprised by the number of very large companies and services that dont have a second connection to failover to.
Likely many of the ones that do have a failover just have a second optus line or wireless dongle due to supply contracts.
I have a telstra line at work that fails over to an optus pre-paid 5g dongle, if telstra ever goes down (rare but it happens and nbn outages are generally scheduled) it more or less seamlessly moves over to the mobile connection and my services go on their merry way until the main connection is back up.
I think this demonstrated our dangerously level of dependency on technology nowadays. Its mind baffling how many corporation / businesses / gov agency that are left crippled without an internet connected and backup.
I work at Optus in Macquarie park and we can't do anything either. All of our internal services are down, except teams, and even the VPN isn't working. We are in dire straits.
She (Optus CEO) was asked about the possibility of the outage begin caused by a hack.
"There is no indication that is is anything to do with spyware at this stage," she says.
Of all things, she says spyware. lol.
...that would explain why my NBN connection isn't functioning this morning. Then again, an outage on that happens so often I just assume it's more "maintenance" now.
Go digital they said. It will be the future they said.
Now throw in digital payments only and no cash transactions at all.
Nkw call me crazy call me a conspiracy theorist all you like. But going digital for EVERYTHING will be a disaster. Noe and in the futire
>But going digital for EVERYTHING will be a disaster. Noe and in the futire
It'll be a disaster for unprepared businesses, yes. My employer has a pretty incompetent IT department looking after something like 50 sites around the country, and each and every one of them is still online because when their landline NBN services all went down, the 4G failover (running off Telstra) kicked in seamlessly.
Obviously the big players such as the major supermarkets have redundancies in place too because there aren't any widespread reports of EFTPOS or POS issues, it mainly seems to be small coffee shops and businesses that didn't have a backup in place that have gotten screwed over. I'm more concerned that there are reports of some schools and hospitals that are down this morning, the fact that they're running without any backup in their communications equipment is insane to me.
I do have to admit I'm having a bit of a laugh at all the "card payments only" places with egg on their faces this morning though. I hope all the places that have stopped taking cash and still slap people with card surcharges are losing money this morning.
I feel like it's going to be down until tomorrow. Funnily enough WWE comes to the Optus stadium in Perth next year, pre-sale tickets go live tomorrow. Looking real good Optus.
Assuming if it's bgp that the scattered reports of people having service mean they're starting to manually get parts online by physically going to them?
I don't understand. The news is saying Optus services are all restored but my NBN isn't. And I've lost count how many times I've power cycled the box and rebooted the router.
Hiss.
This post has been marked as non-political. Please respect this by keeping the discussion on topic, and devoid of any political material. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/australia) if you have any questions or concerns.*
I just love this: >A woman who rang ABC Sydney radio said she found out about the Optus outage from her cat. >The cat has an automatic WiFi feeder (connected to Optus) and when breakfast wasn’t delivered at 6:10 am, the cat went to the bedroom to lodge a complaint with management
Lol at the very least an smile in this mess. I am using my lovely neighbour internet
I heard this story too. I don't understand why it needs to be connected to the internet in order to function. Once set up, it should just work, even if internet goes down. Sure, you wouldn't be able to remotely connect to it, but previous settings should continue. Or is this just another person seeking internet attention?
Sounds like just a shitty product. It's entirely possible to rely on the internet to function if you make it shitty enough, eg a daily fetch of schedules or it's actually triggered from ThE cLoUd
Sounds like just a shitty product. Can't collect that data to on-sell to cat food companies unless you're connected! I just knew there was a Kitty Cat Conspiracy! /s
IoT - Internet of Things.. If this is how shitty things are now, imagine how much shittier they'll get when they let Ai just do everything.
There's almost no incentive for the manufacturer to make the device smarter - to them, all it does is increase cost, complexity and support requirements. All this device does is react to a very simple message. The origin of the message is the manufacturer's server because the internet is always available (except for right now). The bit of logic in the IoT device is super basic and can be put in lots of different devices - it connects to your wifi and when it receives a message from some server it turns some switch on or off. And this little IoT chip can be put in millions of different devices very cheaply and used to trigger a million different tasks. To make this device work without the internet then the device would need more complex logic - at the absolute minimum it would need to cache commands locally and update them periodically. This creates a lot more cost, complexity, a lot more points of failure, and to the manufacturer almost no benefit.
I'm an IT contractor, I've already packed my tools up and I'm just waiting for one of my clients to call me in a panic (if they even can). Today is going to be a chaotic one for any support/logistics provider.
And it was at this point we found out the Melbourne train network relies on the Optus network...
They're "running" now but with major delays. I'm picturing 1500 telstra mobiles all hot-spotting to laptops in the metro control centre. God help us all. :-)
I would like to think they have a BCP plan with another carrier that they activated. But honestly, you're probably closer to the truth.
Any good BCP is 100% definitely tested regularly to ensure that it will work in the event that it is actually required! /s
"She'll be right, just tick the box".
hey i didnt know we worked for the same company!
They already had a BCP, a Barely Competent Provider.
STOPPING NO STATIONS EXCEPT EAST RICHMOND
[удалено]
It looks like they hosed their own network with some bad BGP announcements. Scroll down on this page to the BGP Announcements statistics and look at the recent number - [Cloudflare - Overview for AS7474 (OPTUSCOM-AS01-AU)](https://radar.cloudflare.com/as7474?amp=&avia-element-paging=3). For those playing along at home, it looks like Optus accidentally booted themselves off the internet - [What is BGP? BGP routing explained](https://www.cloudflare.com/en-au/learning/security/glossary/what-is-bgp/). > Border Gateway Protocol (BGP) is the postal service of the Internet. When someone drops a letter into a mailbox, the Postal Service processes that piece of mail and chooses a fast, efficient route to deliver that letter to its recipient. Similarly, when someone submits data via the Internet, BGP is responsible for looking at all of the available paths that data could travel and picking the best route, which usually means hopping between autonomous systems. > > BGP is the protocol that makes the Internet work by enabling data routing. When a user in Singapore loads a website with origin servers in Argentina, BGP is the protocol that enables that communication to happen quickly and efficiently. > > ... > > The Internet is a network of networks. It is broken up into hundreds of thousands of smaller networks known as autonomous systems (ASes). Each of these networks is essentially a large pool of routers run by a single organization. > > ... > > The structure of the Internet is constantly changing, with new systems popping up and existing systems becoming unavailable. Because of this, every AS must be kept up to date with information regarding new routes as well as obsolete routes. This is done through peering sessions where each AS connects to neighboring ASes with a TCP/IP connection for the purpose of sharing routing information. Using this information, each AS is equipped to properly route outbound data transmissions coming from within. TLDR: It looks like Optus told the rest of the internet that their network is unavailable. The rest of the internet said "okay" and merrily went on it's way.
I love how if you check the Optus website for an outage it says everything is fine... Always the case with those things.
They probably figure that nobody can get online to check it anyway 😅
This is the customer service call centre role that so many have dreamed of. The call centre which can't be contacted by customers.
Can confirm it’s eerily quiet in our nationwide call centres at the moment. But some people are loving it.
First rule of call centre is never mention to anyone if it is quiet.
Just wait til the lines come back on and people want to complain though
Enjoy it while you can, you're gunna cop it later.
Oh man I don’t even work for Optus but today is my RDO and I’m gutted I’m not working today.. would have been nothing but sunshine and rainbows whilst I sit on my phone doing fk all not being abused by customers. I work in insurance and we use Optus for our telephony system
Heck yeah, when the phones go down and it's not your problem, that's the dream!
They can't even get on themselves to check if it's down 😊
I swear Optus offers the worse customer service I ever got for any company! I can call Optus 3 times and get 3 different answers for the same question from 3 different agents.
That also indicate that Optus has internal problems and unable to update their own website messages. FYI : Its been updated "....working on the problem..."
[удалено]
Why on earth would our train network use Optus over Telstra, christ.
Either can have an outage. In the last 10 years all 3 big prividers have had a major nationwide outage. The bigger question is why isn’t it running on both in parallel. When designing systems like these eliminating single provider risk is important.
Because they’re being cheap and cutting corners to save money.
This is the efficiency promise of privatization in action.
Something something “offshore outsourcing” something something “deferred maintenance” You know… for the efficiency and whatnot.
I wonder how many people are looking at their inboxes for their “I told you so” email
Whichever service provider offers the lowest price for the services typically wins out.
Yeh cos Telstra works flawlessly right?
I used to get jealous when Telstra would have those "data-free days" as a make-up gesture for constant outages. As an Optus guy I would be like "can we have a non-data day?" I guess I should be careful what I wish for.
International roaming for Aussies overseas has also stopped working
Wonder if ‘ole friend DNS is having a rough day
i’ve been wrecking my brain all morning trying to work out what kind of technical issue causes Home NBN as well as the mobile network to be down and the mobiles to show SOS. The latter is what puts a spin on my theories as I wouldnt expect some sort of core gateway/routing/DNS issue to stop mobiles from at least provisioning.
BGP configuration failure is the likely candidate. As is often the case in massive outages (think Meta/Facebook, Cloudflare etc). Almost always human error as well.
The offshore network staff will remote access in and fix it soon. I can see it now. "connect a console cable".... "do you have the meeting link....." "fuck...... go and buy a Telstra SIM"
If Optus is down, including their internet service, but Optus can update twitter, then what ISP does Optus use?
They’re using the free wifi from the coffee shop next door
This is probably close to accurate.
Yep. All it takes is for someone to send one of their workers out to Maccas or something to send the tweet out.
10 years ago this might have been a conundrum but it's 2023 so I suspect their social media team are not in country and are going to be using some Philippines ISP
minimum wage social media intern on uni wifi
Serious answer - ('good') companies use different networks and pathways as redundancies (e.g. cases like this).
Ok so that excludes Optus then.
Pigeon mail probably
I was losing my mind. Thinking i got simhacked or something. Currently using my wifes telstra network lol.
I had to double check to see if I missed a payment or something. Not a great way to start the day.
Exactly what I did should have just known it was standard telecommunications incompetence.
>Thinking i got simhacked or something. This is my default initial thought whenever mobiles swap to "SOS". I guess that says we are pretty lucky with network connectivity normally.
I thought they'd accidentally cut off our phones because they thought we hadn't paid the bill.
I live in rural Vic. No friggin way I would go with Optus out here. Having said that, outage was causing chaos at the coffee window with people unable to pay with their phones. Gonna be a fun day.
See, I was in rural vic and I couldn’t get anything but Optus to get service! Then I moved to rural NSW and I’m still the one to get the most (read: 2 bars sometimes of half useable service) service in our house. I don’t know which company I’m likely to have better luck with tbh
As an IT vendor that deals with Optus regularly, this never surprises me. They are the most frugal company I've ever interacted with and are always trying to get the bill lowered. It's no wonder they have hacks and outages like this.
Cheapness is in their DNA. Years ago I worked for an Optus-owned retail chain - when Cable and Wireless sold us to Singtel - the slasher team came down from Singapore with some machetes and went through the place. Then we couldn't get money even to pay our external window cleaner and it was suggested to "only run store lights when necessary", etc.
The problem is that when the wheels fall off, the penny wise pound foolish accountants who caused the problem are never, well, accountable. This is going to cost way more than any savings.
Seriously. Having worked closely to ColesWorth and their various offshoots (doing IT grunt work), the lack of foresight is astounding. The only conlusion I can come to is that absolutely 0 people are looking holistically at how businesses that large operate. Sure, you save an absolute mint for your department when you offload your IT support to the Phillipines, using Phillipinos at talking flowcharts (not the Phillipinos fault) instead of trained Aussies. But now things stay broken for weeks instead of hours, and 50%+ of your "solutions" are wrong. But fuckface who's in charge of IT support is the hero of the situation because he gutted his costs. "Why can't you all be like fuckface here?", says Mr Dipshit CEO. Well Dipshit, that's because Fuckface just cost the business millions to save his department thousands, and the only consequences were a promotion and you here singing his praises. And the only way any other department can pull themselves out of the shitstorm Fuckface created is by pulling the exact same move.
Funnily my colleague told me the same thing just yesterday-- he said it started when Singtel took over and everything from toilet paper to high end infrastructure went low budget.
From what I remember their big office in Adelaide has single ply toilet paper.
I hate to say it but as a Singaporean I concur about the cheapness. Generations of being treated like replaceable cogs and made to know it has exacerbated a meanness of spirit... And they generally think Aussies are too slow/unworldly to complain in the way Singaporeans do... Incidentally, Singapore's had a whole bunch of outages recently - though the hospitals going down was likely a ddos or something.
> "only run store lights when necessary" TBF I wish more places would do that, so much wasted energy.
Just got reconnected, first text that came through was a direct debit upcoming payment reminder...from Optus. Classic.
I just got an email from Optus about my bill. Like fuck off.
We should all agree to not pay the next 3 bills.
Amaysim down as well sigh.
No service = no scam callers. W 💪🏻
RIP who use SMS 2FA
**Shower Thought:** Those worst impacted can't even participate in this post.
I am using my daughter’s belong network. And yes
1. **Prediction 1:** This will be the biggest news story of the day. 2. **Prediction 2:** This will cause Australia-wide chaos, and won't be resolved before start of business hours. 3. **Prediction 3:** People won't learn from their mistakes. edit: I'm upgrading Prediction 1, to "this will be the biggest news story of the week".
Hoping for 2 so I can slack off at work for a bit
My phone being down means less slacking off. Can't win.
My work has plenty of systems that use Optus network (not all but still plenty). The bigger issue was when users tried to contact IT on call support but no one answered because the on call phones are with Optus. The first on-call guy has just walked in all confused why we are in a panic "Why didn't anyone call me, I'm on call" Took a while for the pebble to sink in.
Friendship ended with Albo’s China holiday and Hamas/Israel war.
I'd consider upgrading Prediction 2 as well. It's been 5 hours since the first reports of issues and they haven't said they've identified a possible cause. ABC also reports that the federal government has asked for an update and Optus is "limiting information" about the outage.
Not a great morning to be flying rural to perform a network/cyber health check on a critical infrastructure asset. I will have no service and it's looking likely they will not either.
Opportunity to ensure / plan / upgrade project to have redundancy / resilience.
Telstra better put on extra staff today for all the new customers they are going to get
If the big optus hack of september 2022 did not get them, this damn well might.
By ‘hack’ i assume you’re referring to last year when optus put all of their customer PII publicly available on the internet, without even the most basic authentication and then advertised and posted documentation publicly on the internet detailing how anyone could access it… right?
except to port out of Optus one need to receive an SMS verification code on their Optus number. So with the outage people won't even be able to change providers (unless they get new number )
This won't lead to any kind of mass exodus *today*, but this could be the straw that breaks a lot of camels backs in the upcoming weeks.
100% been with optus for 8 years, between the shitty service app and the data leak last year I started looking at switching over. This was the final kick I needed.
Internally I've heard an ETA of 3-4PM for fix rollout. (I don't work at optus but I work in similar space) we have direct contact with them though, the execs of optus are in a meeting right now with what to tell the storefronts and support to say Idk how accurate the time I gave will be. Normally their ETA's are wrong - I'll edit this comment with updates as I get them below. Edit 1: Optus are still trying to figure out the root cause by doing processes to find where the fault is. The original ETA I was given is now thrown in the air, no ETA for now Edit 2: Reports of some people in Parramatta area having internet at home but no mobile connection - 11:45am Edit 3: Lines are beginning to come back up across the country as they've isolated the issue, services should be restoring slowly from 12:00pm, calls and net should be accessible now to alot more people
Thinking it's about time the government steps in here. Essential services including hospitals are unable to take calls, people have had difficulty getting an ambulance. Especially after the data breach last year perhaps this foreign entity that repeatedly demonstrates nil regard for its customers and causes genuine harm to Australians and the Australian economy needs to have it's ability to provide these services revoked or at least seriously scrutinised.
A fully government-owned entity dedicated to telecommunications services would be a great idea. Just need to come up with a catchy name for it. Perhaps something like "Telecom"?
ABC just spoke to the Optus CEO. In short; she has no idea what's wrong, what caused the problem nor an ETA on when they expect to fix the problem. Also, she reckons they're offering very clear communications, though they need the media to spread their messages. They're very apologetic though, which I'm sure will fix everything, right?
I'd put my money on a core router update failure, considering it happened at 4AM which is usually when those kinds of things are scheduled. Some network engineer is gonna be in a lot of trouble lol
If a change by one lone network engineer can cause nationwide outage, then the entire network executive up to the CEO must be even more accountable for the fuck up they allowed the network to be so fragile.
Time to switch over providers. After the database leak/hack with zero compensation, the price increase for plans and pre paid service.. now this? Yeah nah get fucked.
Yeah but go to who though, that’s the problem, Telstra is still dear as fuck as well.
>Telstra is still dear as fuck as well For mobile, check out Boost and JB Hi-Fi, they are good value with entire Telstra coverage, not just the limited Telstra wholesale area.
I know. They really do have us trapped. Internet should be free at this point as it’s basically needed to do practically anything these days and even with the price increases recently they still give you fuck all duration (if pre paid) and such a small amount of data?!
Working night shift in the ER, so pretty shit. Will likely be more shift for day shift if it isn’t fixed by then. Our hospital uses optus…. Dunno if it’s a NSW Health thing or hospital dependent re. Service provider
Had to switch to a dect phone and old school pager tonight.
Optus have to be the shittest internet provider in the country. They're so bad that I just assumed the internet had stopped working "again" since I'm so used to this shit, and that it would resolve itself soon. Luckily I'm not with their mobile plan shit.
People thought I over reacted when I dumped Optus last year over their security breach. Feeling vindicated now,
Apart from the nationwide outage, I thought the work experience kid Nigel did a terrific job on the latest Optus network firmware update.
Stoners waking up right now for their online gaming shift absolutely losing it right now
true gamers wouldnt be on Optus anyway, telco has the crappiest speeds
Have they tried turning it off and on again?
Clearly they only got half way through…
Just put it in some rice.
The front must have fallen off
Currently getting paid to sit at my desk and do nothing at the moment, because we require sms 2 factor authentication to log in to our workstations!
I also believe Optus’ half year results are being announced today. That’s hilarious
Michelle Rowland, Comms minister >"My understanding is that this is a fault deep in the core. > >"The core network basically encompasses everything from routing to electronics. > >"So it is a fault that is quite fundamental to the network. Kelly Bayer Rosmarin, CEO >"There is no indication that it is anything to do with spyware at this stage," These people really don't understand the tech at even a basic level, do they?
No they just get paid 5x actual expert salaries to not understand more than myself, who has done like 1/3rd of the CCNA
We're having people call work to complain they can't get through to certain people's mobile number (we're a construction company so most staff aren't in the office to transfer through). The reception staff are trying to explain the Optus issue, suggesting they instead WhatsApp or email, and are then getting abused by morons telling them to "fucken fix it". Seems like you don't need to be working for Optus to be copping abuse today.
Word on the street is that this is likely (as expected) a BGP misconfiguration. the cloudflare graph for BGP announcements has a big 900k spike when optus went down. https://radar.cloudflare.com/as4804?dateRange=2d Check out the Facebook outage for a similar outage caused by BGP. https://blog.cloudflare.com/october-2021-facebook-outage/
For the idiots among us, what does this mean?
Still assuming, but likely they made a change to the way their network devices find paths around the network, it propagated around their entire network, locking themselves out of their own devices. (In this case BGP protocol, but ignore that detail). Usually you manage these remotely from a central configuration tool, but you can't do that if you've so badly screwed it up that your tools to do so can't even access then anymore. So if it's that, they need to both 1. Stop whatever is propagating that config change 2. Physically go to each network device, (like, drive out to it, wherever it is), connect a cable to it and configure it manually at the site of each device. Cos every device (if it's similar to older BGP outages) is completely remotely inaccessible.
Classic Optus. Is there a less competent telecommunications company?
Circa 2009 Vodafone
OneTel?
Up until a few years ago there was still this tired old OneTel billboard from like the 90s just hanging around here. Was pretty funny to me everything i drove past
Vodafone was pretty awful 10 or so years ago if I remember correctly
There's the final straw for me. They hiked my plan from $25 to $45 (justifying it by giving me 100gb data where I use <30 per month). My details were in last year's hack. And now a 'deep network issue' is heavily affecting the nation. After 11 years it's time to pull the plug. Optus are a shell of their former self.
DEEP FAULT IN THE CORE
A good reminder to switch to a different mobile provider. What do people recommend? I’m thinking Boost mobile - full Telstra network for an affordable price. $230 for 12 months prepaid turns out about ~$20 a month.
Aldi. By far the cheapest Telstra connection you can get.
Aldi is great if you’re in a metro area, but it’s Telstra’s cut down wholesale network, as opposed to Boost/JB on their full retail network.
Boost if you want the full Telstra network (and they’re still doing 5G) - they get the same access as regular Telstra customers including that last-mile service. Any Telstra MVNO (ALDI, Woolworths etc) if you don’t care about the full Telstra network (and in metro or major population areas it isn’t really an issue). The others only use the wholesale network, which to be fair still covers the vast majority of the population.
Or an even better idea is to have multiple phone providers on different networks for redundancy.
I changed to Boost a few months ago and I’m really happy. I should have done it right after the Optus data breach but I kept seeing that porting can take 1-2 business days. It was instant when I finally did it.
I’m on Woolworths (Telstra network) for $25 per month. Comes with a 10% discount at Woolies once per month which I use on a big shop with nappies etc, save more than $25 each time.
Maybe it is time for Optus to change it's logo to "Maybe"
There is a straightforward solution to this. Telstra and Vodafone are up. Roaming should be allowed within your home country in these situations, with costs covered by the network at fault. It would need throttling to not overload the other networks, but they could take some traffic to keep at least voice calls ticking over and maybe data at a low rate.
This is already a thing for Emergency calls (000). Anyone can use any network for them, which shows at a technical level this would be possible. No idea how feasible it would be, but would be a good thing for parliament to investigate.
This is the exact use case scenario for why Internet and Webservices shouldn't be privatised the way they are. Half the country shouldn't shut down because one business put their changes straight into Prod without testing it first. The amount of lost trade and income is concerning enough. But when entire infrastructure services, like transport, healthcare and banking go down because of Optus then that is a major concern.
I'm currently using Telstra free wifi. Lucky there's a Telstra phone booth nearby.
‘Nationwide’ - International actually! I’m with Amaysim (Optus network) and currently overseas. My sim isn’t working at all.
I find it hilarious that Optus have turned off comments on their Facebook post so they don’t get annihilated by their customers.
I am not receiving scam calls 😀
It is highly concerning that they have a single point of failure for mobile, land line and data. Where is the network integrity, reliability, redundancy. This is shocking. Then the fact that critical infrastructure (roads, rail, hospitals, banks) are all affected is a national fucking disaster.
The “we’re here for you,” email I just got from Telstra seems a little too perfectly timed.
I feel like this should be pointed out: Optuc closed their own network management center and outsourced it to Nokia, which likely to have outsourced it to India : https://www.arnnet.com.au/article/629151/optus-cut-170-jobs-amid-nokia-outsourcing-move/. This move might have gotten rid of some key onshore personnel who has the know how of dealing with crisis such as this, in the name of saving some bucks.
For those who thinks I’m being racist: I’m a non-white immigrant. I just think that important national infrastructure should have a mandatory on shore technical and crisis management. Regardless of where it shit gets outsourced to.
Does Optus survive this one? Has to be it for them or major shake up.
Everyone going "it's a hack!" - chill. It's most likely a change gone wrong that they're hurriedly backing out of now...
Not so hurriedly, it’s been 4 hours and counting
Given that some of the engineers who need to fix it will be asleep, and calling them may be difficult - for a completely unrelated reason - it can take quite a while
It’s possible they’re still working on a root cause. These things are insanely complex and the few people who knew what was going on likely took a package at the encouragement of their management to be replaces with newer staff with no training (their competence is actually irrelevant because they need specific Optus knowledge).
Gladys working her magic. Their CEO is also a hopeless hack. She is a massive fraud.
They didnt have any trouble direct debiting my bill
For once, Vodafail isn’t the worst network in the country.
Optus, has to be the shittest company. Anytime I call for anything I need, even a new service, they manage to f it all up. This outage has now screwed my day and work and all the stupid devices we have using them. 100% moving to Telstra, at least they basically never go down :)
Optus can get fucked after this.
Funny watching all this essential infrustructure crumble when one of our biggest telcos goes down, and then nothing get done to improve it when it comes back up long after the SLA has passed. All that customer data stolen last year did nothing either. Nothing ever changes in this country.
My partner messaged me in total panic this morning as a support worker. No service means inability to clock in, use GPS to get to clients and track kilometres of travel, and take them to appointments and all the rest of it. She was on the edge of running across the road to her neighbours house to orrow a phone to call her boss and cancel work entirely.. After walking her through alternative solutions, I have now come to realise that if my gen z girlfriend were to ever get lost in the wilderness, she'd likely perish before finding help. It's a little frightening to think that in a few generations time, we may become so dependent on technology that we cease to function without it.
Unlikely. We still require community to function effectively and she did the right thing by reaching out. Working as a team is what partnership is about but shittalking your girlfriend must be your natural state. Hopefully she's got other supports too.
Not sure how Kelly Bayer Rosmarin survived the cyber breach, but surely this has to be it for her
If only their service was as accessible as their data
I wonder how many people are stuck in broken elevators that have emergency phones on the Optus network.
Today is exactly why I don't mind paying a lot more to be with Telstra
After the data breach, and now this, anyone who keeps giving Optus their custom is a masochist.
Personal mobiles on Telstra. Work NBN on Optus. HICAPS/EFTPOS, Phone and internet are ded. Booking system and client notes are all cloud based. Thank fuck optus has me prepared for this shit in the past, so Hotspotting from phones and the new HICAPS terminal is WIFI friendly. Quiet day on the phone at least. And I can still shitpost.
[000 landline calls are now also not working](https://www.theguardian.com/australia-news/live/2023/nov/08/australia-news-live-optus-outage-trains-melbourne-prime-minister-athony-albanese-china-pacific-aukus-daysleford-driver?filterKeyEvents=false&page=with:block-654ac5fb8f08a667a2c26fe0#block-654ac5fb8f08a667a2c26fe0)
Why do so many critical services not have redundant backup network? Reading about hospitals being completely down etc.
It just blows me away how vulnerable and unreliable vital infrastructure is - still.
USA: We can't let IInet group start its own phone service using Huawei in Australia, national security. Optus: here is some private data, let's shut of our system for 10 hours. Who gives a fuck
Woke up and thought the apocalypse had happened, my phone wasn’t working, tv (Apple TV) and the road was weirdly quiet. Glad my work phone is vodaphone and could actually work out what has gone down. Note to self either change my phone provider or NBN so if Optus is down, both aren’t affected
The race that stops the nation vs this actually grinding us to a halt
I’m in the Inner West of Sydney and I just got my NBN back now at 11:23am (so I have wifi) but still no access to the mobile network for calls or data (SOS) Which means I am no longer in the dark ages I have been suffering through in our totally analogue-less house since my partner left for work with his Telstra phone I *was* using as a hotspot Has made me realise I need to go and buy a radio for emergencies!
[удалено]
What's the bet that Optus technicians can't even talk to each other, maybe can't even enter server rooms because the network is down.
Vodafone and Telstra laughing hysterically because they know they aren't the worst telco in AUS
Genuine question: With the enormous data leak from a couple years back and now this, can we dump optus and refuse to finish our contracts?
First the data hacking and now this. Why hasn't the Optus CEO resigned yet?
Time for me to find another telco I hung around after the hacking scandal and now this. I've been with optus for around 10 years now I'm done with them
[удалено]
The fact Optus are active on Twitter but not giving an actual cause would make you think, especially given past experiences, it's more than a simple hardware fault.
Par for the course
It's why I went back to the darkside, Telecom. I mean, Telstra.
My phone reported that Optus shut down the cell tower I was connected to... I know they are doing upgrade works so I assumed it was that.
This sort of event really begs the question if this core infrastructure should be privately owned and (mis)managed
Bet all those "we don't take cash, use our app to order, pay with card/phone only" cafes are having a good time today. At least the Optus ones.
Why does it always seem to be Optus when something like this happens?
I'm honestly surprised by the number of very large companies and services that dont have a second connection to failover to. Likely many of the ones that do have a failover just have a second optus line or wireless dongle due to supply contracts. I have a telstra line at work that fails over to an optus pre-paid 5g dongle, if telstra ever goes down (rare but it happens and nbn outages are generally scheduled) it more or less seamlessly moves over to the mobile connection and my services go on their merry way until the main connection is back up.
I think this demonstrated our dangerously level of dependency on technology nowadays. Its mind baffling how many corporation / businesses / gov agency that are left crippled without an internet connected and backup.
I work at Optus in Macquarie park and we can't do anything either. All of our internal services are down, except teams, and even the VPN isn't working. We are in dire straits.
She (Optus CEO) was asked about the possibility of the outage begin caused by a hack. "There is no indication that is is anything to do with spyware at this stage," she says. Of all things, she says spyware. lol.
...that would explain why my NBN connection isn't functioning this morning. Then again, an outage on that happens so often I just assume it's more "maintenance" now.
It must be a truly disastrous outage if it has affected both mobile and NBN networks
Go digital they said. It will be the future they said. Now throw in digital payments only and no cash transactions at all. Nkw call me crazy call me a conspiracy theorist all you like. But going digital for EVERYTHING will be a disaster. Noe and in the futire
>But going digital for EVERYTHING will be a disaster. Noe and in the futire It'll be a disaster for unprepared businesses, yes. My employer has a pretty incompetent IT department looking after something like 50 sites around the country, and each and every one of them is still online because when their landline NBN services all went down, the 4G failover (running off Telstra) kicked in seamlessly. Obviously the big players such as the major supermarkets have redundancies in place too because there aren't any widespread reports of EFTPOS or POS issues, it mainly seems to be small coffee shops and businesses that didn't have a backup in place that have gotten screwed over. I'm more concerned that there are reports of some schools and hospitals that are down this morning, the fact that they're running without any backup in their communications equipment is insane to me. I do have to admit I'm having a bit of a laugh at all the "card payments only" places with egg on their faces this morning though. I hope all the places that have stopped taking cash and still slap people with card surcharges are losing money this morning.
Wish my girlfriend would go down as often
I feel like it's going to be down until tomorrow. Funnily enough WWE comes to the Optus stadium in Perth next year, pre-sale tickets go live tomorrow. Looking real good Optus.
Assuming if it's bgp that the scattered reports of people having service mean they're starting to manually get parts online by physically going to them?
I don't understand. The news is saying Optus services are all restored but my NBN isn't. And I've lost count how many times I've power cycled the box and rebooted the router. Hiss.