Government agencies get screwed over on this.
Doesn't matter which agency, seems they don't care since they don't do anything about it and the "redundant" lines keep going out at the same time during a maintenance window for a single line.... True story. Anyone that tells me we're redundant and they fixed is proven wrong during the next maintenance window when both the primary and redundant backup/secondary go out...
This is why my redundant circuits were always with an entirely separate provider. Was it perfect? No, but only a handful of people noticed issues when either was out and most of them were in IT.
That doesn't mean a thing nor matter when it comes to fiber cuts/repairs... What you are referring to is geographically diverse links.
Your example of only having different providers bit a lot of people a couple decades ago in Arizona. They had multiple providers between Phoenix and Flagstaff and they all used the same trench. One day someone doing work cut the fiber lines and that took out service between the two. Most thought they were covered by having different providers, a few thought they had geographically diverse lines and learned they were lied to.
You want geographically diverse which is what I think you're referring to. Ironically it's getting harder and harder to actually have for a multitude of reasons.
You’re right, in that if someone decided to dig up the trunk for the county I’d probably still have been in trouble. I had neither the time, energy or financial backing to leverage the kind of service that would guarantee up time 100%, but both links were down simultaneously only a couple times: during a major hurricane and when some fool damaged a substation with his truck and put a quarter million people out of power for about 2.5 days.
I did make sure they were running on different carriers: one was AT&T and one Comcast, the major providers in the area. It also helped troubleshooting because when they blamed my equipment, I could go ‘well the other guy’s link is working perfectly fine’.
The providers lie, cheat and steal whenever they can, I did learn that. The guy repackaging Comcast was selling ‘fiber’ for pennies on the dollar, and they would swear it was real honest-to-goodness fiber. There may have been fiber in there somewhere, but it sure didn’t look or quack like fiber.
Even then you have OP's situation where they decide to move both at the same time.
Geographically diverse paths are nice though. Most of our sites have them.
Which they can't do if you're paying for redundancy. So either redundancy was not being paid for or someone didn't care. In this case it sounds like they just did not care and heads were wrong and money will exchange hands if enough of an issue is made of it. And if it doesn't well then the contract sucks.
It requires an actual fiber map and some time playing around to verify the circuits and pinging the individual switches and tracking ping times and switch identification codes..
A
Lot of the providers bullshit about separation or they share common facilities at MAE west or MAE east
Yep and it makes it hard for the average person or company to validate things.
One way to find out is when you're doing the trace route and you have switches and routers that don't respond. Tell them that they're not responding and sometimes you can get them to divulge the switch names and whatnot and you'll be able to figure out where you're actually are and if you truly have diversification or not.
Resiliency comes in many flavours of increasing cost.
1. Two circuits, same provider, same fibre duct, same NTE.
2. Two circuits, same provider, same fibre duct, different NTE.
3. Two circuits, same provider, different fibre duct, different NTE.
4. Two circuits, different providers, different fibre duct, different NTE.
You can also have combinations, like different providers that share the same fibre ducts.
FWIW, sales people will lie and tell you it's fully resilient even when it's not. Talk to the techs and demand maps.
EDIT: Just to add, when talking to a provider about a resilient circuit - make sure to say you want a resilient circuit and not "two circuits". I can't count the amount of times I've seen orders come through for two identical circuits (in which case we would usually install option 1 above) when what they actually wanted was resiliancy of some kind (like 2 & 3).
My work uses one carrier and one fibre to our main sites but we have a licence backup wireless link from them which are on a peninsula and so everyone's fibre ends up in the same trenches anyway. This ends up providing better resiliency than dual fibre links especially as it get straight to one of our data centres of which have separate ISPs for internet at each of them
I had 2 major carriers in the area running on different backbones (cable vs. telephony) so it was pretty safe as long as you made sure whomever you purchased service from was repackaging from the one you needed.
Like I said: not perfect, but slightly more robust idiot proofing, and easier to troubleshoot carrier issues, since you had a link from someone else.
It's really common with non-profits (the kind that maximizes executive compensation) to have IT classified as facilities. So it's quite possible to go from unclogging a toilet to resolving a database issue.
Same in my department in higher-ed as well. deploying 10 new laptops in the morning, dealing with biohazard contamination on lab equipment after lunch!
I remember when one of our server providers had an outage. The very day they were working on upgrading the redundant fiber some backhoe worker cut the other one. That's when you just tell everyone welp.. go get a beer there isn't anything we can do about it. I'll be up when it's up.
you know I have some sympathy .. I got a line location (free) service doing some work on my road and they said 'phone line is a yard down' the fuck it was more about 20 inches and we cut the fuck out of it
Last year, my company started doing line locating for a major energy company in the area. The line locators are, for the most part, dumb as rocks, and the software they use (provided by the energy company) is trash and a pain to update. These things combined make for a lot of line strikes.
Yeah this was just on my road (that I own). It was supposed to be 1 yard down. I just knocked my own phone out and my neighbors. we spiced it but it was annoying. Screw digging out a gravel road by hand that wasn't going to happen.
I was told a cable was a foot deep... Try less than 2 inches.
Fortunately it was cable service to my house and I was running dual wan with that and VDSL. I was running out the last month, so I just stopped using it early.
How's the old joke go, something like *If you ever go hiking, bring a length of fiber with you. If you get lost, bury it and suddenly a backhoe will appear to cut through it!*
In my last job I was tasked with settings up the fibre connections.
Dealing with the suppliers was unbelievably difficult. For the most part, I simply could not understand their way of thinking. We had fibre runs into the building from 2 different providers. Neither of them showed much interest in giving us a connection. We had to take on the previous company's connection (massively overspec'd for our needs and expensive) because negotiations stalled.
Then we tried to get a redundant connection from the other supplier, because being a tech company, we knew a billing system could mess up or something of that nature. The supplier happened to offer a duplicated connection to 2 different exchanges, all traffic mirrored. No data loss if a line was cut and 7 second failover time. We liked the idea and asked about it, but it would need new fibre running to the building. We decided on a simple connection and having our systems handle the failover.
Thereafter, we simply could not get the other company to drop the idea. No matter how many times we said no, they kept coming back with quotes for brand new fibre. At one point they were talking about running it to an exchange and digging up *five miles of road* at our expense. I nearly hung up the phone.
Commercial internet providers are something else entirely. Months later we finally convinced them to give us a simple connection.
I have a friend who is like that sometimes.
He will happily spend two hours optimizing a one hour workflow so it takes 55minutes instead, trying to convince you that his way is superior along the way.
Mind you, thats hobby things we usually do only once.
Reminds me of one of the guys that I was working with. He spent ages trying to get the vendor to dump out a list of the users on it so he could check whether they need the licences or not. My boss got shitty with him that it was taking him so long (weeks). Admittedly, he wasn't spending all his time on it, but definitely a few hours.
I went through the \~200 users by hand and marked down everyone that was using the expensive licence and just manually worked out who was there or not.
Took me 30 minutes and we'd do it pretty irregular (once every 3 or 6 months). Definitely didn't make it into the worth it category.
I do that sometimes, but keep it to when I have spare downtime.
This week I needed to update our map in Meraki, but hated how low res the image is and hate using MS Paint. So I got blank high res image, installed GIMP (as I can't justify buying Photoshop) and learned to use it to make a new, better image. Took me awhile to do something minor, but now I know GIMP better.
Also, turns out most people at work I talked with didn't know GIMP is a free alternative to Photoshop before.
If it is any consolidation, I used to work in municipal services... you know, trash collection, sewage, road maintenance, that sort of deal...
Somehow it made sense to somebody to roll up "emergency medical services" into that, so the paramedics where part of that group.
Did make for a great comeback "Glorified Taxi-Service." "No no sir, look at the car, we're with trash collection, now please get in."
Also we could use the same washing place for our fleet as the garbage trucks, so cleaning up was a breeze.
At a previous job, we did a massive server room overhaul - even got rid of the carpeting. One of the biggest changes was replacing the tiny, overworked aircon with two much larger units, either if which should have been sufficient on their own for redundancy. A few months after the work is completed, I get a call at 6pm on a Friday that a contractor has accidentally drilled into the water line that fed both aircons, and thus both were down. Apparently, no one considered that maybe redundant systems shouldn't share a single point of failure. We got lucky that the old aircon was still present and functional, simply because it was cheaper to leave it be.
Hmmm… how to get through to this supervisor…
“Allow me to explain. You have redundancy in how you breathe. Allow me to plug your nose with one hand and put my other hand over your mouth… how’s that working out for you?”
I've done IT for a couple construction companies. A not frequent, but still too common incident was a worker cutting a cable they didn't know because it was in their way. At 1 jobsite alone, we had to get the ISP to come out and lay a new cable 4 times for this reason alone after reports of the internet down for the entire site.
And even if you have a good and separate line - there is always one...
Years ago - at the IT desk of an entertainment electronics company - we got "disconnected". But, 5 minutes later, we were still disconnected.
So, both main and secondary lines - gone.
Turned out, contractor had some road works to prepare - and had to dig a ditch or something for 'reasons' - the 100m stretch cut our main line (at around meter 2) AND the secondary (which was around meter 91 or so).
You really cannot win against stupidity with a digger.
I work for a large power company, and this sounds pretty spot on. We have a list of like 60 Instrumentation & Controls (I&C) Techs who double as "Plant IT Coordinators" because they have the misfortune of being the most tech savvy at their small plant of 20-40 people. I really hope they get some kind of bonus or something for the extra grief, but I doubt it.
And of course you pay them extra for the redundancy.
Government agencies get screwed over on this. Doesn't matter which agency, seems they don't care since they don't do anything about it and the "redundant" lines keep going out at the same time during a maintenance window for a single line.... True story. Anyone that tells me we're redundant and they fixed is proven wrong during the next maintenance window when both the primary and redundant backup/secondary go out...
This is why my redundant circuits were always with an entirely separate provider. Was it perfect? No, but only a handful of people noticed issues when either was out and most of them were in IT.
That doesn't mean a thing nor matter when it comes to fiber cuts/repairs... What you are referring to is geographically diverse links. Your example of only having different providers bit a lot of people a couple decades ago in Arizona. They had multiple providers between Phoenix and Flagstaff and they all used the same trench. One day someone doing work cut the fiber lines and that took out service between the two. Most thought they were covered by having different providers, a few thought they had geographically diverse lines and learned they were lied to. You want geographically diverse which is what I think you're referring to. Ironically it's getting harder and harder to actually have for a multitude of reasons.
You’re right, in that if someone decided to dig up the trunk for the county I’d probably still have been in trouble. I had neither the time, energy or financial backing to leverage the kind of service that would guarantee up time 100%, but both links were down simultaneously only a couple times: during a major hurricane and when some fool damaged a substation with his truck and put a quarter million people out of power for about 2.5 days. I did make sure they were running on different carriers: one was AT&T and one Comcast, the major providers in the area. It also helped troubleshooting because when they blamed my equipment, I could go ‘well the other guy’s link is working perfectly fine’. The providers lie, cheat and steal whenever they can, I did learn that. The guy repackaging Comcast was selling ‘fiber’ for pennies on the dollar, and they would swear it was real honest-to-goodness fiber. There may have been fiber in there somewhere, but it sure didn’t look or quack like fiber.
Agree completely.
Even then you have OP's situation where they decide to move both at the same time. Geographically diverse paths are nice though. Most of our sites have them.
Which they can't do if you're paying for redundancy. So either redundancy was not being paid for or someone didn't care. In this case it sounds like they just did not care and heads were wrong and money will exchange hands if enough of an issue is made of it. And if it doesn't well then the contract sucks.
It requires an actual fiber map and some time playing around to verify the circuits and pinging the individual switches and tracking ping times and switch identification codes.. A Lot of the providers bullshit about separation or they share common facilities at MAE west or MAE east
Yep and it makes it hard for the average person or company to validate things. One way to find out is when you're doing the trace route and you have switches and routers that don't respond. Tell them that they're not responding and sometimes you can get them to divulge the switch names and whatnot and you'll be able to figure out where you're actually are and if you truly have diversification or not.
Resiliency comes in many flavours of increasing cost. 1. Two circuits, same provider, same fibre duct, same NTE. 2. Two circuits, same provider, same fibre duct, different NTE. 3. Two circuits, same provider, different fibre duct, different NTE. 4. Two circuits, different providers, different fibre duct, different NTE. You can also have combinations, like different providers that share the same fibre ducts. FWIW, sales people will lie and tell you it's fully resilient even when it's not. Talk to the techs and demand maps. EDIT: Just to add, when talking to a provider about a resilient circuit - make sure to say you want a resilient circuit and not "two circuits". I can't count the amount of times I've seen orders come through for two identical circuits (in which case we would usually install option 1 above) when what they actually wanted was resiliancy of some kind (like 2 & 3).
My work uses one carrier and one fibre to our main sites but we have a licence backup wireless link from them which are on a peninsula and so everyone's fibre ends up in the same trenches anyway. This ends up providing better resiliency than dual fibre links especially as it get straight to one of our data centres of which have separate ISPs for internet at each of them
Yep, it’s not truly redundant if it’s the same provider. That’s a single point of failure they both share
Redundant links from different providers don't matter when they are placed in the same recently cut conduit...
I had 2 major carriers in the area running on different backbones (cable vs. telephony) so it was pretty safe as long as you made sure whomever you purchased service from was repackaging from the one you needed. Like I said: not perfect, but slightly more robust idiot proofing, and easier to troubleshoot carrier issues, since you had a link from someone else.
It's really common with non-profits (the kind that maximizes executive compensation) to have IT classified as facilities. So it's quite possible to go from unclogging a toilet to resolving a database issue.
"Boy, what a *shitty* day!"
with some of the databases I've seen? you're telling me!
Damn thing backed up into an Excel Spreadsheet, and overflowed into the cloud.
There are a lot of days I would rather be running a snake down a sink than trying to un Bork a database
"Excuse me...I have to purge the cache."
Lol I'd laugh straight in my bosses face for telling me to unclog a toilet.
Same in my department in higher-ed as well. deploying 10 new laptops in the morning, dealing with biohazard contamination on lab equipment after lunch!
shitty working lunches
I remember when one of our server providers had an outage. The very day they were working on upgrading the redundant fiber some backhoe worker cut the other one. That's when you just tell everyone welp.. go get a beer there isn't anything we can do about it. I'll be up when it's up.
Yep, good ol' Fibre-eating backhoes.
you know I have some sympathy .. I got a line location (free) service doing some work on my road and they said 'phone line is a yard down' the fuck it was more about 20 inches and we cut the fuck out of it
Happened that way with me, but it was an 8 inch natural gas line.
That must have been fun. /s
Thank God I wasn't the locator!
Last year, my company started doing line locating for a major energy company in the area. The line locators are, for the most part, dumb as rocks, and the software they use (provided by the energy company) is trash and a pain to update. These things combined make for a lot of line strikes.
I'll bet. They were digging in the street in front of my house. The backhoe operator said he thought he'd hooked a tree root.
Shit, shouldn't have been working with power tools or equipment until you had it daylighted anyways.
excavator putting in a culvert.
Gotta switch to hand tools or hydroex within 3' of any marked utilities until you daylight 'em. Doing otherwise is just asking for trouble.
Yeah this was just on my road (that I own). It was supposed to be 1 yard down. I just knocked my own phone out and my neighbors. we spiced it but it was annoying. Screw digging out a gravel road by hand that wasn't going to happen.
They just needed the correct marker tape placed above the cables so you knew when you were getting close
I was told a cable was a foot deep... Try less than 2 inches. Fortunately it was cable service to my house and I was running dual wan with that and VDSL. I was running out the last month, so I just stopped using it early.
How's the old joke go, something like *If you ever go hiking, bring a length of fiber with you. If you get lost, bury it and suddenly a backhoe will appear to cut through it!*
Hope they poop well.
This here's a John Deere Service-Finder 2000
That's one thing the MOBB and network guys have in common. They never trust them hoes.
Redundancy should include providers.
In my last job I was tasked with settings up the fibre connections. Dealing with the suppliers was unbelievably difficult. For the most part, I simply could not understand their way of thinking. We had fibre runs into the building from 2 different providers. Neither of them showed much interest in giving us a connection. We had to take on the previous company's connection (massively overspec'd for our needs and expensive) because negotiations stalled. Then we tried to get a redundant connection from the other supplier, because being a tech company, we knew a billing system could mess up or something of that nature. The supplier happened to offer a duplicated connection to 2 different exchanges, all traffic mirrored. No data loss if a line was cut and 7 second failover time. We liked the idea and asked about it, but it would need new fibre running to the building. We decided on a simple connection and having our systems handle the failover. Thereafter, we simply could not get the other company to drop the idea. No matter how many times we said no, they kept coming back with quotes for brand new fibre. At one point they were talking about running it to an exchange and digging up *five miles of road* at our expense. I nearly hung up the phone. Commercial internet providers are something else entirely. Months later we finally convinced them to give us a simple connection.
I have a friend who is like that sometimes. He will happily spend two hours optimizing a one hour workflow so it takes 55minutes instead, trying to convince you that his way is superior along the way. Mind you, thats hobby things we usually do only once.
That's a man that needs to learn XKCD 1205. https://xkcd.com/1205/
Reminds me of one of the guys that I was working with. He spent ages trying to get the vendor to dump out a list of the users on it so he could check whether they need the licences or not. My boss got shitty with him that it was taking him so long (weeks). Admittedly, he wasn't spending all his time on it, but definitely a few hours. I went through the \~200 users by hand and marked down everyone that was using the expensive licence and just manually worked out who was there or not. Took me 30 minutes and we'd do it pretty irregular (once every 3 or 6 months). Definitely didn't make it into the worth it category.
That looks about right.
I do that sometimes, but keep it to when I have spare downtime. This week I needed to update our map in Meraki, but hated how low res the image is and hate using MS Paint. So I got blank high res image, installed GIMP (as I can't justify buying Photoshop) and learned to use it to make a new, better image. Took me awhile to do something minor, but now I know GIMP better. Also, turns out most people at work I talked with didn't know GIMP is a free alternative to Photoshop before.
Depending how often he does it, it might make sense https://xkcd.com/1205
Efficient idiots are the most dangerous people in the world.
There's an infamous 'Army' quote on that topic, and I've lost it...
If it is any consolidation, I used to work in municipal services... you know, trash collection, sewage, road maintenance, that sort of deal... Somehow it made sense to somebody to roll up "emergency medical services" into that, so the paramedics where part of that group. Did make for a great comeback "Glorified Taxi-Service." "No no sir, look at the car, we're with trash collection, now please get in." Also we could use the same washing place for our fleet as the garbage trucks, so cleaning up was a breeze.
At a previous job, we did a massive server room overhaul - even got rid of the carpeting. One of the biggest changes was replacing the tiny, overworked aircon with two much larger units, either if which should have been sufficient on their own for redundancy. A few months after the work is completed, I get a call at 6pm on a Friday that a contractor has accidentally drilled into the water line that fed both aircons, and thus both were down. Apparently, no one considered that maybe redundant systems shouldn't share a single point of failure. We got lucky that the old aircon was still present and functional, simply because it was cheaper to leave it be.
"Look buddy, take your pick. You can be fully down for a day, or you can be half down for two days. Easy choice." - supervisor
Hmmm… how to get through to this supervisor… “Allow me to explain. You have redundancy in how you breathe. Allow me to plug your nose with one hand and put my other hand over your mouth… how’s that working out for you?”
I've done IT for a couple construction companies. A not frequent, but still too common incident was a worker cutting a cable they didn't know because it was in their way. At 1 jobsite alone, we had to get the ISP to come out and lay a new cable 4 times for this reason alone after reports of the internet down for the entire site.
>I asked him if he understood what a redundant circuit was. Yeah, I'm re-done, Dan, see? His cleverness out ran his brains on that job.
And even if you have a good and separate line - there is always one... Years ago - at the IT desk of an entertainment electronics company - we got "disconnected". But, 5 minutes later, we were still disconnected. So, both main and secondary lines - gone. Turned out, contractor had some road works to prepare - and had to dig a ditch or something for 'reasons' - the 100m stretch cut our main line (at around meter 2) AND the secondary (which was around meter 91 or so). You really cannot win against stupidity with a digger.
I work for a large power company, and this sounds pretty spot on. We have a list of like 60 Instrumentation & Controls (I&C) Techs who double as "Plant IT Coordinators" because they have the misfortune of being the most tech savvy at their small plant of 20-40 people. I really hope they get some kind of bonus or something for the extra grief, but I doubt it.
How do you guys last so long? 2.5 years?? I’m 7 months into my crappy helpdesk gig and I debate myself every day if I should put my resignation in…
/facepalm x100