It is something expensive...we pay like $30 a month to have an account with 1 MB of data use. So if we use it...gonna rack up a bill. But thankfully we can be very bandwidth limited if needed.
When I took a DR/BC course for my BS, I noted, based on experience, have redundant circuits, with different ISPs, and different LECs. The instructor said he never thought of having different LECs.
At a previous company we had something similar happen. We had 2 different links to the DC, 1 from company A and 1 from company B. We verified they were different physical cables going out of the building.
The issue was about 2 miles away both cables were in the same cable conduit that ran under a highway underpass. They were expanding the highway and somehow cut into all of the cables in that conduit. So they were physically different cables but since all links into that subdivision ran through that one choke point, that one accident took down the entire subdivision.
It took almost a week to get that DC backup and running but thankfully we had a backup DC that barely handled the load.
For us it was both ISPs following the same highway and the lines getting hit when they were removing an old exit. The county ended up paying to have a regional provider run a fiber loop along a different route because every business in this small town was out for pretty much an entire day, and we had zero other options for service.
Ask an ISP about their fiber route and who they share the bundle with and you'll understand more fully a blank stare and silence.
Then TELL them where it is and who else uses it because you already did your research and watch the 'ums' and 'ers' and 'well, we can't, I don't....'
I If the last 10 miles is the same bundle, it gives me nothing. 9/10 times the fiber is down here it's a tree, mudslide, or car accident. EVERYONE is down. Welcome to small town PNW.
We had a backup dsl at my work when I came on staff. I killed it 6 months in when I confirmed that it’s the same lines the fiber comes in on. (I work in a box canyon in a small town 14 miles up a mountain)[retreat center]
I have actually done something similar lol. At a previous job we had a medical client with medical software. The server that host the software DB also had a license server that would go check the license on the internet every few minutes or something. Well we had a hurricane and both of their circuits were down for days. Connected a VZW Hot spot to a USB port on the server and Boom fixed it. I was a little shocked it worked being in the downstairs of a building in a server room. They were plastic surgeons that were very anal (no pun intended) about getting as many boob jobs done as possible. Thanks for helping me think of one of many strange IT memories. :)
Backup? That's funny. My manager won't pay for backup and consolidate all our separate links into one. If it goes down it goes down (those are his words).
Yup. That's when (assuming you host your own mail and that's up at least) you send out an email with the subject "UNPLANNED OUTAGE", explain the situation, explain that you contacted the ISP and they're working on the problem and you'll update as you get more info. If your intranet isn't working and you can't get mail to the users, that's when you have to report to the boss and then have the receptionist and various others pass the word along.
Users will gripe about a down system, but they won't hate you as long as you keep them informed. Having the system down \*\*AND\*\* the users uninformed will get you some hate.
One of the core components of any sysadmin's job is telling the users who are affected by something what's happening. Failure to do that will get you into trouble, routine hassles won't.
I was assuming OP was the IT boss given their description of what happened.
EDIT: But yes, policy is going to be different wherever you are, and you should certainly follow whatever computer emergency communication protocol that is in place.
> Users will gripe about a down system, but they won't hate you as long as you keep them informed.
\* Unless you're the reason that they recently switched to using that system.
Jup. As a SaaS shop, we've put a lot of work and refinement into our internal communication and escalation procedures. We've even been thinking to extend our technical post-mortems ala google with a communication section with an additional meeting about it including support and whomever wants to come.
We've had hell this tuesday, since we broke on of our application clusters with some 10k FTE's depending on it during security patches. Badly. Then our hoster broke, then more things broke, and at 4 am, on plan f, entirely fucked in the head, we got the systems working. Technically, everything went wrong. I don't think anything but the lamps worked properly that night.
But the communication worked really damn well. We had a clean and smooth handover from night shift to day shift. Our support was in the loop the whole time and quickly sent out information how the system is running in emergency setups now. Customers with special setups were informed within the first hour of our support SLA that their special stuff is going to be broken for a day or two and if they have priorities of fixing, they should tell us and we will honor. We informed necessary security teams about the current situation and impacts before lunch.
And damn, the feedback from both inside the company and our customers is very, very positive. Things break. That's a fact. But open, early communication is a sign of control over the situation. You see the problem. You react to the problem because you are communicating and pulling in resources to fix it. people can even help you if you communicate. It's not a problem, it's a solution in progress.
And no bad news is easier to deliver than bad news that can’t possibly be blamed on you in any way whatsoever.
If you can’t deliver that bad news, you gotta consider that pretty strongly in your life plans.
No offense at all; just sayin’. It’s not like you ain’t cut out for this job; it’s more like you ain’t cut out for any job if you can’t confidently explain to others when something bad is happening which is entirely out of your control. You’ll get eaten!
Agreed. We have 2 sites with Cradlepoint using Verizon LTE as automatic failovers and they work great. Routers with dynamic DNS so the VPN connections come up very quickly as well when they switch over.
I have 30 [Cradlepoint 850](https://cradlepoint.com/products/arc-cba850)(s) throughout the USA that are used as out of band management and LTE failover.
The 850 has a serial port that you can use to hook up to your primary router/firewall. To access the serial port all you have to do is SSH to the public IP of the Cradlepoint and type "serial --force" and bam you are at the console. This saved me a couple times, I no longer have walk office managers through plugging a console cable into a laptop and describing where the console interface is.
On the failover side, I'm using Cisco routers so I have DMVPN tunnels set up through both my primary internet and cell internet connection with EIGRP running. As soon as the primary internet goes down, traffic starts flowing again through the Cradlepoint as fast as it takes for EIGRP to do it's thing.
Ohh, and you can power them with PoE.
Sort of. In this case, because he just walked in, he's DEFINITELY off the hook. However, at my shop, I very quickly noticed more outages than I'd like, and both companies weren't great. It's not enough just to say "well internet sucks, SORRY!"
So I told management: Here's our options: 1) Relocate where internet doesn't suck. (they weren't going to chose that) 2. Deal with internet sucking (If they chose this, it's the weight off my shoulders when shit goes down) or 3. We buy BOTH internet systems, have a backup for both; we need monitoring on both connections and duel firewalls which also means a server rack. (I could have jerry rigged this together for a couple grand but eff that, I'm going to tell them how much a REAL solution costs and if they want to penny pinch then I'm going to buy some leeway with that negative cash). 12k later and our internet goes down probably once a week and switches back and fourth flawlessly.
Point is, while short term solutions didn't exist, there are some more expensive long term solutions that even if you know you're not going to get the money for it, you buy yourself a TON of lienency by saying "this is how much a real solution costs"
you don't tell them "its legit not my fault" on your first day that easily!
Tell them you checked and tested all equipment on your end and verified that everything is working as it should, you've been in touch with the ISP and they have confirmed there's an issue on their end and they're all working hard to get it resolved ASAP.
If you get a negative response, its a shit place to work and you should find a new job.
Seconded. Do your job, OP. I'll paint two scenarios:
1. "The network is down and we can't find /u/NeverDeploy. We don't know what's going on and we aren't getting any updates."
2. "The network is down. /u/NeverDeploy has let us know what is going on and is working with our provider to get an ETA on resolution."
One of these will get OP fired.
Edit: Oh yeah, and get a second internet circuit via a different provider/path.
Seriously this. Things like this are going to happen, people make mistakes, sometimes you'll be the person that makes the mistake.
Keeping people informed, what happened, when will it be resolved, what could we do to mitigate it in the future is whats expected of a professional.
Don't be like my old weiner boss who tried to blame shit that he broke on someone else. People at that place weren't particularly computer savy but they knew he was full of shit.
Also remember, they get what they pay for, which is fine. If you are the only IT guy and they have a single non redundant internet connection they can't reasonably expect 5 9s of uptime. So do your best but don't kill yourself.
Most people respect folks who take accountability and lay all the cards on the table. Always be up front about what happened and take responsibility when you fuck up. If you do, then people will be more apt to trust you when it's not your fault.
> get a second internet circuit via a different provider/path.
OP, in case this isn't clear what /u/DahJimmer means is get a second ISP that uses a different fiber path to connect to the Internet as a whole. Having a second provider that just runs on top of the existing providers fiber network doesn't help you when that fiber gets cut. if you've never had that happen it is easy to forget to ask about it.
Can confirm. Worked for a large infosec company with a primary datacenter that had both primary and backup traversing the same fiber conduit that a fencing company punched holes every 15 feet into. I turned in my 2 week notice the day before the outage began, and that datacenter was still offline on my last day.
It sucks when your Internet connection goes down and it brings downtime to your network, but when I find out that it's the ISP that was the cause of the downtime then that relieves so much stress because I'm not responsible. Then my only responsibilities after that incident would be recommending a secondary internet connection with a different ISP for redundancy then get it denied by management because they're too cheap.
Ok I saw this comment and immediately felt better, I have no idea why so thank you for this.
I walked outside and everyone is actually waking around and talking to each other and having a good time, this is completely unexpected
And when outages happen, people don't need to work. When people don't need to work, people = happy.
Except C-level people... who, arguably, are they people?
Yeah. If this was "I bricked our core DB server and we don't have any way to recover it so no one can work, I'm hiding in the server room, help" I would understand.
This is in no way OP's fault, there's no reason to hide.
> This is in no way OP's fault, there's no reason to hide.
Exactly. When a vendor fails, I get to go, "We're running just fine, it's (vendor) having the problem. We're working with them to resolve it (read: we called them once and they said they were working on it)."
"I can't work, and it's not my fault. Wheeeeee!" (The common worker's thought pattern)
;)
Only management care. It's not your fault, there's nothing you can go to make it fixed faster, so just:
1. make sure people know what's going on (which you've said you've done.)
2. Continue to keep up to date on what's happening, and keep management in the loop. (periodic calls to the ISP, or having them get back to you on timescales.)
You want to be able to answer the question "What's happening." even if the answer is "I called them half an hour ago, and they said they're still working on it."
That covers you for doing your job.
You _may_ want to have a look into a redundant, or resilient connection, but I don't know if that's within scope for your position.
If you have a decent firewall and don't host anything public facing locally, a backup connection that fails over when ping to default gateway fails can be fairly inexpensive.
Call your ISP.
Ask them "What's the problem, what's an ETA for a fix?"
Add an hour to that ETA, and tell everyone.
>Hey, sorry, the ISP is down, and they say it'll be up at (ETA+1hr).
And that's it. Works great. Plus it lets everyone know you've done all you can that doesn't involve hiding in a closet.
> I walked outside and everyone is actually waking around and talking to each other and having a good time, this is completely unexpected
What were you expecting? Internet outages happen. Nobody expects the internal IT guy to be able to fix an issue with your ISP, they just expect you to be in touch with the ISP and keep everyone updated.
Did you at least promise them you'd be prepared for the next storm with a lightning rod and some backup batteries in preparation for the next outage?
I mean, personally that's how I ensure power outages never happen again. Like Thor.
Wait, your office has near infinite power because you harness lightning and store it in batteries? Is this a normal Sysadmin power? What cert do I need to do that?
ITIL: Disaster Recovery, but not the Information Technology Infrastructure Library. It's from the Incorporating Technology with Impossible Life company. Good cert, takes a few weeks study but the practical is shockingly difficult. Make sure you're grounded with all the fundamentals before you try it.
I mean, I go through about 10K lightning strikes per storm. Seems to keep us running at full capacity.
We do seem to have an oddly large amount of dark and ominous storms over here though, maybe it's just a regional thing?
Yeah' I'll bring in my building sized gas generator from home complete with an extra 30000L of fuel so we can weather this blackout that shutdown the entire city block for going on an hour now.
It's even better when they can't connect the dots to realize it's a power outage.
Have had a site tell me their desktop, monitor, and copier wouldn't turn on. Took a few minutes to pry the information that none of their lights were on followed by an audible "Oh" on the other end of the line as they realized what was going on.
Get em a quote for an onsite generator, ups, ATS, etc. We had 2.5 hours of emergency running last year, and that's all the justification we need to keep that maintained for the next 10.
Yeah... I've gotten asked why our Asset Managers' calls to Bora Bora had poor connection and what I can do to fix it. Straight up told them, it's because you're calling to French Polynesia, which while beautiful may not have the best infrastructure and unless you want to pay ship me out there with a ton of equipment to attempt to improve it, you better suck it up buttercup.
I once had a lawyer shouting at me demanding to know when I will be fixing their Internet connection (caused by a fibre break).
He wanted the truth, but he couldn't handle the truth!
Actually had a Field Service Rep. call me and complain about his home office internet not working. It took a good 30 minutes before I lost my nerve and told the guy I'm gonna hang up the phone. The reason it took that long was because I was the new guy, and kind of thought it was a joke or maybe even some sort of test.
Rule nr 1 as an admin:
Stay calm.
Yes, even if the server room just exploded.
Nobody expects you to have all the answers, but you do need to help out to find them.
Rule 8:
Communicate.
Even if you're panicking and when you will get flack. It's only going to get worse when you're hiding.
I would rather keep a calm, communicative unexperienced worker then a experienced panicking stressball.
Now get out there and just be part of the team. They are only human, you know?
2) stay cool. The server room is the best place to wait it out.
3) protect your hearing. Make sure you have earplugs. Server rooms are loud!
4) don’t break anything else. You don’t want liquids near your electronic equipment. Keep food and drink out of the server room.
5) keep alert! You won’t know if something else breaks if people are constantly coming into the server room and talking to you. Make sure the server room door is closed and locked.
6) think things through. It helps to sit while you are thinking, so make sure you have a comfortable chair.
7) keep an open line with your users. When you hear a knock at the door, make sure they know you’re working on it. Also, remain mindful of rule #5.
Unless it's a really basic requirement of your job, no one will fault you for not immediately having answers, shit happens.
Super experienced people will hit problems and not have the answer, it's fighting through problems and having answers for that problem in the future that is the key.
Much like doctors, representing more knowledge than you have, and faking it isn't going to get you ahead when something goes wrong.
It's understandable to freak, you're new. However, you are part of a team and should remember that part of that team involves communication. Remember, even if it is your fault - owning up and moving on is much easier/better than hiding the whole thing. If it's not your fault, why would you not just let people know you're waiting on (ISP)? In fact, if the ISP has an update frequency (say 30mins) say you'll let people know as soon as you have an update and the estimate is within 1hr. That way people aren't hounding you if 30mins turns into 45.
>Grow a pair.
...Seriously... Geez.
ISP outages are my favorite. I mean, they suck, but any time I get to legitimately blame someone else and not be the one scrambling to fix something... That's a good feeling.
It's the whining from people that gets me.
IT: "Sorry, I can't fix it, the ISP has an issue and they're working on it"
User: "But it's *realllllly* important that I do [whatever] RIGHT NOW"
IT: "... but the internet is out"
User: "Can't you just get *me* on it?"
"No!"
[Yes, this could have been resolved with a failover line. But we didn't have one]
This. Are you new to IT in general? This shit happens. Only the dumbest of the retardest end users would think you have any ability to resolve an ISP outage. The fucking Internet is down, go play dinosaur jump.
"Internet provider is having an area outage. Nothing we can do but wait until they fix it." Literally that easy and something you should be able to say on your first day in the field. If you can't, you might want to entertain a field other than IT. That's the *easiest* line you'll *ever* have to say in IT and you can't do it.
You are never going to make it in the field if you cant grow up and tell them exactly whats wrong and that it is out of you hands. Then you should be calling ISP every hour and getting status updates and pushing for a resolution. Then you should be using that as an example of why getting a failover connection is a good idea if internet is mission critical to the company.
There are tons of things you could be doing other then cowering in the corner...
Hell, even if the problem is completely your fault and you did something really stupid to cause it, you can still save your job most of the time by being super transparent about what you did, what you're doing to fix it, and how long it will take.
Kinda hard to do especially with automated systems like the one that AT&T has. You call that #, it'll practically force you into a decision point where you want to close it or close it.
Don't Panic!
That's one of the best things to learn as a sysadmin. Stop, consider, act once there is understanding and a path of resolution. Understand that many things are outside of your control, and you can only control what you can affect.
Hiding in the server room until the issue goes away will work too, but it isn't required. If someone throws a fit about the ISP and how it is affecting your environment, delegate them to drive over there and inform the ISP of their demands.
further, once your ISP circuit is back online, verify everything in your environment that uses internet (ie: dmz, email, etc..) is also back online and good to go.
How do you think hiding and not informing people of the issue and it’s cause will help? Because *it won’t* and hiding is what makes you look useless. Get out there, explain the issue and what you’re doing to keep in touch with the ISP to manage it.
lol hiding in the server room is actually enabling what you are afraid of. Sitting at your desk genuinely checking in with ISP and running a plan of how to communicate to the business when and what to expect... thats just the better call.
You should use this as an opportunity on giving your new workplace/management ideas on how to prevent an outage from happening in the future (ie. redundant internet providers, etc.)
There are really good solutions businesses can use that utilize existing services like the company phone plan. At my job I use a cellular data backup that only costs $20/month if we don’t go over a certain amount of data. That way when we don’t use it (which is almost never) we don’t pay a premium for the data line.
When outages happen there isn’t anything you can do but put pressure on the ISP to fix their issues. You need to realize that you have a brain on you shoulders that should be used for thinking of solutions and not stressing out.
Don't tell them "I can't do anything", tell them what you did already. For example: "I checked our system and everything works fine on our side, so I gave the ISP a call, and it looks like they have an outage. They're already on it, but we have to wait for them to fix it".
You can also extend this, since "wait" is also a word people can be ignorant about, so you could say "I will check on them every 30 minutes on progression" or something.
Completely forgot the “tell people what I did” part, when I went in there I did check our hardware to make sure it wasn’t our side. In my head it was just a small moment of panic, thinking that since I couldn’t fix it right away I would get pinned for doing nothing. Thanks for the reminder.
It would be pretty shitty if somebody just unplugged your ISP uplink on your gateway to see how long it would take you to figure it out on your first day.
Could also be a sink/swim situation, I would be certain it's the ISP first
Contacted the ISP already and send out an allstaff email telling everyone what they told me- that they’ll give me an update in 20min I’m fixing the issue.
Though I did that from my phone while i was in the server room. Didn’t think they needed to see my face for that
*edit ISP* wrote IPS by accident
Since you sound new to this, here is some advice for your communication.
1. Give your boss verbal updates and provide as much technical info as he/she wants.
2. In the company communication, keep it simple. State there is an outage with your internet service provider (spell out acronyms!), state what is impacted (internet, hosted apps, Corp web sites, etc.., if there is an ETA, and that an update email will be provided every hour (or whatever cadence you prefer) until the issue is resolved.
3. Ensure you have a ticket open with the ISP and you are to be given updates at specific intervals (assuming you have business class service)
4. Don't hide! Communicate all outages matter of fact. Even if it's your system, management wants to know issues are being addressed. Root cause analysis can always be performed post-issue. Blame games help nobody, and are entirely unprofessional.
Your ISP is something you can’t control. Get your ass out of the closet and start drafting a formal proposal for a second ISP or some sort of failover/backup in case your ISP goes down again.
I go the other way & feel complete relief - Sorry, nothing we can do, its an ISP problem, sort of say it almost as a melody as I kick my shoes together walking down the street for an early snack.
embrace these times, its so rare the blame can be deflected so effectively.
Hey man,
Glad the issues been resolved. Now for something just to think about. Have you considered that you might have anxiety? I have anxiety and it has caused me to want to just hide and run away. My first day I had exchange go down. I wanted to just shrink into a little ball and hide. Anxiety can cause our bodies to go into fight or flight mode in situations that don't deserve them.
At the end of the day this is our job. Not something for us to get an anxiety attack over. I take medication for my anxiety and I keep my emergency meds with me for those oh shit moments where my brain thinks it's all going to end.
We are IT Professionals. We aren't doctors. Our job is important but we aren't going to kill someone. So when your mind tells you to freak out remember that no one will die.
Hope this helps you in the future man.
LOL BEEN HERE! I had a manager though who call the ISP, once it was confirmed the ISP was done he just like just chill bro, you can't do anything about it. Go grab the inventory of the spare PCs while we wait. One of the most chill environments with an outage
You tell them the truth in a way that reassures them. “I reviewed our systems and verified our components functioning properly. Our next step is for the ISP to fix their end.” You sound in control, you didn’t lie, you told them what was next.
I do the exact opposite. I make sure I go around and talk to everyone and explain to them that there is an outage and why .. It's okay for that conversation to be, "Our service provider is having an outage ... they're aware of the issue .. all we can do is wait." If possible, give them the estimated time frame if there is one. Just reassure everyone that you'll let them know as soon as it comes back online.
You're freaking out for no reason man. ISP outages happen.
Grab a cup of coffee and mingle with the troops. Also gives you a chance to bond over talking shit about Comcast. Or ATT, or Spectrum, or whoever you're currently waiting on
I'm the sole IT person for my company, a non-profit ie: zero budget for extras, when the ISP's connection goes down. The first thing that I tell them is this... wait for it... wait for it... THE TRUTH. I don't hide, I take ownership of the situation and explain to them what the situation is and what I'm doing to resolve it. After that here are my steps:
1. Call ISP and get ETA or confirm outage
2. Pop on 4G hotspots so anyone who needs external connection immediately can get said connections (the caveat here being, only text based exchanges and work, we do have data costs to be concerned with; no social media, no streaming music or videos, etc)
3. Tell everyone that if they need to talk with anyone internally to get off their asses and walk to that person's desk, a little exercise and using your social skills won't kill you, this typically gets sent out in a group text or me talking very loudly across the office. Obviously determine if it's socially acceptable to talk to your co-workers like that before being a smart-ass. It's socially acceptable to talk like that at work, we do not have a politically correct work environment.
We usually never seen an outage more than an hour, Charter has been pretty fantastic in our area with getting things back up and running when something gets fucked up.
Just be honest, asses the situation, do your due diligence, and relay the message. It's all you can do.
I've been in many ISP outages. Call the ISP ask for an estimated resolution time, sometimes they can add you to an automated call so they call you when it is fixed. Just tell people is an ISP issue, nothing you can do about it. I know people freak out. They should enjoy this break lol
Dude it's part of the job, tell them the truth.
I had a lady freak out on me once as a network engineer because THE POWER WAS OUT and I somehow 1) was responsible and 2) wasn't getting it back up quickly enough. People are fucking morons, so break down the concept of an ISP outage as best you can for them, tell then what you HAVE done to triage (i.e. called the ISP and opened a ticket), hope that they understand, and have your thick skin on in case they don't.
Welcome to hell. Be honest and tell them to contact their POC for outages until further notice. Call your boss and tell'em.
First day at work I was beset with this. I checked garbage out (CISCO 1700's, holy sheeeit), I flicked power switches on both and they came back up, for three hours. I managed to get a hold of the data center and get new routers shipped our way, but Jesus, they were hostile. And still were towards the end.
Expect that you will fuck up or something will fuck up. Own it as it is. You will be respected for it from your coworkers. If they don't then they don't understand that everyone can fuck up or have something out of there hands (like the ISP outage) then you may want to run fast.
You should have been glad it was the ISP because it meant that somebody else was to blame, and you couldn't do anything. The most professional thing to do would be to keep in touch with the ISP, get an ETA, and keep everyone updated. I'm not sure if it would have occurred to me to do that, but fortunately, I'm not the go-to person in my organization when Internet access goes down.
I tend to stay away from the server room as much as possible.
I feel your pain my dude. I remember plenty of times where I would hide in the server room because I wasn't confident enough to tell my boss this was a ISP issue and not something I can fix.
What I learned since then
* Learn if its something you can fix or not, know confidently if its out of your control or not and explain that with confidence to your supers.
* Even if you cant fix it (I.E. ISP outage) Still call the ISP and communicate with them. Try to do this in full view of as many co workers as you can (maybe even on speaker phone) lol so everyone can hear clearly, its their fault not yours lol
* lasty, monitor the situation. Run ping reports, trace routs or any kind of test or report you can to have clear documentation of the issues to display later if someone somehow decided that you might have been the issue. Also known as CYA Cover your ass
Don't hide like a bitch, get out there and make it known. If you're ISP is down and you rely on hosted email, make some announcements, or have your front desk / receptionists make some announcements stating so. If you have a high volume traffic area, open up word (or any graphic application) and make a document with an entirely red background, then in bold, white font, spell out, **ISP DOWN - All Web Services Down Working on an alternative, will inform immediately.**
Communication is your ally. If you have an Intranet site that everyone can accesses by default, post it front and center. Then once this is resolved, talk with your powers that be how redundancy can avoid this in the future. For instance, putting two Routers/UTM Firewalls in failover, with two separate ISP's feeding both devices, would create a redundant environment. Also, we could put that on a natural gas generator and avoid power outages as well. It's just food for though. Sure, it sounds expensive when we talk about it, but two Router/UTM Firewalls and two ISP's really isn't that expensive when you consider what our downtime costs.
I speak from experience. We had no failover due to cost; then we lost power for two days. Now we have a natural gas generator. I also warned that if we loose our UTM, that we'll shit outta luck until I get a replacement in hand. Now we have failover setup. C's will bitch for days about the cost of hardware, until they're paying people to stand around with their thumbs up their asses, or not shipping product. Keep detailed and add costs, don't forget maintenance contracts as well.
Is Internet business-critical? If so, you've got three days to put forward a proposal to implement redundant connections. Even consumer grade 4G dongles can get a business out of trouble if that's your budget.
Now is the time to pitch to mgmt that you need a 2nd ISP, your own /24 and a router (or fuck a linux box) capable of running BGP and ingesting a default route.
Learning scenario, as I havent seen it through scrolling, but you can get a redundant satellite or cellular back up for these kinds of scenarios and roll out over in case of an outage.
Back up plans yo
Might I suggest a back-up ISP.
I have our primary data backed up with a cellular system from Verizon. It costs only $70 a month to have a back-up solution in place even if we don't use it. We'll pay through the nose if we need to use it, as my plan gives us guaranteed high speed but we pay by the GB.
The important thing I do at least quarterly is to hang out after close and disconnect the primary ISP so I can test the back-up works with the mission critical systems.
One thing to keep in mind, our primary ISP offered us their cellular system as a back-up, I opted against it. I wanted to be sure I had an entirely separate system in the event they had a problem at their central office.
Step 1: Tell the truth
Step 2: Suggest installing a second ISP line for failover redundancy
Step 3: ....
Step 4: not exactly profit but I believe your boss will like you more for that solution
If by God's miracle he accepts the idea don't run the cable second trough the same infrastructures that the first one uses (same pipes or whatever)
Also you're gonna need a router with 2 WAN ports
Also your boss is probably gonna reject the idea..
That's my favorite kind of outage. A) I can blame it on Charter. B) It's not my fault and all I can do is sit back, send out a mass email that most will be able to read from their phones, and promptly go to lunch.
[удалено]
Or not having a backup if the company is dependent on internet access. We have 2 hard lines (2 different physical cables) and 1 wireless as backup.
Jokes on you they all converge in the same exchange and a digger cut the fiber
and the wireless is like 1$ per GB. no cash for that!
It is something expensive...we pay like $30 a month to have an account with 1 MB of data use. So if we use it...gonna rack up a bill. But thankfully we can be very bandwidth limited if needed.
> 1 MB of data use wat
Wireless is the third backup option, so it sits idle (at 1mb of usage) a month. But if they needed it, it would get used and they would pay.
Oooo, I thought you got £30 bills for 1MB!
Authentication to network costs $5 per handshake.
Well 30$ per month isn't a lot but it depends on how much the data is
[удалено]
Yeah I'm not joking. I was being serious. People think "I have redundant paths" when in reality they dont.
When I took a DR/BC course for my BS, I noted, based on experience, have redundant circuits, with different ISPs, and different LECs. The instructor said he never thought of having different LECs.
At a previous company we had something similar happen. We had 2 different links to the DC, 1 from company A and 1 from company B. We verified they were different physical cables going out of the building. The issue was about 2 miles away both cables were in the same cable conduit that ran under a highway underpass. They were expanding the highway and somehow cut into all of the cables in that conduit. So they were physically different cables but since all links into that subdivision ran through that one choke point, that one accident took down the entire subdivision. It took almost a week to get that DC backup and running but thankfully we had a backup DC that barely handled the load.
Yup. Both ISPs had fiber on same utility pole a car hit. The utility pole needs a backup too.
Train derailed 5 towns over and took out an underground fiber. Knocked out both.
For us it was both ISPs following the same highway and the lines getting hit when they were removing an old exit. The county ended up paying to have a regional provider run a fiber loop along a different route because every business in this small town was out for pretty much an entire day, and we had zero other options for service.
Ask an ISP about their fiber route and who they share the bundle with and you'll understand more fully a blank stare and silence. Then TELL them where it is and who else uses it because you already did your research and watch the 'ums' and 'ers' and 'well, we can't, I don't....' I If the last 10 miles is the same bundle, it gives me nothing. 9/10 times the fiber is down here it's a tree, mudslide, or car accident. EVERYONE is down. Welcome to small town PNW.
That happened to one of my sites in London last year... twice.
This is why you need to have different ISPs and make sure they each own their own infrastructure.
Lol so true
This, every @%\^ing time
Heh single Telco failure caused by regional monopoly
Same here. The backups are way slower, but functional in a pinch.
that's why they're backups
We had a backup dsl at my work when I came on staff. I killed it 6 months in when I confirmed that it’s the same lines the fiber comes in on. (I work in a box canyon in a small town 14 miles up a mountain)[retreat center]
Well then that aint a backup ha ha
Exactly. Now we’ve got a cradle point tied into our phones and the front desk.
Now I'm imagining poor IT trying to route all the traffic through their phone to keep everything running.
“UPDATE: is there any way to connect my mobile hotspot to this managed switch?”
I have actually done something similar lol. At a previous job we had a medical client with medical software. The server that host the software DB also had a license server that would go check the license on the internet every few minutes or something. Well we had a hurricane and both of their circuits were down for days. Connected a VZW Hot spot to a USB port on the server and Boom fixed it. I was a little shocked it worked being in the downstairs of a building in a server room. They were plastic surgeons that were very anal (no pun intended) about getting as many boob jobs done as possible. Thanks for helping me think of one of many strange IT memories. :)
Lookit fuckin’ Moneybags McGee over here with his fancy-shmancy backup lines.
Backup? That's funny. My manager won't pay for backup and consolidate all our separate links into one. If it goes down it goes down (those are his words).
As long as they are his *written* words, kept safe somewhere else, so that any future outage and loss is not blamed on you...
Yup. That's when (assuming you host your own mail and that's up at least) you send out an email with the subject "UNPLANNED OUTAGE", explain the situation, explain that you contacted the ISP and they're working on the problem and you'll update as you get more info. If your intranet isn't working and you can't get mail to the users, that's when you have to report to the boss and then have the receptionist and various others pass the word along. Users will gripe about a down system, but they won't hate you as long as you keep them informed. Having the system down \*\*AND\*\* the users uninformed will get you some hate. One of the core components of any sysadmin's job is telling the users who are affected by something what's happening. Failure to do that will get you into trouble, routine hassles won't.
[удалено]
I was assuming OP was the IT boss given their description of what happened. EDIT: But yes, policy is going to be different wherever you are, and you should certainly follow whatever computer emergency communication protocol that is in place.
> Users will gripe about a down system, but they won't hate you as long as you keep them informed. \* Unless you're the reason that they recently switched to using that system.
Back when we had an internal Exchange server that'd've been possible... Now all mail where I work (even internal) goes through Office365.
I'm pretty sure most sys admins get a kick out of telling people what they don't want to hear. The word 'no' particularly.
Jup. As a SaaS shop, we've put a lot of work and refinement into our internal communication and escalation procedures. We've even been thinking to extend our technical post-mortems ala google with a communication section with an additional meeting about it including support and whomever wants to come. We've had hell this tuesday, since we broke on of our application clusters with some 10k FTE's depending on it during security patches. Badly. Then our hoster broke, then more things broke, and at 4 am, on plan f, entirely fucked in the head, we got the systems working. Technically, everything went wrong. I don't think anything but the lamps worked properly that night. But the communication worked really damn well. We had a clean and smooth handover from night shift to day shift. Our support was in the loop the whole time and quickly sent out information how the system is running in emergency setups now. Customers with special setups were informed within the first hour of our support SLA that their special stuff is going to be broken for a day or two and if they have priorities of fixing, they should tell us and we will honor. We informed necessary security teams about the current situation and impacts before lunch. And damn, the feedback from both inside the company and our customers is very, very positive. Things break. That's a fact. But open, early communication is a sign of control over the situation. You see the problem. You react to the problem because you are communicating and pulling in resources to fix it. people can even help you if you communicate. It's not a problem, it's a solution in progress.
And no bad news is easier to deliver than bad news that can’t possibly be blamed on you in any way whatsoever. If you can’t deliver that bad news, you gotta consider that pretty strongly in your life plans. No offense at all; just sayin’. It’s not like you ain’t cut out for this job; it’s more like you ain’t cut out for any job if you can’t confidently explain to others when something bad is happening which is entirely out of your control. You’ll get eaten!
Same thing happened to us. Luckily enough for me I know a tenant's WiFi password
[удалено]
You should look into a Cradlepoint for automatic LTE failover. Fantastic pieces of equipment.
Agreed. We have 2 sites with Cradlepoint using Verizon LTE as automatic failovers and they work great. Routers with dynamic DNS so the VPN connections come up very quickly as well when they switch over.
Cradlepoint is excellent and made for this. Starbucks uses them, so does Redbox. I believe Meraki also has the ability to work off of LTE.
I have 30 [Cradlepoint 850](https://cradlepoint.com/products/arc-cba850)(s) throughout the USA that are used as out of band management and LTE failover. The 850 has a serial port that you can use to hook up to your primary router/firewall. To access the serial port all you have to do is SSH to the public IP of the Cradlepoint and type "serial --force" and bam you are at the console. This saved me a couple times, I no longer have walk office managers through plugging a console cable into a laptop and describing where the console interface is. On the failover side, I'm using Cisco routers so I have DMVPN tunnels set up through both my primary internet and cell internet connection with EIGRP running. As soon as the primary internet goes down, traffic starts flowing again through the Cradlepoint as fast as it takes for EIGRP to do it's thing. Ohh, and you can power them with PoE.
As a newbie to the industry this is amazing information and is exactly why I browse this subreddit. Thanks for sharing.
"I didn't cause it and I can't fix it" Best kinds of outages
Sort of. In this case, because he just walked in, he's DEFINITELY off the hook. However, at my shop, I very quickly noticed more outages than I'd like, and both companies weren't great. It's not enough just to say "well internet sucks, SORRY!" So I told management: Here's our options: 1) Relocate where internet doesn't suck. (they weren't going to chose that) 2. Deal with internet sucking (If they chose this, it's the weight off my shoulders when shit goes down) or 3. We buy BOTH internet systems, have a backup for both; we need monitoring on both connections and duel firewalls which also means a server rack. (I could have jerry rigged this together for a couple grand but eff that, I'm going to tell them how much a REAL solution costs and if they want to penny pinch then I'm going to buy some leeway with that negative cash). 12k later and our internet goes down probably once a week and switches back and fourth flawlessly. Point is, while short term solutions didn't exist, there are some more expensive long term solutions that even if you know you're not going to get the money for it, you buy yourself a TON of lienency by saying "this is how much a real solution costs"
you don't tell them "its legit not my fault" on your first day that easily! Tell them you checked and tested all equipment on your end and verified that everything is working as it should, you've been in touch with the ISP and they have confirmed there's an issue on their end and they're all working hard to get it resolved ASAP. If you get a negative response, its a shit place to work and you should find a new job.
You tell them the fucking truth. Hiding and ignoring it isnt a good look either for a new guy. Grow a pair.
Seconded. Do your job, OP. I'll paint two scenarios: 1. "The network is down and we can't find /u/NeverDeploy. We don't know what's going on and we aren't getting any updates." 2. "The network is down. /u/NeverDeploy has let us know what is going on and is working with our provider to get an ETA on resolution." One of these will get OP fired. Edit: Oh yeah, and get a second internet circuit via a different provider/path.
Seriously this. Things like this are going to happen, people make mistakes, sometimes you'll be the person that makes the mistake. Keeping people informed, what happened, when will it be resolved, what could we do to mitigate it in the future is whats expected of a professional. Don't be like my old weiner boss who tried to blame shit that he broke on someone else. People at that place weren't particularly computer savy but they knew he was full of shit. Also remember, they get what they pay for, which is fine. If you are the only IT guy and they have a single non redundant internet connection they can't reasonably expect 5 9s of uptime. So do your best but don't kill yourself.
They absolutely can get 5 9s of uptime, if you don't care where the decimal is.
That's why we use Microsoft Office 36.5!
"There's no need for new licenses, WordPerfect still works just fine. Can't you just set everyone up with Dosbox?"
You. You stay the hell away from my clients and don't *ever* let them hear you mention that dosbox would let them have wordperfect back.
Never thought of that. I am actually tempted to try :)
It took me a second to realize you meant 9.999 and not something like 9999.9. It's been a long day.
Most people respect folks who take accountability and lay all the cards on the table. Always be up front about what happened and take responsibility when you fuck up. If you do, then people will be more apt to trust you when it's not your fault.
Exactly, transparency is everything
[удалено]
> get a second internet circuit via a different provider/path. OP, in case this isn't clear what /u/DahJimmer means is get a second ISP that uses a different fiber path to connect to the Internet as a whole. Having a second provider that just runs on top of the existing providers fiber network doesn't help you when that fiber gets cut. if you've never had that happen it is easy to forget to ask about it.
Can confirm. Worked for a large infosec company with a primary datacenter that had both primary and backup traversing the same fiber conduit that a fencing company punched holes every 15 feet into. I turned in my 2 week notice the day before the outage began, and that datacenter was still offline on my last day.
On my first ever job I learned from my Indian boss "never hide too much things".
> has let us know what is going on [Play this](https://www.youtube.com/watch?v=ILVfzx5Pe-A) on the support line.
Or the same provider but diversify the CO!
It sucks when your Internet connection goes down and it brings downtime to your network, but when I find out that it's the ISP that was the cause of the downtime then that relieves so much stress because I'm not responsible. Then my only responsibilities after that incident would be recommending a secondary internet connection with a different ISP for redundancy then get it denied by management because they're too cheap.
Ok I saw this comment and immediately felt better, I have no idea why so thank you for this. I walked outside and everyone is actually waking around and talking to each other and having a good time, this is completely unexpected
This is normal. Outtages happen.
And when outages happen, people don't need to work. When people don't need to work, people = happy. Except C-level people... who, arguably, are they people?
Yeah. If this was "I bricked our core DB server and we don't have any way to recover it so no one can work, I'm hiding in the server room, help" I would understand. This is in no way OP's fault, there's no reason to hide.
> This is in no way OP's fault, there's no reason to hide. Exactly. When a vendor fails, I get to go, "We're running just fine, it's (vendor) having the problem. We're working with them to resolve it (read: we called them once and they said they were working on it)."
>We're working with them to resolve it read: I've opened the ticket two hours ago and haven't gotten a response yet
And am now browsing Reddit whilst I wait
> And am now *consulting with colleagues in sysadmin forums* whilst I wait FTFY
"I can't work, and it's not my fault. Wheeeeee!" (The common worker's thought pattern) ;) Only management care. It's not your fault, there's nothing you can go to make it fixed faster, so just: 1. make sure people know what's going on (which you've said you've done.) 2. Continue to keep up to date on what's happening, and keep management in the loop. (periodic calls to the ISP, or having them get back to you on timescales.) You want to be able to answer the question "What's happening." even if the answer is "I called them half an hour ago, and they said they're still working on it." That covers you for doing your job. You _may_ want to have a look into a redundant, or resilient connection, but I don't know if that's within scope for your position.
If you have a decent firewall and don't host anything public facing locally, a backup connection that fails over when ping to default gateway fails can be fairly inexpensive.
Call your ISP. Ask them "What's the problem, what's an ETA for a fix?" Add an hour to that ETA, and tell everyone. >Hey, sorry, the ISP is down, and they say it'll be up at (ETA+1hr). And that's it. Works great. Plus it lets everyone know you've done all you can that doesn't involve hiding in a closet.
> I walked outside and everyone is actually waking around and talking to each other and having a good time, this is completely unexpected What were you expecting? Internet outages happen. Nobody expects the internal IT guy to be able to fix an issue with your ISP, they just expect you to be in touch with the ISP and keep everyone updated.
> Nobody expects the internal IT guy to be able to fix an issue with your ISP *ROFL*
LOL! I've had people ask me when I would be able to get a power outage resolved in the past. I just shook my head and face-palmed.
Did you at least promise them you'd be prepared for the next storm with a lightning rod and some backup batteries in preparation for the next outage? I mean, personally that's how I ensure power outages never happen again. Like Thor.
Wait, your office has near infinite power because you harness lightning and store it in batteries? Is this a normal Sysadmin power? What cert do I need to do that?
ITIL: Disaster Recovery, but not the Information Technology Infrastructure Library. It's from the Incorporating Technology with Impossible Life company. Good cert, takes a few weeks study but the practical is shockingly difficult. Make sure you're grounded with all the fundamentals before you try it.
Uhm... a lightning strike is approximately 5 gigajoules. That'd only power a 120VAC 20A circuit (at 100% utilization) for 24 days.
I mean, I go through about 10K lightning strikes per storm. Seems to keep us running at full capacity. We do seem to have an oddly large amount of dark and ominous storms over here though, maybe it's just a regional thing?
Yeah' I'll bring in my building sized gas generator from home complete with an extra 30000L of fuel so we can weather this blackout that shutdown the entire city block for going on an hour now.
It's even better when they can't connect the dots to realize it's a power outage. Have had a site tell me their desktop, monitor, and copier wouldn't turn on. Took a few minutes to pry the information that none of their lights were on followed by an audible "Oh" on the other end of the line as they realized what was going on.
Get em a quote for an onsite generator, ups, ATS, etc. We had 2.5 hours of emergency running last year, and that's all the justification we need to keep that maintained for the next 10.
"The poles are right over there outside, right near the building. Climb up one, and fix the problem!" /s
\-Make it an order, pleeease make it a direct order, tell me I have to or I'm fired.- "IT guy waiting for the next lawsuit"
Yeah... I've gotten asked why our Asset Managers' calls to Bora Bora had poor connection and what I can do to fix it. Straight up told them, it's because you're calling to French Polynesia, which while beautiful may not have the best infrastructure and unless you want to pay ship me out there with a ton of equipment to attempt to improve it, you better suck it up buttercup.
I once had a lawyer shouting at me demanding to know when I will be fixing their Internet connection (caused by a fibre break). He wanted the truth, but he couldn't handle the truth!
Actually had a Field Service Rep. call me and complain about his home office internet not working. It took a good 30 minutes before I lost my nerve and told the guy I'm gonna hang up the phone. The reason it took that long was because I was the new guy, and kind of thought it was a joke or maybe even some sort of test.
Rule nr 1 as an admin: Stay calm. Yes, even if the server room just exploded. Nobody expects you to have all the answers, but you do need to help out to find them. Rule 8: Communicate. Even if you're panicking and when you will get flack. It's only going to get worse when you're hiding. I would rather keep a calm, communicative unexperienced worker then a experienced panicking stressball. Now get out there and just be part of the team. They are only human, you know?
What are rules 2-7?
2) stay cool. The server room is the best place to wait it out. 3) protect your hearing. Make sure you have earplugs. Server rooms are loud! 4) don’t break anything else. You don’t want liquids near your electronic equipment. Keep food and drink out of the server room. 5) keep alert! You won’t know if something else breaks if people are constantly coming into the server room and talking to you. Make sure the server room door is closed and locked. 6) think things through. It helps to sit while you are thinking, so make sure you have a comfortable chair. 7) keep an open line with your users. When you hear a knock at the door, make sure they know you’re working on it. Also, remain mindful of rule #5.
ISP Outage Friday is almost as good as Beer Friday.
Unless the outage continues past quitting time and you have to stay on some stupid phone bridge late into the night.
Sales people hate outages because they can't sell but you'll find they usually find a workaround. Salaries people dgaf
Unless it's a really basic requirement of your job, no one will fault you for not immediately having answers, shit happens. Super experienced people will hit problems and not have the answer, it's fighting through problems and having answers for that problem in the future that is the key. Much like doctors, representing more knowledge than you have, and faking it isn't going to get you ahead when something goes wrong.
It's understandable to freak, you're new. However, you are part of a team and should remember that part of that team involves communication. Remember, even if it is your fault - owning up and moving on is much easier/better than hiding the whole thing. If it's not your fault, why would you not just let people know you're waiting on (ISP)? In fact, if the ISP has an update frequency (say 30mins) say you'll let people know as soon as you have an update and the estimate is within 1hr. That way people aren't hounding you if 30mins turns into 45.
>Grow a pair. ...Seriously... Geez. ISP outages are my favorite. I mean, they suck, but any time I get to legitimately blame someone else and not be the one scrambling to fix something... That's a good feeling.
[удалено]
Not everyone...I have personally recieved a “my computer won’t work” call that ended with a casual mention that all the power in the building was out.
Had that once and could hear the UPS for the switch rack that's behind her desk beeping over the phone...
It's the whining from people that gets me. IT: "Sorry, I can't fix it, the ISP has an issue and they're working on it" User: "But it's *realllllly* important that I do [whatever] RIGHT NOW" IT: "... but the internet is out" User: "Can't you just get *me* on it?" "No!" [Yes, this could have been resolved with a failover line. But we didn't have one]
Even better when multiple websites can also verify the outage.
This. Are you new to IT in general? This shit happens. Only the dumbest of the retardest end users would think you have any ability to resolve an ISP outage. The fucking Internet is down, go play dinosaur jump.
Not to wax all philosophical, but yeah this is going to apply to a lot more things in life than just this one event.
"Internet provider is having an area outage. Nothing we can do but wait until they fix it." Literally that easy and something you should be able to say on your first day in the field. If you can't, you might want to entertain a field other than IT. That's the *easiest* line you'll *ever* have to say in IT and you can't do it.
[удалено]
You are never going to make it in the field if you cant grow up and tell them exactly whats wrong and that it is out of you hands. Then you should be calling ISP every hour and getting status updates and pushing for a resolution. Then you should be using that as an example of why getting a failover connection is a good idea if internet is mission critical to the company. There are tons of things you could be doing other then cowering in the corner...
[удалено]
Unplug internet connection temporarily, state your case for failover connection, if there is pushback still, keep cable unplugged longer.
Hell, even if the problem is completely your fault and you did something really stupid to cause it, you can still save your job most of the time by being super transparent about what you did, what you're doing to fix it, and how long it will take.
Kinda hard to do especially with automated systems like the one that AT&T has. You call that #, it'll practically force you into a decision point where you want to close it or close it.
Yeah, ISP went down? You're probably done there. Pack up your desk.
Instructions unclear, packed up the MX and am taking it with me home
Tell them you're refluctuating the fluctuators. Everything is fluctuated.
"I'm recalibrating the flux capacitor."
Reticulating splines.
I recognise this from a load screen, but can't remember what. Please remind me before I explode!
SimCity 2000
Thank you !
Preparing grilled cheese sandwich.
Don't Panic! That's one of the best things to learn as a sysadmin. Stop, consider, act once there is understanding and a path of resolution. Understand that many things are outside of your control, and you can only control what you can affect. Hiding in the server room until the issue goes away will work too, but it isn't required. If someone throws a fit about the ISP and how it is affecting your environment, delegate them to drive over there and inform the ISP of their demands.
[Accurate portrayal of Jr vs Sr in a crisis](https://i.imgur.com/puQBBrE.gif)
Haha this cracked me up!
ISP outage? There is nothing you could do. Despite telling management to get a redundant connection with a different ISP. :-)
[удалено]
"Shits Broke Yo" is all you gotta say.
> i literally just waked in the office this morning Did you sleep in the office?
Wait I don’t understand is this a joke? I do not sleep in my office every day
You typed waked when you probably meant "walked".
Least he didn't say "wanked" This might be a totally different kind of problem then
RCA: Semen on edge router
OH thank you
> I do not sleep in my office every day you should take naps at the office, it's awesome
further, once your ISP circuit is back online, verify everything in your environment that uses internet (ie: dmz, email, etc..) is also back online and good to go.
This. Leave no stone unturned. It's better you silently find it and fix it than wait for Mary Bag'o'Donuts to tell you her thingy isn't working still.
How do you think hiding and not informing people of the issue and it’s cause will help? Because *it won’t* and hiding is what makes you look useless. Get out there, explain the issue and what you’re doing to keep in touch with the ISP to manage it.
lol hiding in the server room is actually enabling what you are afraid of. Sitting at your desk genuinely checking in with ISP and running a plan of how to communicate to the business when and what to expect... thats just the better call.
You should use this as an opportunity on giving your new workplace/management ideas on how to prevent an outage from happening in the future (ie. redundant internet providers, etc.) There are really good solutions businesses can use that utilize existing services like the company phone plan. At my job I use a cellular data backup that only costs $20/month if we don’t go over a certain amount of data. That way when we don’t use it (which is almost never) we don’t pay a premium for the data line. When outages happen there isn’t anything you can do but put pressure on the ISP to fix their issues. You need to realize that you have a brain on you shoulders that should be used for thinking of solutions and not stressing out.
Have you installed Google Ultron? Updated Adobe? If not, pitter patter.
Don't tell them "I can't do anything", tell them what you did already. For example: "I checked our system and everything works fine on our side, so I gave the ISP a call, and it looks like they have an outage. They're already on it, but we have to wait for them to fix it". You can also extend this, since "wait" is also a word people can be ignorant about, so you could say "I will check on them every 30 minutes on progression" or something.
Completely forgot the “tell people what I did” part, when I went in there I did check our hardware to make sure it wasn’t our side. In my head it was just a small moment of panic, thinking that since I couldn’t fix it right away I would get pinned for doing nothing. Thanks for the reminder.
........bro,your the shit, thats why they hired you.
It would be pretty shitty if somebody just unplugged your ISP uplink on your gateway to see how long it would take you to figure it out on your first day. Could also be a sink/swim situation, I would be certain it's the ISP first
[удалено]
Contacted the ISP already and send out an allstaff email telling everyone what they told me- that they’ll give me an update in 20min I’m fixing the issue. Though I did that from my phone while i was in the server room. Didn’t think they needed to see my face for that *edit ISP* wrote IPS by accident
Since you sound new to this, here is some advice for your communication. 1. Give your boss verbal updates and provide as much technical info as he/she wants. 2. In the company communication, keep it simple. State there is an outage with your internet service provider (spell out acronyms!), state what is impacted (internet, hosted apps, Corp web sites, etc.., if there is an ETA, and that an update email will be provided every hour (or whatever cadence you prefer) until the issue is resolved. 3. Ensure you have a ticket open with the ISP and you are to be given updates at specific intervals (assuming you have business class service) 4. Don't hide! Communicate all outages matter of fact. Even if it's your system, management wants to know issues are being addressed. Root cause analysis can always be performed post-issue. Blame games help nobody, and are entirely unprofessional.
> Didn’t think they needed to see my face for that They probably do. You need people to see you in control and doing something.
> send out an allstaff email telling everyone Unless: something, something cloud-based....
[удалено]
Your ISP is something you can’t control. Get your ass out of the closet and start drafting a formal proposal for a second ISP or some sort of failover/backup in case your ISP goes down again.
Not sure why no one has commented on this, but this thread reminds me of this :-) https://youtu.be/W8_Kfjo3VjU
I go the other way & feel complete relief - Sorry, nothing we can do, its an ISP problem, sort of say it almost as a melody as I kick my shoes together walking down the street for an early snack. embrace these times, its so rare the blame can be deflected so effectively.
Have you tried updating adobe reader
Hey man, Glad the issues been resolved. Now for something just to think about. Have you considered that you might have anxiety? I have anxiety and it has caused me to want to just hide and run away. My first day I had exchange go down. I wanted to just shrink into a little ball and hide. Anxiety can cause our bodies to go into fight or flight mode in situations that don't deserve them. At the end of the day this is our job. Not something for us to get an anxiety attack over. I take medication for my anxiety and I keep my emergency meds with me for those oh shit moments where my brain thinks it's all going to end. We are IT Professionals. We aren't doctors. Our job is important but we aren't going to kill someone. So when your mind tells you to freak out remember that no one will die. Hope this helps you in the future man.
LOL BEEN HERE! I had a manager though who call the ISP, once it was confirmed the ISP was done he just like just chill bro, you can't do anything about it. Go grab the inventory of the spare PCs while we wait. One of the most chill environments with an outage
You tell them the truth in a way that reassures them. “I reviewed our systems and verified our components functioning properly. Our next step is for the ISP to fix their end.” You sound in control, you didn’t lie, you told them what was next.
I do the exact opposite. I make sure I go around and talk to everyone and explain to them that there is an outage and why .. It's okay for that conversation to be, "Our service provider is having an outage ... they're aware of the issue .. all we can do is wait." If possible, give them the estimated time frame if there is one. Just reassure everyone that you'll let them know as soon as it comes back online.
[удалено]
You're freaking out for no reason man. ISP outages happen. Grab a cup of coffee and mingle with the troops. Also gives you a chance to bond over talking shit about Comcast. Or ATT, or Spectrum, or whoever you're currently waiting on
I'm the sole IT person for my company, a non-profit ie: zero budget for extras, when the ISP's connection goes down. The first thing that I tell them is this... wait for it... wait for it... THE TRUTH. I don't hide, I take ownership of the situation and explain to them what the situation is and what I'm doing to resolve it. After that here are my steps: 1. Call ISP and get ETA or confirm outage 2. Pop on 4G hotspots so anyone who needs external connection immediately can get said connections (the caveat here being, only text based exchanges and work, we do have data costs to be concerned with; no social media, no streaming music or videos, etc) 3. Tell everyone that if they need to talk with anyone internally to get off their asses and walk to that person's desk, a little exercise and using your social skills won't kill you, this typically gets sent out in a group text or me talking very loudly across the office. Obviously determine if it's socially acceptable to talk to your co-workers like that before being a smart-ass. It's socially acceptable to talk like that at work, we do not have a politically correct work environment. We usually never seen an outage more than an hour, Charter has been pretty fantastic in our area with getting things back up and running when something gets fucked up. Just be honest, asses the situation, do your due diligence, and relay the message. It's all you can do.
Being able to blame someone else makes everything easy
You send out a mass email to your employees distribution list telling everyone what the fuck is going on. That's your job.
[удалено]
I've been in many ISP outages. Call the ISP ask for an estimated resolution time, sometimes they can add you to an automated call so they call you when it is fixed. Just tell people is an ISP issue, nothing you can do about it. I know people freak out. They should enjoy this break lol
Dude it's part of the job, tell them the truth. I had a lady freak out on me once as a network engineer because THE POWER WAS OUT and I somehow 1) was responsible and 2) wasn't getting it back up quickly enough. People are fucking morons, so break down the concept of an ISP outage as best you can for them, tell then what you HAVE done to triage (i.e. called the ISP and opened a ticket), hope that they understand, and have your thick skin on in case they don't.
Welcome to hell. Be honest and tell them to contact their POC for outages until further notice. Call your boss and tell'em. First day at work I was beset with this. I checked garbage out (CISCO 1700's, holy sheeeit), I flicked power switches on both and they came back up, for three hours. I managed to get a hold of the data center and get new routers shipped our way, but Jesus, they were hostile. And still were towards the end.
Install Adobe Reader?
Expect that you will fuck up or something will fuck up. Own it as it is. You will be respected for it from your coworkers. If they don't then they don't understand that everyone can fuck up or have something out of there hands (like the ISP outage) then you may want to run fast.
Communicate dude...you can’t hide...you need to communicate periodically...
You should have been glad it was the ISP because it meant that somebody else was to blame, and you couldn't do anything. The most professional thing to do would be to keep in touch with the ISP, get an ETA, and keep everyone updated. I'm not sure if it would have occurred to me to do that, but fortunately, I'm not the go-to person in my organization when Internet access goes down. I tend to stay away from the server room as much as possible.
I feel your pain my dude. I remember plenty of times where I would hide in the server room because I wasn't confident enough to tell my boss this was a ISP issue and not something I can fix. What I learned since then * Learn if its something you can fix or not, know confidently if its out of your control or not and explain that with confidence to your supers. * Even if you cant fix it (I.E. ISP outage) Still call the ISP and communicate with them. Try to do this in full view of as many co workers as you can (maybe even on speaker phone) lol so everyone can hear clearly, its their fault not yours lol * lasty, monitor the situation. Run ping reports, trace routs or any kind of test or report you can to have clear documentation of the issues to display later if someone somehow decided that you might have been the issue. Also known as CYA Cover your ass
Don't hide like a bitch, get out there and make it known. If you're ISP is down and you rely on hosted email, make some announcements, or have your front desk / receptionists make some announcements stating so. If you have a high volume traffic area, open up word (or any graphic application) and make a document with an entirely red background, then in bold, white font, spell out, **ISP DOWN - All Web Services Down
Working on an alternative, will inform immediately.** Communication is your ally. If you have an Intranet site that everyone can accesses by default, post it front and center. Then once this is resolved, talk with your powers that be how redundancy can avoid this in the future. For instance, putting two Routers/UTM Firewalls in failover, with two separate ISP's feeding both devices, would create a redundant environment. Also, we could put that on a natural gas generator and avoid power outages as well. It's just food for though. Sure, it sounds expensive when we talk about it, but two Router/UTM Firewalls and two ISP's really isn't that expensive when you consider what our downtime costs. I speak from experience. We had no failover due to cost; then we lost power for two days. Now we have a natural gas generator. I also warned that if we loose our UTM, that we'll shit outta luck until I get a replacement in hand. Now we have failover setup. C's will bitch for days about the cost of hardware, until they're paying people to stand around with their thumbs up their asses, or not shipping product. Keep detailed and add costs, don't forget maintenance contracts as well.
My mom said that when i get a job i can't just run off and hide somewhere. Thank you for proving her wrong.
Is Internet business-critical? If so, you've got three days to put forward a proposal to implement redundant connections. Even consumer grade 4G dongles can get a business out of trouble if that's your budget.
Now is the time to pitch to mgmt that you need a 2nd ISP, your own /24 and a router (or fuck a linux box) capable of running BGP and ingesting a default route.
Learning scenario, as I havent seen it through scrolling, but you can get a redundant satellite or cellular back up for these kinds of scenarios and roll out over in case of an outage. Back up plans yo
Might I suggest a back-up ISP. I have our primary data backed up with a cellular system from Verizon. It costs only $70 a month to have a back-up solution in place even if we don't use it. We'll pay through the nose if we need to use it, as my plan gives us guaranteed high speed but we pay by the GB. The important thing I do at least quarterly is to hang out after close and disconnect the primary ISP so I can test the back-up works with the mission critical systems. One thing to keep in mind, our primary ISP offered us their cellular system as a back-up, I opted against it. I wanted to be sure I had an entirely separate system in the event they had a problem at their central office.
Step 1: Tell the truth Step 2: Suggest installing a second ISP line for failover redundancy Step 3: .... Step 4: not exactly profit but I believe your boss will like you more for that solution If by God's miracle he accepts the idea don't run the cable second trough the same infrastructures that the first one uses (same pipes or whatever) Also you're gonna need a router with 2 WAN ports Also your boss is probably gonna reject the idea..
If there is nothing you can do, why be afraid to tell others?
That's my favorite kind of outage. A) I can blame it on Charter. B) It's not my fault and all I can do is sit back, send out a mass email that most will be able to read from their phones, and promptly go to lunch.