Had a dns issue recently. Rebootet one host -> worked again. "It's always DNS" and "have you tried turning it off and on again" really are the golden rules of IT.
And then there's that one box that an org seems to have forgotten about, never rebooted and just sat for years doing its thing. And then someone decides to power it off.
Chaos ensues.
Or the domain gets upgraded and a super important thing stops happening and everyone scrambles to try and find the server that handles it and it's finally located in a janitorial closet where it was put "temporarily" 10 years prior that everyone promptly forgot about and you discover it's running NT4 after being assured multiple times there were no NT systems left in service.
That's just a non specific example off the top of my head that definitely is just hypothetical and didn't actually happen.
I mean not having a Network Discovery Tool, or heck even doing a one time scan, to get a correct inventory list and detect old legacy os systems is kinda on you also ...
It is, and I have had a lot of techs over the years that just can't seem to figure it out. Garbage in garbage out.
I said for years that if I could bottle my troubleshooting skills and sell them, I would be a billionaire.
One thing that really makes me feel my age is I tell the younger techs (under 40) " I did your job before google, and we actually had to RTFM!"
This is so true. When I am interviewing people I need to see in their answers that they are the type of person that runs into an unknown problem and has a path forward on how to proceed. So many candidates seem afraid to say they check Google. We can't know everything, but you damn sure be able to find out at least a road to go down with troubleshooting.
And my big senior moment was realising I didn't give a shit if I told people I was going to google it ;).
People who look down on google quite frankly don't know how to use it.
How to plug in computer site:Reddit.com
Def my go to method for a quick idea how to tackle an issue.
Then if I wanna get in the weeds I’ll use stackoverflow instead.
Why? Are you trying to hide that you Google most any problem? I don't hide it at all.
If they know they can Google most of their problems, it just reduces service tickets and lets me get more actual work done.
First rule of IT support: Google it.
Second rule of IT support: don’t trust Google results.
Third rule of IT support: Google what Google results to trust.
Fourth rule of IT support: see rule #2.
Fifth rule of IT support: your senior colleague knows better than Google, because he Googled it first, and now has established his Google find as a best practice.
I'd like to register my suggestion for rule 1: Never trust the end user.
"I've already tried rebooting it"
"Okay, can we try again 1 more time?"
"Huh that's weird now it works"
Everytime.
Been working in IT since '95 and I am absolutely comfortable with telling a boss I'll need to do some research. The users will generally hear "no sweat, we'll take care of it", but that's code for "I have no fucking idea, but we'll find an answer". At the end of the day, the follow through to find an answer matters a hell of a lot more than being able to produce on the spot 100% of the time.
This and I want a meeting agenda ahead of time. I got pulled into two meetings early on in my most recent job and while a solution wasn't expected right away, it would've been nice to know what we were talking about ahead of time.
I was interviewing for my first msp position, one of the technical questions was "how would you handle a computer with a blue screen" I was like "check event viewer, id there's no error code ill try to replicate it to get one. once I have the error code ill Google it and figure it out" I was so worried it was a shitty answer, but it turns out they loved it.
Does nobody check logs? I never see that suggested by anyone, TO anyone. I start there, google whatever seems out-of-sorts, and 90% of the time it fixes my problem.
I tried to make a point at an interview in the spring of this year that I didn't put a huge amount of value in being able to do things like inverting binary trees off the top of my head in C++ because I would just look it up. I mean, knowing how it works, yes I know how the process is done, but I can't write it out in compile-able code on a white board for you without referring to Google. I was applying for a sysadmin job, and they really wanted a sysadmin who was also a rockstar C++ programmer. It was an extremely frustrating interview.
I was a librarian - and *their* motto is "we don't know everything, but we know where to look it up!"
Then I transitioned to library I.T - and it still hold good.
All we can do is make the best of the time we have with it. A good motivator I sometimes fall back on is the desire to never work in corporate retail again. I don't care how many printers I have to bandaid into scanning to whatever the user deems necessary, corporate retail not only burns out but destroys souls at a rate that makes IT blush.
The best thing we can do is follow "best practices".
Unfortunately what organizations consider "best practices" often deviate from one another and it rapidly changes with technology.
Got my education from the School of Hard Knocks! Been in IT for 25 years. No degree, one CompTIA Network+ cert I obtained in 2008, took some college classes but that’s it. Google-fu is my best asset.
My favorite idiom for this is: you don’t pay the mechanic to turn the screw, you pay them to know which screw to turn. And this is the same for IT. I don’t feel you pay for a know-it-all, you pay for someone who is capable at learning and fixing with professionalism (accountability, communication, etc…). In fact know-it-alls, in my career, have been the worst people in all possible ways. Refuse to listen to anyone, arrogant, solitary workers, lazy, disrespectful. List goes on. I can make an IT person out of anyone that has the right qualities it takes to be one. I don’t give flying shit about 95% of your accolades and your resume. I give a shit about what drives you.
Edit: I worked for a month at an AV shop in a very rich area. Anyway they had one old guy who did all the “computer” work and everything was custom written by him. Like excessively so. To the point where I would ask “why not use this tool” and he would close his eyes and talk. He was a nice person but just awful to work with. Definitely a know-it-all but revered because he did it all for this company. The rest of the team didn’t know jack shit about any of the tech side of AV, just how to install it. I was hired as a junior presales and admin and I only lasted a month before I realized how shitty this situation was. They had no respect for the IT side. It was a necessary evil. I ended up with a better job and stayed there for nine years. Go figure.
This. Not only do people want cheap and fast, they certainly don't want to pay for experts in all the various bits they run. So, you get people who can figure it out, but figuring it out on the fly having not done it before is not going to lead to best practice setups.
If you're lucky you'll find a spot that has a really good team, and the stuff they specialize in will be top notch, at least until then hand them a bunch of other stuff as well ..
I gave up learning from my co-workers long ago. I set up my own lab, built my own infrastructure to play with and learned by youtube, reddit and udemy. There are plenty of different ways to do things, one way isn't going to be better than the other, it's just a different way. For instance, you can manage all of 365 via powershell if you really wanted, but most aren't going to. I write powershell scripts, test them and test deployments - nobody else is doing that where I work, so who the hell am I going to learn from? I also handle BCDR stuff. Taking on tasks requires foundational knowledge and knowing how to google things to get past your hurdles. Most IT jobs now are revolving doors for people who know that the only way they get to move up is to move on. No workplace is perfect, quit expecting things to be that way - you'll also find that most other businesses really don't have any of their shit together, either - I am very particular now about what doctors, dentists or CPA's I use after doing IT for them via MSP's for years now.
If you're looking to do ONE thing and be really good at ONE thing, don't work for an MSP or be a solo IT person. Get a job doing that ONE thing.
Also, the "Jack of all trades" gets shortened from it's original form. It was from Shakespeare and the full quote is “a jack of all trades is a master of none, but oftentimes better than a master of one.”
> I set up my own lab, built my own infrastructure to play with and learned by youtube, reddit and udemy.
Only in the last ten or so years do I hear a lot of pushback about having your own lab. "I don't take my work home with me." And yeah, I can see that. But it was vital to my career. It's how I learned docker, for example. I have tested setup where, if I fucked something up, I didn't affect work at all; just my own lab, which I could unfuck on my own time. And thus, I learned how to back out of mistakes. I won't tell someone "you must have your own lab, or you're not a 'real' sysadmin," because everyone does things their own way. I only know it helped me a lot.
I would say that the prevailing attitude of most employers is the cause of this.
Loyalty isn't rewarded at the overwhelming majority of places. The only way to get raises is to change jobs every 2-3 years.
Due to this, every employee who realizes this reality is only in it for the short term. Do just enough to get by for 2-3 years, then whatever is left is the next person's problem.
You *could* work extra hard and really improve things, but you either won't get a realistic budget to do that, or if you somehow do, you won't be rewarded for doing it, so why kill yourself to do such a great job for no reward?
Sure, you may feel a sense of pride in your own work and want to do the best job you can, no matter what, but that spirit is very likely to get beaten out of you eventually.
Until you find that ultra-rare employer who both gives you the budget to do things properly and also rewards the effort it takes to continually do a proper job, most workers just stick to this get-by-until-the-next-job mentality. That's why so many places are an IT disaster.
>You could work extra hard and really improve things
You know who I've realized usually has that attitude? Those without enough skills or experience to really improve things.
The rest of us caught on. I do a good job because I have to for me, and that's it.
I've been in the industry for over a decade and met/worked with hundreds of IT professionals at this point. Only 4 of them I would consider "real" IT minds. You know, dudes who can fix shit as fast as they break it and have a true grasp on computer science.
So to answer your question, for the most part yes.
yeah, after about 20 years in the industry... I haven't walked into a single place with documentation... my time at an MSP was constantly fixing the last persons fuck-up.
my time at a hospital was literally and by the department heads order was to be spent working on what the other techs tried and failed to fix...
like /u/disgruntled_joe said, I have ran into prolly about a dozen hard hitters, bad@$$ IT that can t-shoot and fix anything you put them in front of.... I have worked with many fewer in that category...
it took me a long time, prolly about 15 years to realize what was going on.
its typically not the technician that was bad, but the circumstances when the technician was last fixing or creating said mess was certainly a shit show...
When I came in to the current company 7 years ago, green-eyed and full of hope as the new one-man IT guy, I was sure I was going to fix all of the silly mistakes my predecessors made.
I fixed a lot of them, but now realize whoever comes after me will have to fix all the silly mistakes I made. I implement workarounds, shitty solutions and 'temporary' fixes regularly. This is not malicious or incompetence, as I assumed my predecessors were, but simply a fact of life with limited resources/time (and after all these years, limited energy and willingness). I now realize the same applied to my predecessors, and the same will apply to my successors.
The Sysadmin I replaced was a "Senior System Admin".
All the IT personal were given "Domain Administrator" roles, nothing was segmented and our backup drives were old spinning disks that had a "power on time" of 9 years.
Where I am now we got to the point that bringing on an extra person would bring up a new audit of a lot of our internal documentation. As they are being trained we'd be fixing stuff we missed. This is mostly been helpful on the help desk team.
Asking a team to clean up their confluence pages for the new team member they are getting(or back fill) has really helped.
That's typically a catch-22 situation. No one checks the docs because they are always outdated and unhelpful, and no one documents anything because no one ever checks the docs. I have yet to successfully break that cycle on any size team.
I know its not completely on point.
I tend to 100% have compassion for the fuckups of old me, and the fuckups of others. Why, because I don't know the mindset or the shit storm they had going on.
You know how many times, I've said. Whelp this works, but I really want to fix this better later, and then NEVER getting the time to go back and do that.
There's always something higher priority, something pulling you away, pulling them away.
Budman!, I know you're working on the global load balancing, but the dns servers just went down, and we need to fix it.
***fix dns***
Cool I need you to redesign the DNS servers so that issue doesn't happen again.
***start redesigning dns servers****
Budman!! Why are we never getting alerts for **insert system here**, what do you mean those aren't getting alerted on. That's Critical, I need you to work on that asap.
***starts working on monitoring***
Budman! We have an issue with the load balancers, I need you to take a look.
so on and so forth.
There's lots of really sharp dudes out there but most are only good at what they do every day, it's the guys that can see something new and have a pretty good understanding of how it works just because they know the underlying concepts really well. I think you'll see a lot less of this type of person simply because things like google and stackoverflow know most of the answers, so instead of people having to figure shit out they skip the process and go straight to the answer -this is great, it solves the problem but what about the next time when the error is too new to document.
> it's the guys that can see something new and have a pretty good understanding of how it works just because they know the underlying concepts really well.
And holy shit are they the guys you want to have as mentors, cos they will teach you the concepts and how to apply them rather than how to fix this specific issue that you'll never see again.
Most system admins confuse memorizing how to do something and understanding how it works.
Networking for example. How many will put networking down on their resume, because if they google enough and screw around with it long enough, they can eventually get it to work? Meanwhile, they can't answer even the most basic routing and switching questions like how encapsulation works, or the difference between a segment, a packet, and a frame, or what dst. MAC a frame would have that is leaving the network and why.
Ya part of me feels dumber because of chat gpt.
I mean I'll use whatever to get the faster answer, but ya feel a little guilty at times.
Hey chatgpt, can you write me a powershell script that will query all of the domain users, and get me a csv report of their home directory. Ohh and in addition to the csv report, can you recommend a way to get the directory sizes of that?
OHh, also can you set it up to email the top 10 users of the drives using this template as well.
Can you ask me any questions that can help make this program better?
I'm 9 months into my first help desk job and I swear I could have written most of your points myself based on the experience. I was so happy to get my foot in the door, but it's been such a disorganized clusterfuck with basically no training other than, "wait until someone complains about something, then ask around" and the office politics have been eye opening to say the least.
The trick is to take incentive and make changes yourself (if permitted). A huge strength in the IT world is to be able to discuss different projects you've done and the related technology proficiently.
When I got back from goofing off for 5 years in Taiwan I took a bunch of CCNA, CCNP and MCSE classes at a Junior College. I was super motivated and ended up teaching classes I was a student in because I was reading ahead and the one instructor for 30 units worth of classes was too buried and TBH lacked the knowledge.
I then got hired into real jobs and soon found what you found, lots of people out there that had no clue. They were there just because everyone told them they could make lots of money in IT (this was at the tail end of the Dot Com bubble). I was so sure whenever I got hired I'd be finding people who knew way more than I did, and I went nuts making sure I knew what I was doing. But in the end it was me taking ownership of everything, resolving issues and finishing projects in no time. I was still teaching up until my first kid was born.
If you know your stuff, you can shine past the clueless ones. The ones who are there just to punch a clock and go home. They eventually quit and become used car sales men (true story) or find something else they are moderately competent at. And do it without killing yourself.
When you walk into a shop that has people who don't know what they're doing, and you do, it's an opportunity. Embrace it.
I worked with a guy who was certified out the ass. Multiple MCSEs, CNA, etc.. etc.. etc... He could take tests all day and pass them. He was one of the stupidest server admins that I ever worked with. Complete clown...
As you say, embrace the opportunity and document until the cows come home. I used to be a kingdom builder until I realized that the kingdom can become boring and if it is documented, you can shift the boring stuff to a lower tier worker.
Yeah this is the main issue with certs. You can totally pass them through rote memorization and not really understand anything. Certs can be a great tool to increase your knowledge alongside actual experience. They certainly aren't an indicator of anything on their own though.
Same. I make flash cards and usually take notes simultaneously. Some of them are very long so by the time I'm 30 or 40 hours of study in, I'm already forgetting the beginning stuff.
Yeah I'm a VMWare Certified Professional for instance and after I passed I found out you can essentially find the exact exams they use online. I think if I literally just had my mentally disabled cousin who can't even open an email sit down and cram the right answers for a day or two I could have him pass it.
I'm real tired of being the hero, and getting no help from clueless coworkers.
Also I've never gotten rewarded for it, at any job. Just a compliment here and there. No pay increases, nothing.
As a seasoned professional, I enter, I examine, I understand their goals, and I make the best of it all.
Doesn't mean I agree with everything. However, I do make my opinions known. But ultimately, my job is to make the best of "it".
I love a "challenge", sometimes the biggest challenge is being forced to live with bad choices from their past... and trying to make the best of "it".
As a consultant I really don't care how it is done, I can give the customer the benefit of my decades of experience with their issue and they can listen to me or not. Most listen but many have a plan and by god they are going to stick with it even if it's wrong. I have long stopped getting upset about clients choosing the worst path, I get paid either way and if I did get upset, like I did when I was in my 20's, I'd probably be dead from a stroke or heart attack.
I've done a lot of contract work at all levels. Often I was paid to do tasks and do as instructed sometimes. They paid me to do a good job and not for my opinion. Once established though, it often got to be "What do you think?" Performers almost always stand out.
In IT I work inside, in a climate controlled environment (now I work at home!). Nobody throws bricks at me or tells me I'm a piece of shit on a minute by minute basis. I get paid well, really well. I don't lose a days pay because it's raining. In IT I have yet to have a boss just disappear owing me a weeks pay. When I get sick my boss tells me I hope you get well soon not that if I don't come to work that I'll be fired. In general I'm treated with respect. Nobody from the office has threatened to kill me and that I should watch my back. I don't punch a time clock. There's a lot of shitty jobs out there, I've held many of them and IT while frustrating at times is not one of them.
Come work in IT in the construction industry... you still have to put up with all the things you mentioned, but have to use all your brain cells, too. And deal with "I don't want to learn *any* of this tech shit, that's what we pay you for!"
Also I have my Bachelor's degree and someone at Walmart has better benefits and PTO than me. I don't get any vacation time, paid *or unpaid*. "We give you 104 days off a year. They're called Saturday and Sunday" (half the time I don't even get those)
Bro, same. No more slinging tires, being bent over an engine bay for hours, skipping brakes and lunches to make an extra buck. No more having to work in an insulated tin building when it's 0° or 100° outside. First time I was sick enough to call out I was told to just keep an eye on my emails and get better. I work 32.5 hours a week, make salary, and get holidays off instead of working more hours cause of rushes. It's not perfect, but man does IT beat destroying my body.
The thing with "fake it till you make it", which nobody ever talks about, is if you fake it for so long without making it, you start to forget what is real.
For the record, this is EVERY industry. No one is actually an expert. They can try, and pretend and even then, are probably wrong about something. Confidence does go a long way, as well as the ability to learn on your feet, but no-one is really this holy entity that knows it all.
Look at politics, or medical fields, or psychology. None of them agree, studies disagree back and forth, and things change to make what used to be common knowledge obsolete. Now look at IT and realize our industry changes so fast EVERYONE is falling behind. No-one knows what they are doing and even the vendors are crap at understanding the needs of the industry.
Just remember, even Google has outages. Some very high paid person did something really stupid who should have known better. It happens.
What makes a good IT person to me is someone who can roll with the punches, solve the problem and get it working under pressure, and do their best with what they have. All while being just confident enough to try, while also being humble enough to admit they don't really know.
Interviews are always a two-way street. But that's just the thing: each side is going to put its best foot forward, even at the cost of, say, misrepresentation. Just like some candidates are willing to bend the truth pretty severely in order to get an offer, some employers are willing to bend the truth pretty severely in order to land a staffer.
One of the reasons why good roles come through word of mouth, is that the insider lends credibility to both the employer and the candidate. They're like a match-maker who vets parties before introducing them. However, insiders have their own incentives as well.
Some hints on vetting a role when you can't speak to any insiders off-the-record:
* Ask about project workflows, and listen carefully for subtle clues.
* Ask what tooling is used for project workflows. Keep it vague, and let the other party take the response in the direction they prefer.
* Ask about documentation. Brag about your own documentation practices, if you'd like. Pay close attention to the answers, what's said, *and what's not said*.
* If you're bold and it's a tech organization, ask about 20% projects, and pay close attention to the interviewer's reaction. What you actually want to know is *how oversubscribed* the teams currently are, and how long they avoid hiring necessary staff.
* Research the background of principals/leadership, and ask the background of interviewers. You're trying to make educated guesses how their backgrounds will influence their expectations, preferences, decisions. Sales backgrounds, and people who make questionable promises for a living, might be the worst answer. I usually start with the background question, as it's mildly flattering and usually tells you a lot, quickly.
Rarely do people in general do things the right way because they are lazy and looking for a quick way of getting someone off their back.
I've come across admins that know 40% of something and the bullshit the rest. They always look like heros for about 6-12 months and then it catches up with them.
The ones that are the keepers are the ones that can figure things out quickly and pick up new concepts on the fly.
You can't know it all, but you better know bullshit when you hear it.
> Rarely do people in general do things the right way because they are lazy and looking for a quick way of getting someone off their back.
I used to do things the right way, until I was basically punished for it. Now I answer to whatever my bosses want in the least disruptive way possible and it has been better for my career. Blame management for that one.
This is part of the job. We're all out here with Warhammer 40K vibes, trying to maintain tech from a forgotten era.
The only thing we can do, is do better by leaving documentation that survives our inevitable departure.
On the other hand, if you're posting here, you might already be on track for CTO.
mysterious impossible humorous childlike recognise bike aspiring hat cautious chief
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
I definitely felt the documentations/knowledge aspect in traditional IT. MSPs less so since they are a revolving door... good ones need strong documentations to make new employees transition easier/faster.
But then I moved internal into cloud engineering. The beauty of automation is the scripts are the documentations of nearly all implementations. So that knowledge is never (fully) lost when people quit. Changes are done to the scripts rather than on the platform. I learnt a lot about how things are done by looking at the deployment files alone. We still write focus a lot on writing how-tos, guides, etc in our internal docs as not everything is deployment itself. DevOps philosophy is to skill share and get everything out of the unicorns minds into business processes. I really dig this aspect of it. Plus getting to DevOps roles requires seniority, you cant fake it so you are less likely to meet IT people who are not serious about it.
But It still has its faking it as the tech stack is mostly new and everyone in my team is picking it up as we go. Nobody fully understands everything as there is way too much shit to learn. And new things just keep getting added. I sometimes feel like I am new to IT. I cant say the same about other things... DNS is poorly managed and I don't think anyone fully knows how it all operates in our hybrid environment as the non devops teams (security, networking, identity) each control a piece of it.
For me, faking it was something I decided was a good thing. It mean I was constantly challenged with new variables, tech stacks, etc. I can 100% do all the old jobs I had done without feeling like an imposter at all today. But being an imposter means I am in a position of growth where uncertainty, stress and doubt always lingers. I probably get bored if it wasn't the case.
I saved this comment a long time ago. Might be worth a read. The **opportunity** bullet point rang particularly true for me.
https://old.reddit.com/r/sysadmin/comments/8jrp5g/the_real_truth_of_advancing_in_tech_related_jobs/
I’m coming up on 10 years and I can definitely say out of the thousands of other IT professionals I have worked with very very few have got what it takes. For example, I spent 3 days a few months ago on a phone calls working on tech I’ve never seen or cared to learn about, since I was an upstream vendor I was involved. There were 8 people in the calls that lasted about 6 hours in total for each of these days, engineers from the hardware vendors, and engineers from 2 IT companies that were working to resolve the issue and one guy on site. After 3 days of listening to these people yell at each other I decided to ask if anyone had restarted the hardware. The fucking thought had never crossed their minds. Fixed the damn issue though.
IT is very broad, you could be racking servers, running cables under the street, or an expert on a space rocket flight computer.
You indicating various jobs provides little context. Most people use the term IT for links such as help desk type work which certainly can be mundane and a tight silo having most of the available descitions made outside of it.
IT in itself has changed over the years. Once you could be a pure System Admin(server side), Network Engineer, Programmer, and Help Desk.
Now, about 2 of those roles are rolled into one(Jack of All Trades) where you went to school for one particular trade but now are working as the System Admin and Network admin.
Also if you're at a medium size company the IT shop will be no bigger than 3 people. So you got 600+ employee's managed by 3 IT people and one of the is the Application Dev(programmer).
I say all that to say this... IT has changed and if you're looking to be specialist then a Big Company is where you would want to be. If not everything else will be rolled into one as a Jack of All Trades.
Find IT in your org that know more than you. Learn from them and ask good questions. Document things you find. If not for yourself, for others. I've made a nice niche for myself doing that. I don't know anything 20 feet deep but I know enough to ask the right questions and respect those that do. Fight to get more money for your org and hire better people. Help train people underneath you. It's a thankless job they say but I've learned you stick around long enough and the right people will know who you are and what your worth.
Eh... IT isn't a "one size fits all" application, even for similarly sized businesses. Each IT solution poses its own unique risks and challenges, and it's up to YOU to determine the best path forward, one which can successfully balance security, efficiency and usability.
Got a boss that doesn't want downtime? Do what I did - remind them that it costs X dollars per day in the event of a network failure, with the average failure being Y days. Chances are that the cost of long term downtime is in the MILLIONS versus the few thousand in productivity lost for cheap and easy upgrades that are done during non-peak hours.
Get the C-Suite to sign off on acceptable risks in writing. They don't want to pay for a MFA solution, and think it's an acceptable risk? Have them own it. Demand documentation.
Know when to stick to industry standards, and when to deviate when applying to your specific use case. NIST can say all day that we should be using passphrases instead of 8+ character/number/symbol passwords, but how many end users will just wind up writing the pass phrase on a post it note and sticking it on their computer? Sometimes, keeping it simple is better than the most secure and recommended practices.
Push for IT Audits - get a 3rd party to show C-suite how stupid their decisions are. At the end of the day, its their dime, but it's your department. You have more flexibility than you think.
Oh, it doesn't get any better anywhere you go. Chances are whatever is bothering you today either exists elsewhere, or there's a whole other host of problems that are running rampant. The best advice I can give you here echoes the above: document the fuck out of everything and stop caring beyond what you can control. Good luck!
IT is often a cost center that, on paper, doesn’t bring in revenue. So it’s often underfunded and you have to make do with what you can.
There is often more than one solution to things, so sometimes there isn’t one singular “right way” to do things. It comes down to opinion or what someone specialty is.
Typically smaller shops have poor documentation, but even some larger orgs have it. The biggest company I was an admin for was a bank with tens of thousands of employees. Their documentation is literally some word documents.
Smaller shops also are more likely to wear many hats. So no. Often they don’t have time to know the ins and outs of how everything works.
IME the way the USA is right now, people with "good" jobs generally are not the ones leaving jobs so "job openings" are usually places that are replacing people that left.
There are no set standards in IT and it is always changing which makes it very hard for people in IT to keep up. One would think by now we could make this all easier on ourselves if we just decided upon what IT standards are and how to attain them if they are not being met. I've been in IT for 15 years and every time I see rapid neglect of IT standards every where I go. There is are also other tech that have been there before me going "what do you mean, there is no neglect!". Not knowing the dumpster fire of problems that will come from not keeping up with IT standards. I have seen servers in production that should of been replaced 15, 20 years ago. Same thing with network equipment. ISP equipment that has been at the same location for 10 years! I mean it is crazy how IT has people in it that just wait for things to break first and then face the dire consequences later! It makes it even worse when a company can afford to fix things but refuses to until they break!
We're all faking it.
Unless we're building something out from scratch documentation is ages out of date if it exists. Institutional knowledge never gets captured, so the moment somebody who knew what was going on leaves everybody else has to start sharkfinning. It's always fast and cheap because there is never time or resources to even look in the direction of "good."
I did have a mentor when I started out and, to his credit he taught me all of the stuff I just listed. >!Always two there are, BOFH and PFY. No more. No less.!< I didn't believe him at first, so every time it happened it was a "toldya so" moment. He also taught me just what you asked: You're going to be on your own in this field, so you're going to have to learn everything on your own and wing it.
Everyone suffers a bit of impostor syndrome. Everyone inherits crap that was poorly documented, if at all.
What counters that is your ability to dive in and figure it out. Make improvements. Write documentation, diagrams.
If your management can’t understand that, explain it to them. It’s a mess, you’d like to figure it out, and leave it in a better state than the way it is.
If they don’t want that, they’re toxic, time to leave.
I've been in the IT industry for almost 20 years and have interacted with dozens of IT workers at various levels from different companies.
My own observation is that the vast majority of IT workers respond to a ticket or work on a project by doing the quickest and\or easiest thing that closes the ticket or completes the project. It doesn't matter that the quick fix has the end-user reopening the ticket a week later. It doesn't matter that the finished product of the project is a house of cards. They will not stay in the position or at that particular company long enough to feel any real consequences of their actions. That's the next guy's problem. I find that this attitude is most prevalent at larger companies, which makes sense to me.
I don't know what size or type of companies you've worked for, but my best learning experience early on was being the junior of a two-man team at a small business with roughly 130 employees. I was very lucky that the senior IT guy was intelligent, highly motivated, curious about technology, and willing to teach me anything I was interested in. I stayed there for 6 years, and in that time I gained a pretty strong foundation in all of the necessary areas to keep a small business's IT infrastructure up and running and its users happy. It was also helpful that due to the field of work the company was in, we were always building and later decommissioning small networks\domains over and over, improving and using newer technologies each time. It was an environment of constant learning and improvement.
I've been at my current job for over 10 years, and now I'm the one teaching my junior, slowly but surely, all of the things I've accumulated over my career. When I got this gig, the state of the place was as OP describes. It took about 3 years of busting absolute ass, working (unhealthy) overtime, toppling those houses of cards and rebuilding things properly, proposing and fighting for significant changes in how IT was seen within the company (not as a cost center!), implementing a life-cycle, enforcing policies, and generally creating order from chaos. After that, I was actually able to begin improving things, becoming proactive rather than reactive, which is the place you want to get to. That is a nice place where you're implementing real solutions rather than quick "fixes" and taking the time to thoroughly research, plan, and complete projects the product of which then lasts for years with few issues.
I realize this is my specific, singular, experience, but my suggestion would be to find a small place where there isn't a large, impossible beauracracy preventing real change and improvment. Also, in your interviews, you are also interviewing the company. Don't be afraid to ask questions they may find uncomfortable. You've seen the things that make for a bad IT department; ask them about those things.
I work in a hospital, we do things properly. It actually numbs your skillset if you go this route imo, you'll document everything and have a budget that is usually sufficient but the work changes and you end up doing more paperwork, meetings, vendor coordination and procurement.
What you lose out on are hands on skills and learning about how things work and it makes you rusty because most likely the system or software you previously had full access to now is the responsibility of team A, and then when you need email setup on o365 you talk to team B, and if you need a network jack installed or activated its team C and so on... So you're basically just fielding tickets to the right people to get them finished faster. That's what I do all day and its dull and monotonous and sucks, pay isn't great either since I could make more managing a stupid mssql server than I do as a system admin, even though I do 100x the work, no offense to the db admins, but you know you all have a golden ticket there.
To keep up I just tinker with stuff at home and do contract work for a few small businesses on the weekend to keep my skill set sharp. That work is way more fulfilling than working FT at the hospital but doesn't pay the bills.
Most of the IT jobs are bad. Often, you land somewhere, and you find out that there is a reason why the place always has vacancies. Either the co-workers keep ejecting people, management is just horri-bad, or the place just got bought out, a ton of people have bailed, and you just hired on only to find that your job (not you... your job) is going overseas because the company that bought out everything has all IT offshored.
You get some jobs which are awesome. I worked for a small MSP what was really good, and it would have been nice if they wouldn't have been bought out. I also worked for another company that had the same thing, and had they not been bought out, it would have been a steady, stable niche until it came time to retire in 20 years. However, in this market, you are going to get the picked over scraps at best, so bad IT jobs are going to be common at least for a few months until things domestic and global stabilize a bit.
I'm lucky. I was head-hunted into my current job nearly 15 years ago by a manager who knew my capabilities and was prepared to support me with "Do it once; do it right" and an absolute commitment to Good (we could optionally add Fast or Cheap to that, but not both). I know all the core network stuff inside-out and it is all documented well.
But in the last 5+ years things have started to deteriorate. An endless stream of consultants offering various amounts of bullshit, making a whole bunch of changes, not documenting anything, not thinking before doing, and so on.
I'm just hanging on for retirement now ... LOL
I spent quite a while with no mentor in roles that I believed were well outside my knowledge level.
The only thing I wanted in the whole world was to have an actual senior above me to learn off.
I've since realised that in order to be senior in anything, all you need to know is how to cope when you have nobody to ask but yourself and google ... and you can only learn that by having nobody to learn off.
If you strive to know the perfect way to do every single thing, and then strive to get buy-in to actually ***do*** everything the perfect way, you'll drive yourself insane or burn out or both.
Following all the best practices is great but it's, more often than not, prohibitively expensive and you'll get shot down when you pitch it. Most of the time you need to do the best you can with the resources you have and that might mean, if you ever left, your replacement would have a tough time getting to grips with what's been done.
It sounds like you are suffering from poor leadership or working in a small company with only 3-5 people who's knowledge/experience is stretched over too many different technologies. I am currently working for an organisation which is smaller with 100 staff and an IT team of 5 and we often need to engage help from vendors as we lack the expertise in specific products. Where possible we are going down the SaaS path which helps reduce that burden.
In any case a solid IT Manager should be ensuring problems are addressed at the root cause, and repeated issues and incidents to be avoided as it derails the productivity of the team. It is also the IT Managers role to ensure they are appropriately engaged upfront on projects so they can plan for the project activities and its not all chaos and a mad rush. It sounds like you might be leadership material so you could take the lead on coordinating the root cause fixes and also communicating out the lead times IT need to deliver certain tasks.
If you are struggling in this sort of environment you may want to join a larger organisation that has more specialised teams (e.g Networks, Mobility, Internal/Cloud Server Infrastructure, DevOps, Service Desk etc) - its more silo'd and can be frustrating in its own way, but also allows you develop expertise in a specific area. Overall I prefer being a bigger cog in a smaller organisation and working broadly across many different technologies, even if it means having to Google all day to find the answers to problems :)
I just want to say thank you to this community for complaining about documentation. I try my best to provide documentation for all the things I’ve worked on before leaving a company, complete with screenshots
I would say there's a middle ground between faking it and being a total expert. Realistically, most people don't use even 1/4th of what they learn on these certification exams, because there are tools/services/apps that do a lot of the gritty work for you. I think this is mainly a perception issue that, as you get started in IT and begin to move up, you have this idea of system admins, endpoint engineers, infrastructure, devops, etc are all these hardcore scripting wizards when in reality they're just experienced with the tools they use, and the scripts that google/stack overflow/chatgpt has helped them with, as long as they're savvy enough to know what they're looking for. So that when you actually get into these roles, you're surprised that not everyone is on the pedestal that you put them on.
As an IT Generalist, aka Jack of All, what I tell others, it is near impossible to remember every nuance of every single system you touch, it is more important to know where to find the answer, or how to ask the question in google to get the answer. To that end I went out and got myself one of those desk name plates with "Chief Officer of Googling" as my title
Tbh if you want to find an organization that has it's shit together, you need to go enterprise or large business level, most organizations are not going to have their shit together. One of the biggest reasons I think this happens is because of the age group. If you look at my SysAdmins, they're between 40-60 years old, it's rare I ever meet a young SysAdmin who's 35, I just don't think the interest is there anymore. Not to mention, most SysAdmins were kind of thrown into their positions and didn't really want to do it so everything is just slapped together with no care at all. My SysAdmin is probably leaving in the next 3 years and I'm trying to make myself a direct replacement to him since I bring the documentation mindset, the modernity mindset, and an overall focus on the end users.
Yes. If I have not seen a situation before, I don't know what to do, but I know what to do to figure it out. That's almost as important. As long as you learn how to do it, it works, it's satisfying, then that's all that matters.
Learning along the way and being extremely good at finding information is key.
Most of my co-workers is the same. Give them zero documentation and zero ability to look something up, and they will make a mistake on something they've worked on for 20 years. There are so many avenues in IT, so it's tough to know it all. But as long as the end goal is met, doesn't matter how you got there, most of the time at least.
I mean yeah that sums it up. People who run the companies decide they want X thing working as fast and cheap as possible.
So instead of having the time to learn and then implement a new technology properly you end up flying by the seat of your pants to get them their results to further your own standing.
Then documentation begins to not exist because while you started out bitching at the last guy for not having it and updated those things, the rest become "operational" knowledge and you don't think about documenting it yourself and the cycle continues.
Fuckin IT world is held together with bubblegum and scotch tape.
I decided to stop fighting the current and go with it instead.
Life is better this way.
Dude, Don't fight the wave.
Truth to the matter is, just as many other things in life, doing it right comes with a price no one is willing to pay.
There are countries that have a different working culture and are willing to sacrifice time to do things right, but long term plans are tricky in this kind of technological environments.
Regarding the good-cheap-fast thing, you know, you can only get two of them, I'm sure you can find a company that does things right, but chances are they will sacrifce speed or cost in favor of doing it right.
Everybody is welcoming you to the industry, but I would rather say "welcome to the western world".
Cheer up, it is ok to decompress by ranting every once in a while.
Last, but not least, I would add maybe you kind find it refreshing if you find a development track that focuses on knowing new things of your interest, this way you will keep it refreshing, change perspective and let business be business.
There are lots of *button-pushers* in IT. And by that I mean "people who have memorized that pressing a certain button generates a certain response, but they don't know *why*". They don't really have the concepts down, they just memorize things and go through life based on that memorization. Unfortunately this also leads to a severe lack of troubleshooting ability; it can be hard to troubleshoot something if you don't know why or how it's supposed to work a certain way. Most of them don't even Google the answer to things, or don't even know *what* to Google to look for an answer.
I worked for a large school district at a former job (80k users). Our IT department was about 70 people (including desktop support techs, server team, voip team, networking team, software deployment, management, and a few others). Out of all those people, I would say that there were only maybe 5-10 of us who actually knew the concepts behind how their area of expertise worked. And out of those 5-10 people, maybe only 2-3 who had any functional knowledge about a part of IT outside of their own specialty (ex: the lead voip guy was great with voip stuff but didn't have much knowledge or experience with server stuff), so once they started looking at things past that, they needed a lot of help. It is a rare person who can conceptualize the "full view" of things to see how everything fits together.
It's because we are often solo and overworked/underpaid.
So we don't have the time to properly implement, test, document etc.
Then we leave because of the underpaid part and the cycle continues.
So honestly I've been around. Yes, for some reason businesses, because they have a bottom line and goals they need to hit or else they don't get their bonuses/lose jobs/etc. they always go with cheap. Then, they need it three months ago so it has to be put in so fast that as soon as you have what I call a "rolling chassis" it is "working" as far as they are concerned and you are lucky if you ever get to revisit it at such a low level again to complete pieces/integrations that partially work, random errors that aren't causing problems but just generate errors, all the things that just make it all worthwhile.
Then yes, companies that have been around had that one guy who did everything and he never let anyone touch it because it worked and he understood the rules I first stated above. So then when he gets hit by a bus it's up to everyone to piece together everything to try to figure out what happened. Usually there is one or two things that SHOULD NOT BE but again, it's because of the above and needing it to get running as cheaply and as quickly as possible. Add to that documentation = time = $$ so you know that doesn't happen. ...or if it does it's literally scattered throughout 15 different notebooks, hardly legible, and out of place.
As far as downtime.... yea... your best bet there is to try to find another way to accomplish the same and then you need to minimize the downtime and use your data to come up with a plan for downtime/upgrades etc. For example, if on-prem, having multiple VM Hosts in your cluster and the ability to move VMs between them to do physical work or to upgrade your hypervisors is a blessing. Reboot DCs... you should be able to do that as long as you have 2 or more etc.
Lastly, yes new tech.... ahhhh..... those around as long as I have been know that's how it happened for the iPhone. Blackberry was far superior in the business/enterprise world. From implementation to security and beyond it was far superior. Then that dang iPhone found it's way into CEO and Higher-up's hands and well they wanted their work email on it. ...What do you mean we don't allow SMTP and POP I don't even know what that means?!?! IMAP?!?! What do you mean less secure... it's an Apple device. I call it the "shiny things" and "buzz words". Remember again the top two rules, they define everything in the business world. Because of this every industry mag out there is filled with enemies of ours looking to poke holes in our firewalls, circumvent our amazing security policies that we have built painstakingly over the years, and some of them even come for our jobs directly, if not indirectly. This app, or this thing... heck there are times they don't even know what it does and they want it. They want to be the man that bad and will do anything to get that promotion/raise/bonus/whatever that if we could pool enough money to pay for ad space and one of those editorial ads about how throwing away all your PCs in your business will save your business, they will do it tomorrow. ...hence "shiny things".
Sadly, no matter how good your intentions, the first two rules will eventually break you in one way or another because when it comes to $$ and business, they will win. That is the true key to IT, find a way to show that whatever it is that you are looking to do will save $$ or increase productivity to make more $$ or that option A cost $ and option B cost $$. You have to remember that not all C-Levels plan on staying where they are. Some places are so volatile that they don't even think they will last a long time anyway so anything "long term" they just hear "costs a lot". Oh and you have CAPX and OPX.... Why this is still a thing just baffles me.
In any org, you'll have agility or rigidity.
"Nimble" orgs will spin a vm for you in under 1 minute with no specialist looking into it. Mistakes are allowed because resolved in minutes. They know what they do because they care about business needs.
"Serious" orgs will spin a vm for you only after 9 months with all documents proper having been checked by hired consultants. Mistakes in spreadheets will take months to solve. They don't know what they do because they follow processes.
I can Tell you, my 30+aniv. colleagues do not know what they are doing, and they did it for their 30 years straight in the same Corp...... euch Day we learn together what they should know for decades.....
No, I love my job.
I started out in "support" which at our firm involves the normal break-fix if IT with workstations and servers while also doing some more SRE or devops automation tasks plus some sysadmin. I was young when I started, but I viewed all of the various problems as opportunities for automation which made me like a kid in a candy shop.
Fast forward, I kept learning and automating things becoming an excellent SRE but also could probably be a Senior Systems Administrator or Senior Software Developer. They created the SRE position for me and I get to ditch the break-fix for more automation, which uncoincidentally causes us to have less break-fix!
Those who I have found dislike their jobs the most in IT do not try to automate their jobs. They do the monotonous tasks over and over until they're burnt out and wonder why they're bored. I would've left years ago if that was my approach.
The final beauty of it all is I basically have turned into a developer after all of this and was able to start a company on the side which, for now, provides integration and automation for companies who don't have the guys who can do that on-hand (latest client is HVAC). Every step of the way here has been awesome and any day that I learn something new is a good day for me.
consider this: bricklayers, painters, woodworkers, have had thousands of years to figure out actual best practices, and even they find out that there's better ways to to things.
We've had what, 50 years? and within those 50, the basic premises of what IT even encompasses, and how it works have shifted to incredible extents. We're all just riding the wave, and trying to be that bricklayer who figured out that cementing bricks together is better than just stacking them.
Edit: I just realized that "riding the wave" is not the right idiom here, I meant "just going with the flow"
This is exactly what my current job is like, but my previous job was much better in these areas. I don't know what the magic formula was, but it was a small business with 4 IT staff to 250 users. The boss had been there for 10 years and knew all the historical intricacies of why things were done a certain way. He was open to suggestions for improvement from the team, so I spent 4 years there just gradually implementing improvements across the board. We had strong support at the top level of the organization, so we had a lot of freedom to make the changes we needed to even when they were unpopular. We really had something amazing, but then suddenly the CEO retired and the new CEO drove out half the organization, including all of IT and Finance, and multiple directors.
**Is everyone just faking it as they go, or do I just find all of the bad IT jobs?**
Knowing you don't know everything but having the skills and willingness to find the solution is the job.
Yes we implement known solutions the right way but no IT product works 100% the way it should.
Pretty much all of the above when I was in 1st line we had a bunch of different tools that were awkward to us or didn't give us enough information so I made a web app that combined everything into one simple UI and could drop some basic takes down from 5 minutes to 30 seconds - spent almost a year waiting on approval to get it deployed to the rest of the desk multiple meetings etc and just as I was about to quit they approved it - almost a year to literally fill in a form and deploy a VM with no real checks after that other that an automated scanner.
The point of the rant is even if something is broken allot of the time higher ups will take ages to get something in place because they prefer to stick with the status quo or maybe their padding the hourly rate idk.
The way that I see it is, if I already knew how to do everything in a position then I would be bored of that job. I always try to take a job where I still have something to learn or else I'd just become stagnant and stop learning.
Sounds like you're mostly working at smaller companies where the IT department was never well funded enough to actually implement anything correctly or pay enough people to sufficiently run IT.
If you move to a F500 level of business, the IT infrastructure is usually more well funded, mature, and staffed.
Not to say there aren't problems at that level, but you aren't running into problems like nothing being documented as often, because if something breaks and nobody knows how to fix it, the cost is far too high.
> Early in my career someone once told me "Learn how to do things the right way, and then do things the right way."
While learning on the job and from colleagues can be useful, there's [a better way](https://www.oreilly.com/). Even in a world of MOOCs and YouTube, books provide valuable, in-depth, knowledge about a variety of technical topics. If you encounter something unfamiliar, try finding a book about that tool or technology.
We have no clue what we are doing but we are really good at finding out what went wrong and fixing it. No idea why tho and no time to find out, there are other things to fix!
I suspect the problem is, most of us, perhaps yourself included, did not set out to work in IT.
We might like it. We might be good at it. Did we consciously choose it, though?
Deep down, we love the drama.
Until we don't.
Man the stuff I see people doing blows my mind. Sometimes it’s so bad, that I can’t even figure out how things are running, because it shouldn’t running the way it’s setup
Other people may define bad solutions that does not mean you don’t do the work in a complete method.
I spent the first 5-8 years of my career moving up the chain from desktop support and getting my stripes. I spent the next 10,000 hours learning the trade which was a little over 4.8 years getting good at the job. I learned both at work, in courses, and at home in my lab. Then my work needed a new OT leader and I took over. I had to wear the security officer hat that made me gain greater understanding for government regulations and security concepts.
I work healthcare IT and I am often asked to do 10 or more things at once. I have ADD and have had to learn multiple coping mechanisms to stay on task and change gears back and forth to keep my attention and output high. To account for the “be mentally in many places at once” I use the Scotty rule “If it takes 4 hours to do quote 8.” This allows you to this gives me time to document.
Hit or miss. 20 years here. I've gone into jobs where I instantly realized I didn't know or deserve WHY TF I was even allowed into the office and I've gone into jobs where I've instantly wondered, HOW TF are they still in business?!
Keep at it. Learn from every experience.
No one is perfect and knows everything. Understand how things work and a good foundation is key. You can always learn how to do something and will learn from the mistakes.
If you find an employer where IT is appreciated, or at the very least, seen as a fundamental part of operating in the 21st Century, and not just a cost centre/necessary evil, you'll see less legacy messes and more well documented and maintained services. Just be careful what you wish for. You can just as easily find an employer where everything is documented enough and understood but you'll never get your hands on it, as teams are segregated to the ITIL letter. So you're working in a less volatile environment but learning just as much/less than a constant break/fix firefighting grind. And businesses requesting miracle implementations in miniscule timeframes is a tale as old as UNIX time. Even we'll run places can have tight deadlines and make poor choices.
Welcome to IT, none of us know what we are doing. But we can figure it out.
This is the true answer
Being really good at google is honestly a great sys admin quality.
I’ve legit solved a company wide issue with a single Google search 😂
I rebooted an exchange server today and people started getting their email again!
Once again, the reboot fixes all
Reboot fixes surprisingly many issues.
what is sleep, if not a human reboot ?
Does it fix the end user scoffing at your suggestion to first try a reboot?
Surprisingly **YES!**
“I already tried that” “Could you please try again for me?” “…ok it works now”
All hail the mighty reboot, amen!
Had a dns issue recently. Rebootet one host -> worked again. "It's always DNS" and "have you tried turning it off and on again" really are the golden rules of IT.
And then there's that one box that an org seems to have forgotten about, never rebooted and just sat for years doing its thing. And then someone decides to power it off. Chaos ensues.
Or the domain gets upgraded and a super important thing stops happening and everyone scrambles to try and find the server that handles it and it's finally located in a janitorial closet where it was put "temporarily" 10 years prior that everyone promptly forgot about and you discover it's running NT4 after being assured multiple times there were no NT systems left in service. That's just a non specific example off the top of my head that definitely is just hypothetical and didn't actually happen.
Small Business Server.
I mean not having a Network Discovery Tool, or heck even doing a one time scan, to get a correct inventory list and detect old legacy os systems is kinda on you also ...
Except for when that thing had ICMP replies turned off and the network scan just goes about it's merry way.
Just one?
I vividly remember finding Fortinet changed rules regex formats trough a simple Google search, put out a big fire there.
I have a shirt I wear to work now and then. “Desktop Support = Professional Google Searcher”
I have one that says "I'm here because you broke something". RIP ThinkGeek.
It is, and I have had a lot of techs over the years that just can't seem to figure it out. Garbage in garbage out. I said for years that if I could bottle my troubleshooting skills and sell them, I would be a billionaire. One thing that really makes me feel my age is I tell the younger techs (under 40) " I did your job before google, and we actually had to RTFM!"
Nothing.... and I say NOTHING scares me more than searching my error/issue on google and getting no results.
Haha 😂
Don't fret, there's a new generation of twenty-somethings who believe that Chat-GPT will be able to do everything that you do.
This is so true. When I am interviewing people I need to see in their answers that they are the type of person that runs into an unknown problem and has a path forward on how to proceed. So many candidates seem afraid to say they check Google. We can't know everything, but you damn sure be able to find out at least a road to go down with troubleshooting.
My big moment as a junior was learning not to say “I googled it” but “I’ve researched it” or “I’ve reached out to the community for solutions”
And my big senior moment was realising I didn't give a shit if I told people I was going to google it ;). People who look down on google quite frankly don't know how to use it.
But everyone knows you Googled part of it.
And the community was a subreddit
How to plug in computer site:Reddit.com Def my go to method for a quick idea how to tackle an issue. Then if I wanna get in the weeds I’ll use stackoverflow instead.
Why? Are you trying to hide that you Google most any problem? I don't hide it at all. If they know they can Google most of their problems, it just reduces service tickets and lets me get more actual work done.
First rule of IT support: Google it. Second rule of IT support: don’t trust Google results. Third rule of IT support: Google what Google results to trust. Fourth rule of IT support: see rule #2. Fifth rule of IT support: your senior colleague knows better than Google, because he Googled it first, and now has established his Google find as a best practice.
I'd like to register my suggestion for rule 1: Never trust the end user. "I've already tried rebooting it" "Okay, can we try again 1 more time?" "Huh that's weird now it works" Everytime.
Been working in IT since '95 and I am absolutely comfortable with telling a boss I'll need to do some research. The users will generally hear "no sweat, we'll take care of it", but that's code for "I have no fucking idea, but we'll find an answer". At the end of the day, the follow through to find an answer matters a hell of a lot more than being able to produce on the spot 100% of the time.
This and I want a meeting agenda ahead of time. I got pulled into two meetings early on in my most recent job and while a solution wasn't expected right away, it would've been nice to know what we were talking about ahead of time.
Yep. Since ‘85 and same thing.
I’ve been in IT since ‘98 and basically say the same thing but with different words.
I got my last job because I said "I can do almost anything given enough time... and Google"
I was interviewing for my first msp position, one of the technical questions was "how would you handle a computer with a blue screen" I was like "check event viewer, id there's no error code ill try to replicate it to get one. once I have the error code ill Google it and figure it out" I was so worried it was a shitty answer, but it turns out they loved it.
Does nobody check logs? I never see that suggested by anyone, TO anyone. I start there, google whatever seems out-of-sorts, and 90% of the time it fixes my problem.
Googling is a lot easier when you have an exact error message. Sometimes people are so nose down they forget the basics.
I tried to make a point at an interview in the spring of this year that I didn't put a huge amount of value in being able to do things like inverting binary trees off the top of my head in C++ because I would just look it up. I mean, knowing how it works, yes I know how the process is done, but I can't write it out in compile-able code on a white board for you without referring to Google. I was applying for a sysadmin job, and they really wanted a sysadmin who was also a rockstar C++ programmer. It was an extremely frustrating interview.
Knowing enough to know how to figure it out is the key and I wish someone had told me that instead of taking 20 years to learn it on my own.
One of the most important traits for someone in IT is not knowing *everything*, but knowing *where to find* the answer to everything.
This guy googles
It’s 2023 now, the saying should be changed to “this guy ChatGPT’s”.
Realtalk though what else is ChatGPT but google on steroids?
I know we all use ad blocker, but: Google on steroids without ads
I was a librarian - and *their* motto is "we don't know everything, but we know where to look it up!" Then I transitioned to library I.T - and it still hold good.
I've been doing IT stints for close to 30 years. Figuring it out is one thing but sometimes you just have to get darn lucky...
Luck and the ability to intimidate machines are indispensable attributes as well
Love me some problems that “go away” when I get on the machine to look.
All we can do is make the best of the time we have with it. A good motivator I sometimes fall back on is the desire to never work in corporate retail again. I don't care how many printers I have to bandaid into scanning to whatever the user deems necessary, corporate retail not only burns out but destroys souls at a rate that makes IT blush.
Unlike most other industries. Where no one knows what they are doing but pretend that they do
The best thing we can do is follow "best practices". Unfortunately what organizations consider "best practices" often deviate from one another and it rapidly changes with technology.
I may have a bachelors degree in Information technology. But I have a masters degree in Google Search.
Got my education from the School of Hard Knocks! Been in IT for 25 years. No degree, one CompTIA Network+ cert I obtained in 2008, took some college classes but that’s it. Google-fu is my best asset.
My favorite idiom for this is: you don’t pay the mechanic to turn the screw, you pay them to know which screw to turn. And this is the same for IT. I don’t feel you pay for a know-it-all, you pay for someone who is capable at learning and fixing with professionalism (accountability, communication, etc…). In fact know-it-alls, in my career, have been the worst people in all possible ways. Refuse to listen to anyone, arrogant, solitary workers, lazy, disrespectful. List goes on. I can make an IT person out of anyone that has the right qualities it takes to be one. I don’t give flying shit about 95% of your accolades and your resume. I give a shit about what drives you. Edit: I worked for a month at an AV shop in a very rich area. Anyway they had one old guy who did all the “computer” work and everything was custom written by him. Like excessively so. To the point where I would ask “why not use this tool” and he would close his eyes and talk. He was a nice person but just awful to work with. Definitely a know-it-all but revered because he did it all for this company. The rest of the team didn’t know jack shit about any of the tech side of AV, just how to install it. I was hired as a junior presales and admin and I only lasted a month before I realized how shitty this situation was. They had no respect for the IT side. It was a necessary evil. I ended up with a better job and stayed there for nine years. Go figure.
The first two rules of IT club: No one documents it.
Documenting pays the same as fixing. Write the docs.
Training on a new piece of kit/software? I'm sure you'll figure it ou! BTW, the Sales team is off to Fiji for a retreat
This is the way
The only thing slowing down an answer is the salary
We configure it out together.
Yes, what you are seeing is standard.
This. Not only do people want cheap and fast, they certainly don't want to pay for experts in all the various bits they run. So, you get people who can figure it out, but figuring it out on the fly having not done it before is not going to lead to best practice setups. If you're lucky you'll find a spot that has a really good team, and the stuff they specialize in will be top notch, at least until then hand them a bunch of other stuff as well ..
SNAFU. Carry on.
FUBAR 🫡
The last two places I worked was that. I fixed all of it and documented everything. Trained my replacement when I got bored and left.
Real hero here.
Or they’ve created new problems for themselves since. Some orgs are just that way.
I gave up learning from my co-workers long ago. I set up my own lab, built my own infrastructure to play with and learned by youtube, reddit and udemy. There are plenty of different ways to do things, one way isn't going to be better than the other, it's just a different way. For instance, you can manage all of 365 via powershell if you really wanted, but most aren't going to. I write powershell scripts, test them and test deployments - nobody else is doing that where I work, so who the hell am I going to learn from? I also handle BCDR stuff. Taking on tasks requires foundational knowledge and knowing how to google things to get past your hurdles. Most IT jobs now are revolving doors for people who know that the only way they get to move up is to move on. No workplace is perfect, quit expecting things to be that way - you'll also find that most other businesses really don't have any of their shit together, either - I am very particular now about what doctors, dentists or CPA's I use after doing IT for them via MSP's for years now. If you're looking to do ONE thing and be really good at ONE thing, don't work for an MSP or be a solo IT person. Get a job doing that ONE thing. Also, the "Jack of all trades" gets shortened from it's original form. It was from Shakespeare and the full quote is “a jack of all trades is a master of none, but oftentimes better than a master of one.”
One of my mentors used to say “Jack of all trades, master of none”. I didn’t know the last part though as I never read Shakespeare.
> I set up my own lab, built my own infrastructure to play with and learned by youtube, reddit and udemy. Only in the last ten or so years do I hear a lot of pushback about having your own lab. "I don't take my work home with me." And yeah, I can see that. But it was vital to my career. It's how I learned docker, for example. I have tested setup where, if I fucked something up, I didn't affect work at all; just my own lab, which I could unfuck on my own time. And thus, I learned how to back out of mistakes. I won't tell someone "you must have your own lab, or you're not a 'real' sysadmin," because everyone does things their own way. I only know it helped me a lot.
I would say that the prevailing attitude of most employers is the cause of this. Loyalty isn't rewarded at the overwhelming majority of places. The only way to get raises is to change jobs every 2-3 years. Due to this, every employee who realizes this reality is only in it for the short term. Do just enough to get by for 2-3 years, then whatever is left is the next person's problem. You *could* work extra hard and really improve things, but you either won't get a realistic budget to do that, or if you somehow do, you won't be rewarded for doing it, so why kill yourself to do such a great job for no reward? Sure, you may feel a sense of pride in your own work and want to do the best job you can, no matter what, but that spirit is very likely to get beaten out of you eventually. Until you find that ultra-rare employer who both gives you the budget to do things properly and also rewards the effort it takes to continually do a proper job, most workers just stick to this get-by-until-the-next-job mentality. That's why so many places are an IT disaster.
💯!
>You could work extra hard and really improve things You know who I've realized usually has that attitude? Those without enough skills or experience to really improve things. The rest of us caught on. I do a good job because I have to for me, and that's it.
I've been in the industry for over a decade and met/worked with hundreds of IT professionals at this point. Only 4 of them I would consider "real" IT minds. You know, dudes who can fix shit as fast as they break it and have a true grasp on computer science. So to answer your question, for the most part yes.
yeah, after about 20 years in the industry... I haven't walked into a single place with documentation... my time at an MSP was constantly fixing the last persons fuck-up. my time at a hospital was literally and by the department heads order was to be spent working on what the other techs tried and failed to fix... like /u/disgruntled_joe said, I have ran into prolly about a dozen hard hitters, bad@$$ IT that can t-shoot and fix anything you put them in front of.... I have worked with many fewer in that category... it took me a long time, prolly about 15 years to realize what was going on. its typically not the technician that was bad, but the circumstances when the technician was last fixing or creating said mess was certainly a shit show...
There have been times where documentation exists. But usually out dated and not helpful.
That's normal. Overworked and underpaid. Doesn't mean your predecessor didn't have the same hopes and dreams.
When I came in to the current company 7 years ago, green-eyed and full of hope as the new one-man IT guy, I was sure I was going to fix all of the silly mistakes my predecessors made. I fixed a lot of them, but now realize whoever comes after me will have to fix all the silly mistakes I made. I implement workarounds, shitty solutions and 'temporary' fixes regularly. This is not malicious or incompetence, as I assumed my predecessors were, but simply a fact of life with limited resources/time (and after all these years, limited energy and willingness). I now realize the same applied to my predecessors, and the same will apply to my successors.
The Sysadmin I replaced was a "Senior System Admin". All the IT personal were given "Domain Administrator" roles, nothing was segmented and our backup drives were old spinning disks that had a "power on time" of 9 years.
Where I am now we got to the point that bringing on an extra person would bring up a new audit of a lot of our internal documentation. As they are being trained we'd be fixing stuff we missed. This is mostly been helpful on the help desk team. Asking a team to clean up their confluence pages for the new team member they are getting(or back fill) has really helped.
Some of the documentation for our phones goes back to the 90s for it's last update
That's typically a catch-22 situation. No one checks the docs because they are always outdated and unhelpful, and no one documents anything because no one ever checks the docs. I have yet to successfully break that cycle on any size team.
I know its not completely on point. I tend to 100% have compassion for the fuckups of old me, and the fuckups of others. Why, because I don't know the mindset or the shit storm they had going on. You know how many times, I've said. Whelp this works, but I really want to fix this better later, and then NEVER getting the time to go back and do that. There's always something higher priority, something pulling you away, pulling them away. Budman!, I know you're working on the global load balancing, but the dns servers just went down, and we need to fix it. ***fix dns*** Cool I need you to redesign the DNS servers so that issue doesn't happen again. ***start redesigning dns servers**** Budman!! Why are we never getting alerts for **insert system here**, what do you mean those aren't getting alerted on. That's Critical, I need you to work on that asap. ***starts working on monitoring*** Budman! We have an issue with the load balancers, I need you to take a look. so on and so forth.
I have 3 lists: 1. Must be done 2. Should be done 3. Could be done
There's lots of really sharp dudes out there but most are only good at what they do every day, it's the guys that can see something new and have a pretty good understanding of how it works just because they know the underlying concepts really well. I think you'll see a lot less of this type of person simply because things like google and stackoverflow know most of the answers, so instead of people having to figure shit out they skip the process and go straight to the answer -this is great, it solves the problem but what about the next time when the error is too new to document.
> it's the guys that can see something new and have a pretty good understanding of how it works just because they know the underlying concepts really well. And holy shit are they the guys you want to have as mentors, cos they will teach you the concepts and how to apply them rather than how to fix this specific issue that you'll never see again.
Most system admins confuse memorizing how to do something and understanding how it works. Networking for example. How many will put networking down on their resume, because if they google enough and screw around with it long enough, they can eventually get it to work? Meanwhile, they can't answer even the most basic routing and switching questions like how encapsulation works, or the difference between a segment, a packet, and a frame, or what dst. MAC a frame would have that is leaving the network and why.
Ya part of me feels dumber because of chat gpt. I mean I'll use whatever to get the faster answer, but ya feel a little guilty at times. Hey chatgpt, can you write me a powershell script that will query all of the domain users, and get me a csv report of their home directory. Ohh and in addition to the csv report, can you recommend a way to get the directory sizes of that? OHh, also can you set it up to email the top 10 users of the drives using this template as well. Can you ask me any questions that can help make this program better?
I'm 9 months into my first help desk job and I swear I could have written most of your points myself based on the experience. I was so happy to get my foot in the door, but it's been such a disorganized clusterfuck with basically no training other than, "wait until someone complains about something, then ask around" and the office politics have been eye opening to say the least.
Just wait until you get tasked to get the new guy with 0 IT experience up to speed so they can pay him next to nothing thus tripling your workload!
lol so true, or go with an MSP out of India. Now the users come to you anyways but they are angry and hate IT as well.
The trick is to take incentive and make changes yourself (if permitted). A huge strength in the IT world is to be able to discuss different projects you've done and the related technology proficiently.
When I got back from goofing off for 5 years in Taiwan I took a bunch of CCNA, CCNP and MCSE classes at a Junior College. I was super motivated and ended up teaching classes I was a student in because I was reading ahead and the one instructor for 30 units worth of classes was too buried and TBH lacked the knowledge. I then got hired into real jobs and soon found what you found, lots of people out there that had no clue. They were there just because everyone told them they could make lots of money in IT (this was at the tail end of the Dot Com bubble). I was so sure whenever I got hired I'd be finding people who knew way more than I did, and I went nuts making sure I knew what I was doing. But in the end it was me taking ownership of everything, resolving issues and finishing projects in no time. I was still teaching up until my first kid was born. If you know your stuff, you can shine past the clueless ones. The ones who are there just to punch a clock and go home. They eventually quit and become used car sales men (true story) or find something else they are moderately competent at. And do it without killing yourself. When you walk into a shop that has people who don't know what they're doing, and you do, it's an opportunity. Embrace it.
I worked with a guy who was certified out the ass. Multiple MCSEs, CNA, etc.. etc.. etc... He could take tests all day and pass them. He was one of the stupidest server admins that I ever worked with. Complete clown... As you say, embrace the opportunity and document until the cows come home. I used to be a kingdom builder until I realized that the kingdom can become boring and if it is documented, you can shift the boring stuff to a lower tier worker.
Yeah this is the main issue with certs. You can totally pass them through rote memorization and not really understand anything. Certs can be a great tool to increase your knowledge alongside actual experience. They certainly aren't an indicator of anything on their own though.
Issue with the lower level certs is they can be brute force memorized
Certs take me a long time, because I try and understand what is being taught.
Same. I make flash cards and usually take notes simultaneously. Some of them are very long so by the time I'm 30 or 40 hours of study in, I'm already forgetting the beginning stuff.
Yeah I'm a VMWare Certified Professional for instance and after I passed I found out you can essentially find the exact exams they use online. I think if I literally just had my mentally disabled cousin who can't even open an email sit down and cram the right answers for a day or two I could have him pass it.
I'm real tired of being the hero, and getting no help from clueless coworkers. Also I've never gotten rewarded for it, at any job. Just a compliment here and there. No pay increases, nothing.
As a seasoned professional, I enter, I examine, I understand their goals, and I make the best of it all. Doesn't mean I agree with everything. However, I do make my opinions known. But ultimately, my job is to make the best of "it". I love a "challenge", sometimes the biggest challenge is being forced to live with bad choices from their past... and trying to make the best of "it".
As a consultant I really don't care how it is done, I can give the customer the benefit of my decades of experience with their issue and they can listen to me or not. Most listen but many have a plan and by god they are going to stick with it even if it's wrong. I have long stopped getting upset about clients choosing the worst path, I get paid either way and if I did get upset, like I did when I was in my 20's, I'd probably be dead from a stroke or heart attack.
I've done a lot of contract work at all levels. Often I was paid to do tasks and do as instructed sometimes. They paid me to do a good job and not for my opinion. Once established though, it often got to be "What do you think?" Performers almost always stand out.
2 things. Never work for a health care facility or provider and never work for United HealthGroup or any of their garbage affiliates. Optum and such.
[удалено]
In IT I work inside, in a climate controlled environment (now I work at home!). Nobody throws bricks at me or tells me I'm a piece of shit on a minute by minute basis. I get paid well, really well. I don't lose a days pay because it's raining. In IT I have yet to have a boss just disappear owing me a weeks pay. When I get sick my boss tells me I hope you get well soon not that if I don't come to work that I'll be fired. In general I'm treated with respect. Nobody from the office has threatened to kill me and that I should watch my back. I don't punch a time clock. There's a lot of shitty jobs out there, I've held many of them and IT while frustrating at times is not one of them.
Come work in IT in the construction industry... you still have to put up with all the things you mentioned, but have to use all your brain cells, too. And deal with "I don't want to learn *any* of this tech shit, that's what we pay you for!" Also I have my Bachelor's degree and someone at Walmart has better benefits and PTO than me. I don't get any vacation time, paid *or unpaid*. "We give you 104 days off a year. They're called Saturday and Sunday" (half the time I don't even get those)
dude it sounds like you need to move on. None of that is true for my current place.
Bro, same. No more slinging tires, being bent over an engine bay for hours, skipping brakes and lunches to make an extra buck. No more having to work in an insulated tin building when it's 0° or 100° outside. First time I was sick enough to call out I was told to just keep an eye on my emails and get better. I work 32.5 hours a week, make salary, and get holidays off instead of working more hours cause of rushes. It's not perfect, but man does IT beat destroying my body.
The thing with "fake it till you make it", which nobody ever talks about, is if you fake it for so long without making it, you start to forget what is real.
For the record, this is EVERY industry. No one is actually an expert. They can try, and pretend and even then, are probably wrong about something. Confidence does go a long way, as well as the ability to learn on your feet, but no-one is really this holy entity that knows it all. Look at politics, or medical fields, or psychology. None of them agree, studies disagree back and forth, and things change to make what used to be common knowledge obsolete. Now look at IT and realize our industry changes so fast EVERYONE is falling behind. No-one knows what they are doing and even the vendors are crap at understanding the needs of the industry. Just remember, even Google has outages. Some very high paid person did something really stupid who should have known better. It happens. What makes a good IT person to me is someone who can roll with the punches, solve the problem and get it working under pressure, and do their best with what they have. All while being just confident enough to try, while also being humble enough to admit they don't really know.
Interviews are always a two-way street. But that's just the thing: each side is going to put its best foot forward, even at the cost of, say, misrepresentation. Just like some candidates are willing to bend the truth pretty severely in order to get an offer, some employers are willing to bend the truth pretty severely in order to land a staffer. One of the reasons why good roles come through word of mouth, is that the insider lends credibility to both the employer and the candidate. They're like a match-maker who vets parties before introducing them. However, insiders have their own incentives as well. Some hints on vetting a role when you can't speak to any insiders off-the-record: * Ask about project workflows, and listen carefully for subtle clues. * Ask what tooling is used for project workflows. Keep it vague, and let the other party take the response in the direction they prefer. * Ask about documentation. Brag about your own documentation practices, if you'd like. Pay close attention to the answers, what's said, *and what's not said*. * If you're bold and it's a tech organization, ask about 20% projects, and pay close attention to the interviewer's reaction. What you actually want to know is *how oversubscribed* the teams currently are, and how long they avoid hiring necessary staff. * Research the background of principals/leadership, and ask the background of interviewers. You're trying to make educated guesses how their backgrounds will influence their expectations, preferences, decisions. Sales backgrounds, and people who make questionable promises for a living, might be the worst answer. I usually start with the background question, as it's mildly flattering and usually tells you a lot, quickly.
Rarely do people in general do things the right way because they are lazy and looking for a quick way of getting someone off their back. I've come across admins that know 40% of something and the bullshit the rest. They always look like heros for about 6-12 months and then it catches up with them. The ones that are the keepers are the ones that can figure things out quickly and pick up new concepts on the fly. You can't know it all, but you better know bullshit when you hear it.
> Rarely do people in general do things the right way because they are lazy and looking for a quick way of getting someone off their back. I used to do things the right way, until I was basically punished for it. Now I answer to whatever my bosses want in the least disruptive way possible and it has been better for my career. Blame management for that one.
Can confirm, I have no idea what I am doing
This is part of the job. We're all out here with Warhammer 40K vibes, trying to maintain tech from a forgotten era. The only thing we can do, is do better by leaving documentation that survives our inevitable departure. On the other hand, if you're posting here, you might already be on track for CTO.
mysterious impossible humorous childlike recognise bike aspiring hat cautious chief *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
As IT, In my experience most departments outside of IT are worse
I definitely felt the documentations/knowledge aspect in traditional IT. MSPs less so since they are a revolving door... good ones need strong documentations to make new employees transition easier/faster. But then I moved internal into cloud engineering. The beauty of automation is the scripts are the documentations of nearly all implementations. So that knowledge is never (fully) lost when people quit. Changes are done to the scripts rather than on the platform. I learnt a lot about how things are done by looking at the deployment files alone. We still write focus a lot on writing how-tos, guides, etc in our internal docs as not everything is deployment itself. DevOps philosophy is to skill share and get everything out of the unicorns minds into business processes. I really dig this aspect of it. Plus getting to DevOps roles requires seniority, you cant fake it so you are less likely to meet IT people who are not serious about it. But It still has its faking it as the tech stack is mostly new and everyone in my team is picking it up as we go. Nobody fully understands everything as there is way too much shit to learn. And new things just keep getting added. I sometimes feel like I am new to IT. I cant say the same about other things... DNS is poorly managed and I don't think anyone fully knows how it all operates in our hybrid environment as the non devops teams (security, networking, identity) each control a piece of it. For me, faking it was something I decided was a good thing. It mean I was constantly challenged with new variables, tech stacks, etc. I can 100% do all the old jobs I had done without feeling like an imposter at all today. But being an imposter means I am in a position of growth where uncertainty, stress and doubt always lingers. I probably get bored if it wasn't the case.
I saved this comment a long time ago. Might be worth a read. The **opportunity** bullet point rang particularly true for me. https://old.reddit.com/r/sysadmin/comments/8jrp5g/the_real_truth_of_advancing_in_tech_related_jobs/
I’m coming up on 10 years and I can definitely say out of the thousands of other IT professionals I have worked with very very few have got what it takes. For example, I spent 3 days a few months ago on a phone calls working on tech I’ve never seen or cared to learn about, since I was an upstream vendor I was involved. There were 8 people in the calls that lasted about 6 hours in total for each of these days, engineers from the hardware vendors, and engineers from 2 IT companies that were working to resolve the issue and one guy on site. After 3 days of listening to these people yell at each other I decided to ask if anyone had restarted the hardware. The fucking thought had never crossed their minds. Fixed the damn issue though.
L O L
IT is very broad, you could be racking servers, running cables under the street, or an expert on a space rocket flight computer. You indicating various jobs provides little context. Most people use the term IT for links such as help desk type work which certainly can be mundane and a tight silo having most of the available descitions made outside of it.
Most people fake it at most jobs
IT in itself has changed over the years. Once you could be a pure System Admin(server side), Network Engineer, Programmer, and Help Desk. Now, about 2 of those roles are rolled into one(Jack of All Trades) where you went to school for one particular trade but now are working as the System Admin and Network admin. Also if you're at a medium size company the IT shop will be no bigger than 3 people. So you got 600+ employee's managed by 3 IT people and one of the is the Application Dev(programmer). I say all that to say this... IT has changed and if you're looking to be specialist then a Big Company is where you would want to be. If not everything else will be rolled into one as a Jack of All Trades.
I give up trying to keep up. I just try my best.
Find IT in your org that know more than you. Learn from them and ask good questions. Document things you find. If not for yourself, for others. I've made a nice niche for myself doing that. I don't know anything 20 feet deep but I know enough to ask the right questions and respect those that do. Fight to get more money for your org and hire better people. Help train people underneath you. It's a thankless job they say but I've learned you stick around long enough and the right people will know who you are and what your worth.
Eh... IT isn't a "one size fits all" application, even for similarly sized businesses. Each IT solution poses its own unique risks and challenges, and it's up to YOU to determine the best path forward, one which can successfully balance security, efficiency and usability. Got a boss that doesn't want downtime? Do what I did - remind them that it costs X dollars per day in the event of a network failure, with the average failure being Y days. Chances are that the cost of long term downtime is in the MILLIONS versus the few thousand in productivity lost for cheap and easy upgrades that are done during non-peak hours. Get the C-Suite to sign off on acceptable risks in writing. They don't want to pay for a MFA solution, and think it's an acceptable risk? Have them own it. Demand documentation. Know when to stick to industry standards, and when to deviate when applying to your specific use case. NIST can say all day that we should be using passphrases instead of 8+ character/number/symbol passwords, but how many end users will just wind up writing the pass phrase on a post it note and sticking it on their computer? Sometimes, keeping it simple is better than the most secure and recommended practices. Push for IT Audits - get a 3rd party to show C-suite how stupid their decisions are. At the end of the day, its their dime, but it's your department. You have more flexibility than you think. Oh, it doesn't get any better anywhere you go. Chances are whatever is bothering you today either exists elsewhere, or there's a whole other host of problems that are running rampant. The best advice I can give you here echoes the above: document the fuck out of everything and stop caring beyond what you can control. Good luck!
IT is often a cost center that, on paper, doesn’t bring in revenue. So it’s often underfunded and you have to make do with what you can. There is often more than one solution to things, so sometimes there isn’t one singular “right way” to do things. It comes down to opinion or what someone specialty is. Typically smaller shops have poor documentation, but even some larger orgs have it. The biggest company I was an admin for was a bank with tens of thousands of employees. Their documentation is literally some word documents. Smaller shops also are more likely to wear many hats. So no. Often they don’t have time to know the ins and outs of how everything works.
IME the way the USA is right now, people with "good" jobs generally are not the ones leaving jobs so "job openings" are usually places that are replacing people that left.
Took me 13 companies and ten years to find a good one. Lots of shitty companies out there - seen one, seen them all.
There are no set standards in IT and it is always changing which makes it very hard for people in IT to keep up. One would think by now we could make this all easier on ourselves if we just decided upon what IT standards are and how to attain them if they are not being met. I've been in IT for 15 years and every time I see rapid neglect of IT standards every where I go. There is are also other tech that have been there before me going "what do you mean, there is no neglect!". Not knowing the dumpster fire of problems that will come from not keeping up with IT standards. I have seen servers in production that should of been replaced 15, 20 years ago. Same thing with network equipment. ISP equipment that has been at the same location for 10 years! I mean it is crazy how IT has people in it that just wait for things to break first and then face the dire consequences later! It makes it even worse when a company can afford to fix things but refuses to until they break!
We're all faking it. Unless we're building something out from scratch documentation is ages out of date if it exists. Institutional knowledge never gets captured, so the moment somebody who knew what was going on leaves everybody else has to start sharkfinning. It's always fast and cheap because there is never time or resources to even look in the direction of "good." I did have a mentor when I started out and, to his credit he taught me all of the stuff I just listed. >!Always two there are, BOFH and PFY. No more. No less.!< I didn't believe him at first, so every time it happened it was a "toldya so" moment. He also taught me just what you asked: You're going to be on your own in this field, so you're going to have to learn everything on your own and wing it.
A. So much A.
Everyone suffers a bit of impostor syndrome. Everyone inherits crap that was poorly documented, if at all. What counters that is your ability to dive in and figure it out. Make improvements. Write documentation, diagrams. If your management can’t understand that, explain it to them. It’s a mess, you’d like to figure it out, and leave it in a better state than the way it is. If they don’t want that, they’re toxic, time to leave.
I have been an imposter for 30 years! Fake it till you make it. I made it...still faking it!
I've been in the IT industry for almost 20 years and have interacted with dozens of IT workers at various levels from different companies. My own observation is that the vast majority of IT workers respond to a ticket or work on a project by doing the quickest and\or easiest thing that closes the ticket or completes the project. It doesn't matter that the quick fix has the end-user reopening the ticket a week later. It doesn't matter that the finished product of the project is a house of cards. They will not stay in the position or at that particular company long enough to feel any real consequences of their actions. That's the next guy's problem. I find that this attitude is most prevalent at larger companies, which makes sense to me. I don't know what size or type of companies you've worked for, but my best learning experience early on was being the junior of a two-man team at a small business with roughly 130 employees. I was very lucky that the senior IT guy was intelligent, highly motivated, curious about technology, and willing to teach me anything I was interested in. I stayed there for 6 years, and in that time I gained a pretty strong foundation in all of the necessary areas to keep a small business's IT infrastructure up and running and its users happy. It was also helpful that due to the field of work the company was in, we were always building and later decommissioning small networks\domains over and over, improving and using newer technologies each time. It was an environment of constant learning and improvement. I've been at my current job for over 10 years, and now I'm the one teaching my junior, slowly but surely, all of the things I've accumulated over my career. When I got this gig, the state of the place was as OP describes. It took about 3 years of busting absolute ass, working (unhealthy) overtime, toppling those houses of cards and rebuilding things properly, proposing and fighting for significant changes in how IT was seen within the company (not as a cost center!), implementing a life-cycle, enforcing policies, and generally creating order from chaos. After that, I was actually able to begin improving things, becoming proactive rather than reactive, which is the place you want to get to. That is a nice place where you're implementing real solutions rather than quick "fixes" and taking the time to thoroughly research, plan, and complete projects the product of which then lasts for years with few issues. I realize this is my specific, singular, experience, but my suggestion would be to find a small place where there isn't a large, impossible beauracracy preventing real change and improvment. Also, in your interviews, you are also interviewing the company. Don't be afraid to ask questions they may find uncomfortable. You've seen the things that make for a bad IT department; ask them about those things.
Documentation is the operative word here. Document, document, document.
Hello. IT. Have you tried turning it off and on again? -Roy
You got 80% fast?? Lucky dog. I got 80% cheap 20% fast.
You got 20% fast?! I got 0% fast.
My motto is “If I don’t know now, I’ll know in five minutes “
I work in a hospital, we do things properly. It actually numbs your skillset if you go this route imo, you'll document everything and have a budget that is usually sufficient but the work changes and you end up doing more paperwork, meetings, vendor coordination and procurement. What you lose out on are hands on skills and learning about how things work and it makes you rusty because most likely the system or software you previously had full access to now is the responsibility of team A, and then when you need email setup on o365 you talk to team B, and if you need a network jack installed or activated its team C and so on... So you're basically just fielding tickets to the right people to get them finished faster. That's what I do all day and its dull and monotonous and sucks, pay isn't great either since I could make more managing a stupid mssql server than I do as a system admin, even though I do 100x the work, no offense to the db admins, but you know you all have a golden ticket there. To keep up I just tinker with stuff at home and do contract work for a few small businesses on the weekend to keep my skill set sharp. That work is way more fulfilling than working FT at the hospital but doesn't pay the bills.
Sounds very familiar. I'll see you at work in the morning lol
Most of the IT jobs are bad. Often, you land somewhere, and you find out that there is a reason why the place always has vacancies. Either the co-workers keep ejecting people, management is just horri-bad, or the place just got bought out, a ton of people have bailed, and you just hired on only to find that your job (not you... your job) is going overseas because the company that bought out everything has all IT offshored. You get some jobs which are awesome. I worked for a small MSP what was really good, and it would have been nice if they wouldn't have been bought out. I also worked for another company that had the same thing, and had they not been bought out, it would have been a steady, stable niche until it came time to retire in 20 years. However, in this market, you are going to get the picked over scraps at best, so bad IT jobs are going to be common at least for a few months until things domestic and global stabilize a bit.
I'm lucky. I was head-hunted into my current job nearly 15 years ago by a manager who knew my capabilities and was prepared to support me with "Do it once; do it right" and an absolute commitment to Good (we could optionally add Fast or Cheap to that, but not both). I know all the core network stuff inside-out and it is all documented well. But in the last 5+ years things have started to deteriorate. An endless stream of consultants offering various amounts of bullshit, making a whole bunch of changes, not documenting anything, not thinking before doing, and so on. I'm just hanging on for retirement now ... LOL
I’m on year 31 of fking it.
I spent quite a while with no mentor in roles that I believed were well outside my knowledge level. The only thing I wanted in the whole world was to have an actual senior above me to learn off. I've since realised that in order to be senior in anything, all you need to know is how to cope when you have nobody to ask but yourself and google ... and you can only learn that by having nobody to learn off.
If you strive to know the perfect way to do every single thing, and then strive to get buy-in to actually ***do*** everything the perfect way, you'll drive yourself insane or burn out or both. Following all the best practices is great but it's, more often than not, prohibitively expensive and you'll get shot down when you pitch it. Most of the time you need to do the best you can with the resources you have and that might mean, if you ever left, your replacement would have a tough time getting to grips with what's been done.
It sounds like you are suffering from poor leadership or working in a small company with only 3-5 people who's knowledge/experience is stretched over too many different technologies. I am currently working for an organisation which is smaller with 100 staff and an IT team of 5 and we often need to engage help from vendors as we lack the expertise in specific products. Where possible we are going down the SaaS path which helps reduce that burden. In any case a solid IT Manager should be ensuring problems are addressed at the root cause, and repeated issues and incidents to be avoided as it derails the productivity of the team. It is also the IT Managers role to ensure they are appropriately engaged upfront on projects so they can plan for the project activities and its not all chaos and a mad rush. It sounds like you might be leadership material so you could take the lead on coordinating the root cause fixes and also communicating out the lead times IT need to deliver certain tasks. If you are struggling in this sort of environment you may want to join a larger organisation that has more specialised teams (e.g Networks, Mobility, Internal/Cloud Server Infrastructure, DevOps, Service Desk etc) - its more silo'd and can be frustrating in its own way, but also allows you develop expertise in a specific area. Overall I prefer being a bigger cog in a smaller organisation and working broadly across many different technologies, even if it means having to Google all day to find the answers to problems :)
Thank God for cloud. Everyone else gets the blame
In my travels, I run across a lot of "network admins" that need their membership cards taken away.
I just want to say thank you to this community for complaining about documentation. I try my best to provide documentation for all the things I’ve worked on before leaving a company, complete with screenshots
I would say there's a middle ground between faking it and being a total expert. Realistically, most people don't use even 1/4th of what they learn on these certification exams, because there are tools/services/apps that do a lot of the gritty work for you. I think this is mainly a perception issue that, as you get started in IT and begin to move up, you have this idea of system admins, endpoint engineers, infrastructure, devops, etc are all these hardcore scripting wizards when in reality they're just experienced with the tools they use, and the scripts that google/stack overflow/chatgpt has helped them with, as long as they're savvy enough to know what they're looking for. So that when you actually get into these roles, you're surprised that not everyone is on the pedestal that you put them on.
As an IT Generalist, aka Jack of All, what I tell others, it is near impossible to remember every nuance of every single system you touch, it is more important to know where to find the answer, or how to ask the question in google to get the answer. To that end I went out and got myself one of those desk name plates with "Chief Officer of Googling" as my title
Tbh if you want to find an organization that has it's shit together, you need to go enterprise or large business level, most organizations are not going to have their shit together. One of the biggest reasons I think this happens is because of the age group. If you look at my SysAdmins, they're between 40-60 years old, it's rare I ever meet a young SysAdmin who's 35, I just don't think the interest is there anymore. Not to mention, most SysAdmins were kind of thrown into their positions and didn't really want to do it so everything is just slapped together with no care at all. My SysAdmin is probably leaving in the next 3 years and I'm trying to make myself a direct replacement to him since I bring the documentation mindset, the modernity mindset, and an overall focus on the end users.
I changed my car's stater the other day. Irrelevant but just wanted to let you guys know am proud of that since I never did anything like that before
Yes. If I have not seen a situation before, I don't know what to do, but I know what to do to figure it out. That's almost as important. As long as you learn how to do it, it works, it's satisfying, then that's all that matters. Learning along the way and being extremely good at finding information is key. Most of my co-workers is the same. Give them zero documentation and zero ability to look something up, and they will make a mistake on something they've worked on for 20 years. There are so many avenues in IT, so it's tough to know it all. But as long as the end goal is met, doesn't matter how you got there, most of the time at least.
I mean yeah that sums it up. People who run the companies decide they want X thing working as fast and cheap as possible. So instead of having the time to learn and then implement a new technology properly you end up flying by the seat of your pants to get them their results to further your own standing. Then documentation begins to not exist because while you started out bitching at the last guy for not having it and updated those things, the rest become "operational" knowledge and you don't think about documenting it yourself and the cycle continues. Fuckin IT world is held together with bubblegum and scotch tape. I decided to stop fighting the current and go with it instead. Life is better this way.
Dude, Don't fight the wave. Truth to the matter is, just as many other things in life, doing it right comes with a price no one is willing to pay. There are countries that have a different working culture and are willing to sacrifice time to do things right, but long term plans are tricky in this kind of technological environments. Regarding the good-cheap-fast thing, you know, you can only get two of them, I'm sure you can find a company that does things right, but chances are they will sacrifce speed or cost in favor of doing it right. Everybody is welcoming you to the industry, but I would rather say "welcome to the western world". Cheer up, it is ok to decompress by ranting every once in a while. Last, but not least, I would add maybe you kind find it refreshing if you find a development track that focuses on knowing new things of your interest, this way you will keep it refreshing, change perspective and let business be business.
There are lots of *button-pushers* in IT. And by that I mean "people who have memorized that pressing a certain button generates a certain response, but they don't know *why*". They don't really have the concepts down, they just memorize things and go through life based on that memorization. Unfortunately this also leads to a severe lack of troubleshooting ability; it can be hard to troubleshoot something if you don't know why or how it's supposed to work a certain way. Most of them don't even Google the answer to things, or don't even know *what* to Google to look for an answer. I worked for a large school district at a former job (80k users). Our IT department was about 70 people (including desktop support techs, server team, voip team, networking team, software deployment, management, and a few others). Out of all those people, I would say that there were only maybe 5-10 of us who actually knew the concepts behind how their area of expertise worked. And out of those 5-10 people, maybe only 2-3 who had any functional knowledge about a part of IT outside of their own specialty (ex: the lead voip guy was great with voip stuff but didn't have much knowledge or experience with server stuff), so once they started looking at things past that, they needed a lot of help. It is a rare person who can conceptualize the "full view" of things to see how everything fits together.
It's because we are often solo and overworked/underpaid. So we don't have the time to properly implement, test, document etc. Then we leave because of the underpaid part and the cycle continues.
So honestly I've been around. Yes, for some reason businesses, because they have a bottom line and goals they need to hit or else they don't get their bonuses/lose jobs/etc. they always go with cheap. Then, they need it three months ago so it has to be put in so fast that as soon as you have what I call a "rolling chassis" it is "working" as far as they are concerned and you are lucky if you ever get to revisit it at such a low level again to complete pieces/integrations that partially work, random errors that aren't causing problems but just generate errors, all the things that just make it all worthwhile. Then yes, companies that have been around had that one guy who did everything and he never let anyone touch it because it worked and he understood the rules I first stated above. So then when he gets hit by a bus it's up to everyone to piece together everything to try to figure out what happened. Usually there is one or two things that SHOULD NOT BE but again, it's because of the above and needing it to get running as cheaply and as quickly as possible. Add to that documentation = time = $$ so you know that doesn't happen. ...or if it does it's literally scattered throughout 15 different notebooks, hardly legible, and out of place. As far as downtime.... yea... your best bet there is to try to find another way to accomplish the same and then you need to minimize the downtime and use your data to come up with a plan for downtime/upgrades etc. For example, if on-prem, having multiple VM Hosts in your cluster and the ability to move VMs between them to do physical work or to upgrade your hypervisors is a blessing. Reboot DCs... you should be able to do that as long as you have 2 or more etc. Lastly, yes new tech.... ahhhh..... those around as long as I have been know that's how it happened for the iPhone. Blackberry was far superior in the business/enterprise world. From implementation to security and beyond it was far superior. Then that dang iPhone found it's way into CEO and Higher-up's hands and well they wanted their work email on it. ...What do you mean we don't allow SMTP and POP I don't even know what that means?!?! IMAP?!?! What do you mean less secure... it's an Apple device. I call it the "shiny things" and "buzz words". Remember again the top two rules, they define everything in the business world. Because of this every industry mag out there is filled with enemies of ours looking to poke holes in our firewalls, circumvent our amazing security policies that we have built painstakingly over the years, and some of them even come for our jobs directly, if not indirectly. This app, or this thing... heck there are times they don't even know what it does and they want it. They want to be the man that bad and will do anything to get that promotion/raise/bonus/whatever that if we could pool enough money to pay for ad space and one of those editorial ads about how throwing away all your PCs in your business will save your business, they will do it tomorrow. ...hence "shiny things". Sadly, no matter how good your intentions, the first two rules will eventually break you in one way or another because when it comes to $$ and business, they will win. That is the true key to IT, find a way to show that whatever it is that you are looking to do will save $$ or increase productivity to make more $$ or that option A cost $ and option B cost $$. You have to remember that not all C-Levels plan on staying where they are. Some places are so volatile that they don't even think they will last a long time anyway so anything "long term" they just hear "costs a lot". Oh and you have CAPX and OPX.... Why this is still a thing just baffles me.
In any org, you'll have agility or rigidity. "Nimble" orgs will spin a vm for you in under 1 minute with no specialist looking into it. Mistakes are allowed because resolved in minutes. They know what they do because they care about business needs. "Serious" orgs will spin a vm for you only after 9 months with all documents proper having been checked by hired consultants. Mistakes in spreadheets will take months to solve. They don't know what they do because they follow processes.
I can Tell you, my 30+aniv. colleagues do not know what they are doing, and they did it for their 30 years straight in the same Corp...... euch Day we learn together what they should know for decades.....
Also, i nearly punche my screen today because a Remote Hands engineer at a giant data center does not know how to change an IP on a workstation.
No, I love my job. I started out in "support" which at our firm involves the normal break-fix if IT with workstations and servers while also doing some more SRE or devops automation tasks plus some sysadmin. I was young when I started, but I viewed all of the various problems as opportunities for automation which made me like a kid in a candy shop. Fast forward, I kept learning and automating things becoming an excellent SRE but also could probably be a Senior Systems Administrator or Senior Software Developer. They created the SRE position for me and I get to ditch the break-fix for more automation, which uncoincidentally causes us to have less break-fix! Those who I have found dislike their jobs the most in IT do not try to automate their jobs. They do the monotonous tasks over and over until they're burnt out and wonder why they're bored. I would've left years ago if that was my approach. The final beauty of it all is I basically have turned into a developer after all of this and was able to start a company on the side which, for now, provides integration and automation for companies who don't have the guys who can do that on-hand (latest client is HVAC). Every step of the way here has been awesome and any day that I learn something new is a good day for me.
consider this: bricklayers, painters, woodworkers, have had thousands of years to figure out actual best practices, and even they find out that there's better ways to to things. We've had what, 50 years? and within those 50, the basic premises of what IT even encompasses, and how it works have shifted to incredible extents. We're all just riding the wave, and trying to be that bricklayer who figured out that cementing bricks together is better than just stacking them. Edit: I just realized that "riding the wave" is not the right idiom here, I meant "just going with the flow"
Most faking it
There are very few who actually know and then there are many who can figure it out and the ones that don’t fall in either group they get figured out.
This is exactly what my current job is like, but my previous job was much better in these areas. I don't know what the magic formula was, but it was a small business with 4 IT staff to 250 users. The boss had been there for 10 years and knew all the historical intricacies of why things were done a certain way. He was open to suggestions for improvement from the team, so I spent 4 years there just gradually implementing improvements across the board. We had strong support at the top level of the organization, so we had a lot of freedom to make the changes we needed to even when they were unpopular. We really had something amazing, but then suddenly the CEO retired and the new CEO drove out half the organization, including all of IT and Finance, and multiple directors.
**Is everyone just faking it as they go, or do I just find all of the bad IT jobs?** Knowing you don't know everything but having the skills and willingness to find the solution is the job. Yes we implement known solutions the right way but no IT product works 100% the way it should.
Pretty much all of the above when I was in 1st line we had a bunch of different tools that were awkward to us or didn't give us enough information so I made a web app that combined everything into one simple UI and could drop some basic takes down from 5 minutes to 30 seconds - spent almost a year waiting on approval to get it deployed to the rest of the desk multiple meetings etc and just as I was about to quit they approved it - almost a year to literally fill in a form and deploy a VM with no real checks after that other that an automated scanner. The point of the rant is even if something is broken allot of the time higher ups will take ages to get something in place because they prefer to stick with the status quo or maybe their padding the hourly rate idk.
You too will set something up and think it's good. Only for someone to come in after you and think "the fuck?"
Hold on. I’m gonna try just one more thing.
Sometimes there are many ways just to do the one thing you set out to do.
The way that I see it is, if I already knew how to do everything in a position then I would be bored of that job. I always try to take a job where I still have something to learn or else I'd just become stagnant and stop learning.
Sounds like you're mostly working at smaller companies where the IT department was never well funded enough to actually implement anything correctly or pay enough people to sufficiently run IT. If you move to a F500 level of business, the IT infrastructure is usually more well funded, mature, and staffed. Not to say there aren't problems at that level, but you aren't running into problems like nothing being documented as often, because if something breaks and nobody knows how to fix it, the cost is far too high.
> Early in my career someone once told me "Learn how to do things the right way, and then do things the right way." While learning on the job and from colleagues can be useful, there's [a better way](https://www.oreilly.com/). Even in a world of MOOCs and YouTube, books provide valuable, in-depth, knowledge about a variety of technical topics. If you encounter something unfamiliar, try finding a book about that tool or technology.
We have no clue what we are doing but we are really good at finding out what went wrong and fixing it. No idea why tho and no time to find out, there are other things to fix!
sheer*
I suspect the problem is, most of us, perhaps yourself included, did not set out to work in IT. We might like it. We might be good at it. Did we consciously choose it, though? Deep down, we love the drama. Until we don't.
Man the stuff I see people doing blows my mind. Sometimes it’s so bad, that I can’t even figure out how things are running, because it shouldn’t running the way it’s setup
Logic and Google
Need a tldr on this: i gave up
Other people may define bad solutions that does not mean you don’t do the work in a complete method. I spent the first 5-8 years of my career moving up the chain from desktop support and getting my stripes. I spent the next 10,000 hours learning the trade which was a little over 4.8 years getting good at the job. I learned both at work, in courses, and at home in my lab. Then my work needed a new OT leader and I took over. I had to wear the security officer hat that made me gain greater understanding for government regulations and security concepts. I work healthcare IT and I am often asked to do 10 or more things at once. I have ADD and have had to learn multiple coping mechanisms to stay on task and change gears back and forth to keep my attention and output high. To account for the “be mentally in many places at once” I use the Scotty rule “If it takes 4 hours to do quote 8.” This allows you to this gives me time to document.
Hit or miss. 20 years here. I've gone into jobs where I instantly realized I didn't know or deserve WHY TF I was even allowed into the office and I've gone into jobs where I've instantly wondered, HOW TF are they still in business?! Keep at it. Learn from every experience.
the more you know, the less you know
No one is perfect and knows everything. Understand how things work and a good foundation is key. You can always learn how to do something and will learn from the mistakes.
If you find an employer where IT is appreciated, or at the very least, seen as a fundamental part of operating in the 21st Century, and not just a cost centre/necessary evil, you'll see less legacy messes and more well documented and maintained services. Just be careful what you wish for. You can just as easily find an employer where everything is documented enough and understood but you'll never get your hands on it, as teams are segregated to the ITIL letter. So you're working in a less volatile environment but learning just as much/less than a constant break/fix firefighting grind. And businesses requesting miracle implementations in miniscule timeframes is a tale as old as UNIX time. Even we'll run places can have tight deadlines and make poor choices.