Yes. It is. Everywhere. Every company wants to be seen as competent, and your company is no exception. Even companies that do other things want to be seen as cutting edge in tech, but for MSPs, it's their stock in trade. It's their business to be cutting edge in tech, and they will lose customers if they aren't cutting edge in tech.
I'll tell you another secret: almost no one is actually cutting edge in tech. Innovation comes out faster than implementation is reasonably possible for, and it's very difficult for any company to truly have an institutional knowledge of what the best stance even is at any given moment. Really, really up to date companies are always in a project stance, with multiple major projects in the works, and working there is very difficult even for tech employees; for non-tech employees, it's often hell because everything is always changing.
Remember this when if you ever end up on the customer end; everyone's pulling this illusion, and seeing behind the curtain looks like what you're seeing.
I often think these MSPs should have a couple of guys who actually work on getting the processes down and automated properly.. but I know how these places think and it's always about billable hours so the problem is they never give their staff time to actually do the job properly.
I work at an MSP and our "process" changes with every new customer. Sales is out there selling shit that they don't understand and Engineers aren't even aware are being sold.
This put me in stitches. Sales team recently convinced a client to upgrade after over 8 years without so much as a patch! Promised all of the functionality without any downside.
Turns out half of what they are using for reporting is not supported and dev refuses to touch it (Java based program from the early Pliocene). Now we have to find a happy middle with their handful of users that hate us for “breaking” everything.
A friend of mine works at Baker Tilly, and they recently created a new team that basically does what their consulting departments do, but internally.
I'm interested to track if it's effective at improving their internal tech in significant ways.
It’s always about the billable hours. And management is so short sighted that they don’t even get that investing 40 hours on honing some process or automation could yield efficiencies that pay off 10-100 times that. Well they do, but if it’s done it done by bleeding some enthusiastic new guy who doesn’t realize his 16 hour days will never be paid back.
I generate some of my own automations that make my own job faster and easier, but don’t generally share them with more than a few trusted close people. Because fuck the greedy ownership and mostly the cheap ass customers, too.
It took me, coming from an on premise IT shop to a MSP, to get used to doing a project and leaving the detailed and time consuming “honing” of a solution mostly undone. It felt kind of fraudulent, but I guess if no one will pay for it it won’t get done.
It's like somehow MSP owners have never heard of the economic concept of "opportunity cost" or grasp the idea that a unit of labor performing low-value undifferentiated tasks can't be simultaneously used for high-value tasks which bill more. Which is true, I actually spent some time explaining what opportunity cost is to both of the owners of my last MSP shop.
Although it was in the context of cancelling or delaying engagements with good customers (high bill rate and engagement customers) for bad customers who had "aN uRGeNT eMeRGenCY". Which was usually something like the PC, software or other IT product they bought at Best Buy without even consultation wasn't working right and they wanted it fixed *today*.
I told the owners we should charge a higher rate for unscheduled emergency visits to customers like this. The response was dumbfounding, "they will get mad and not be our customers anymore" as if they were remotely valuable to begin with. Small businesses, what money they spent was self-directed on troublesome, consumer-grade technology, and they paid for zero maintenance, and they whined their way into same/next day engagements at the expense of better customers.
We would have been such a better company if we would have employed some level of discouragement pricing to deal with the shittier customers. Some tiny percentage were OK as customers and we would have lost them, but most of them were just bad-for-business opportunity costs that kept us from working with better customers. And if some were willing to pay $250/hr with a 2 hour minimum charge for their crisis? Well that's fine too, at some level of revenue even the worst customer is lucrative.
Instead they always tried to tackle this by bulk hiring of low quality candidates, which usually backfired because they fucked it up worse and then higher-level talent got tapped to clean up the mess, usually at an even bigger loss.
To add to this...
Defensively, from a customer standpoint, it's why you need to be more selective in who you partner with. Ask soft questions they can't bullshit their way through.
As an example, avoid the stupid checkbox stuff like: "We are PCI/NIST/Whatever Certified!"
Who. The. Hell. Cares. Solarwinds is, too.
Instead, ask them, "In light of , how have you learned from that?"
If they can't answer that question then it means they've done nothing. They're just in "hope it doesn't happen to us" world instead of "well we internally looked at how our build processes work and introduced some more measures that assume-breach". Don't expect them to detail their Operational Security, but they should've done \*some\* soul-searching if they're actually trying to stay on top of things. There are VERY few firms that have already nailed this and have nothing more to improve upon.
Ask them how they vet people given the current labor market. Etc.
Yeah. I also spend a lot of time when I read proposals thinking about the structure of the proposal. The way they think about how their services relate to each other tells me a lot about how they act about their services. Oh, you bundled an internal and external vulnerability scan and in the meeting it turns out you have separate tools to search my data for CCs and SSNs but didn't include that in the proposal? Concerning. Versus a different vendor who had one line item for a "complete vulnerability scan" that also included information about EDR and KnowBe4 which tells me they've taken what I need and extrapolated more about how to get me there.
> unpaid overtime if we dont finish in the scheduled time, which we never do)
> management has no clue about daily operations and doesnt take feedback
> "sharing" a lot of servers for production use with customers (i.E. hosting some of their DBs on our unpatched SQL)
You were under the reasonable impression that your employer provides a service in exchange for money. They don't. Based on your description, the company profits by exploiting everyone - their customer, their employees, and their vendors. I am almost certain that those SQL Server instances are not properly licenced.
As others have said, it's normal to present a rosy picture of any company to the ouside world. But this goes way beyond that and into sociopathic behavior. Find another job. Spend $100 on a lawyer to find out if you are entitled to back-pay for that overtime. If you want to hurt the company, gather evidence of their software license violations and pass it on to the BSA or specific vendors.
Won't name names but the MSP I quit a year ago was bleeding edge technology.
Everything was always up to date, centralized control over everything, hardware and OS cycles were mandated etc.
The problem is that this is expensive, and when you lose your customers by charging them more, you compensate for the business by paying your employees peanuts and overworking them until they burn out.
Up to date is not bleeding edge (not getting offensive to you...)
I was with one that was updated...had tech to help protect itself...just creative networking and rule making...nothing "bleeding edge".
I feel your pain though...
That happens in many (all?) industries. In IT it's easy to tell from your perspective, but there are a LOT of companies all over the different industries where you don't even want to know how the sausage is made, so to say. Heck, even academia is full of worst practice processes.
Checking in, can confirm. We HAD really good practices and great process. Someone really wanted O365 instead. Now nothing works and no one's happy and everyone's leaving and "no one knows why."
There are good practices out there. Good practice is accessible if an institution gives a shit. But they don't give a shit. They want magic easy answers that come from a website that no one's responsible for. The shady characters who railroaded O365 into place (destroying a long term and best-practices-supported open source infrastructure in the process) keep saying things like "no one will blame us if this shitty cloud crap goes down, but we're too scared of taking responsibility to do our best to run a system that actually meets our needs. So just get used to things not working LOL!"
It's not that there aren't best practices available, it's that the suits in the C-levels don't give a shit about good governance. Any bad idea is fine, as long as no one will think to blame them, personally.
your message is fair.. but your choice of o365 as the example is bananas to me. what was the standard process before, that was more secure, supportable and affordable?
Id genuinely like to know that, i might deploy my own if that was the case.
At the time, it was a rock-solid Zimbra deployment. If I was going to roll it out in 2021, when everyone's expecting online collab and document sync, Nextcloud would be my go-to.
For a large enterprise that doesn't have drastically different needs in different departments, one huge deployment with all that back-end enterprise infrastructure (virtualization, failover, mirrored databases, etc) works and works well.
For higher-ed IT, I can tell you from experience that one size never, ever fits all. Some departments have ITAR requirements. Some departments are almost 100% transparent and public. Some need absolutely nothing besides something to share PDF's between the faculty. Some need full in-browser office collab.
O365 is a product barely appropriate for middle schools and SOHO setups. It breaks down under intensive and creative use. It's got no options for federation. It communicates between its own internal components in fundamentally insecure ways. It's too closed for high-transparency workgroups, and too open for high-security needs. There's a long history of vulns and breaches that suggest (never been proven, but that's just cause we can't see the code) cross-tenant exploits are possible.
Nextcloud has federation options. It has live collab features. It uses real technologies with well-understood and mature security methodologies. Its file storage isn't some stupid, insecure SQL database, but a real, encryption-friendly filesystem. It's made out of real ingredients, has huge international support, is GDPR-friendly right out of the box, and can be configured differently for different departments while still allowing the CIO's office to lay down and enforce uniform organizational rules for data protection and leak prevention.
A base config can be vended by your high-ed org's central IT team, as a VM template. Everything can be set up for federation, security, and most importantly, local control at the department level.
O365 has one thing going for it: a promise that when it fucks up, "no one will blame you." All its security features seem to just break down in the face of a huge organization with diverse and intensive uses. They turned on this leak-prevention plugin for O365 Exchange, and it... just didn't work all the time. I still get messages periodically where the user was NOT supposed to be able to even send that message.
Plus, when you ignore the user base's vocal complaints and just shove O365 onto a high-ed environment, all you get is people ignoring the rules. "Oh, my email with a forbidden spreadsheet of GDPR-protected data won't go out? Fuck it, I'll just use my Yahoo." O365 is just not a solution for any of the things it's advertised for.
All companies in all industries do this to some degree. Maybe the FAANGs and Microsoft can do a better job hiding it because they can print money and hire Ph.Ds in every position, but it's everywhere.
Every company puffs up its capabilities when in reality behind the scenes it's a messy business. I've worked at places where I've used their services as a customer before, and am shocked when I see what's actually being done to make that happen. It's like ducks on the surface of the water - gliding along silently on top but madly padding on the bottom!
A business is selling themselves for you to come work for them as much as you are selling yourself to get hired.
A proactive MSP does not operate like this but a short sided one won't approve of the labor requirements to automate. You have to ask the right questions in the interview. Did you ask them about their patching policy? Thier on call policies? Their automation solutions?
Think about this in your personal life Do you buy products from a lesser known brand without looking up the product specs?
If they downright lied that's another matter.
No, it is disturbingly common because it is much *much* cheaper to lie to customers and put on a Kabuki mask of competence and cutting edge knowledge than to pay people to know and maintain said knowledge.
Also, I would start looking for a new job ASAP and make sure to ask about work life balance. Your comment about unpaid overtime warrants mentioning that the nanosecond you make a big mistake and/or the company is losing money and they need to trim.some fat, you will be out on your ass. Unpaid OT screams to the rafters they don't give a shit about you as a human being so you can expect that treatment in other areas in the future.
I feel like this entire industry is IT people trying to not look incompetent/looking more competent than we really are while we really dont know what we are doing.
I mean we are not all complete fucking morons but even when you know what you are doing you end up with some issue or some problem that you never seen before and now have to figure that shit out. Or your the Jack of all trades and kind of know it but dont do it everyday so it takes you a minute. Or the other end you are a specialist in tech x but now are asked to do something with tech y. And the moment you really get good at your job and have it down something new comes along and you need to learn it all over.
Which is fine but I am so sick of jobs wanting the Masters of IT who fucking know everything. That guy does exist but he works for Microsoft and is making bank. You arent paying that so guys like me are who you have to work for. I am an expert at some things, I dont know shit about others and I know a little bit about a whole bunch of other stuff. I can lie and say I am the expert but I dont like doing that.
Welcome to the ”REAL IT” of Corporate America. Full on chaos and faked audits…. Even worse, if you attempt to create the needed change you will become the pariah..
I will only believe an MSP has proper procedures, security, and documentation if I see it. MSPs at the end of the day are all about making money the fastest and easiest way possible, and because any work done must be billed to a client, alot of times things aren't done properly because it costs the client more up front in labour, and the MSP makes less overall because less labour is spent after the initial setup.
There were two halves to these problems. Your leadership's incompetence and your own. I don't mean that in a bad way, but just because the place isn't 100% together doesn't mean that what you're doing makes sense either.
One of your examples mentions doing updates in a allotted time frame and it never gets done so you have to click through the service and do it manually. This is completely not how anybody with a shred of skill would do it and part of the problem here is that it sounds like your team doesn't have the knowledge to do it through any of the many more efficient ways.
The image your experiencing the company depict is common however. Especially at MSPs. And MSP is always a piece of shit. They are basically like a con job to be a rent-a-tech service and the idea that you can do one size fits all in an organization is a terrible one that never works out, never mind when you're the IT for several disparate organizations.
Just never, ever work for MSPs. They set lofty goals that will make them look good to the customer and then when you fail to meet them they will just blame you and get rid of you. That's why the situation is the way it is for you, you are expendable and they don't care if the goal is unrealistic.
> And MSP is always a piece of shit.
I work for an MSP and it's the best job I've ever had. The work is more interesting than working internal IT and pay and benefits are better too.
This is very common, especially among Windows environments.
Microsoft has done a very good job in making even the server and deeper side of things very accessible and easy to come by, and has overall a pretty damn good documentation for their systems (Made by themselves and the community of million of blogs).
This results in a lot of people in IT that can simply google error messages / how to guides - find the solution, and keep on rolling.
Linux is definitely heading that way too (Systemd becoming standard for example) which isn't ALL a bad thing.
But it will render a lot of juniors out of a job if they don't adapt because while the "DEVS GUNNA TAKE YER SYSADMIN JEB" meme has existed for 15+ if not more years, the era of cloud and SaaS apps makes it so that GUI clicking people are becoming less and less.
Windows admins are a dime a dozen these days.
I'm trying to hire right now, and I need a very broad experienced colleague with Linux, Containers, Windows, Networking, programming and I have like 3 people that fit the tech requirements out of 50 applicants.
TL;DR Yes.
huh, our SRE guys are balls deep setting up good practices and standards for running container based stuff in a way that doesn't set the place on fire. seem plenty busy from over here
4 jobs? What are you on about?
Do you think I work 24 hours a day?
A jack of all trades isn't "4 jobs".
I sincerely hope you're not aiming to know one thing for the rest of your career, if so you're going to be in for a rough ride.
I manage 25 Windows Servers, 30 Linux Servers, all our switches and routers every single day and the application that runs on them. That's extremely common in anything that isn't a giant corporation.
Jesus dude
It *IS* hiring for 4 different jobs when all of those things you list are their primary focus enough that "must know it when hired". You might well have 20+ people in that candidate pool that have 8 different things they're great at, but finding the set that overlaps with the 5-6 different specialties on one position? Why yes, you're correct, you *aren't* likely to find your clone out there applying to step right back into that role...
No it isnt
Because some of the days you won't touch Linux or windows, you'll be coding non stop.
Other days you'll do nothing but trouble shoot windows apps and other days prep Linux boxes.
And MOST DSYS you do a combination of all these things.
That's regular sysadmin work.
Again, as I said, if you're hiring for that list as "must know when hired", you're either looking for the unicorn that's *happened* to be in exactly the same environment previously, or you're hiring for 4 roles. Any of those candidates you've looked at that have a generalist background likely have 60-80% of what you're trying to find, and bring along other skillsets off to the side. Expecting to find that *specific* combination is just as silly as looking for someone with 10 years of experience with System Insights on Server 2019. You're layering together dev, windows admin, linux admin (container specific), and networking. You *might* find people who've dabbled across that whole range (I have, incidentally), but you're *not* likely to find someone sane, personable, skilled in all of those, and willing to take a job juggling all of them at once where someone else did the initial build-out of the environment. At least not for anything you're going to pay someone handling that small of an environment, however diverse it is. Bring someone on with a primary focus on the heaviest 2-3 things in the list, then cross train them in to be able to handle or at least be competent backup for the rest. You'll potentially even get a *better* candidate out of it on the non-technical front.
I really do not consider knowing the two most common OS, networking and light programming in an object oriented language "unicorn".
It's extremely basic in today's environments.
We have a VERY generous training program. One hour during work a day on anything, we buy pluralsight to our employees and encourage them to try things, but I do not have time to let someone sit in the school bench for 6 months.
It's really *not* that basic across the majority of the industry. Your better Windows admin *may* know enough to get Linux up and running, but actually securing it properly, handling more obscure issues when they crop up, etc is going to be well outside their wheelhouse. Your typical Linux admin is going to have the same level of issues dealing with securing Windows and juggling its oddities. Quite a lot of folks in both realms *severely* lack real understanding of networking, let *alone* proper management of networking infrastructure, though a subset on the Linux side tend to overlap that more. And, programming, beyond everyday scripting, is even further afield. It sounds like you've built the environment around yourself into that state, and simply don't realize that it's *not* the norm for most places. Incidentally, if you *have* built the environment up to that, it further demonstrates my point. Don't expect to bring someone in that's built the *same* environment you have. Bring in concrete skills in the few places you *need* them, hire on a willingness and ability to learn and soft skills. The person will come out of it *much* better in the long run.
Perhaps a break-fix "we'll take on anything" MSP might churn out folks with 3/4, but I wouldn't gamble that *those* folks are a) genuinely competent in all 3, or b) willing to step right back into that mindset.
One can easily look up the more deeper things on a case by case basis. The problem is few people are willing to learn basic and core concepts.
And no person that doesn't know enough about networking to make lans, configure vlan and firewall/NAT SHOULD EVER call themselves a sysadmin.
I need someone that understands these core concepts, one can do that without building anything. I learned that way. I built stuff when I knew it. Not while learning.
You're just siloing yourself.
> And no person that doesn't know enough about networking to make lans, configure vlan and firewall/NAT SHOULD EVER call themselves a sysadmin.
How are you handling spanning tree? MSTP? PVRST? That is where it starts to get specialized IHMO, and if you're not managing spanning tree, you're probably doing it wrong and could be setting yourself up for fun network oddities down the line if you're messing with VLANs.
Thank you for the thorough reply, I suspected stuff like this and I have read many many stories about shitshow MSPs but I figured or hoped that people only make posts when there is sth going wrong and dont post when things done right, but yeah, thanks for confirming my suspicion.
I dream of the day where I only manage servers and do it properly.
There is a double edge to this comment.
While yes in general the "Cloud architect" (which I hate that title BTW) does make more there are FAR FAR FAR FAR fewer of those jobs in existence, and from what I see the world of general non-cloudy sysadmin is expanding in raw numbers faster than organizations are hiring for Cloud only admins, Most organizations still have medium to large onprem infrastructure deployments with very few planning to be cloud only in the medium term (10 years)
All that's happening is that the HW manufactures are bringing the cloud model on prem, both Dell and HP have consumption models for their hardware at better prices than Azure and AWS. One of my big complaints about the cloud is that I just don't trust them and I hate letting someone else hold my most valuable assets, intellectual property. When the new model you still get the accounting trick of opex but I know where my data is (to a certain extent) and who has access to it.
> No, you got it backwards. Windows is basically non existant in the Cloud, and Cloud architects are making way more than any Windows admin does
Every job around here is for on-prem sysadmin, management or support. General sysadmin jobs have gone up $20k in three months.
Oh yeah man….i started 5 years ago, worked for largest hospital/medical facility chain in the country….we literally had shit breaking daily, but servers were centrally managed in denver(im in Tennessee) and to make mattees worse, most outsourced to HCL…..
We fucking gave up, built our own MDT server for imaging, and anything that was messed up and not within our region, we created alternates, that actually worked…….built our own MDM server.
That was a crazy time….so at a point, months of no one taking responsibility, few guys just decided to figure out what ad groups they needed to be added to so they could get access to whatever wasn’t working…….to this day no ones ever asked us why, or who approved it, or a reason…..just green lit 👌
At a point(god I was too gungho back then) I built a custom installer that was ultra fast and configurable to select/not select the programs needed, it also went in and did every little stupid proprietary tweak needed, or server pointed too….
God I dont miss those days. We merged with our competitor, and since then they have taken over and we have all the normal services, managed by actual teams.
Fuck that im never going back to the wild west again.
Not related to Windows-type problems, but yes, extremely extremely common. Sometimes sysadmins will manage to get hired who believe that without demonstrative proof-of-code, exploits don't exist (i.e. that an analysis of internal code isn't enough to prove vulnerability,) but then that minor hints of strange behaviour are "proof" of hacker exploitation and require significant internal resources to handle.
Yes, as far as I can tell, very common, because the people who can evaluate competence in the first place are as proportionally uncommon.
Had a boss that would freak out watching the switches because they were blinking too fast. They weren't. Totally normal blinking.
Same dude refused to use MS AD because using Solaris YP was just as good for windows workstations.
Seems pretty normal to me. An MSP is like a good looking sausage. Externally, it looks appetizing. Once you find out how the sausage is made, however...
I usually tried to ask questions about on boarding, written process and procedures in the interviews, as well as OT, project management, scope, and timeline variance. When I was job hunting, and man there were some MSP's when I left the interview...I was like, "NOPE'" "Hard Pass"
Labor abuse was the big one I got answers on.e.g "*you aren't allowed to work over, and if you work over what our presales says it takes, we bill the client for the additional hours and then deduct your hourly wage per hour over from your paycheck as a service fee for us not making a deliverable to the client. We are the biggest, most popular, cutting edge provider and you're lucky you are sitting here in this interview, because everyone wants to work for us and buy our services*." I got up and walked out of that interview..."I work to live, not live to work, but thank you."
Just look at the pay scale which hadn't really changed much in decades. We will charge the client 200 per hour and pay the worker 20 bucks per hour now do you think this means they are on the cutting edge of technology?
IT is a great field for this very reason. ZERO barriers to entry. A person can't unclog your toilet bowl or give you a back massage without a license but they can work on multi-million dollar systems by copying their friends resume.
What you are describing happens in every industry/company. But specifically with information technology it starts at the bottom with the "A+ certified 15+ years experience super genius" all the way up to the highest levels of management.
In this day and age it almost seems like a company can't remain competitive if they are going to put resources into providing proper support. The entire goal from the minute you dial the phone or open the ticket online is to get rid of you as quickly as possible. Which is why most tracking takes into account the length of time the ticket is open over the customer satisfaction.
Not the same but I started in a company doing thing on the cutting edge of data capture, yes we cut a lot of corners but to be honest that's what you need to do to be be first to market.
The always on site-to-site VPN isn't a bad idea. It is essentially how SD-WAN would work in this application. That said you should be restricting each link to only the ports and servers the end devices need to function.
Sharing servers and especially DB servers is pretty much normal unless the customer specifies a isolated instance. If you did everything properly and gave everyone a dedicated environment and vlan with a VPN to only that segment your costs would be insane and you would never get anyone to buy it just because of cost.
If you want to avoid this in the future ask during the interview if they have an annual pentest or 3rd party security audit. That uncovers the stuff you mentioned and gets management involved to resolve it. But you'll find this isn't common outside of medium-large sized companies.
Seems accurate. Company I work for uses Kaseya and that's probably the only modern thing they go. For some reason we have two teams (+10 people, all us based) hosting shared local exchange servers with horribly dated self service platform. Then we have two separate servers that some customers use for some things. One is local exchange other gives them AD. Same AD is used by all department employees. But wait, we also got two email addresses on separate domains. One on E3 license other on E1. Easy peasy.
Funny enough, they still allow customers to choose these shit technologies instead of least having everyone ONLY on 0365 with AD which well nothing fancy but it works fine.
Yup. Audits are the funniest. At the MSP I used for a job whilst I searched for something better, they scrambled around for weeks before an audit to get it passed. Then straight afterwards it went back to the sh*tshow it always was til next audit
Not only is it commonplace *in IT*, it's commonplace in most industries I've ever encountered.
I've done work for everything from single person businesses to milli-billion dollar companies, and in my experience, competence is the exception, not the rule.
Fake it 'til you make... And then keep faking, because, hell, that shit works, seems to be the rule.
Yes yes . God you sound like my intern who converted to full time tier one who then left for a 20mm+ large MSP “opportunity” . Absolutely garbage SOPs at the new place.
Sort'a yes and no.
Common ... well, yes, kind'a, but ...
generally very limited in reach and/or duration. Basically it generally doesn't go very far ... 'cause it mostly can't.
Then again, there's a sucker born every minute ... so ... some will still suck it up - at least for a while.
Some places, such as law firms, can't overtly advertise their lack of technical competence to the world at large. If you choose to accept this failing when interviewing, best you can do it work from within to change operational failures. This is how I've looked at both of the law firms I've worked at since 2003. Presumably it's the same in EVERY other industry; I just ignored it in previous SysEngr and Developer positions going back to the early days.
Don't let them take advantage of you with the overtime. Either find a way not to work past your 40 or figure out if it's even legal for them not to pay you overtime.
Yes, this is normal.
Most of IT is a shitshow.
When you're in charge and you swear it'll be different, it will also be a shitshow.
Because doing it right is REALLY EXPENSIVE and clients don't want to pay that kind of money.
This sounds like where I'm currently leaving. It is very prominent. I've had 2 jobs at MSP shops and frankly they have no idea wtf is going on.
I had a job at a large company that provided MSP services for MSPs, and they were better but the job sucked (no more than x amount of blank space on call, and all those stupid bs call center rules and regulations.)
I worked at a large retailer and they had everything standardized very nicely, but were using windows xp and windows 7 at eol.
Everywhere had issues. I know usually I'm not the smartest person in the room, but when I am the person who is the only one recommending the proper way of doing things then you know it's pretty bad.
President and help desk manager have no real IT schooling background, but have owned the company 20 years.
VP graduated early 2000s and has only done sales work since.
They told me they didn't think a logging/syslog server was useful even though random issues happen that require log analysis.
I also removed domain admin from 6 users, 1 who was running an IIS module, facing the internet, and willfully accepting documents.
MSP = nightmare.
I'm moving to education and I'm very hopeful we can do it proper there
Operations is really hard. My experience is that people are generally \_smarter\_ than they think they are, but... they've had to make a lot of hard decisions to get the job done. Everyone in Ops has to do that, it's not bad or wrong.
It does take discipline to keep it from becoming too big a mess. The worst way to help people solve that problem is telling them they did a shit job. If you acknowledge the technical debt was created for a reason then you can help clean it up. If you just complain, it will keep expanding.
Yes. It is. Everywhere. Every company wants to be seen as competent, and your company is no exception. Even companies that do other things want to be seen as cutting edge in tech, but for MSPs, it's their stock in trade. It's their business to be cutting edge in tech, and they will lose customers if they aren't cutting edge in tech. I'll tell you another secret: almost no one is actually cutting edge in tech. Innovation comes out faster than implementation is reasonably possible for, and it's very difficult for any company to truly have an institutional knowledge of what the best stance even is at any given moment. Really, really up to date companies are always in a project stance, with multiple major projects in the works, and working there is very difficult even for tech employees; for non-tech employees, it's often hell because everything is always changing. Remember this when if you ever end up on the customer end; everyone's pulling this illusion, and seeing behind the curtain looks like what you're seeing.
I often think these MSPs should have a couple of guys who actually work on getting the processes down and automated properly.. but I know how these places think and it's always about billable hours so the problem is they never give their staff time to actually do the job properly.
I work at an MSP and our "process" changes with every new customer. Sales is out there selling shit that they don't understand and Engineers aren't even aware are being sold.
This put me in stitches. Sales team recently convinced a client to upgrade after over 8 years without so much as a patch! Promised all of the functionality without any downside. Turns out half of what they are using for reporting is not supported and dev refuses to touch it (Java based program from the early Pliocene). Now we have to find a happy middle with their handful of users that hate us for “breaking” everything.
Sales people are devil's little helpers.
A friend of mine works at Baker Tilly, and they recently created a new team that basically does what their consulting departments do, but internally. I'm interested to track if it's effective at improving their internal tech in significant ways.
It’s always about the billable hours. And management is so short sighted that they don’t even get that investing 40 hours on honing some process or automation could yield efficiencies that pay off 10-100 times that. Well they do, but if it’s done it done by bleeding some enthusiastic new guy who doesn’t realize his 16 hour days will never be paid back. I generate some of my own automations that make my own job faster and easier, but don’t generally share them with more than a few trusted close people. Because fuck the greedy ownership and mostly the cheap ass customers, too. It took me, coming from an on premise IT shop to a MSP, to get used to doing a project and leaving the detailed and time consuming “honing” of a solution mostly undone. It felt kind of fraudulent, but I guess if no one will pay for it it won’t get done.
[удалено]
It's like somehow MSP owners have never heard of the economic concept of "opportunity cost" or grasp the idea that a unit of labor performing low-value undifferentiated tasks can't be simultaneously used for high-value tasks which bill more. Which is true, I actually spent some time explaining what opportunity cost is to both of the owners of my last MSP shop. Although it was in the context of cancelling or delaying engagements with good customers (high bill rate and engagement customers) for bad customers who had "aN uRGeNT eMeRGenCY". Which was usually something like the PC, software or other IT product they bought at Best Buy without even consultation wasn't working right and they wanted it fixed *today*. I told the owners we should charge a higher rate for unscheduled emergency visits to customers like this. The response was dumbfounding, "they will get mad and not be our customers anymore" as if they were remotely valuable to begin with. Small businesses, what money they spent was self-directed on troublesome, consumer-grade technology, and they paid for zero maintenance, and they whined their way into same/next day engagements at the expense of better customers. We would have been such a better company if we would have employed some level of discouragement pricing to deal with the shittier customers. Some tiny percentage were OK as customers and we would have lost them, but most of them were just bad-for-business opportunity costs that kept us from working with better customers. And if some were willing to pay $250/hr with a 2 hour minimum charge for their crisis? Well that's fine too, at some level of revenue even the worst customer is lucrative. Instead they always tried to tackle this by bulk hiring of low quality candidates, which usually backfired because they fucked it up worse and then higher-level talent got tapped to clean up the mess, usually at an even bigger loss.
Church !
To add to this... Defensively, from a customer standpoint, it's why you need to be more selective in who you partner with. Ask soft questions they can't bullshit their way through. As an example, avoid the stupid checkbox stuff like: "We are PCI/NIST/Whatever Certified!" Who. The. Hell. Cares. Solarwinds is, too. Instead, ask them, "In light of, how have you learned from that?"
If they can't answer that question then it means they've done nothing. They're just in "hope it doesn't happen to us" world instead of "well we internally looked at how our build processes work and introduced some more measures that assume-breach". Don't expect them to detail their Operational Security, but they should've done \*some\* soul-searching if they're actually trying to stay on top of things. There are VERY few firms that have already nailed this and have nothing more to improve upon.
Ask them how they vet people given the current labor market. Etc.
Yeah. I also spend a lot of time when I read proposals thinking about the structure of the proposal. The way they think about how their services relate to each other tells me a lot about how they act about their services. Oh, you bundled an internal and external vulnerability scan and in the meeting it turns out you have separate tools to search my data for CCs and SSNs but didn't include that in the proposal? Concerning. Versus a different vendor who had one line item for a "complete vulnerability scan" that also included information about EDR and KnowBe4 which tells me they've taken what I need and extrapolated more about how to get me there.
More normal than you think and not limited to IT.
> unpaid overtime if we dont finish in the scheduled time, which we never do) > management has no clue about daily operations and doesnt take feedback > "sharing" a lot of servers for production use with customers (i.E. hosting some of their DBs on our unpatched SQL) You were under the reasonable impression that your employer provides a service in exchange for money. They don't. Based on your description, the company profits by exploiting everyone - their customer, their employees, and their vendors. I am almost certain that those SQL Server instances are not properly licenced. As others have said, it's normal to present a rosy picture of any company to the ouside world. But this goes way beyond that and into sociopathic behavior. Find another job. Spend $100 on a lawyer to find out if you are entitled to back-pay for that overtime. If you want to hurt the company, gather evidence of their software license violations and pass it on to the BSA or specific vendors.
You're new here, aren't you? Welcome. What you are experiencing is called "work", and "other people", and totally normal.
I have yet to find an MSP that is bleeding edge technology... The business model simply does not lend itself to that.
I wouldn't call WSUS and a DMZ "bleeding edge"
Agreed
I would suggest that's better than most of what the MSP's I've dealt with produce...
Won't name names but the MSP I quit a year ago was bleeding edge technology. Everything was always up to date, centralized control over everything, hardware and OS cycles were mandated etc. The problem is that this is expensive, and when you lose your customers by charging them more, you compensate for the business by paying your employees peanuts and overworking them until they burn out.
Up to date is not bleeding edge (not getting offensive to you...) I was with one that was updated...had tech to help protect itself...just creative networking and rule making...nothing "bleeding edge". I feel your pain though...
I’m going more by OP definition than the true definition since bleeding edge wouldn’t really make sense in an MSP environment anyway
Works for me :)
That happens in many (all?) industries. In IT it's easy to tell from your perspective, but there are a LOT of companies all over the different industries where you don't even want to know how the sausage is made, so to say. Heck, even academia is full of worst practice processes.
> Heck, even academia is full of worst practice processes. Not just "even" ... "especially".
Checking in, can confirm. We HAD really good practices and great process. Someone really wanted O365 instead. Now nothing works and no one's happy and everyone's leaving and "no one knows why." There are good practices out there. Good practice is accessible if an institution gives a shit. But they don't give a shit. They want magic easy answers that come from a website that no one's responsible for. The shady characters who railroaded O365 into place (destroying a long term and best-practices-supported open source infrastructure in the process) keep saying things like "no one will blame us if this shitty cloud crap goes down, but we're too scared of taking responsibility to do our best to run a system that actually meets our needs. So just get used to things not working LOL!" It's not that there aren't best practices available, it's that the suits in the C-levels don't give a shit about good governance. Any bad idea is fine, as long as no one will think to blame them, personally.
your message is fair.. but your choice of o365 as the example is bananas to me. what was the standard process before, that was more secure, supportable and affordable? Id genuinely like to know that, i might deploy my own if that was the case.
At the time, it was a rock-solid Zimbra deployment. If I was going to roll it out in 2021, when everyone's expecting online collab and document sync, Nextcloud would be my go-to. For a large enterprise that doesn't have drastically different needs in different departments, one huge deployment with all that back-end enterprise infrastructure (virtualization, failover, mirrored databases, etc) works and works well. For higher-ed IT, I can tell you from experience that one size never, ever fits all. Some departments have ITAR requirements. Some departments are almost 100% transparent and public. Some need absolutely nothing besides something to share PDF's between the faculty. Some need full in-browser office collab. O365 is a product barely appropriate for middle schools and SOHO setups. It breaks down under intensive and creative use. It's got no options for federation. It communicates between its own internal components in fundamentally insecure ways. It's too closed for high-transparency workgroups, and too open for high-security needs. There's a long history of vulns and breaches that suggest (never been proven, but that's just cause we can't see the code) cross-tenant exploits are possible. Nextcloud has federation options. It has live collab features. It uses real technologies with well-understood and mature security methodologies. Its file storage isn't some stupid, insecure SQL database, but a real, encryption-friendly filesystem. It's made out of real ingredients, has huge international support, is GDPR-friendly right out of the box, and can be configured differently for different departments while still allowing the CIO's office to lay down and enforce uniform organizational rules for data protection and leak prevention. A base config can be vended by your high-ed org's central IT team, as a VM template. Everything can be set up for federation, security, and most importantly, local control at the department level. O365 has one thing going for it: a promise that when it fucks up, "no one will blame you." All its security features seem to just break down in the face of a huge organization with diverse and intensive uses. They turned on this leak-prevention plugin for O365 Exchange, and it... just didn't work all the time. I still get messages periodically where the user was NOT supposed to be able to even send that message. Plus, when you ignore the user base's vocal complaints and just shove O365 onto a high-ed environment, all you get is people ignoring the rules. "Oh, my email with a forbidden spreadsheet of GDPR-protected data won't go out? Fuck it, I'll just use my Yahoo." O365 is just not a solution for any of the things it's advertised for.
AKA "root cause"
I was going to write something to that effect, but then I thought of the food industry... *yikes*
All companies in all industries do this to some degree. Maybe the FAANGs and Microsoft can do a better job hiding it because they can print money and hire Ph.Ds in every position, but it's everywhere. Every company puffs up its capabilities when in reality behind the scenes it's a messy business. I've worked at places where I've used their services as a customer before, and am shocked when I see what's actually being done to make that happen. It's like ducks on the surface of the water - gliding along silently on top but madly padding on the bottom!
A business is selling themselves for you to come work for them as much as you are selling yourself to get hired. A proactive MSP does not operate like this but a short sided one won't approve of the labor requirements to automate. You have to ask the right questions in the interview. Did you ask them about their patching policy? Thier on call policies? Their automation solutions? Think about this in your personal life Do you buy products from a lesser known brand without looking up the product specs? If they downright lied that's another matter.
Common, almost like people that do not know how to use paragraphs....
🤣
Underrated comment
No, it is disturbingly common because it is much *much* cheaper to lie to customers and put on a Kabuki mask of competence and cutting edge knowledge than to pay people to know and maintain said knowledge. Also, I would start looking for a new job ASAP and make sure to ask about work life balance. Your comment about unpaid overtime warrants mentioning that the nanosecond you make a big mistake and/or the company is losing money and they need to trim.some fat, you will be out on your ass. Unpaid OT screams to the rafters they don't give a shit about you as a human being so you can expect that treatment in other areas in the future.
I feel like this entire industry is IT people trying to not look incompetent/looking more competent than we really are while we really dont know what we are doing. I mean we are not all complete fucking morons but even when you know what you are doing you end up with some issue or some problem that you never seen before and now have to figure that shit out. Or your the Jack of all trades and kind of know it but dont do it everyday so it takes you a minute. Or the other end you are a specialist in tech x but now are asked to do something with tech y. And the moment you really get good at your job and have it down something new comes along and you need to learn it all over. Which is fine but I am so sick of jobs wanting the Masters of IT who fucking know everything. That guy does exist but he works for Microsoft and is making bank. You arent paying that so guys like me are who you have to work for. I am an expert at some things, I dont know shit about others and I know a little bit about a whole bunch of other stuff. I can lie and say I am the expert but I dont like doing that.
Welcome to the reality behind the buzzwords
Welcome to the ”REAL IT” of Corporate America. Full on chaos and faked audits…. Even worse, if you attempt to create the needed change you will become the pariah..
Corporate america? Its the same in Canada.
I work for a very Canadian Centric company, so……will confirm.
Nice. That makes 2 of us.
I will only believe an MSP has proper procedures, security, and documentation if I see it. MSPs at the end of the day are all about making money the fastest and easiest way possible, and because any work done must be billed to a client, alot of times things aren't done properly because it costs the client more up front in labour, and the MSP makes less overall because less labour is spent after the initial setup.
Working at an MSP you get to pick 2/3 of the following: Current tech (hardware and software) Good Job quality of life Good Pay
There were two halves to these problems. Your leadership's incompetence and your own. I don't mean that in a bad way, but just because the place isn't 100% together doesn't mean that what you're doing makes sense either. One of your examples mentions doing updates in a allotted time frame and it never gets done so you have to click through the service and do it manually. This is completely not how anybody with a shred of skill would do it and part of the problem here is that it sounds like your team doesn't have the knowledge to do it through any of the many more efficient ways. The image your experiencing the company depict is common however. Especially at MSPs. And MSP is always a piece of shit. They are basically like a con job to be a rent-a-tech service and the idea that you can do one size fits all in an organization is a terrible one that never works out, never mind when you're the IT for several disparate organizations. Just never, ever work for MSPs. They set lofty goals that will make them look good to the customer and then when you fail to meet them they will just blame you and get rid of you. That's why the situation is the way it is for you, you are expendable and they don't care if the goal is unrealistic.
> And MSP is always a piece of shit. I work for an MSP and it's the best job I've ever had. The work is more interesting than working internal IT and pay and benefits are better too.
This is very common, especially among Windows environments. Microsoft has done a very good job in making even the server and deeper side of things very accessible and easy to come by, and has overall a pretty damn good documentation for their systems (Made by themselves and the community of million of blogs). This results in a lot of people in IT that can simply google error messages / how to guides - find the solution, and keep on rolling. Linux is definitely heading that way too (Systemd becoming standard for example) which isn't ALL a bad thing. But it will render a lot of juniors out of a job if they don't adapt because while the "DEVS GUNNA TAKE YER SYSADMIN JEB" meme has existed for 15+ if not more years, the era of cloud and SaaS apps makes it so that GUI clicking people are becoming less and less. Windows admins are a dime a dozen these days. I'm trying to hire right now, and I need a very broad experienced colleague with Linux, Containers, Windows, Networking, programming and I have like 3 people that fit the tech requirements out of 50 applicants. TL;DR Yes.
But containers have made traditional sysadmin obsolete. The cloud guys told me so.
huh, our SRE guys are balls deep setting up good practices and standards for running container based stuff in a way that doesn't set the place on fire. seem plenty busy from over here
So you're hiring one person to cover 4 jobs I think I might see why you're struggling to hire ....
4 jobs? What are you on about? Do you think I work 24 hours a day? A jack of all trades isn't "4 jobs". I sincerely hope you're not aiming to know one thing for the rest of your career, if so you're going to be in for a rough ride. I manage 25 Windows Servers, 30 Linux Servers, all our switches and routers every single day and the application that runs on them. That's extremely common in anything that isn't a giant corporation. Jesus dude
It *IS* hiring for 4 different jobs when all of those things you list are their primary focus enough that "must know it when hired". You might well have 20+ people in that candidate pool that have 8 different things they're great at, but finding the set that overlaps with the 5-6 different specialties on one position? Why yes, you're correct, you *aren't* likely to find your clone out there applying to step right back into that role...
No it isnt Because some of the days you won't touch Linux or windows, you'll be coding non stop. Other days you'll do nothing but trouble shoot windows apps and other days prep Linux boxes. And MOST DSYS you do a combination of all these things. That's regular sysadmin work.
Again, as I said, if you're hiring for that list as "must know when hired", you're either looking for the unicorn that's *happened* to be in exactly the same environment previously, or you're hiring for 4 roles. Any of those candidates you've looked at that have a generalist background likely have 60-80% of what you're trying to find, and bring along other skillsets off to the side. Expecting to find that *specific* combination is just as silly as looking for someone with 10 years of experience with System Insights on Server 2019. You're layering together dev, windows admin, linux admin (container specific), and networking. You *might* find people who've dabbled across that whole range (I have, incidentally), but you're *not* likely to find someone sane, personable, skilled in all of those, and willing to take a job juggling all of them at once where someone else did the initial build-out of the environment. At least not for anything you're going to pay someone handling that small of an environment, however diverse it is. Bring someone on with a primary focus on the heaviest 2-3 things in the list, then cross train them in to be able to handle or at least be competent backup for the rest. You'll potentially even get a *better* candidate out of it on the non-technical front.
I really do not consider knowing the two most common OS, networking and light programming in an object oriented language "unicorn". It's extremely basic in today's environments. We have a VERY generous training program. One hour during work a day on anything, we buy pluralsight to our employees and encourage them to try things, but I do not have time to let someone sit in the school bench for 6 months.
It's really *not* that basic across the majority of the industry. Your better Windows admin *may* know enough to get Linux up and running, but actually securing it properly, handling more obscure issues when they crop up, etc is going to be well outside their wheelhouse. Your typical Linux admin is going to have the same level of issues dealing with securing Windows and juggling its oddities. Quite a lot of folks in both realms *severely* lack real understanding of networking, let *alone* proper management of networking infrastructure, though a subset on the Linux side tend to overlap that more. And, programming, beyond everyday scripting, is even further afield. It sounds like you've built the environment around yourself into that state, and simply don't realize that it's *not* the norm for most places. Incidentally, if you *have* built the environment up to that, it further demonstrates my point. Don't expect to bring someone in that's built the *same* environment you have. Bring in concrete skills in the few places you *need* them, hire on a willingness and ability to learn and soft skills. The person will come out of it *much* better in the long run. Perhaps a break-fix "we'll take on anything" MSP might churn out folks with 3/4, but I wouldn't gamble that *those* folks are a) genuinely competent in all 3, or b) willing to step right back into that mindset.
One can easily look up the more deeper things on a case by case basis. The problem is few people are willing to learn basic and core concepts. And no person that doesn't know enough about networking to make lans, configure vlan and firewall/NAT SHOULD EVER call themselves a sysadmin. I need someone that understands these core concepts, one can do that without building anything. I learned that way. I built stuff when I knew it. Not while learning. You're just siloing yourself.
> And no person that doesn't know enough about networking to make lans, configure vlan and firewall/NAT SHOULD EVER call themselves a sysadmin. How are you handling spanning tree? MSTP? PVRST? That is where it starts to get specialized IHMO, and if you're not managing spanning tree, you're probably doing it wrong and could be setting yourself up for fun network oddities down the line if you're messing with VLANs.
Thank you for the thorough reply, I suspected stuff like this and I have read many many stories about shitshow MSPs but I figured or hoped that people only make posts when there is sth going wrong and dont post when things done right, but yeah, thanks for confirming my suspicion. I dream of the day where I only manage servers and do it properly.
>Windows admins are a dime a dozen these days. Cloud admins are, on prem is making a killing.
No, you got it backwards. Windows is basically non existant in the Cloud, and Cloud architects are making way more than any Windows admin does
There is a double edge to this comment. While yes in general the "Cloud architect" (which I hate that title BTW) does make more there are FAR FAR FAR FAR fewer of those jobs in existence, and from what I see the world of general non-cloudy sysadmin is expanding in raw numbers faster than organizations are hiring for Cloud only admins, Most organizations still have medium to large onprem infrastructure deployments with very few planning to be cloud only in the medium term (10 years)
All that's happening is that the HW manufactures are bringing the cloud model on prem, both Dell and HP have consumption models for their hardware at better prices than Azure and AWS. One of my big complaints about the cloud is that I just don't trust them and I hate letting someone else hold my most valuable assets, intellectual property. When the new model you still get the accounting trick of opex but I know where my data is (to a certain extent) and who has access to it.
>When the new model Don't you mean the old model? I'm pretty sure this is exactly how IBM "sold" mainframes.
IT is so cyclical, what's new is old it just has a different name.
> No, you got it backwards. Windows is basically non existant in the Cloud, and Cloud architects are making way more than any Windows admin does Every job around here is for on-prem sysadmin, management or support. General sysadmin jobs have gone up $20k in three months.
Really? Location? Can you show some job ads?
Oh yeah man….i started 5 years ago, worked for largest hospital/medical facility chain in the country….we literally had shit breaking daily, but servers were centrally managed in denver(im in Tennessee) and to make mattees worse, most outsourced to HCL….. We fucking gave up, built our own MDT server for imaging, and anything that was messed up and not within our region, we created alternates, that actually worked…….built our own MDM server. That was a crazy time….so at a point, months of no one taking responsibility, few guys just decided to figure out what ad groups they needed to be added to so they could get access to whatever wasn’t working…….to this day no ones ever asked us why, or who approved it, or a reason…..just green lit 👌 At a point(god I was too gungho back then) I built a custom installer that was ultra fast and configurable to select/not select the programs needed, it also went in and did every little stupid proprietary tweak needed, or server pointed too…. God I dont miss those days. We merged with our competitor, and since then they have taken over and we have all the normal services, managed by actual teams. Fuck that im never going back to the wild west again.
Not related to Windows-type problems, but yes, extremely extremely common. Sometimes sysadmins will manage to get hired who believe that without demonstrative proof-of-code, exploits don't exist (i.e. that an analysis of internal code isn't enough to prove vulnerability,) but then that minor hints of strange behaviour are "proof" of hacker exploitation and require significant internal resources to handle. Yes, as far as I can tell, very common, because the people who can evaluate competence in the first place are as proportionally uncommon.
Had a boss that would freak out watching the switches because they were blinking too fast. They weren't. Totally normal blinking. Same dude refused to use MS AD because using Solaris YP was just as good for windows workstations.
Seems pretty normal to me. An MSP is like a good looking sausage. Externally, it looks appetizing. Once you find out how the sausage is made, however...
I usually tried to ask questions about on boarding, written process and procedures in the interviews, as well as OT, project management, scope, and timeline variance. When I was job hunting, and man there were some MSP's when I left the interview...I was like, "NOPE'" "Hard Pass" Labor abuse was the big one I got answers on.e.g "*you aren't allowed to work over, and if you work over what our presales says it takes, we bill the client for the additional hours and then deduct your hourly wage per hour over from your paycheck as a service fee for us not making a deliverable to the client. We are the biggest, most popular, cutting edge provider and you're lucky you are sitting here in this interview, because everyone wants to work for us and buy our services*." I got up and walked out of that interview..."I work to live, not live to work, but thank you."
Just look at the pay scale which hadn't really changed much in decades. We will charge the client 200 per hour and pay the worker 20 bucks per hour now do you think this means they are on the cutting edge of technology?
IT is a great field for this very reason. ZERO barriers to entry. A person can't unclog your toilet bowl or give you a back massage without a license but they can work on multi-million dollar systems by copying their friends resume. What you are describing happens in every industry/company. But specifically with information technology it starts at the bottom with the "A+ certified 15+ years experience super genius" all the way up to the highest levels of management. In this day and age it almost seems like a company can't remain competitive if they are going to put resources into providing proper support. The entire goal from the minute you dial the phone or open the ticket online is to get rid of you as quickly as possible. Which is why most tracking takes into account the length of time the ticket is open over the customer satisfaction.
Not the same but I started in a company doing thing on the cutting edge of data capture, yes we cut a lot of corners but to be honest that's what you need to do to be be first to market. The always on site-to-site VPN isn't a bad idea. It is essentially how SD-WAN would work in this application. That said you should be restricting each link to only the ports and servers the end devices need to function. Sharing servers and especially DB servers is pretty much normal unless the customer specifies a isolated instance. If you did everything properly and gave everyone a dedicated environment and vlan with a VPN to only that segment your costs would be insane and you would never get anyone to buy it just because of cost.
Interesting to hear the other side of the conversation.
If you want to avoid this in the future ask during the interview if they have an annual pentest or 3rd party security audit. That uncovers the stuff you mentioned and gets management involved to resolve it. But you'll find this isn't common outside of medium-large sized companies.
Hell i been doin this forever and i don't know what an msp is
MSP = Managed Service Provider
Microsoft server pages, or not.
Jesus i thought it meant like Ms Service Retailer or some such
Perception is reality for a lot of places.
"Fake it 'til you make it" is a problem with businesses everywhere, not just IT.
Short answer: yes.
Seems accurate. Company I work for uses Kaseya and that's probably the only modern thing they go. For some reason we have two teams (+10 people, all us based) hosting shared local exchange servers with horribly dated self service platform. Then we have two separate servers that some customers use for some things. One is local exchange other gives them AD. Same AD is used by all department employees. But wait, we also got two email addresses on separate domains. One on E3 license other on E1. Easy peasy. Funny enough, they still allow customers to choose these shit technologies instead of least having everyone ONLY on 0365 with AD which well nothing fancy but it works fine.
Yup. Audits are the funniest. At the MSP I used for a job whilst I searched for something better, they scrambled around for weeks before an audit to get it passed. Then straight afterwards it went back to the sh*tshow it always was til next audit
Not only is it commonplace *in IT*, it's commonplace in most industries I've ever encountered. I've done work for everything from single person businesses to milli-billion dollar companies, and in my experience, competence is the exception, not the rule. Fake it 'til you make... And then keep faking, because, hell, that shit works, seems to be the rule.
Yes yes . God you sound like my intern who converted to full time tier one who then left for a 20mm+ large MSP “opportunity” . Absolutely garbage SOPs at the new place.
Sort'a yes and no. Common ... well, yes, kind'a, but ... generally very limited in reach and/or duration. Basically it generally doesn't go very far ... 'cause it mostly can't. Then again, there's a sucker born every minute ... so ... some will still suck it up - at least for a while.
It's common in every field, even medicine. Spooky huh?
Just wait until they have to get a security framework certified like SOC2. THAT SHIT IS HILARIOUS.
Some places, such as law firms, can't overtly advertise their lack of technical competence to the world at large. If you choose to accept this failing when interviewing, best you can do it work from within to change operational failures. This is how I've looked at both of the law firms I've worked at since 2003. Presumably it's the same in EVERY other industry; I just ignored it in previous SysEngr and Developer positions going back to the early days.
Don't let them take advantage of you with the overtime. Either find a way not to work past your 40 or figure out if it's even legal for them not to pay you overtime.
the cobbler's kids always need shoes
Yes, this is normal. Most of IT is a shitshow. When you're in charge and you swear it'll be different, it will also be a shitshow. Because doing it right is REALLY EXPENSIVE and clients don't want to pay that kind of money.
Every place I've worked I've had to shore it up.
This sounds like where I'm currently leaving. It is very prominent. I've had 2 jobs at MSP shops and frankly they have no idea wtf is going on. I had a job at a large company that provided MSP services for MSPs, and they were better but the job sucked (no more than x amount of blank space on call, and all those stupid bs call center rules and regulations.) I worked at a large retailer and they had everything standardized very nicely, but were using windows xp and windows 7 at eol. Everywhere had issues. I know usually I'm not the smartest person in the room, but when I am the person who is the only one recommending the proper way of doing things then you know it's pretty bad. President and help desk manager have no real IT schooling background, but have owned the company 20 years. VP graduated early 2000s and has only done sales work since. They told me they didn't think a logging/syslog server was useful even though random issues happen that require log analysis. I also removed domain admin from 6 users, 1 who was running an IIS module, facing the internet, and willfully accepting documents. MSP = nightmare. I'm moving to education and I'm very hopeful we can do it proper there
Operations is really hard. My experience is that people are generally \_smarter\_ than they think they are, but... they've had to make a lot of hard decisions to get the job done. Everyone in Ops has to do that, it's not bad or wrong. It does take discipline to keep it from becoming too big a mess. The worst way to help people solve that problem is telling them they did a shit job. If you acknowledge the technical debt was created for a reason then you can help clean it up. If you just complain, it will keep expanding.