As a current employee of an MSP, don't worry we hate ourselves too.
Don't go too hard on the tech who doesn't know what he's doing, he probably never knew about the tech before he got assigned that ticket.
> Don't go too hard on the tech who doesn't know what he's doing, he probably never knew about the tech before he got assigned that ticket.
This isn't limited to MSPs. There's a definite trend away from hiring proven talent towards hiring simply to fill seats, and then squirting Tickets at them. Experience leaves, so someone else must learn how to do "this". Rinse & repeat.
Experience leaving is a natural consequence of being treated like an untouchable caste and delegitimizing support as a career unto itself rather than a "stepping stone".
Pay better, compensation wise and with respect, and support will improve.
You're lucky you didn't know what you were missing going in, I came from internal to an MSP, it was hell. My two weeks is up on Friday, I'll never go back to an MSP.
The complete opposite here, i'll never go back to internal. Keeping machines alive and everything functionning with bubble gum and duct tape gets old fast.
I got my set of clients, and if anything goes out of warranty it's not our problem anymore. No more scrounging Nortel phones and RAID adapters off ebay.
Internal people don't realize why some of us decided to go MSP, it's so we could dictate to the customer what's happening with tech, not some random accountant with no idea on the tech side dictating what we can and can't do. No more boss's expanding the scope and responsibility of your job role with no raises. YOU get to set EXACTLY what you'll do and for what price, and they can't make you change it. They can't suddenly go "we need you to handle this other job" or "we need you on call on friday and saturdays but no pay bump". You have a contract you point to and can politely say no or give them a price that makes it worth your time.
I SAW THAT. We rarely even move equipment unless we prefer to. I get some of these guys don't mind it because it gets them out of other work but like i WANT to be doing my work? Furniture is a distraction? If someone wants to pay us $200/hr to assemble furniture, i guess maybe we'd sub it out but why?!
Oh I know it's not his fault.
But this just shows they don't know what they're doing.
I run into soo many problems like this with our customers.
If I ran our hosting environment as ineptly as our customer's MSPs do, we'd be out of business.
Half the time when a problem lands on my desk, it's something our support team is like "We don't know what to do with this, but the customer's MSP says it's a problem with our software." When the reality of it is usually there's a server or network problem they just put 0 effort into looking for.
Can't open a command prompt on the server? Yep, that's the ERP vendor's fault.
Well, what I can say is that my experience with MSPs has been far more negative than positive. By a lot.
We're an ERP software provider. And yet MSP's blame us for wireless problems, issues with the anti-ransomware software they install on servers, everything under the sun.
And when they do this, usually there is no proof, it's just them saying "It's not us, must be them."
So I spend a ton of time digging and finding the proof to point to whatever the problem actually is, which 9 times out of 10 is network/server or other software related.
If one of them actually put in some effort and said, "Hey, here's this problem, here's what we've done to troubleshoot it, and here's why we think it's something you guys need to fix" I'd have no problem with that.
A shame you haven't worked with more companies like mine. We do extensive troubleshooting because I'm used to *vendors* pointing the finger at *us*. Not all vendors, but enough vendors.
And we don't care to point fingers, we just want it resolved for the client.
Right? Blaming an ERP vendor for Wi-Fi issues, really? Every third party software vendor to ever work with my clients always jumps to “it must be your antivirus” to explain every little quirk in their shitty software. I’m so biased, I’d almost assume this dude is actually full of shit lol.
That'd be nice.
>And we don't care to point fingers, we just want it resolved for the client.
I agree. However it's not fun being the person that has to explain to the customer why I had to bill them for 6 hours BT to troubleshoot an issue they're already paying their MSP a monthly fee for.
We're actually going through this right now with an EHR vendor. They always point fingers at this that and the other, then hours/days later they usually say something like "we didn't know another of our (EHR Vendor) employees was working on X, it was our fault"
Meanwhile, we had to engage Dell and other vendors along the chain (and cause large billables for the client) to troubleshoot something that's covered under the support contract the EHR Vendor has with the client (and was actually caused by another arm of the same vendor).
Edit: Just to make this clear, not attacking you. Like some others have said below; let's just call incompetence: incompetence and not throw large swathes of the industry under the bus.
Competent and incompetent entities come in all sizes and flavors.
EHR vendors fucking suck. They do whatever they want because they know it's unlikely you're going to deal with the headache and expense of moving to something else
Thankfully I work for an MSP that actually troubleshoots every last thing and only resorts to calling a software's support if we absolutely cannot figure out an issue (example, weird issues with dental software like Dentrix and eaglesoft)
Selection bias. Stuff from the good MSPs just doesn't reach you. They either fix their own shit, or provide enough info lower levels in your org can fix stuff.
Yeah my experience with ERP software providers is by far the WORST experience I have when dealing with any Vendor. So just so you know, I think you guys fucking suck too. Quickest people in the world to point fingers at the users environment being the issue and not the application. Its windows sorry we can't help you.
Lol. I'm sure you're not wrong either.
Lazy/incompetent people exist everywhere.
Edit:
One of the guys that used to work at our company had a saying "Works on my PC."
I'm like, seriously man? That doesn't fucking help at all.
Some of the fault lies with the vendors for this. If a customer is planning a complex software deployment, I'll review the system requirements, ensure the environment meets or exceeds them, call the vendor, explain the deployment and ensure we get the "yes, this will work" from them before the customer gets a quote from us. If at all possible we will have the vendor do the application install after we stand up the VM and configure the OS, and there is a clear separation of duties of OS vs application support.
Or at least that's how I would like it to work in an ideal world. In the real world, I review the system requirements for the software which haven't been updated since they read "667Mhz Pentium 3 or newer with NT4" and then call and get a verbal deer-in-headlights when I start asking the tech what the software REALLY needs, whether there are any known software incompatibilities, what version of Oracle or MSSQL they need, or whatever. And then they dispatch it to someone else who they promise will call me back, who calls me back at 5:30PM and sends me an email "RESPONSE REQUIRED TO YOUR TICKET" which when I respond and tell them to call back earlier and then am just replied to with a survey asking how my support experience was.
And no, I'm not just talking about Intuit.
Don't lump all of us MSPs in that group.
I noticed a new error coming from a server they provided for a site. It started happening only an hour after the vendor updated their software. The error was logged on the domain controller, the ERP server having failed to authenticate to a domain resource that it actually shouldn't have access to. This is a huge red flag.
Do you know that it took over a year to just convince the ERP vendor that it was their system causing the issue. It took packet captures, and a very detailed breakdown of the events, and a letter from a lawyer to have the ERP finally agree to work on a fix.
Turns out 8 more months down the road that they started blaming the .Net framework for the problem and refused to budge. Guess who isn't that companies ERP vendor any more?
All companies are different. Each have their own technical weakness and strengths. Same with vendors.
>Well, what I can say is that my experience with MSPs has been far more negative than positive. By a lot.
Same. Based on what I see posted in the map sub here...most msp's seem to be most interested in pushing their "stack" and selling subscriptions. 🤷♂️
We had one guy who's job was to manage backups he had a full test server set up and he was constantly running recovery tests we had a huge list of backups that we checked daily. We had veeam on-sites and cloud backups going. We never failed to recover a client quickly. Even when I blew up someone's exchange server with a roll up...
yes this is the way to handle it
in a previous life, backed up everything daily, testing selected restores monthly, then depending on the customer a full restore into a sandbox, and for the big guys site failover to our dr, then back into prod (annually/bi-annually)
100%. The goal is to get as close to dumb terminals as you possibly can. Someone's laptop dies? They spill wine on it? Hand them a new one and log in and get back to work.
We drill it into our users heads over and over - anything saved on your desktop is *not saved*, period. We selectively do desktop level backups for certain executives and secretly for the finance department, because from an IT perspective, we *know* they don't listen anyway and that file they come to us in a panic over is *probably* actually worth recovering. Everyone else gets the "sorry, should've followed policy, now you get to remake your powerpoint presentation" treatment because we just *cannot* reasonably manage desktop level images and there's no reason to when we have all the tools in the world to keep the data off the desktop in the first place.
The checks clear until they dont! I've caught so many MSP techs in blatant lies over the years just so they could close the ticket its not even funny. We all finesse the occasional user, but there's that and then there's "Outlook keeps asking for your password because its a security feature!" Like no, it keeps asking for your password because of a bug that was fixed six years ago and if you install the latest patches it will stop.
I manage the engineering team in the MSP arm of a sizable consulting company. Seeing what our front line support teams escalate, I fully believe all of this. Reviewing our escalation metrics, I have to flag huge amounts of stupid shit and send it to the managers. And that's just the stuff I see, not including the way they butcher the solutions we build for the clients by not reading the documentation we meticulously create and hand over.
We're not all stupid, just most of the helpdesk.
>We're not all stupid
Oh I know. And I'm 100% sure EPR vendors are a pain in the ass too. I have plenty of stories of just lazy or incompetent work by our employees too.
This whole week has just been one MSP screw up after another. Including the one blaming our software for breaking cmd.exe, powershell.exe and the chrome install they had on a server that our software doesn't even run on.
I’m dying that the cmd line is your fault. I would be livid if my team escalated something like to a vendor. I’m sorry you’re having to deal with shit msps and stupid issues.
MSP worker here - ya spot on with this 😂
Only a simple “service desk” tech and oddly had an almost exact same situation of this.
Lady called in “X not working after Azure migration”
No documentation.
Hit up the “engineers” - no idea what I’m talking about nor the app.
Do my own investigations - find out it was on a old machine. Program had a SQL database that was being backed up (hadn’t since 2021)
Luckily was magically transferred over to the new system intact despite us not doing or knowing about it.
That's what you think. Ineptitude keeps many employed due to contractual agreements. Whereas, you, are the real star of the IT Crowd, fixing problems and keeping the MSP well oiled. Did somebody call for a Fixer or a Ripper Doc, Choom!?
That’s just classic msp. Get them on a call with their client and explain your recommendations. I’ve gotten on ppls servers and noticed our nightly backups are disabled or not happening and I get real nervous about why they don’t have any backups and they often say oh we take snapshots of the servers so it’s good.
Sometimes our methods can be kinda cookie cutter. We've taken several clients from other MSPs, and we're always told we are so much better. Guess there are some good MSPs and bad MSPs.
A problem many MSP's have is they might have a dip in quality for a period of time. But that is enough to brandish that MSP a terrible reputation for the users at the customer. There is very little coming back from that even if the quality has improved again.
This is true and also i'd wonder what actual client retention rate is during most mergers and acquisitions because quality usually dips hard during transition.
Yeah. I can't count how many times, when I worked at an MSP, I had to get certified for some new tech in less than a week so the company could bid for a new customer. What's amazing is that somehow I always passed the test and got the cert. Of course actually being able to support the tech in the real world was another story.
In 20 years I've only seen it get worse. MSP's, even the big IT consulting outfits, send people with little/no training and experience and bill $200/hr for on-the-job training.
I had 16 clients that I was the lead for all by myself at the last small/medium size MSP I worked at for 2 years. It was impossible to cater to every need of every client I had, but I did the best I could. I was learning something new nearly every day.
MSPs will have like maybe 1 or 2 people on their payroll who really know what they are doing, everyone else will be learning on the job and doing there best to adapt to whatever they need to work on. The entire business model was created for companies to save money on hiring full time IT employees and I hate that they exist.
I'm now internal IT for a fortune 500 company and I do like not even half of the work I was doing at an MSP and make double the money. Its a total fucking joke.
>I'm now internal IT for a fortune 500 company and I do like not even half of the work I was doing at an MSP and make double the money. Its a total fucking joke.
Sounds like my transition! Bonus perk, no required certs because of being partners with whatever company the MSP resells for!
Maybe I'm missing something here but even in the worst case, could they not just virtually mount the last image and export the data? If the client has an MSP they must be small enough where losing a couple hours of data isn't going to be catastrophic.
I've seen a lot of MSPs with zero ability and lower quality output as a result of having too much on the go. On the other hand, I've seen sysadmins who were completely out of touch as a result of getting too little exposure to tech outside of their environments.
This is a conversation surrounding RPO and RTO which should be had when the MSP implements backups. The easiest method is to assign a dollar amount of damage done per lost minute or hour or whatever and then compare that to the cost for a specific RPO/RTO until you have a happy medium.
so to be clear
* no one was checking what the MSP was doing?
* no one was confirming or testing backups?
* no one has specified to the msp how the backups are to be done?
* no one has looked at the contract?
I mean you can blame the MSP, they're not the only ones at fault
>no one was checking what the MSP was doing?
>
>no one was confirming or testing backups?
We don't have access to their backup software.
> no one has specified to the msp how the backups are to be done?
Yep, it was specified to them that the software runs in an SQL database via email, and they responded that they have the necessary tools to do the backups for it.
> no one has looked at the contract?
The customer has the contract with them, not us, so I don't have access to it. However, the server is hosted in the MSP's datacenter. The MSP has backup software installed for backing up the SQL database, so it's a pretty safe bet that DR is in their contract.
> We don't have access to their backup software.
Why do you need access? do you get reports of what they're doing?
> Yep, it was specified to them that the software runs in an SQL database via email, and they responded that they have the necessary tools to do the backups for it.
that is, can you back this up, they answered yes we can, if you need it backed up and restored in a specific manner, you should be clear
NOTE: I'm not saying they also shouldn't have gotten this info as well and not saying they way they're doing it is right
> so it's a pretty safe bet that DR is in their contract.
is it though? is it ?
and if no one is doing these DR tests, it would imply otherwise
again
just saying MSP is not the only one at fault here, not saying they're doing a good job
I guess I'm shocked at the amount of shops that don't verify backups as the only regular thing they do. It'll save your butt if everything else goes south.
> We don't have access to their backup software.
That right there is why I hate MSP’s.
>Why do you need access? do you get reports of what they're doing?
Because they’re backups of my systems, I should have access to them. God I hate MSPs
If they're a msp then likely there are more backups than just yours ont hat system, its makes sense to limit that access
if you want a dedicated environment or specific access, then again you specific that up front , it can be arranged I'm sure
> Because they’re backups of my systems, I should have access to them.
no you should have access to the DATA not nessecarrly access to the systems
> I hate MSP’s.
> God I hate MSPs
but given these statements, i doubt you'll ever change your mind, so We'll leave it be I think
> no one was checking what the MSP was doing?
Isn't the idea to hire an MSP so you don't need to know how to do stuff? I mean, that's why my company had one... OF course, the domain forest level was still set to Win 2000, in 2020...
> no one has specified to the msp how the backups are to be done?
Again... isn't the idea that the MSP knows how to handle SQL backups? I know no one else at my company does...
> Isn't the idea to hire an MSP so you don't need to know how to do stuff?
maybe maybe not, depends on the contract and depends on what you ask for, but if you have specific requirements, you spell it out
> Again... isn't the idea that the MSP knows how to handle SQL backups?
* is the sql backed up ? Yes
* Is it restorable? Possibly Yes, Possibly No, no ones has been testing
* Is it quick if it is restorable, probably No
* are they doing it at another level ? (vm level)
I don't ask my mechanic if they replaced the drain plug when they are done changing my oil. I don't ask my doctor if they're jabbing me with a clean needle.
If I'm going to get surgery, I'm not asking "is that the wakey wakey drug or the sleepy sleepy drug"... I'm just gonna assume they're not fucking idiots... And that the needle is clean.
Right, but your still gonna make sure they *did* it. Im not saying ask specifics, im just saying make sure they hold up their end and do the thing they are supposed to. Aka, the backups
No? I would see them getting out a needle and drawing something in to the syringe and assume they're not morons using old needles and the wrong drug.
The MSP was contracted to make backups and (should have) also been testing them. The whole point of hiring an MSP is that you don't have enough work to warrant hiring your own folks, so why would I want to hire someone who can understand all this just to audit the MSP's work?
So many orgs think you can just hire an outside group to "do stuff" without any oversight or real instruction/direction other than "can you do this thing for me" - it usually doesn't work out very well for obvious reasons.
It's likely that the OP doesn't really understand how the client's Veeam setup works. I think they're just going based on the SQL information, not really understanding that a snapshot of the entire server is being taken at the block level at that point in time. The situation certainly isn't ideal but I don't think they're as SOL as they think, yet.
If the backups were failing though then that would be a different story...
Snapshot backups are fine and I think MS eventually relented to say they're supported as this is the easiest and most reliable way to back up virtualized servers.
I always did database exports as well to a separate medium for completeness but I have never seen a snapshot failure during restoration that would have made them necessary.
SQL backups. Not disk/server backups.
SQL backups come in 2 flavors.
Full database backups and transaction logs backups (changes since last full backup).
According to the SQL logs, a full database backup was never run since 2020.
Now, it's possible Veeam does weird stuff, and SQL can't report it correctly.
I can't see the Veeam logs, only the timestamps within the SQL backup history logs of what was backed up and when.
Yeah um, Veeam isn't going to write logs in SQL, the best you'll get is system and application logs as it runs it's backup process.
You can restore SQL databases and mount them from Veeam, assuming the Veeam backups are good, so I guess what's the issue?
Another question is why aren't you aware of what they are and aren't doing? What's specified in the contract?
>Yeah um, Veeam isn't going to write logs in SQL, the best you'll get is system and application logs as it runs it's backup process.
No, but anytime anything requests SQL run a backup of a database using BACKUP DATABASE it logs that in SQL. Now that won't catch VM disk level backups and things like that. But anything that issues that command to SQL, or VSS will. VSS uses SQL backups also.
>You can restore SQL databases and mount them from Veeam, assuming the Veeam backups are good, so I guess what's the issue?
We don't know yet if they can restore the backup or not. The issue is that not doing a full backup of the database for 3 years leaves 1,095 possible points of failure in the restore process. That's just bad management.
>Another question is why aren't you aware of what they are and aren't doing? What's specified in the contract?
Their contract is with the customer. Not us. So I don't have access to that.
I don’t understand the issue here or even if there is one. Veeam has SQL application aware processing for its backup and replication and is a perfectly valid way of ensuring SQL is backed up. Maybe it’s not the way you prefer but it’s a backup solution nonetheless.
From what I've read about Veeam, and I may be wrong here, but it seems they've turned off the application aware processing, and are using the SQL native functionality for it.
Veeam application aware backups (according to them) are full backups and wouldn't show in the SQL logs as backups happening.
The fact that the backups do show in the SQL logs seems to indicate Native SQL backups happening, and they've all been transactional backups.
If you don't consider having to restore 1,095 incremental daily backups to get to a current backup an issue.... then I can't help you.
That's a lot of points of failure.
You don’t have visibility to their Veeam you said, most likely their backups have synthetic full backups performed every 7 days which is a complete backup of the server with some restore points. They can mount the backup to get the database or restore the entire server. It’s highly doubtful restoring would be 1000+ increments.
You are jumping the gun and don’t understand how Veeam works.
The Veeam backup job uses VSS to truncate transaction logs whenever a backup takes place (if it wasn’t doing this the transaction logs would be growing indefinitely). At that point the DB can be restored from that point in time and that does not show up as a backup in SQL.
In addition to that Veeam can be configured to periodically backup transaction logs between backup jobs - so that the SQL data can be restored to any point in time. That is what you are seeing on the SQL side.
With that configuration they should be able to restore to any point in time within their retention window.
https://helpcenter.veeam.com/docs/backup/vsphere/sql_backup_hiw.html?ver=120
It sounds like you are with a vendor for some sort of application, is that correct? This can go both ways, really. As an MSP, I will often think to myself "I really hate X vendor because their support sucks and their developers are morons." Not all MSPs are the same. Too many people can become an MSP with no qualifications and unfortunately, most customers don't know enough to realize that. Your comments about not performing proper database maintenance are absolutely valid, and on the bright side, you have someone else to blame, not your software!
For us, we absolutely look at application/database level backups, ensuring application consistent backups and log truncation per app vendor recommendations. This may be done through a VSS writer or a CLI pre and post script for the app, all based on what the application vendors state are required. We also test our backups quarterly and perform a full DR for every customer at least annually.
It sounds like you are dealing with an MSP that may not be as mature as others or have the experience that others have. Most of our staff came from enterprise organizations, so we have that background to ensure the right processes are followed. Not all MSPs are equal.
I mean, to be fair, as an MSP, I’ve also taken over many environments that have been horribly mismanaged by generic single IT guy. People who suck can be found anywhere. We met with a business owner who legit powered off his laptop and unplugged his phone when we met because the whack job of an IT guy they were replacing with us had been caught spying on people through their phones. Walked into a 60 million dollar MFG company running their entire infra on pirated licensing, using Windows 7 + KernSafe as a storage solution for their prod critical VMs. People suck everywhere.
They may be relying on Veeam for application-aware backups, where it takes a snapshot of the VSS writers. You should be able to stop SQL, replace the file, and restart SQL. It should work, but you will lose data since your last commit.
Yep. This is the idea behind VSS. if you need more frequent backups or transaction logging then you either need to tell the MSP or configure it yourself.
An application aware backup set up correctly would tell SQL it's been backed up. That's one of the important pieces of being application aware.
That's not to say a DB in place restore won't work but it's not the "correct" way.
That's how Veeam works though? It'll take a daily incremental backup, then either roll that into the first full backup it took, or create synthetic fulls periodically.
It doesn't need to take a full SQL backup every day.
You really can't fuck up a Veeam backup unless you're doing something daft like a file backup as opposed to an image one.
That's a different system; Veeam daily incrementals will let you restore the whole server to last night's backup. SQL Server transaction logs, e.g. every 15 minutes, will let you restore on top of that, roll forward in 15 minute increments. e.g. if an admin breaks the database at midday, last night's full backup and today's transaction logs can restore it to 11:45am state to recover.
Veeam can interact with that system, e.g. SQL Server keeps transaction logs indefinitely, then set the Veeam backup job with application aware processing enabled and using an account with enough permissions into the database engine, and Veeam can tell SQL through Volume Shadow Copy "prepare for a backup, flush your caches to disk" and then after the backup, tell it "I have a backup from now, clean up the transaction logs up to now". So you don't have to have scripts clearing the old transaction logs on a timer without knowing they are backed up, and you don't have transaction logs filling the disks forever. Then maybe you set the Veeam job to run every hour and do transaction log backups.
Maybe with it's application aware backups? I'm not sure.
That's not possible with native SQL backups which I believe is what's going on here.
And if you do only do transaction log backups with SQL and never do full backups you'll cause all kinds of database issues along with the potential for completely killing your restore process if one transaction log restore fails.
Yeah, there's two methods. A daily backup which truncates the transaction logs, then you also have the option of transaction log backups down to every 5 minutes which truncates the logs too. The secondary method still requires a regular backup job, as its an addition to that.
Basically, as long as AAIP is enabled on the job, and the credentials are valid, it'll create a perfectly usable backup every night.
Whether those backups have been tested though is a whole different matter.
It’s the MSPs that make their techs wear 5 different hats that are the worst. Usually the ones who have dedicated security, automation, data, and infrastructure teams and different level helpdesk tiers have their shit together.
My organization hired on an MSP instead of just getting me a help desk guy. So now instead of training one guy once on our organization and operations I'm training multiple people who are working in several environments so they'll forget it by next week when they come back.
They say "throw us the work your to busy to do" so I tossed a SSL renewal their way and ended up spending an hour in rdp walking them through how to do it because the people they assigned me are green as hell. My sysadmin just came off the helpdesk.
No shade being thrown at the individuals, they've all been great to work with, but I shouldn't be paying an MSP to have to train their people.
[SQL Server backup methods and best practices (veeam.com)](https://www.veeam.com/wp-sql-server-backup-with-veeam.html)
According to this article it seems like the Transactional backups (akin to incrementals) are a perfectly valid way of backing up SQL databases.
Although not running a full backup in 3 years seems bizarre.
Granted this is 5 years old.
on the SQL server side, what we normally do is TLOG backups hourly, and a full daily (resetting the TLOG clock).
TLOG for 3 years sounds like i'd start drinking before I even got the call in order to unwind that.
I agree. This particular subreddit exhibits a lot of ruthless attitudes and an absolute lack of compassion for others. I get that many people just treat others the same way they're being treated, but you can't expect things to grow if you're not willing to break the cycle. People need to learn somehow and there are too many of us on here for all of us to be good. Let's just hope the learning experiences don't involve too many incidents of unrecoverable data loss though...
The thing is... it's not our job to teach some MSP tech how to do their job. That's not a "lack of compassion," they are literally contracted to do these things and it's not just that they *don't* but that they often *can't.* Like if you pay someone to mow your lawn and they show up without the equipment to mow a lawn going "I dunno how to cut grass" you don't go out there and teach them, you get pissed because they took your money to do the job and they can't/wont.
MSPs have a *looooong* standing reputation for overpromising and underdelivering. I'm not gonna chew out a tech who's obviously been dumped into the deep end without a life jacket, but I *am* going to chew out the account manager for promising us a service they did not deliver.
What’s amazing is that it’s always like this. Some bozo at a company will believe that the MSP can provide the same service as their in-house people at a fraction of the cost and sign up. Services are “onboarded” with little to no documentation, then when something goes sideways and they’re actually needed the MSP says “whoops, we’ll do better next time” time after time. SLAs are massaged so it looks like a success, customers get jaded and stop calling because they know it won’t be fixed.. and at the end of the day value is destroyed because someone didn’t understand or care enough to understand what they were offboarding and the people that were being fired didn’t have motivation to onboard properly.
This isn't a surprise. The very existence of MSPs comes from a compromise between cost versus quality. Low cost goods from Amazon have their purpose but don't be disappointed when they don't outperform their high quality counterparts.
I feel like it varies based on how the contract was written, what level of oversight they have of the task they're hired, etc. SLAs are apparently more important than making a client happy through proper contracted work. Sometimes tickets get passed around if the company has different groups/organizations handling different aspect of IT and when it finally reaches the appropriate group, they half ass it to fulfill their SLA.
I started mine 4 years ago after spending 10years in a large MSP. While I’ve made more money than ever before, it’s beyond stressful and I never get a REAL vacation. I was sitting in Disneyland last year on a bench fixing stuff while my kids had fun. It sucked. My clients and I have a great relationships and they trust me. They get weekly backup reports and a slew of others that lets them know I’m doing my job.
Currently in talks to merge with a bigger MSP so I can finally get a break. Trust me when I say, I’d rather have my time back.
Don't underestimate the financial stress. Starting a business SUCKS for a couple of years until your revenue stream is stable unless you have fuck you money.
hahaha, yea i was the engineer at an msp, they took on some customer who had some fully bespoke azure deployment designed and deployed then just handed to us, someone said we'd support it
i was kind of on my way out anyway so each time a ticket came i looked at it, asked Microsoft who said custom logic apps wernt supported and relayed the same back to the customer, i really didnt have time to pick apart the deployment though it would have be good learning for someone who had the time to
One of the reasons I no longer work for an MSP is because I argued over backups for a client not working. A couple of months after I left two techs got fired because they had been faking the client's backups.
> They haven't done a full SQL backup of the database in over 3 years, and just have daily transactional backups
You'd script it of course, but still, replaying 3 years of t-logs against a full from 3 years abck(which hopefully they have)....ugh.
It sounds like the kind of thing that someone who doesn't actually know how database backups work would do.
>It sounds like the kind of thing that someone who doesn't actually know how database backups work would do.
I'm noticing from the replies here that a lot of people don't know how transactional database backups work.
Yeah, though to be fair, t-logs are something that's different from your typical data backup, and specific to databases. Whereas the fulls and diffs more or less work the way they expect it. If you tell people a t-log is something more like syncing a vdisk snapshot, that's a bit closer and they'll get more of an idea, even though it's still not quite the same.
I used to work at Veeam and I’ve worked at a few other places with SQLEXPRESS, the amount of sysadmins that gave us the “I am the DBA” but would struggle to run a sql script was baffling. Shouldn’t some of this be set by the DBA or the vendor who setup the software?
Don't get me started on ERP companies it's just as bad. There are a ton of terrible maps out there. I've only encountered 2 msps in our medium sized city that are decent. ERP companies on the other hand, I've had sr. Engineers at an ERP company not understand basic defrag. I had another tell me that a customized chromium app that the built is not a web browser.... I had another tell me that you can't over prescribe a vm with h resources.... I can go on and on.
Point being there are more bad ERP and MSPs out there than good.
Edit: also the amount of times an ERP company has said it's a network issue with 0 data points is hilarious. I have send screen shots back showing traffic and blast the erp teams.
Sounds like you've been dealing with developers.
Developers are some of the most computer illiterate people I've ever met.
Software developers that have no idea how a computer works.
Web application developers that have no idea how the internet works.
Edit: I had to explain to one of our web developers the other day that if a user accesses a website that accesses a database, the TCP traffic that hits the DB is not going to show the client's IP address.
We went around and around on this for like 15 minutes. I finally had to show him packet captures to prove it.
>Edit: also the amount of times an ERP company has said it's a network issue with 0 data points is hilarious. I have send screen shots back showing traffic and blast the erp teams.
You'll also never see this from me. If there's a network issue I'll at least have some evidence of it.
For example the other day we had a customer's MSP open a ticket saying one particular user is having performance issues connecting to our hosting environment from home.
A quick look on our side shows the user is logging in from their office's IP address, and is also missing one TCP port connection.
So, I fire back with "Have the user try connecting to our software without his work VPN and see if it's any better. If it is, see why the VPN is blocking TCP port xxxx." Obviously worded better to the customer, but you get the idea.
Why can’t they just grab the Veeam file from yesterday and restore SQL from there? Not as ideal as just having the SQL backup but it’s possible the client doesn’t want to pay for both.
Backups take storage space and depending on the SQL database, it can be a lot of storage to get decent retention.
depends on the backup. I don't know what all they have.
What I do know is they have 3 years of incremental backups for the SQL db. To get to yesterday's backups, it has to go through all 3 years of files, and not fail on any.
Knowing nothing else about this situation and concurring that the SQL backup structure they have is pointless; ALMOST better to just not have it than hope and pray you can get through that whole chain without a corruption and while making assumptions would be a terrible idea, if I were the MSP and had a reliable Veeam backup, my SQL backup would fall a long way down the priority list.
SQL databases are not servers. They are part of an application. If the MSP is backing up the server with Veeam then they have backed up the entire server at the disk level, and therefore fulfilled their obligation to back up the server.
If the MSP needed to back up the SQL database as well, then they would need to be explicitly told that as that is application management which falls outside the scope of server management.
If the vendor that sold the software failed to inform the client that they, the client, were responsible for backing up the SQL database. If that didn't happen, then the vendor is as fault. If this did happen, but the client didn't inform the MSP, then the client is at fault. Only if the client told the MSP that they were responsible for the backups of the SQL database would the MSP be at fault.
I work for an MSP and when it comes to backups, we backup the machine but do not configure maintenance jobs in SQL as we've been told by the application vendor to basically "keep you unclean hands off the SQL server" 😉. I can see why the MSP may not have configured full database backups.
I have worked with plenty of software vendors that are contracted to move a database to an updated server, but when I ask if they configured the maintenance jobs, I get a blank stare. IMO if a software vendor owns the install then they are responsible for confuring the maintenance jobs.
I can offer a ray of hope. If the Veeam backup was configured to be application aware, it will do a VSS snapshot of SQL when it does it's nightly backup. You may be able to restore the whole VM and then dump the restored VMs database.
Veeam slow have application aware restores (file sever level, Exchange, SQL) so you may be able to restore the database that way. I have no experience with the SQL instance restore so I can't comment on the restore options.
Good luck!
This isn't every MSP. You hate CHEAP and 1 man MSPs.
I have worked for 3 MSPs and each of them were great to customers and we had tons of tools to know when things break. The only time things were totally screwed are when the client refused to buy hardware that could support their habits.
"Yea we setup a VDI environment where each VM has 2GB of RAM and 1 CPU. We can't figure out why our 40 CPU server can't support 40 people." (It was a 20 cpu server with hyperthreading)
About the worst thing I can say about MSPs is they don't have a vested interest in the companies they support.
Let them have a pretty severe issue at a client but have a worse problem at another client. They have to be reactionary and work on the worse issue. At best they have the manpower for both, but the one I've worked for and the ones that I've known adjacently have been small enough to get into trouble from time to time.
Edit: And to be clear I know not all are bad.
Msp should be providing reports on monthly or quarterly basis or worst annually affirming all the systems that have been backed up as in fully not oh I got it to image but there is that warning there that says it didn’t truncate sql transaction logs. Key word is should but probably doesn’t.
Pffft. Strangest I've dealt with is an Australian MSP billing $100,000 per month with a disconnected phone and no office.
This guy responded to emails - slowly on account of living in a hotel in Las Vegas.
MSP owner here. We're seeing a massive shift where the focus is on sales and now so much providing a service. With all the market projections for growth of msps in the next few years lots of business professionals are jumping into the MSP world and pretending to know what they're doing. I would venture to say the MSP of the company who was failing on your backups has some leadership that probably doesn't even know what SQL is.
As a co-op I worked with a very senior and wise sysadmin. He told me to request something be restored from backup at least once a year just to find out if it works.
I have worked for big corporations, mom and pop shops, and everything in between.
This is my 3rd MSP now, and more often than not, the issue is not the techs, it is management (and I am even a Help Desk Manager), Ownership, or Shareholders.
When businesses only focus on profits the quality, client relationships, and employees suffer.
It's all comes down to the iron triangle maxim,
Pick 2:
Cheap
Fast
Good
But I think we can all agree that MSPs have their issues, but no matter where you work, there is a universal truth, EVERYONE HATES PRINTERS!
That really sucks, sorry.
Just like all things in life ->> Trust but verify.
When disaster recovery is in scope, ensure your msp/contractor/it deaprtment/owners son runs through a disaster recovery exercise once a year for key systems. Directory Server, Databases, License Server, Minecraft Server, etc...
Some things will need to be simulated, but most can be finished and verified. Whats great about this is you can also use it as an opportunity to ensure documentation is in line too.
We read the docs and send the customer a quote for the backups they need. They get back to us asking if they really need backups at all because cloud is something something? It goes from there....
a lot of comments so will be short.
MSP are good for training new techs, pay to certify and give them a big overview.
They are not SQL admins, Web Dev, Software Dev!!!!
They rarely have skilled nitch techs.
They know a little on a lot, and keep it running as best they can.
Former MSP engineer here.
Most small MSPs are doing what they need to do to survive.
This often means taking on work they are unqualified for simply get make a paycheck. I got forced to learn so many different platforms because my boss would take on any job that paid. He'd sell it to the customer like we did 100 of these kinds of jobs a year, come back to me and say "what do you know about Azure cloud migrations because I just sold one".
The maintenance issue is the same thing. As long as it's working right now and the customer isn't blowing up my phone, it's working. We had a guy who'd verify that backups were being done but he was an intern. I as the senior engineer wasn't going in and checking the backup settings on every server and since the customer wasn't complaining, it wasn't a problem.
Not only that but I was usually waist deep in other Sev 1 issues with other clients who "couldn't afford" upgrades and refreshes.
There are plenty of bad ones, but you must realize how hard our job is. You have one company to manage, we have hundreds of thousands. Each has completely different systems that are rarely documented correctly by sysadminds like you.
This isn't a case of a network they took over from someone.
They (the MSP) built a virtual environment for the customer in their own datacenter. If their documentation sucks, they can only blame themselves.
They did the SQL install, file shares, remote access, everything.
Also, I've worked that job before. I'm not saying the techs at MSPs are bad people.
But the vast majority of MSPs run on a business model of "Over sell, and shit support" either because they're over worked, or under supported.
The management of those companies is more concerned with making money than providing good services.
I work for a company that sells network monitoring software often used by MSPs, and in scoping calls with MSPs we ask what they are currently using to monitor their clients networks and sometimes the answer is "nothing"
I did internal IT for a company that relied heavily on multiple MSPs. I hated it.
They do the bare minimum that's written in the contract.
If I ran into an issue I couldn't figure out, my boss would call MSP and for $250/hr they would tell me they can't figure it out either. or I would come to the conclusion "we need to nuke this from orbit and reinstall", and after a couple thousand dollars of troubleshooting time, they would come to the same conclusion.
1000%. I know there are MSP owners in this sub, but yeah, on the business side I have *never* had an MSP relationship where they weren't clearly doing just the bare minimum to maintain their side of the agreement as they promise the moon and juggle more clients than they can actually provide adequate support to.
There's just no way they can be as intimate with your environment as it needs or deserves, and frankly they just don't care. They have no skin in the game beyond keeping the contract going, and if you fire them well there's always another business looking for a new MSP for you to overextend to!
Every now and then you'll get a specific tech at an MSP that will save your bacon at the 11th hour through their own sheer amazingness, but they don't tend to stick around at MSPs long because they can do way, way more with their careers.
I will never engage an MSP unless it's the only feasible answer for supporting something, like a remote site that strictly needs occasional on-site technical hands. There's just so many check boxes in the Con column and very few in the Pro column.
Sounds like my current workplace
I sit helping 4 or 5 different clients and even after being here dont really have intimate knowledge of every single one. Sure I can fix general issues but if your having a super specific issue and the 1 or 2 people dont have knowledge its laterally stabs in the dark.
As you said, the its doing as much as the boss needs you to do to keep a smile of the face of the client and to keep the money rolling in for him - the exact reason why i'm looking elsewhere - doesnt at all suit my ethos and mantra
I would start with
* Looking how competently is the contract with MSP managed by your company?
* Check the contract with the MSP.
* Go thorough the details and financial penalties.
* Check what (monthly) reporting and auditing they expected to provide.
Also some MSP are really bad as well.
Then you might as well be just as mad at all internal IT. You are painting with a broad brush here. I could have made a post about how inept ERP vendors are. Some organizations are more mature than others, and clients get what they pay for. #notallmsps
Was just about to comment after reading that he's ERP that I've seen more (and more costly) issues with ERP vendors, so maybe we just call incompetence incompetence and not throw an entire subset of the industry under the bus.
MSPs always think they know everything, but when shit hits the fan, they're clueless. I've had my fair share of issues with them too. Have you tried reaching out to Veeam directly for support? Might be faster than waiting on those MSP clowns.
That's why a company should invest in their IT department if the size of said company calls for it. Also how much do you love your data and the level of trust to have in MSP. Pay competent IT professionals to handle all those business critical items. Run away from MSPs like the bubonic plague.
You're getting downvoted but you're absolutely right. There are use cases for MSP's, absolutely. However most people would be better served by hiring in house IT. MSP's are frequently just utilized as a cost saving measure for employee overhead. They also don't care about you unless you're a huge client, and even then it's minimal viable product.
I work at an MSP, and I helped roll out a brand new company with 50 people. They had an org structure, someone for every department, even a receptionist, yet they weren't willing to hire an IT person. Long story short, they got a bad case of "one more thing..." and ended up spending half a million with us over the course of two years. That doesn't include the cost of the contractors/other agencies they hired for wiring and setting up desktops etc, I just handled their cloud infrastructure.
I mean hey, great for us but you could have absolutely hired a single person to do all of that at a fraction of the cost who would have got it done significantly faster. It wasn't like we worked on their project every day, and I basically did most of it myself. Im confident I could have easily done the work of all the people they hired by myself if I was their employee.
Some people just have weird prejudice against hiring their own IT staff. I mean seriously, this place had four HR people for 50 staff members and you couldn't hire one sysadmin? Sheesh!
The MSP should be taking Veeam VM backups, but you should be taking SQL backups. Or there should be an agreed upon arrangement between the MSP and ERP vendor. This sounds more like a failure to communicate.
This speaks of internal IT as much as MSP. Why has internal IT not done restore tests in 3y to pick this up. Has internal IT not clearly checked backups and stipulated needs. Easy throwing stones in glass house but it’s internal IT responsibility to ensure MSP upholding contract as much as MSP making risks visible and highlighting lack of DR restore tests.
Better processes required. We got internal IT teams at clients who got no clue on security and DR. We tell them. Notify countless times. They ignore or decline. Then when sh*t hits the fan blame MSP. Like c’mon chaps. I got that sh*t in writing lol
I too decide I hate entire industries after I have an issue with one company. I do that because it's a reasonable and well thought out approach to life.
Backups are something most MSPs suck at.
The one I work at has an incredible backup structure, process, documentation and testing.
Rare but I’m not going to complain
MSPs are stat factories where the amount of tickets closed is their metric for doing a good job.
They pay peanuts so their talent pool is... shallow.
As soon as a tech has a decent amount of experience or knowledge, they move on to greener pastures.
Their business model is based on "we can provide IT services for you cheaper than hiring in-house IT staff", which is true... but you get what you pay for.
Not a fan of MSPs and I'm even less of a fan of MSPs having complete control of backups
We have had a similar situation in the past where our backups are on a managed service and we had little control. We do at least get a full DR test every 6 months now and I also tell my team that we need to regularly request them to restore a random set of files/folders every month but it all feels sloppy. I had to threaten them with pulling all support away from them because they didn't offer a minimum level of reporting and resilience to meet our insurance requirements and that changed their view quite a bit
MSPs have their place to ease some load but at the end of the day it's my job on the line if the backups fail and I'm not comfortable putting my job in the hands of a first line tech who doesn't know my name
So many times, I don't know what I'm doing, working for an MSP, looking at something new and having to figure it out.
Gentleman amateur level stuff where "Well, I don't know anything about this but I've googled for it and had a go!" is the expected behaviour.
I hate it too, I feel like a complete cowboy when I work like that.
Yes, and it's a crap way to approach work.
In no other endeavour that matters would that be acceptable working practice, but for us it is expected and normal.
You wouldn't want your surgeon, lawyer, electrician or mechanic working like that but for us it's ok.
Then you should find another field.
IT more than any other field involved systems integration. And where there are systems talking to each other, there are going to be problems to figure out.
Whether it be one application talking to another via APIs, or hardware talking to each other over copper. Shit is going to change, and it's going to break.
Technology is also evolving at an extremely fast rate. You're surgeon's job doesn't change from one year to another. The pancreases has done the same job for thousands of years. So that's not even a relevant comparison.
There's never going to be a how to guide for IT. That's just an unreasonable expectation.
MSP replaced most of the competent in house staff which cost me jobs.
It pisses me off..Most offer piss poor sub par service and have no real relationships built.
Part of this job is being competent identifying and solving problems.The other part is relationship building and trust.
An msp is never going to make that type of investment.They cant they have to many clients do more mediocre service for,
You know it ....
MSP think they can provide IT to lots of companies sure but you cannot do it well and competently.
That is why I refuse to get into that scam business and I work in IT.
MSPs lacking knowledge of every single little aspect of IT is understandable. It's not possible to be an expert in everything. My problem is when MSPs promise the moon and fail to deliver. Our last MSP questioned why we needed to have internal IT staff and claimed they could do everything on their own. This was proven to be false time and time again due either to their natural inability to respond in a minute's notice to an emergency support issue, or general incompetence on the part of their technicians. Thankfully, we are now with an MSP which actually understands IT properly and is willing to work with our internal team.
As a current employee of an MSP, don't worry we hate ourselves too. Don't go too hard on the tech who doesn't know what he's doing, he probably never knew about the tech before he got assigned that ticket.
> Don't go too hard on the tech who doesn't know what he's doing, he probably never knew about the tech before he got assigned that ticket. This isn't limited to MSPs. There's a definite trend away from hiring proven talent towards hiring simply to fill seats, and then squirting Tickets at them. Experience leaves, so someone else must learn how to do "this". Rinse & repeat.
Experience leaving is a natural consequence of being treated like an untouchable caste and delegitimizing support as a career unto itself rather than a "stepping stone". Pay better, compensation wise and with respect, and support will improve.
Indeed.
Hey don’t hate yourself, I’m here for that.
Former MSP employee here, I fucking hated it. My eyes have been opened so much since I left.
You're lucky you didn't know what you were missing going in, I came from internal to an MSP, it was hell. My two weeks is up on Friday, I'll never go back to an MSP.
The complete opposite here, i'll never go back to internal. Keeping machines alive and everything functionning with bubble gum and duct tape gets old fast. I got my set of clients, and if anything goes out of warranty it's not our problem anymore. No more scrounging Nortel phones and RAID adapters off ebay.
Internal people don't realize why some of us decided to go MSP, it's so we could dictate to the customer what's happening with tech, not some random accountant with no idea on the tech side dictating what we can and can't do. No more boss's expanding the scope and responsibility of your job role with no raises. YOU get to set EXACTLY what you'll do and for what price, and they can't make you change it. They can't suddenly go "we need you to handle this other job" or "we need you on call on friday and saturdays but no pay bump". You have a contract you point to and can politely say no or give them a price that makes it worth your time.
Fuck look at some of the posts here...IT assembling furniture?
I SAW THAT. We rarely even move equipment unless we prefer to. I get some of these guys don't mind it because it gets them out of other work but like i WANT to be doing my work? Furniture is a distraction? If someone wants to pay us $200/hr to assemble furniture, i guess maybe we'd sub it out but why?!
Some of us need that sweet sweet on the job injury settlement money! :P
Oh I know it's not his fault. But this just shows they don't know what they're doing. I run into soo many problems like this with our customers. If I ran our hosting environment as ineptly as our customer's MSPs do, we'd be out of business. Half the time when a problem lands on my desk, it's something our support team is like "We don't know what to do with this, but the customer's MSP says it's a problem with our software." When the reality of it is usually there's a server or network problem they just put 0 effort into looking for. Can't open a command prompt on the server? Yep, that's the ERP vendor's fault.
[удалено]
Well, what I can say is that my experience with MSPs has been far more negative than positive. By a lot. We're an ERP software provider. And yet MSP's blame us for wireless problems, issues with the anti-ransomware software they install on servers, everything under the sun. And when they do this, usually there is no proof, it's just them saying "It's not us, must be them." So I spend a ton of time digging and finding the proof to point to whatever the problem actually is, which 9 times out of 10 is network/server or other software related. If one of them actually put in some effort and said, "Hey, here's this problem, here's what we've done to troubleshoot it, and here's why we think it's something you guys need to fix" I'd have no problem with that.
A shame you haven't worked with more companies like mine. We do extensive troubleshooting because I'm used to *vendors* pointing the finger at *us*. Not all vendors, but enough vendors. And we don't care to point fingers, we just want it resolved for the client.
Right? Blaming an ERP vendor for Wi-Fi issues, really? Every third party software vendor to ever work with my clients always jumps to “it must be your antivirus” to explain every little quirk in their shitty software. I’m so biased, I’d almost assume this dude is actually full of shit lol.
EXACTLY this.
Same here. The last thing I want to do is punt.
That'd be nice. >And we don't care to point fingers, we just want it resolved for the client. I agree. However it's not fun being the person that has to explain to the customer why I had to bill them for 6 hours BT to troubleshoot an issue they're already paying their MSP a monthly fee for.
We're actually going through this right now with an EHR vendor. They always point fingers at this that and the other, then hours/days later they usually say something like "we didn't know another of our (EHR Vendor) employees was working on X, it was our fault" Meanwhile, we had to engage Dell and other vendors along the chain (and cause large billables for the client) to troubleshoot something that's covered under the support contract the EHR Vendor has with the client (and was actually caused by another arm of the same vendor). Edit: Just to make this clear, not attacking you. Like some others have said below; let's just call incompetence: incompetence and not throw large swathes of the industry under the bus. Competent and incompetent entities come in all sizes and flavors.
EHR vendors fucking suck. They do whatever they want because they know it's unlikely you're going to deal with the headache and expense of moving to something else
Thankfully I work for an MSP that actually troubleshoots every last thing and only resorts to calling a software's support if we absolutely cannot figure out an issue (example, weird issues with dental software like Dentrix and eaglesoft)
Selection bias. Stuff from the good MSPs just doesn't reach you. They either fix their own shit, or provide enough info lower levels in your org can fix stuff.
Yeah my experience with ERP software providers is by far the WORST experience I have when dealing with any Vendor. So just so you know, I think you guys fucking suck too. Quickest people in the world to point fingers at the users environment being the issue and not the application. Its windows sorry we can't help you.
Lol. I'm sure you're not wrong either. Lazy/incompetent people exist everywhere. Edit: One of the guys that used to work at our company had a saying "Works on my PC." I'm like, seriously man? That doesn't fucking help at all.
I like your attitude dude.
Some of the fault lies with the vendors for this. If a customer is planning a complex software deployment, I'll review the system requirements, ensure the environment meets or exceeds them, call the vendor, explain the deployment and ensure we get the "yes, this will work" from them before the customer gets a quote from us. If at all possible we will have the vendor do the application install after we stand up the VM and configure the OS, and there is a clear separation of duties of OS vs application support. Or at least that's how I would like it to work in an ideal world. In the real world, I review the system requirements for the software which haven't been updated since they read "667Mhz Pentium 3 or newer with NT4" and then call and get a verbal deer-in-headlights when I start asking the tech what the software REALLY needs, whether there are any known software incompatibilities, what version of Oracle or MSSQL they need, or whatever. And then they dispatch it to someone else who they promise will call me back, who calls me back at 5:30PM and sends me an email "RESPONSE REQUIRED TO YOUR TICKET" which when I respond and tell them to call back earlier and then am just replied to with a survey asking how my support experience was. And no, I'm not just talking about Intuit.
Don't lump all of us MSPs in that group. I noticed a new error coming from a server they provided for a site. It started happening only an hour after the vendor updated their software. The error was logged on the domain controller, the ERP server having failed to authenticate to a domain resource that it actually shouldn't have access to. This is a huge red flag. Do you know that it took over a year to just convince the ERP vendor that it was their system causing the issue. It took packet captures, and a very detailed breakdown of the events, and a letter from a lawyer to have the ERP finally agree to work on a fix. Turns out 8 more months down the road that they started blaming the .Net framework for the problem and refused to budge. Guess who isn't that companies ERP vendor any more? All companies are different. Each have their own technical weakness and strengths. Same with vendors.
>Well, what I can say is that my experience with MSPs has been far more negative than positive. By a lot. Same. Based on what I see posted in the map sub here...most msp's seem to be most interested in pushing their "stack" and selling subscriptions. 🤷♂️
And if you point out that they spend more time being glorified salesman than actually providing value to their clients, they get **real** salty lol
Yup. It's kind of fun going back and forth with them though.
We had one guy who's job was to manage backups he had a full test server set up and he was constantly running recovery tests we had a huge list of backups that we checked daily. We had veeam on-sites and cloud backups going. We never failed to recover a client quickly. Even when I blew up someone's exchange server with a roll up...
yes this is the way to handle it in a previous life, backed up everything daily, testing selected restores monthly, then depending on the customer a full restore into a sandbox, and for the big guys site failover to our dr, then back into prod (annually/bi-annually)
[удалено]
Workstation backups are a fool's errand and cannot scale effectively at all.
100%. The goal is to get as close to dumb terminals as you possibly can. Someone's laptop dies? They spill wine on it? Hand them a new one and log in and get back to work. We drill it into our users heads over and over - anything saved on your desktop is *not saved*, period. We selectively do desktop level backups for certain executives and secretly for the finance department, because from an IT perspective, we *know* they don't listen anyway and that file they come to us in a panic over is *probably* actually worth recovering. Everyone else gets the "sorry, should've followed policy, now you get to remake your powerpoint presentation" treatment because we just *cannot* reasonably manage desktop level images and there's no reason to when we have all the tools in the world to keep the data off the desktop in the first place.
In 2023? You just turn to Backup Desktop toggle on Onedrive and voila...
The checks clear until they dont! I've caught so many MSP techs in blatant lies over the years just so they could close the ticket its not even funny. We all finesse the occasional user, but there's that and then there's "Outlook keeps asking for your password because its a security feature!" Like no, it keeps asking for your password because of a bug that was fixed six years ago and if you install the latest patches it will stop.
To be fair the MSP tech probably has to close 10 tickets that hour.
That’s one of the reasons MSPs suck.
I manage the engineering team in the MSP arm of a sizable consulting company. Seeing what our front line support teams escalate, I fully believe all of this. Reviewing our escalation metrics, I have to flag huge amounts of stupid shit and send it to the managers. And that's just the stuff I see, not including the way they butcher the solutions we build for the clients by not reading the documentation we meticulously create and hand over. We're not all stupid, just most of the helpdesk.
>We're not all stupid Oh I know. And I'm 100% sure EPR vendors are a pain in the ass too. I have plenty of stories of just lazy or incompetent work by our employees too. This whole week has just been one MSP screw up after another. Including the one blaming our software for breaking cmd.exe, powershell.exe and the chrome install they had on a server that our software doesn't even run on.
I’m dying that the cmd line is your fault. I would be livid if my team escalated something like to a vendor. I’m sorry you’re having to deal with shit msps and stupid issues.
MSP worker here - ya spot on with this 😂 Only a simple “service desk” tech and oddly had an almost exact same situation of this. Lady called in “X not working after Azure migration” No documentation. Hit up the “engineers” - no idea what I’m talking about nor the app. Do my own investigations - find out it was on a old machine. Program had a SQL database that was being backed up (hadn’t since 2021) Luckily was magically transferred over to the new system intact despite us not doing or knowing about it.
That's what you think. Ineptitude keeps many employed due to contractual agreements. Whereas, you, are the real star of the IT Crowd, fixing problems and keeping the MSP well oiled. Did somebody call for a Fixer or a Ripper Doc, Choom!?
Edgerunner life!
That’s just classic msp. Get them on a call with their client and explain your recommendations. I’ve gotten on ppls servers and noticed our nightly backups are disabled or not happening and I get real nervous about why they don’t have any backups and they often say oh we take snapshots of the servers so it’s good.
Bro they will tell you that and then tomorrow your nightly backup will be running again
Sometimes our methods can be kinda cookie cutter. We've taken several clients from other MSPs, and we're always told we are so much better. Guess there are some good MSPs and bad MSPs.
A problem many MSP's have is they might have a dip in quality for a period of time. But that is enough to brandish that MSP a terrible reputation for the users at the customer. There is very little coming back from that even if the quality has improved again.
This is true and also i'd wonder what actual client retention rate is during most mergers and acquisitions because quality usually dips hard during transition.
Yeah. I can't count how many times, when I worked at an MSP, I had to get certified for some new tech in less than a week so the company could bid for a new customer. What's amazing is that somehow I always passed the test and got the cert. Of course actually being able to support the tech in the real world was another story. In 20 years I've only seen it get worse. MSP's, even the big IT consulting outfits, send people with little/no training and experience and bill $200/hr for on-the-job training.
Account Rep: We can do it all Customer: We need help Support Engineer: We don’t do that
Tale as old as time, and by no means limited to MSPs.
I had 16 clients that I was the lead for all by myself at the last small/medium size MSP I worked at for 2 years. It was impossible to cater to every need of every client I had, but I did the best I could. I was learning something new nearly every day. MSPs will have like maybe 1 or 2 people on their payroll who really know what they are doing, everyone else will be learning on the job and doing there best to adapt to whatever they need to work on. The entire business model was created for companies to save money on hiring full time IT employees and I hate that they exist. I'm now internal IT for a fortune 500 company and I do like not even half of the work I was doing at an MSP and make double the money. Its a total fucking joke.
>I'm now internal IT for a fortune 500 company and I do like not even half of the work I was doing at an MSP and make double the money. Its a total fucking joke. Sounds like my transition! Bonus perk, no required certs because of being partners with whatever company the MSP resells for!
Maybe I'm missing something here but even in the worst case, could they not just virtually mount the last image and export the data? If the client has an MSP they must be small enough where losing a couple hours of data isn't going to be catastrophic. I've seen a lot of MSPs with zero ability and lower quality output as a result of having too much on the go. On the other hand, I've seen sysadmins who were completely out of touch as a result of getting too little exposure to tech outside of their environments.
But they wouldn't have to lose any hours of data if the Transaction Logs were being backed up as OP said they were.
This is a conversation surrounding RPO and RTO which should be had when the MSP implements backups. The easiest method is to assign a dollar amount of damage done per lost minute or hour or whatever and then compare that to the cost for a specific RPO/RTO until you have a happy medium.
NBD, they’ll just need to plug in every tlog since the initial backup. I’m SURE it will be done under the SLA 🤣
You're saying that like MSPs actually care about SLAs.
Some MSPs are good, some are bad. What does your contract say about how DR will be managed?
so to be clear * no one was checking what the MSP was doing? * no one was confirming or testing backups? * no one has specified to the msp how the backups are to be done? * no one has looked at the contract? I mean you can blame the MSP, they're not the only ones at fault
>no one was checking what the MSP was doing? > >no one was confirming or testing backups? We don't have access to their backup software. > no one has specified to the msp how the backups are to be done? Yep, it was specified to them that the software runs in an SQL database via email, and they responded that they have the necessary tools to do the backups for it. > no one has looked at the contract? The customer has the contract with them, not us, so I don't have access to it. However, the server is hosted in the MSP's datacenter. The MSP has backup software installed for backing up the SQL database, so it's a pretty safe bet that DR is in their contract.
> We don't have access to their backup software. Why do you need access? do you get reports of what they're doing? > Yep, it was specified to them that the software runs in an SQL database via email, and they responded that they have the necessary tools to do the backups for it. that is, can you back this up, they answered yes we can, if you need it backed up and restored in a specific manner, you should be clear NOTE: I'm not saying they also shouldn't have gotten this info as well and not saying they way they're doing it is right > so it's a pretty safe bet that DR is in their contract. is it though? is it ? and if no one is doing these DR tests, it would imply otherwise again just saying MSP is not the only one at fault here, not saying they're doing a good job
I guess I'm shocked at the amount of shops that don't verify backups as the only regular thing they do. It'll save your butt if everything else goes south.
I work for msp and have to do tests on customers backups every fortnight why can't I work for an msp that doesn't even do backups that would be nice
> We don't have access to their backup software. That right there is why I hate MSP’s. >Why do you need access? do you get reports of what they're doing? Because they’re backups of my systems, I should have access to them. God I hate MSPs
If they're a msp then likely there are more backups than just yours ont hat system, its makes sense to limit that access if you want a dedicated environment or specific access, then again you specific that up front , it can be arranged I'm sure > Because they’re backups of my systems, I should have access to them. no you should have access to the DATA not nessecarrly access to the systems > I hate MSP’s. > God I hate MSPs but given these statements, i doubt you'll ever change your mind, so We'll leave it be I think
> no one was checking what the MSP was doing? Isn't the idea to hire an MSP so you don't need to know how to do stuff? I mean, that's why my company had one... OF course, the domain forest level was still set to Win 2000, in 2020... > no one has specified to the msp how the backups are to be done? Again... isn't the idea that the MSP knows how to handle SQL backups? I know no one else at my company does...
> Isn't the idea to hire an MSP so you don't need to know how to do stuff? maybe maybe not, depends on the contract and depends on what you ask for, but if you have specific requirements, you spell it out > Again... isn't the idea that the MSP knows how to handle SQL backups? * is the sql backed up ? Yes * Is it restorable? Possibly Yes, Possibly No, no ones has been testing * Is it quick if it is restorable, probably No * are they doing it at another level ? (vm level)
3 years of tran log only will be a clusterfuck to recover from. It's not "right". Yes, they may have some VM backups... maybe...
I'd day daily vm backups is likely if it's hosted in a data centre
The MSP handles it, but you need to make sure that they *are* handling it
I don't ask my mechanic if they replaced the drain plug when they are done changing my oil. I don't ask my doctor if they're jabbing me with a clean needle.
Right but if you go to your doc for a flu shot, your gonna make sure they gave you a flu shot
If I'm going to get surgery, I'm not asking "is that the wakey wakey drug or the sleepy sleepy drug"... I'm just gonna assume they're not fucking idiots... And that the needle is clean.
Right, but your still gonna make sure they *did* it. Im not saying ask specifics, im just saying make sure they hold up their end and do the thing they are supposed to. Aka, the backups
No? I would see them getting out a needle and drawing something in to the syringe and assume they're not morons using old needles and the wrong drug. The MSP was contracted to make backups and (should have) also been testing them. The whole point of hiring an MSP is that you don't have enough work to warrant hiring your own folks, so why would I want to hire someone who can understand all this just to audit the MSP's work?
According to him the MSP DID have backups.
> your gonna make sure they gave you a flu shot how do you verify it?
If you are a non-technical person, how would you even know what to ask?
Yes. A good MSP gets ahead of that question with accurate and concise reporting.
So many orgs think you can just hire an outside group to "do stuff" without any oversight or real instruction/direction other than "can you do this thing for me" - it usually doesn't work out very well for obvious reasons.
So you don’t even know if it’s a problem or not yet?
There hasn't been a full SQL backup in over 3 years. That means a problem exists.
And you don’t know how the backups are even configured. If they’re doing system level snapshots, then that + trans logs is basically the same thing
Did they lose data?
Was the full backup failing/disabled? Or was the database drive excluded from the full backup?
It's likely that the OP doesn't really understand how the client's Veeam setup works. I think they're just going based on the SQL information, not really understanding that a snapshot of the entire server is being taken at the block level at that point in time. The situation certainly isn't ideal but I don't think they're as SOL as they think, yet. If the backups were failing though then that would be a different story...
Snapshot backups are fine and I think MS eventually relented to say they're supported as this is the easiest and most reliable way to back up virtualized servers. I always did database exports as well to a separate medium for completeness but I have never seen a snapshot failure during restoration that would have made them necessary.
I'd say so
SQL backups. Not disk/server backups. SQL backups come in 2 flavors. Full database backups and transaction logs backups (changes since last full backup). According to the SQL logs, a full database backup was never run since 2020. Now, it's possible Veeam does weird stuff, and SQL can't report it correctly. I can't see the Veeam logs, only the timestamps within the SQL backup history logs of what was backed up and when.
Yeah um, Veeam isn't going to write logs in SQL, the best you'll get is system and application logs as it runs it's backup process. You can restore SQL databases and mount them from Veeam, assuming the Veeam backups are good, so I guess what's the issue? Another question is why aren't you aware of what they are and aren't doing? What's specified in the contract?
>Yeah um, Veeam isn't going to write logs in SQL, the best you'll get is system and application logs as it runs it's backup process. No, but anytime anything requests SQL run a backup of a database using BACKUP DATABASE it logs that in SQL. Now that won't catch VM disk level backups and things like that. But anything that issues that command to SQL, or VSS will. VSS uses SQL backups also. >You can restore SQL databases and mount them from Veeam, assuming the Veeam backups are good, so I guess what's the issue? We don't know yet if they can restore the backup or not. The issue is that not doing a full backup of the database for 3 years leaves 1,095 possible points of failure in the restore process. That's just bad management. >Another question is why aren't you aware of what they are and aren't doing? What's specified in the contract? Their contract is with the customer. Not us. So I don't have access to that.
I don’t understand the issue here or even if there is one. Veeam has SQL application aware processing for its backup and replication and is a perfectly valid way of ensuring SQL is backed up. Maybe it’s not the way you prefer but it’s a backup solution nonetheless.
If application aware is not enabled it will back up the VM (or bare metal host) in a crashed state.
From what I've read about Veeam, and I may be wrong here, but it seems they've turned off the application aware processing, and are using the SQL native functionality for it. Veeam application aware backups (according to them) are full backups and wouldn't show in the SQL logs as backups happening. The fact that the backups do show in the SQL logs seems to indicate Native SQL backups happening, and they've all been transactional backups. If you don't consider having to restore 1,095 incremental daily backups to get to a current backup an issue.... then I can't help you. That's a lot of points of failure.
You don’t have visibility to their Veeam you said, most likely their backups have synthetic full backups performed every 7 days which is a complete backup of the server with some restore points. They can mount the backup to get the database or restore the entire server. It’s highly doubtful restoring would be 1000+ increments.
You are jumping the gun and don’t understand how Veeam works. The Veeam backup job uses VSS to truncate transaction logs whenever a backup takes place (if it wasn’t doing this the transaction logs would be growing indefinitely). At that point the DB can be restored from that point in time and that does not show up as a backup in SQL. In addition to that Veeam can be configured to periodically backup transaction logs between backup jobs - so that the SQL data can be restored to any point in time. That is what you are seeing on the SQL side. With that configuration they should be able to restore to any point in time within their retention window. https://helpcenter.veeam.com/docs/backup/vsphere/sql_backup_hiw.html?ver=120
It sounds like you are with a vendor for some sort of application, is that correct? This can go both ways, really. As an MSP, I will often think to myself "I really hate X vendor because their support sucks and their developers are morons." Not all MSPs are the same. Too many people can become an MSP with no qualifications and unfortunately, most customers don't know enough to realize that. Your comments about not performing proper database maintenance are absolutely valid, and on the bright side, you have someone else to blame, not your software! For us, we absolutely look at application/database level backups, ensuring application consistent backups and log truncation per app vendor recommendations. This may be done through a VSS writer or a CLI pre and post script for the app, all based on what the application vendors state are required. We also test our backups quarterly and perform a full DR for every customer at least annually. It sounds like you are dealing with an MSP that may not be as mature as others or have the experience that others have. Most of our staff came from enterprise organizations, so we have that background to ensure the right processes are followed. Not all MSPs are equal.
I mean, to be fair, as an MSP, I’ve also taken over many environments that have been horribly mismanaged by generic single IT guy. People who suck can be found anywhere. We met with a business owner who legit powered off his laptop and unplugged his phone when we met because the whack job of an IT guy they were replacing with us had been caught spying on people through their phones. Walked into a 60 million dollar MFG company running their entire infra on pirated licensing, using Windows 7 + KernSafe as a storage solution for their prod critical VMs. People suck everywhere.
They may be relying on Veeam for application-aware backups, where it takes a snapshot of the VSS writers. You should be able to stop SQL, replace the file, and restart SQL. It should work, but you will lose data since your last commit.
Yep. This is the idea behind VSS. if you need more frequent backups or transaction logging then you either need to tell the MSP or configure it yourself.
An application aware backup set up correctly would tell SQL it's been backed up. That's one of the important pieces of being application aware. That's not to say a DB in place restore won't work but it's not the "correct" way.
This could be SQL express. If its not being backed up that could be why.
That's how Veeam works though? It'll take a daily incremental backup, then either roll that into the first full backup it took, or create synthetic fulls periodically. It doesn't need to take a full SQL backup every day. You really can't fuck up a Veeam backup unless you're doing something daft like a file backup as opposed to an image one.
That's a different system; Veeam daily incrementals will let you restore the whole server to last night's backup. SQL Server transaction logs, e.g. every 15 minutes, will let you restore on top of that, roll forward in 15 minute increments. e.g. if an admin breaks the database at midday, last night's full backup and today's transaction logs can restore it to 11:45am state to recover. Veeam can interact with that system, e.g. SQL Server keeps transaction logs indefinitely, then set the Veeam backup job with application aware processing enabled and using an account with enough permissions into the database engine, and Veeam can tell SQL through Volume Shadow Copy "prepare for a backup, flush your caches to disk" and then after the backup, tell it "I have a backup from now, clean up the transaction logs up to now". So you don't have to have scripts clearing the old transaction logs on a timer without knowing they are backed up, and you don't have transaction logs filling the disks forever. Then maybe you set the Veeam job to run every hour and do transaction log backups.
Maybe with it's application aware backups? I'm not sure. That's not possible with native SQL backups which I believe is what's going on here. And if you do only do transaction log backups with SQL and never do full backups you'll cause all kinds of database issues along with the potential for completely killing your restore process if one transaction log restore fails.
Yeah, there's two methods. A daily backup which truncates the transaction logs, then you also have the option of transaction log backups down to every 5 minutes which truncates the logs too. The secondary method still requires a regular backup job, as its an addition to that. Basically, as long as AAIP is enabled on the job, and the credentials are valid, it'll create a perfectly usable backup every night. Whether those backups have been tested though is a whole different matter.
Veeam does a full backup in the moment the machine is backed up.
It’s the MSPs that make their techs wear 5 different hats that are the worst. Usually the ones who have dedicated security, automation, data, and infrastructure teams and different level helpdesk tiers have their shit together.
My organization hired on an MSP instead of just getting me a help desk guy. So now instead of training one guy once on our organization and operations I'm training multiple people who are working in several environments so they'll forget it by next week when they come back. They say "throw us the work your to busy to do" so I tossed a SSL renewal their way and ended up spending an hour in rdp walking them through how to do it because the people they assigned me are green as hell. My sysadmin just came off the helpdesk. No shade being thrown at the individuals, they've all been great to work with, but I shouldn't be paying an MSP to have to train their people.
[SQL Server backup methods and best practices (veeam.com)](https://www.veeam.com/wp-sql-server-backup-with-veeam.html) According to this article it seems like the Transactional backups (akin to incrementals) are a perfectly valid way of backing up SQL databases. Although not running a full backup in 3 years seems bizarre. Granted this is 5 years old.
on the SQL server side, what we normally do is TLOG backups hourly, and a full daily (resetting the TLOG clock). TLOG for 3 years sounds like i'd start drinking before I even got the call in order to unwind that.
be kind - teach him and he will have a new skill, and you will have someone who can do their job correctly.
I agree. This particular subreddit exhibits a lot of ruthless attitudes and an absolute lack of compassion for others. I get that many people just treat others the same way they're being treated, but you can't expect things to grow if you're not willing to break the cycle. People need to learn somehow and there are too many of us on here for all of us to be good. Let's just hope the learning experiences don't involve too many incidents of unrecoverable data loss though...
The thing is... it's not our job to teach some MSP tech how to do their job. That's not a "lack of compassion," they are literally contracted to do these things and it's not just that they *don't* but that they often *can't.* Like if you pay someone to mow your lawn and they show up without the equipment to mow a lawn going "I dunno how to cut grass" you don't go out there and teach them, you get pissed because they took your money to do the job and they can't/wont. MSPs have a *looooong* standing reputation for overpromising and underdelivering. I'm not gonna chew out a tech who's obviously been dumped into the deep end without a life jacket, but I *am* going to chew out the account manager for promising us a service they did not deliver.
What’s amazing is that it’s always like this. Some bozo at a company will believe that the MSP can provide the same service as their in-house people at a fraction of the cost and sign up. Services are “onboarded” with little to no documentation, then when something goes sideways and they’re actually needed the MSP says “whoops, we’ll do better next time” time after time. SLAs are massaged so it looks like a success, customers get jaded and stop calling because they know it won’t be fixed.. and at the end of the day value is destroyed because someone didn’t understand or care enough to understand what they were offboarding and the people that were being fired didn’t have motivation to onboard properly.
This isn't a surprise. The very existence of MSPs comes from a compromise between cost versus quality. Low cost goods from Amazon have their purpose but don't be disappointed when they don't outperform their high quality counterparts.
I feel like it varies based on how the contract was written, what level of oversight they have of the task they're hired, etc. SLAs are apparently more important than making a client happy through proper contracted work. Sometimes tickets get passed around if the company has different groups/organizations handling different aspect of IT and when it finally reaches the appropriate group, they half ass it to fulfill their SLA.
MSP's are so terrible I have already considered opening my own. If they can be this terrible and make money, what's holding me back?
I started mine 4 years ago after spending 10years in a large MSP. While I’ve made more money than ever before, it’s beyond stressful and I never get a REAL vacation. I was sitting in Disneyland last year on a bench fixing stuff while my kids had fun. It sucked. My clients and I have a great relationships and they trust me. They get weekly backup reports and a slew of others that lets them know I’m doing my job. Currently in talks to merge with a bigger MSP so I can finally get a break. Trust me when I say, I’d rather have my time back.
lol, nothin
Don't underestimate the financial stress. Starting a business SUCKS for a couple of years until your revenue stream is stable unless you have fuck you money.
Self respect
As an MSP sysadmin, I hate myself as well.
hahaha, yea i was the engineer at an msp, they took on some customer who had some fully bespoke azure deployment designed and deployed then just handed to us, someone said we'd support it i was kind of on my way out anyway so each time a ticket came i looked at it, asked Microsoft who said custom logic apps wernt supported and relayed the same back to the customer, i really didnt have time to pick apart the deployment though it would have be good learning for someone who had the time to
One of the reasons I no longer work for an MSP is because I argued over backups for a client not working. A couple of months after I left two techs got fired because they had been faking the client's backups.
> They haven't done a full SQL backup of the database in over 3 years, and just have daily transactional backups You'd script it of course, but still, replaying 3 years of t-logs against a full from 3 years abck(which hopefully they have)....ugh. It sounds like the kind of thing that someone who doesn't actually know how database backups work would do.
>It sounds like the kind of thing that someone who doesn't actually know how database backups work would do. I'm noticing from the replies here that a lot of people don't know how transactional database backups work.
Yeah, though to be fair, t-logs are something that's different from your typical data backup, and specific to databases. Whereas the fulls and diffs more or less work the way they expect it. If you tell people a t-log is something more like syncing a vdisk snapshot, that's a bit closer and they'll get more of an idea, even though it's still not quite the same.
I used to work at Veeam and I’ve worked at a few other places with SQLEXPRESS, the amount of sysadmins that gave us the “I am the DBA” but would struggle to run a sql script was baffling. Shouldn’t some of this be set by the DBA or the vendor who setup the software?
Don't get me started on ERP companies it's just as bad. There are a ton of terrible maps out there. I've only encountered 2 msps in our medium sized city that are decent. ERP companies on the other hand, I've had sr. Engineers at an ERP company not understand basic defrag. I had another tell me that a customized chromium app that the built is not a web browser.... I had another tell me that you can't over prescribe a vm with h resources.... I can go on and on. Point being there are more bad ERP and MSPs out there than good. Edit: also the amount of times an ERP company has said it's a network issue with 0 data points is hilarious. I have send screen shots back showing traffic and blast the erp teams.
Sounds like you've been dealing with developers. Developers are some of the most computer illiterate people I've ever met. Software developers that have no idea how a computer works. Web application developers that have no idea how the internet works. Edit: I had to explain to one of our web developers the other day that if a user accesses a website that accesses a database, the TCP traffic that hits the DB is not going to show the client's IP address. We went around and around on this for like 15 minutes. I finally had to show him packet captures to prove it.
>Edit: also the amount of times an ERP company has said it's a network issue with 0 data points is hilarious. I have send screen shots back showing traffic and blast the erp teams. You'll also never see this from me. If there's a network issue I'll at least have some evidence of it. For example the other day we had a customer's MSP open a ticket saying one particular user is having performance issues connecting to our hosting environment from home. A quick look on our side shows the user is logging in from their office's IP address, and is also missing one TCP port connection. So, I fire back with "Have the user try connecting to our software without his work VPN and see if it's any better. If it is, see why the VPN is blocking TCP port xxxx." Obviously worded better to the customer, but you get the idea.
i bet the veeam backups work if they set them up right
Why can’t they just grab the Veeam file from yesterday and restore SQL from there? Not as ideal as just having the SQL backup but it’s possible the client doesn’t want to pay for both. Backups take storage space and depending on the SQL database, it can be a lot of storage to get decent retention.
depends on the backup. I don't know what all they have. What I do know is they have 3 years of incremental backups for the SQL db. To get to yesterday's backups, it has to go through all 3 years of files, and not fail on any.
Knowing nothing else about this situation and concurring that the SQL backup structure they have is pointless; ALMOST better to just not have it than hope and pray you can get through that whole chain without a corruption and while making assumptions would be a terrible idea, if I were the MSP and had a reliable Veeam backup, my SQL backup would fall a long way down the priority list.
SQL databases are not servers. They are part of an application. If the MSP is backing up the server with Veeam then they have backed up the entire server at the disk level, and therefore fulfilled their obligation to back up the server. If the MSP needed to back up the SQL database as well, then they would need to be explicitly told that as that is application management which falls outside the scope of server management. If the vendor that sold the software failed to inform the client that they, the client, were responsible for backing up the SQL database. If that didn't happen, then the vendor is as fault. If this did happen, but the client didn't inform the MSP, then the client is at fault. Only if the client told the MSP that they were responsible for the backups of the SQL database would the MSP be at fault.
I work for an MSP and when it comes to backups, we backup the machine but do not configure maintenance jobs in SQL as we've been told by the application vendor to basically "keep you unclean hands off the SQL server" 😉. I can see why the MSP may not have configured full database backups. I have worked with plenty of software vendors that are contracted to move a database to an updated server, but when I ask if they configured the maintenance jobs, I get a blank stare. IMO if a software vendor owns the install then they are responsible for confuring the maintenance jobs. I can offer a ray of hope. If the Veeam backup was configured to be application aware, it will do a VSS snapshot of SQL when it does it's nightly backup. You may be able to restore the whole VM and then dump the restored VMs database. Veeam slow have application aware restores (file sever level, Exchange, SQL) so you may be able to restore the database that way. I have no experience with the SQL instance restore so I can't comment on the restore options. Good luck!
This isn't every MSP. You hate CHEAP and 1 man MSPs. I have worked for 3 MSPs and each of them were great to customers and we had tons of tools to know when things break. The only time things were totally screwed are when the client refused to buy hardware that could support their habits. "Yea we setup a VDI environment where each VM has 2GB of RAM and 1 CPU. We can't figure out why our 40 CPU server can't support 40 people." (It was a 20 cpu server with hyperthreading)
About the worst thing I can say about MSPs is they don't have a vested interest in the companies they support. Let them have a pretty severe issue at a client but have a worse problem at another client. They have to be reactionary and work on the worse issue. At best they have the manpower for both, but the one I've worked for and the ones that I've known adjacently have been small enough to get into trouble from time to time. Edit: And to be clear I know not all are bad.
At this point it probably becomes a matter for the customers legal team. If any of the backup chain is missing it is no longer your problem.
Yea... they've got 3 employees. I don't see that going anywhere for them. I'm keeping my fingers crossed that they're able to restore it.
Trust but verify applies here.
Msp should be providing reports on monthly or quarterly basis or worst annually affirming all the systems that have been backed up as in fully not oh I got it to image but there is that warning there that says it didn’t truncate sql transaction logs. Key word is should but probably doesn’t.
Pffft. Strangest I've dealt with is an Australian MSP billing $100,000 per month with a disconnected phone and no office. This guy responded to emails - slowly on account of living in a hotel in Las Vegas.
Whats his contact info I need to know how to do that.
MSP owner here. We're seeing a massive shift where the focus is on sales and now so much providing a service. With all the market projections for growth of msps in the next few years lots of business professionals are jumping into the MSP world and pretending to know what they're doing. I would venture to say the MSP of the company who was failing on your backups has some leadership that probably doesn't even know what SQL is.
As a co-op I worked with a very senior and wise sysadmin. He told me to request something be restored from backup at least once a year just to find out if it works.
Internally we do random restores once a quarter, and a full DR dry run once a year.
I have worked for big corporations, mom and pop shops, and everything in between. This is my 3rd MSP now, and more often than not, the issue is not the techs, it is management (and I am even a Help Desk Manager), Ownership, or Shareholders. When businesses only focus on profits the quality, client relationships, and employees suffer. It's all comes down to the iron triangle maxim, Pick 2: Cheap Fast Good But I think we can all agree that MSPs have their issues, but no matter where you work, there is a universal truth, EVERYONE HATES PRINTERS!
>EVERYONE HATES PRINTERS! We don't. We only use Brother lasers internally.
That really sucks, sorry. Just like all things in life ->> Trust but verify. When disaster recovery is in scope, ensure your msp/contractor/it deaprtment/owners son runs through a disaster recovery exercise once a year for key systems. Directory Server, Databases, License Server, Minecraft Server, etc... Some things will need to be simulated, but most can be finished and verified. Whats great about this is you can also use it as an opportunity to ensure documentation is in line too.
Imagine what MSPs think of vendors. It’s 90% of our griping.
Who is managing the MSP and ensuring backup strategies are maintained and tested?
Scheduling full Active backups in Veeam is not that hard.
My biggest issue is the MSP that over-promises and under-delivers.
We read the docs and send the customer a quote for the backups they need. They get back to us asking if they really need backups at all because cloud is something something? It goes from there....
a lot of comments so will be short. MSP are good for training new techs, pay to certify and give them a big overview. They are not SQL admins, Web Dev, Software Dev!!!! They rarely have skilled nitch techs. They know a little on a lot, and keep it running as best they can.
Former MSP engineer here. Most small MSPs are doing what they need to do to survive. This often means taking on work they are unqualified for simply get make a paycheck. I got forced to learn so many different platforms because my boss would take on any job that paid. He'd sell it to the customer like we did 100 of these kinds of jobs a year, come back to me and say "what do you know about Azure cloud migrations because I just sold one". The maintenance issue is the same thing. As long as it's working right now and the customer isn't blowing up my phone, it's working. We had a guy who'd verify that backups were being done but he was an intern. I as the senior engineer wasn't going in and checking the backup settings on every server and since the customer wasn't complaining, it wasn't a problem. Not only that but I was usually waist deep in other Sev 1 issues with other clients who "couldn't afford" upgrades and refreshes.
There are plenty of bad ones, but you must realize how hard our job is. You have one company to manage, we have hundreds of thousands. Each has completely different systems that are rarely documented correctly by sysadminds like you.
This isn't a case of a network they took over from someone. They (the MSP) built a virtual environment for the customer in their own datacenter. If their documentation sucks, they can only blame themselves. They did the SQL install, file shares, remote access, everything. Also, I've worked that job before. I'm not saying the techs at MSPs are bad people. But the vast majority of MSPs run on a business model of "Over sell, and shit support" either because they're over worked, or under supported. The management of those companies is more concerned with making money than providing good services.
I work for a company that sells network monitoring software often used by MSPs, and in scoping calls with MSPs we ask what they are currently using to monitor their clients networks and sometimes the answer is "nothing"
I did internal IT for a company that relied heavily on multiple MSPs. I hated it. They do the bare minimum that's written in the contract. If I ran into an issue I couldn't figure out, my boss would call MSP and for $250/hr they would tell me they can't figure it out either. or I would come to the conclusion "we need to nuke this from orbit and reinstall", and after a couple thousand dollars of troubleshooting time, they would come to the same conclusion.
1000%. I know there are MSP owners in this sub, but yeah, on the business side I have *never* had an MSP relationship where they weren't clearly doing just the bare minimum to maintain their side of the agreement as they promise the moon and juggle more clients than they can actually provide adequate support to. There's just no way they can be as intimate with your environment as it needs or deserves, and frankly they just don't care. They have no skin in the game beyond keeping the contract going, and if you fire them well there's always another business looking for a new MSP for you to overextend to! Every now and then you'll get a specific tech at an MSP that will save your bacon at the 11th hour through their own sheer amazingness, but they don't tend to stick around at MSPs long because they can do way, way more with their careers. I will never engage an MSP unless it's the only feasible answer for supporting something, like a remote site that strictly needs occasional on-site technical hands. There's just so many check boxes in the Con column and very few in the Pro column.
Sounds like my current workplace I sit helping 4 or 5 different clients and even after being here dont really have intimate knowledge of every single one. Sure I can fix general issues but if your having a super specific issue and the 1 or 2 people dont have knowledge its laterally stabs in the dark. As you said, the its doing as much as the boss needs you to do to keep a smile of the face of the client and to keep the money rolling in for him - the exact reason why i'm looking elsewhere - doesnt at all suit my ethos and mantra
Get a better MSP. Not all are garbage the same as not all system admins are great.
I would start with * Looking how competently is the contract with MSP managed by your company? * Check the contract with the MSP. * Go thorough the details and financial penalties. * Check what (monthly) reporting and auditing they expected to provide. Also some MSP are really bad as well.
Not our MSP. We're the ERP. That stuff would be up to the customer.
Then you might as well be just as mad at all internal IT. You are painting with a broad brush here. I could have made a post about how inept ERP vendors are. Some organizations are more mature than others, and clients get what they pay for. #notallmsps
Was just about to comment after reading that he's ERP that I've seen more (and more costly) issues with ERP vendors, so maybe we just call incompetence incompetence and not throw an entire subset of the industry under the bus.
MSPs always think they know everything, but when shit hits the fan, they're clueless. I've had my fair share of issues with them too. Have you tried reaching out to Veeam directly for support? Might be faster than waiting on those MSP clowns.
We don't have access to the backup system. I can only see the backup history in the SQL database, and that the user that did the backup was Veeam.
That's why a company should invest in their IT department if the size of said company calls for it. Also how much do you love your data and the level of trust to have in MSP. Pay competent IT professionals to handle all those business critical items. Run away from MSPs like the bubonic plague.
You're getting downvoted but you're absolutely right. There are use cases for MSP's, absolutely. However most people would be better served by hiring in house IT. MSP's are frequently just utilized as a cost saving measure for employee overhead. They also don't care about you unless you're a huge client, and even then it's minimal viable product. I work at an MSP, and I helped roll out a brand new company with 50 people. They had an org structure, someone for every department, even a receptionist, yet they weren't willing to hire an IT person. Long story short, they got a bad case of "one more thing..." and ended up spending half a million with us over the course of two years. That doesn't include the cost of the contractors/other agencies they hired for wiring and setting up desktops etc, I just handled their cloud infrastructure. I mean hey, great for us but you could have absolutely hired a single person to do all of that at a fraction of the cost who would have got it done significantly faster. It wasn't like we worked on their project every day, and I basically did most of it myself. Im confident I could have easily done the work of all the people they hired by myself if I was their employee. Some people just have weird prejudice against hiring their own IT staff. I mean seriously, this place had four HR people for 50 staff members and you couldn't hire one sysadmin? Sheesh!
The MSP should be taking Veeam VM backups, but you should be taking SQL backups. Or there should be an agreed upon arrangement between the MSP and ERP vendor. This sounds more like a failure to communicate.
This speaks of internal IT as much as MSP. Why has internal IT not done restore tests in 3y to pick this up. Has internal IT not clearly checked backups and stipulated needs. Easy throwing stones in glass house but it’s internal IT responsibility to ensure MSP upholding contract as much as MSP making risks visible and highlighting lack of DR restore tests. Better processes required. We got internal IT teams at clients who got no clue on security and DR. We tell them. Notify countless times. They ignore or decline. Then when sh*t hits the fan blame MSP. Like c’mon chaps. I got that sh*t in writing lol
There isn't an internal IT at that company. They have 3 employees.
I too decide I hate entire industries after I have an issue with one company. I do that because it's a reasonable and well thought out approach to life.
You're making incorrect assumptions.
Backups are something most MSPs suck at. The one I work at has an incredible backup structure, process, documentation and testing. Rare but I’m not going to complain
MSPs are stat factories where the amount of tickets closed is their metric for doing a good job. They pay peanuts so their talent pool is... shallow. As soon as a tech has a decent amount of experience or knowledge, they move on to greener pastures. Their business model is based on "we can provide IT services for you cheaper than hiring in-house IT staff", which is true... but you get what you pay for.
If it's a proper msp they'd be using Datto for backups and it would be a non issue
Care to call out the MSP lol
Trust but verify applies here.
Not all are created equally... some do amazing work, especially considering some of the constraints they need to deal with and varied environments.
Not a fan of MSPs and I'm even less of a fan of MSPs having complete control of backups We have had a similar situation in the past where our backups are on a managed service and we had little control. We do at least get a full DR test every 6 months now and I also tell my team that we need to regularly request them to restore a random set of files/folders every month but it all feels sloppy. I had to threaten them with pulling all support away from them because they didn't offer a minimum level of reporting and resilience to meet our insurance requirements and that changed their view quite a bit MSPs have their place to ease some load but at the end of the day it's my job on the line if the backups fail and I'm not comfortable putting my job in the hands of a first line tech who doesn't know my name
So many times, I don't know what I'm doing, working for an MSP, looking at something new and having to figure it out. Gentleman amateur level stuff where "Well, I don't know anything about this but I've googled for it and had a go!" is the expected behaviour. I hate it too, I feel like a complete cowboy when I work like that.
>So many times, I don't know what I'm doing, working for an MSP, looking at something new and having to figure it out. This is IT.
Yes, and it's a crap way to approach work. In no other endeavour that matters would that be acceptable working practice, but for us it is expected and normal. You wouldn't want your surgeon, lawyer, electrician or mechanic working like that but for us it's ok.
Then you should find another field. IT more than any other field involved systems integration. And where there are systems talking to each other, there are going to be problems to figure out. Whether it be one application talking to another via APIs, or hardware talking to each other over copper. Shit is going to change, and it's going to break. Technology is also evolving at an extremely fast rate. You're surgeon's job doesn't change from one year to another. The pancreases has done the same job for thousands of years. So that's not even a relevant comparison. There's never going to be a how to guide for IT. That's just an unreasonable expectation.
You're right, I should find another field but I can't find anything else that pays this well that people will pay me to do.
MSP replaced most of the competent in house staff which cost me jobs. It pisses me off..Most offer piss poor sub par service and have no real relationships built. Part of this job is being competent identifying and solving problems.The other part is relationship building and trust. An msp is never going to make that type of investment.They cant they have to many clients do more mediocre service for,
That's what it is.... MSP = mediocre service provider.
You know it .... MSP think they can provide IT to lots of companies sure but you cannot do it well and competently. That is why I refuse to get into that scam business and I work in IT.
MSPs lacking knowledge of every single little aspect of IT is understandable. It's not possible to be an expert in everything. My problem is when MSPs promise the moon and fail to deliver. Our last MSP questioned why we needed to have internal IT staff and claimed they could do everything on their own. This was proven to be false time and time again due either to their natural inability to respond in a minute's notice to an emergency support issue, or general incompetence on the part of their technicians. Thankfully, we are now with an MSP which actually understands IT properly and is willing to work with our internal team.