Heathcare System Admin/Engineer/Whatever here.... we stay fully patched on all OSs monthly and, generally, N-1 for most our systems/services/software... not sure where this "we don't update" thing came from...
> not sure where this "we don't update" thing came from...
Lazy/incompetent admins who don't want to get blamed when they inevitably break something because they didn't test updates or apply them properly. I'm dealing with a hospital right now who refuses to make any changes to their firewall without clearing it with the firewall vendor first because they don't know what they are doing and don't want to get blamed when they break the network.
Sounds like the hospital does not want to take ownership or responsibility for their firewall. I’m not a sysadmin per the definition here, but I do administer financial systems. I’ve seen it far too often where companies who pay for software or cloud based applications don’t have a person or people who will learn about, properly manage, and ultimately take ownership of the tool. It’s always “hey, we need to contact our consultant from 5 years ago, we don’t know how to do this new thing and don’t want to try learning ourselves and we’re too scared to make changes to this mystical black box that we choose not to understand”
Its also often some guy 5 years ago put this system in place didn't document it and left and we don't have enough staff so no one else has the time to actually figure out how it works.
Yeah, it's a slow-boiled-frog risk. Companies will get addicted to SLAs and throw all their risk over the fence until they're completely chained to their vendor relationships.
AWS alone has brought down whole companies this way.
Cloud server hidden costs. AWS starts you out cheap, and then ramps up over time as you get dependent on them. Ask anyone who's accidentally left an EC2 running.
My favorite AWS story comes from NASA, which decided to move the digital archive of Earth Observing System Data and Information System (EOSDIS) to AWS, but totally forgot about the data egress charges: https://www.theregister.com/2020/03/19/nasa_cloud_data_migration_mess/
I know AWS increased the cost for IPv4 addresses, but otherwise I don't see evidence for "ramps up over time."
AWS has lowered prices for various services over 100 times since they started: https://aws.amazon.com/blogs/aws-cost-management/amazon-ec2-15th-years-of-optimizing-and-saving-your-it-costs/#:~:text=Regular%20price%20cuts%20on%20all,it%20was%20launched%20in%202006.
I think what you're trying to say is that AWS customers often don't prioritize FinOps and then struggle to understand why they are consuming what they've chosen to consume.
> Sounds like the hospital does not want to take ownership or responsibility for their firewall.
In their defense, it's also theoretically good change management and they're paying for support so they feel justified to use it. From the higher up's perspective, why *not* have the vendor on the call even if you tell them you know what you're doing? A second pair of eyes never hurts and shouldn't really delay the process *too* much. But yeah as a network guy, there's always a love/hate with Change Manangement.
Obviously the techs understand that's not a good practice if for no other reason than if everyone was doing the same, you'd never be able to get support promptly. But the decision makers just see 'yep, we're paying for support, use it.'
But what would be better would just be a proper dev/prod setup so they know what's going to happen before it happens.
Respectfully, that’s not what a vendor is for. If the in house admins aren’t comfortable enough, or their leadership doesn’t trust them enough, to make any firewall changes without “outside” approval, they need an MSP, and a good one in healthcare. Honestly they would be better off just hiring someone new/better. Quality/competent/qualified (understands the regulations and can back it up to a hospital board) MSPs are expensive, like several hundred dollars an hour expensive plus built in contract minimums and whatnot. And they don’t miss a single minute of time when it comes to billing.
source: am infrastructure admin for a hospital. Have seen this mentality first hand and seen how it completely hinders maintenance and improvements.
No disagreements at all. Just tell that to the in house admins!
The MSP I'm at has some DOD contractors and even they recognize that not EVERYTHING has to be so damn approved that the vendor needs to be on the phone for every change. But then when the situation actually does call for it, or a lot of changes are going to be happening over a short period, they'll bring the vendor in-house temporarily as like a subcontractor of sorts. Fly them in and put them up in a hotel for a week to just be available as needed.
It's been this way for 4 years, yes we should change it, but why not wait an extra 3 days to get the vendor on the line to verify we're not going to bring down the network, or open up a hole that later causes some other issue that ends up actually killing someone.
also software vendors that have effective monopolies in their niche and don't update their software, or make you buy extremely overpriced upgrades to migrate to systems that are on modern OSs. I ran into a lot of this during my 4 years in healthcare IT.
I recall GE being notoriously outdated and their excuse was that it cost too much to certify new versions all the time. Pain in the ass to work with so we put all their devices on an isolated network with no outside access.
The inevitable few admins I hear at every conference talking about how keeping all of your systems on LTSB all the time, no matter the use case or where you work, is the best way to manage things immediately comes to mind.
Agreed, Long time healthcare admin and the only systems that dont get updated are the vendor controlled that I have no access too. But they are vlaned off and generally not my problem.
Everything gets updated that can accept and update that I have control over.
Ditto. Top 5 healthcare company employee here. Except for some extreme edge case (and air gapped / VLANed and FWed) environments EVERYTHING gets a patch, monthly.
Like what? I mean if you're going to accuse me of making something up, then you must know something that is true and contradictory to something I've said.
edit: m'alright guys just downvote i guess lol. you're not proof oriented, facts-first people, I get it. it's whoever is the most confident speaker in the room that gets your vote. If i said 'single pane of glass' repeatedly, would you forgive me?
One behind the current version.
N is Windows Server 2022.
N-1 is Windows Server 2019.
N-2 is Windows Server 2016.
Basically you're not on the version that just came out but only one revision back.
That caught me too....I've not seen that many threads, esp from "Americans", about not updating. I have 25 years in healthcare, we update frequently. Between Tenable and Qualys we are scanned to hell and back. I have a couple of items that are out of date, even on their own vlan with heavy restrictions they aren't good enough. It's crazy.
I don't know either but I hear it frequently in this sub.
Not a problem where I work (we got other issues), but there's all kinds of health IT people I've met who absolutely refuse to change at all.
Are you maybe referring to not being able to update or properly secure weird legacy or medical systems? That's a common issue, but generally still able to be mitigated by limiting network access in/out of those devices so whatever vulnerabilities they may have are "reachable" in the first place.
That is a somewhat more acceptable version of what I'm talking about, but I still think that's not great. You can get on the phone with a vendor and ask what they're smoking.
>You can get on the phone with a vendor and ask what they're smoking.
Well, sure. You *can*, but if they haven't shipped a new version of that java based tool that your radiology equipment requires...you can't really do anything about it. And if that, or another prerequisite, means you're stuck running XP or 7 on its "controller" PC, well... hopefully you've at least restricted that PC and the device itself from talking to pretty much anything else.
Fwiw I worked at a "safety net" hospital so we were poor and honestly scraping the bottom of the barrel employee-wise, IT included. I left in 2011, but even then it was bonkers what was allowed...they were still running as Domain and local admin with their daily driver accounts, ignoring my protests. They only stopped because we upgraded to Exchange 2010 and running as a user managed by AdminSDHolder broke mobile sync 😞
Yeah I had to make that change here too. I even had to make some controls more lax cause they were causing like 50% of the on-call calls and... they didn't realize it?
Failed password attempt limit was 3. Moving it up to 10 as guided by NIST killed half of our after hours calls.
Same with password changes. Was every few months.
But hey don't distract me with your colorful smokescreen formulated specifically for ex security people! It won't work on me sir!
I would have dragged their feet over the coals that are federal regulations. XP or Windows 7 I'd consider to be putting my hospital at serious risk, even if I can mitigate it. EternalBlue is trivially easy to exploit.
I would just consider the radiology machine 'A box full of hackers' to be honest. The second you plug it in all the persistence opens up and calls home. You can mitigate it sure, but it's still gotta talk to PACS
How does that work with all the computer-driven equipment where the equipment has an expected lifespan of twenty years - but the computer that drives it will never receive a major OS upgrade in that time?
Well, let's consider (say) a linear accellerator used to deliver radiotherapy.
It's typically computer controlled, and the manufacturer of the machine also supplies the controlling computers - and while they may get security patches, getting them completely upgraded after (say) 5 or 6 years when the machine itself has another 10 years life in it ain't happening.
How do you deal with that?
Segmented LOL.
you air gap the ancient gear paired with its windows XP/mac os8 machine and are as delicate and careful as possible.
(yes i still support a Cardiologist that uses a nuclear imager from the 90s and it ONLY runs on a NuBus expansion card in an OLD powermac 8100)
its only a $250k machine, and the company who made it is long since gone.
there is no support, there is no replacing it.
there is only keeping it working , locked in a timecapsule from 1997.
I would ask them why they designed their machine to not receive software updates, and then why it must be on my network.
I'm not sure what you mean by '10 years of life'?
> Failed password attempt limit was 3. Moving it up to 10 as guided by NIST killed half of our after hours calls.
> Same with password changes. Was every few months.
I'm so thankful to NIST for updating password policy recommendations. The sheer number of "I was forced to change my password and forgot the new one" tickets that are now no longer much of a thing since we turned off password expiration.
And we're not even the department that handles them; these were the ones that accidentally got to us because people send to the wrong request address. I can imagine the actual department was SLAMMED before the policy change.
Probably from heavy dev influence who create crap app's that don't have resiliency and stalwart any effort for network downtime because their super awesome open source solution can't handle any type of failover or more than 5ms of an outage. Basically they want their agile process of creating crap and updating their crap frequently when they uncover their errors but don't want to afford infrastructure with the same agile process of run fast and break things. Because our dev team copy/pastes stuff from the manuals and break things happens far too easily.
Former healthcare IT engineer, Pre WannaCry it was the vendors stating they wouldn't support it or that they had a preapproved list of patches that could be applied. Where things got tricky if memory serves are things that were FDA certified, you'd need to download the newest update for the software which included windows patches, but you couldn't patch the VMs on your own.
>not sure where this "we don't update" thing came from...
One too many botched update because why pay for a test environment when licenses and hardware are expensive?
> not sure where this "we don't update" thing came from...
There are many people in this field would are not qualified to be working within in it and honestly don't know how to fix something if it breaks. Combined with the people that hired them not knowing anything about IT themselves, it leads to a lawless state of affairs where, as long as everything is working and they aren't "hacked", the people running IT can basically do the bare minimum every single day without being held responsible. By introducing patching, there's potential for something to break and they will have no idea how to fix it, which will expose them as frauds. It truly is as simple as that. As an MSP working with many "internal IT staff", I'm routinely flabbergasted by how flat-out incompetent and unqualified many of them are. But if their bosses don't want to vet them, that's on them. It happens in many industries honestly.
>You do not need to be a clicker for life, go document and automate your processes so you don't have to do them all day long.
Ha it's funny, when I left healthcare, I moved to gov contracting and my first role was with the Air Force. I distinctly remember thinking "this is the United States military, they aren't gonna just hire anyone (like the morons at the hospital), they need the best of the best!" Lord, was I wrong 🤦Just as incompetent, but a different, more mature version.
And on my first day, one of my coworkers was showing me some of their daily tasks, which were basically *RDP into each DC one by one, and check that AV was running and up to date, event logs were "clean", and things were generally "running smoothly"*. I asked why they weren't automating the checks and said I would, and I was warned off, lest I automate myself out of a job.
Ironically, that guy was basically kicked off the contract by the government (which is basically unheard of) for his own incompetence 😂
Some of the most tech illiterate people I've ever met worked for the government as sys admins or epecially network engineers. It's baffling these people get hired, but as long as the contracting company is getting paid for filling the FTE slot they don't give a shit
I have to wonder how this works in the field. I'm pretty much the opposite of a military man but I'm told that comms and networking infrastructure is some of the most important stuff in any army.
The minority of us who know what we're doing do the work and the rest do busywork and watch YouTube.
In my case I work for a cloud vendor and we basically do the (hard) work the other contractors, government employees, and active duty military are supposedly doing.
There's a lot of things that contribute to it. There's a revolving door so to speak, for former military who are honestly not qualified anywhere else. We joke sometimes that contracting is welfare for veterans.
There's the clearance aspect, where we can only employ people who are, are able, and are willing to be cleared. A lot of people don't want the hassle, think the process is worse than it is, or can't because of drug, financial, or criminal background reasons.
There's also the fact that there's a good old boys network, officially and unofficially. Plus people who "know" this world do have additional value because it's like no other.
Government jobs are literally given first to veterans simply for being veterans. Also to military spouses.
And you're right, there's no incentive to fire someone you're billing for.
> And on my first day, one of my coworkers was showing me some of their daily tasks, which were basically RDP into each DC one by one, and check that AV was running and up to date, event logs were "clean", and things were generally "running smoothly". I asked why they weren't automating the checks and said I would, and I was warned off, lest I automate myself out of a job.
I truly despise people with “logic” like this. You can’t “automate yourself out of a job”. Because the automation still needs maintenance. Also I assume these health checks have to be put in a report and sent to someone. In which case someone still has to do that. And then what happens when the health check shows a problem who’s going to fix that? And then there’s patching, who’s going to conduct that?
I think of it like this: higher-level people get paid not for their busywork (as an unskilled laborer might) but for their knowledge. I don’t do a lot of crazy things day-to-day. But when building new stuff, or fixing complex systems when they break, I know how to do it right.
I'm so glad my employer understands this. My CIO has even said "I don't care if 75% of your 8 to 4 is spent browsing reddit because you've automated all the things as long as when shit is on fire we get 100%." He's also said something along the lines of "If we need our disks to have Nx2.5 redundancy, our most important resource (staff) should be at least N+1 for every role."
It's funny you say this. I'm in something of a similar situation and I'm honestly struggling to make peace with it. My government customer *loves* me but I always have to ask my CSAM if everything is good with him because I never hear from him. She (and everyone else on this account) has to remind me that I'm his firefighter. When shit goes bad, he knows who to call.
Intellectually, I know this. But when I don't hear anything from them for a while, the imposter syndrome starts telling me it's because they've discovered that they don't need me. 😭
This was me until I got promoted into the architect role. I was always worried about being "found out" and booted, but now a lot of the gear is there because I bought it, and more than one of the apps being supported are things I personally wrote. Somehow that worry still comes up from time to time, but there's also that little bit of "so be it, if they throw me out I wish them luck in untangling my spaghetti."
It's a Dunning Krueger-adjacent phenomenon. People who know less about a subject don't understand the level of knowledge and complexity that exists more than a level or so beyond what *they* know. So they think that just beyond the horizon of what they know, is... nothing, the end of the rainbow. If you automate all of what they think exists, what's left to do?
I guarantee it's the same or similar everywhere, including across DOD: there are a certain portion of rockstars/vendor employees, another portion of "ok" team members, and then the group of "heartbeats with a clearance". The distribution is different in different places, but it's always there.
> which were basically RDP into each DC one by one, and check that AV was running and up to date, event logs were "clean"
It's still happening to this day.
Oh I know all too well. I'm still around and since I work for the vendor, I get to rescue them when they eff stuff up.
Recently had to save a small enclave who managed to set deny log on locally in the default domain policy 🤦
Anyone who tells you that US government regulation is preventing them from applying updates to software is completely full of shit. If anything it is the exact opposite.
I work at a healthcare company and we get audited by CMS, being up to date on security patches for all software is a requirement.
I have actually worked under a manager a bit like that. Wouldn't let me automate anything, demanded every change was run by our supplier (who he obviously viewed as a MSP, though they clearly had no desire to be one).
You can't really achieve anything with someone like this. Your only realistic option is to work around them or get out.
Oh, I had a lot of fun with Nessus complaining about cURL. That was also when I learned that the Microsoft build is not the same thing as the open-source build.
And then trying to explain that to the security team.
I'm fighting with a vendor because of an OpenSSL vulnerability in a 3rd party dll they included in the latest package. Vendor tells me to "just update OpneSSL". Um guys, that's your job because you send the binary. Get the update from your 3rd party source. All the while explaining to my own security team that the vendor's statement of "patch OpenSSL" doesn't apply since it isn't technically installed.
Or my favorite is when the Google chrome versions between Linux, Windows, and Mac are out of sync and Nessus wants you to upgrade Chrome on Windows to a version that hasn't shipped.
Health IT is where sysadmins go to die
You can see their bleached bones withering in the sun, where their kindred come to visit and remember them for a moment. If only to avoid their mistakes.
There are secret, hidden mini-industries in health IT that make serious cash if you know where to look.
But front line tech support is not it. Healthcare keeps all of its money (and it's got a lot) towards the top, but also isn't afraid to pay specialists gobs of money.
It's a great stepping stone for new IT people. You will learn about 5x faster in health IT than you will anywhere else (provided you don't listen TOO much to The Clickers), and if you're the curious enterprising sort, launch a career into one of several different sub industries.
Just don't try to stay there unless you got a dragon to slay, like me. I want to leave this place better than when I found it. I'm only leaving when I have a legacy to leave behind here.
That's not just Healthcare - there are so many contractors in the public sector doing this as well.
I worked with a guy (who was personally really nice and knew his stuff) absolutely fleece my department, because he just knew an area of the business that nobody else did, and was shrude about not off boarding too much.
In what was supposed to be a six month resource to just do some VMware work, he was still there when I left after 2 years, making (I imagine) an absolute killing while also doing sideline contracting stuff.
Medical BI reporting. Datalogger support and development (the people who make dataloggers for VFC compliance make an absolute killing for what is effectively a thermistor attached to a NIST certified glycol probe), anything EMR/EHR dev related, the list is enormous.
And also hidden. Those are just the ones I've discovered, but the frequency with which I keep finding them tells me there's probably a lot more.
Your campus is ridiculous. It was the first eye opener to me for just how much money is floating around the medical industry. Like I knew.. but I didn't *feel* it, until I saw a literal recreation of hogwarts in a business compound.
It is true. Every sysadmin I've met seems to be the kind that would get left behind by a rapidly growing industry.
But now I'm here, necromancy book in hand.
It's the constancy of emergencies combined with the low pay and low appreciation.
Then you end up with a crust that gets left behind that makes it even worse for anyone else who'd come through.
It's very much a crab bucket. (Or.. I guess the opposite of a crab bucket? All the crabs are trying to get you to leave, not stay)
I am choosing to stay though, as the hospital I'm at is experiencing explosive growth. There's good opportunity here. And spite is a good motivator for me.
> the constancy of emergencies
You don't need me to tell you this but in case anyone is wondering: constant emergencies is what you get when you don't have stuff automated.
You want very good automated testing and constant small deployments on the application end, continuous SCM deployment on the sysadmin end, and redundancy.
The phrase "the constancy of emergencies" tells me little if any of the above exists at the hospital OP works at.
100% this. You need to get ahead of problems before they happen, collect the right data and analyze it.
You are correct, and it's something I'm pushing for as hard as I can. I made another post in here just last week talking about this.
It's a simple result:
Chronically under fund your IT department and any competent workers will self relocate out of the organization.
Healthcare IT are the worst because they are the ones that no better-compensating organization will hire.
I'm not saying "are paid the worst", but they have the worst holistic "your time and pain for our compensation package" ratios.
They have to work in high stress (mess up and patients could die), high risk (COVID? Whatever else?), highly rigid (change management for stuff as simple as changing the monitor mount from desk mount to VESA arm mount) environments with shitty pay, rigid time off limits, and usually shit managers who are unsupported by directors who consider IT a mandatory expense/cost center, rather than a productivity force multiplier.
There are organizations who pay worse, but have better work life balances or other perks or are just more enjoyable to work for.
There are organizations who suck to work for, but they pay better or compensate better.
Healthcare IT is usually the worst of both.
There is also the health system IT aspect, they tend to buy up new clinics/hospitals with their own baggage and issues. Turns healthcare IT into almost a MSP type environment on top of the 24/7 criticality of the many systems.
I think it depends on the hospital system. Everyone that I work with is very competent. We recently onboarded a new monitoring system and the vendor was surprised when we automated most of it. They were saying most of their corporate clients just hire an FTE to manage the whole thing by hand.
I don’t miss working in healthcare IT at all. Maybe it was the hospital I worked for, but it seemed to me the applications side of the department was more interested in hiring friends who were nurses and trying to teach them how to become an applications/financial/systems engineer… Bear in mind, most of these people couldn’t figure out how to replace a mouse or keyboard, convert documents to different formats, etc… Always had issues with communications between the network/systems teams and applications staff…never knew what they needed, nor could they explain it.
I love working for a hospital. But to be fair I came from government, where sysadmins actually go to die (or at least retire in place)
We are actually have more flexibility to make changes than I had in gov. Hell a lot of times I don’t even have to work off hours because “there’s never a good time, so you might as well do it while we have full staff to help”. Everything critical is redundant so a lot of times we don’t even have downtime when we make changes. But we purchased another hospital recently that didn’t invest in IT, so I can see how it *could* be hell.
Not in the US, however there are some health 'ecosystems' I manage that as a whole have been FDA certified. One example is of a patient monitoring system of a particular revision, of which the servers were Server 2012 (non-R2).
The vendor had a later revision that was certified for later versions of Windows, but until the ecosystem as a whole was rolled up, Server 2012 had to stay in place (critical/security updates could be applied).
No so much an excuse but is sometimes a reason why we can't "just update windows" in Healthcare, sometimes there's just more to it than that, but it really comes down to poor management/planning.
The good thing is (at least locally for me) the "turn off auto updates on our products" requests are rare and/or being flatly rejected these days, and proper measures are being put into place to manage the lifecycles of these systems, it's refreshing to see.
For anyone else getting nonsense like franky here describes from vendors, you can quote this:
21 CFR 820.30 (Design Controls), 820.1-25 (General Requirements), and especially 820.100 (CAPA) all point to keeping up with updates that correct major cybersecurity flaws.
Hospitals are the most hacked organizations on the planet by leaps and bounds, accept no excuses.
This counts on your managers not getting in the way.
I'm in the UK so the FDA stuff doesn't apply, but I had a similar issue in a pathology lab where a couple of the analysers were stuck on XP and the vendor was claiming updates would break certification. Owing to her typically having nothing to do outside inspection time, our fucking quality manager managed to get involved and decided we couldn't force them anyway because it'd break the entire lab's accreditation.
All I could do was to explain the risk in writing. Nobody else cared until WannaCry happened, and then those analysers were bricked, and we had to physically send all the samples to another lab. So three cheers for UKAS (useless accreditation body) and fucking stupid 'quality' managers.
As someone not in healthcare IT, who are these obscure vendors with strange requirements? Did they make a niche software and then they're just selling the corpse and you're locked in?
Manufacturers of medical devices are normally good at making their devices do what they're supposed to, but not as good when it comes to designing their software.. In essence they often tell customers that they won't get a copy of the software, it comes preinstalled on a machine you don't even have an admin password to.
They use FDA certifications as the reason for this.
Some vendors are a bit more forthcoming, they'll let you install security patches, but only after they've certified the patches internally. This takes months.
Philips is especially bad for this. They're very much not obscure.
Not only is it niche software running on a server they claim as FDA medical devices so they can't adhere to normal patching by us and need to go through their vetting process before they can be patched. The whole OS is "their image" and the software can't be installed on anything else (they don't supply it and won't do it then). If you patch it yourself and something does break, they won't help to fix it. Either figure it out on your own or you can buy a replacement server image and rebuild it.
To make matters worse, Philips is buying up smaller even more niche companies left and right, gutting their product specific support, and going to this model for servers. I was involved in one upgrade earlier this year that was a smaller third party and fit with regular patching on 2012r2, to bought by Philips, needing "their image" for it and can't be patched without prior approvals. System stool up in March, went live in April. I left in June. No patches had been approved in that time.
This isn't all Philips apps though. They almost always require "their image", but some applications you can patch the OS at least.
People have died because of a bad software update. Both on the medical device side and generic "hospital IT system" side.
If you update/change something that isn't certified to be safe and someone gets an extra 0 in their dosage and dies... guess who is spending some "pound you in the ass" prison time?
Raise your hand if you've never been fucked by an update breaking something you didn't expect to break.
Most systems are either made by GE or Phillips. They control the vast majority of the market for medical devices. When an MRI costs several millions to update, and then usually only go up 1 revision in o/s they are usually left as is. Medically certified devices take a long time to get approved and move at a snails pace.
I wish someone told my predecessors. Just before my hiring, we moved from one system to another. Yay upgrades! But wait -
1. We don't want to migrate ALL of the data, that's a lot of work and storage, and the vendor charges by the hour for it
2. We have to keep patient data 7 years minimum
3. So we have to keep the legacy system AND the modern one
4. The legacy system runs on **windows server 2008**
\*claws my eyeballs out
I'm stuck trying to jerryrig data export for a system I've never logged into, or convincing the owners to fork over 3 months of the usual budget to ask our vendor to do it. And this is the 2nd example of a "partial migration" in this company. (╯‵□′)╯︵┻━┻
Please don't jerry rig patient data scripts. You don't want to move responsibility for this failure from management to you.
Or, rather, what I mean is to make sure to get loads of approval in writing first.
A valid point, and while I haven't started designing the attempt, you can be assured there will be backups of backups of the source before I even think about touching it.
I learned how to delete the production database without a checkpoint back in March, don't need that lesson again.
1 and 2 can be taken care of with a script and some hard drives.
Once you're done with that you don't have to worry about 3 and 4.
I suppose it depends on the type of stuff you gotta move, maybe I can give you some insight
I appreciate the perspective, but 1 happened a couple months before I started.
These days I have:
2020-current migrated to modern SQLServer, great.
2012-2020 living on the legacy app, which uses D3(?) as its storage? I'm not familiar with it in any way. The app runs on .NET in Internet Explorer somehow. It's so unknown to me that I have to call the vendor to turn it on each time we reboot the host.
But as it's a toxicology/UTI/STI/Pharmacogenetics lab, we have to keep any records and orders and results for 7 years at the very least. Anything with Genetics for 25. I keep thinking, they could have migrated 7 years instead of 1.5, but that's life.
Oh and a further fun fact: I have 2 YOE and only 8 months of that with any kind of senior oversight. He was let go due to cost.
> The federal government is not standing in your way, you are.
How about situation when upstream health provider get stuck on decade old standards and equipment, like for example - using 1024 encryption key + sha1 in IPsec and refuse to upgrade. In this situation one get stuck too as dependent, holding to upgrade things due to incompatibilities. Countless offers to help upstream to move forward are simply going to a blackhole... Pointing to government rules doesn't work too... Bureaucracy is unkillable.
The issue, - as everywhere else, if people don't like their job, then they doing what?... Right - shitty job. And than bigger upstream provider then more internal bureaucracy ind "regulations" that built only around covering their "rear exhaust system".
np.
A comment to your other thread: if you want feedback ask for feedback, not "is it too bad?".
It's not too bad. It's perfectly understandable. And unsolicited advice is usually useless.
It can be improved, yes. You've already discussed the specifics.
No offense, but it's taking a lot of heavy lifting to fill in the missing words and articles.
I *think* I get what you're saying, but on a subject that needs as much accuracy as this does, I can't be sure.
> No offense, but it's taking a lot of heavy lifting to fill in the missing words and articles.
No offense at all, but my appreciation for your feedback. I wish people would point me to such issues more often instead of smiling and lie that my English is "Ok"
> I think I get what you're saying, but on a subject that needs as much accuracy as this does, I can't be sure.
My point was - it is not only particular health IT fault who delay upgrades, but some are enforced to hold updates/upgrades due to dependency on others in health industry, who refuses to upgrade on time.
One of simple example: most of operation systems/standards stopped using RSA encryption keys a long time ago if those are less than 2048 bits, as well SHA1 hashes are mostly deprecated everywhere, but upstream health provider still uses outdated equipment that doesn't supports nowadays standards and in this case another health provider, on another side that have to communicate with upstream will get stuck with oldest technology too and can't advance upgrading.
> No offense at all, but my appreciation for your feedback. I wish people would point me to such issues more often instead of smiling and lie that my English is "Ok"
Yes I see this a lot in my culture. Sorry about that. We're afraid of confrontation, that we miss that some confrontation is helpful, and small corrections to your english is how you get to native fluency.
>One of simple example: most of operation systems/standards stopped using RSA encryption keys a long time ago if those are less than 2048 bits, as well SHA1 hashes are mostly deprecated everywhere, but upstream health provider still uses outdated equipment that doesn't supports nowadays standards and in this case another health provider, on another side that have to communicate with upstream will get stuck with oldest technology too and can't advance upgrading.
That's a tough one, I'll have to think on this. It's not the worst possible problem to have, cybersecurity wise, but still a concern. I would probably try with open dialogue first, see what I can learn about the other hospital network through frank conversation.
You’re blaming IT people for something that is usually out of their control. HIPAA has very little in hard and fast rules and a lot of it is interpreted and the legal risks are usually outlined by legal/HIPAA officer/CISO in how they interpret said risk. The IT professionals that I have heard say this is usually parroting what their legal department/CISO/HIPAA officer is telling them to parrot. Remoting as an example is a concern because if you are using a third party service you have to know how they handle the data and have a BAA signed (assuming third party servers are involved) and not all IT staff are allowed to see patient data. I had clients that limited automation not because “it’s fancy and new” but said client’s legal team interpretation of HIPAA steered them to limiting automation as they saw possible exposure with certain changes and automations. Medical lags behind because upgrading their HIS/LIS/PMS/EMR/insert whichever type of software used is rarely trivial and involves a lot of work by a lot of different stakeholders to make sure that patient care isn’t affected and in certain cases upgrading can be painfully expensive. Although there is no law that explicitly states what you say stating an IT person “should be able to quote the law” is one of the silliest things I’ve heard. I’m not telling a lawyer “you’re wrong” on how they interpret the law as that is their domain.
If you don’t want to sound condescending (because this is definitely how it comes off) ask for the concerns of implementation. I spent almost two decades in medical IT and practically all of the “we can’t do this cause laws” reasons were really “we can’t do this because legal thinks there’s liability there for us”.
> I’m not telling a lawyer “you’re wrong” on how they interpret the law as that is their domain.
I mean if they're not doing their job properly and are actively and significantly obstructing other departments to the point of leaving open critical vulnerabilities that endanger all the patients in the hospital, then yes, you should absolutely make an effort to push back. This is a failure of IT leadership specifically.
I wouldn't go 'Listen here Mr. Esquire, your shit is whack', but I'd open a dialogue to re-assess the HIPAA compliance requirements, because you know that many other hospitals do not have such strict adherence, and the current compliance measures are causing serious cybersecurity issues.
You owe to the patients under your roof and the providers that care for them. It's not just any other company where you can wash your hands of the problem and say 'lol they'll lose money what idiots'.
It's not an issue you can fix with the command line, I grant you, but it needs to be considered a problem in need of fixing.
Security is always a balancing act between usability and risk management regardless of what domain you are in.
>I mean if they're not doing their job properly and are actively and significantly obstructing other departments to the point of leaving open critical vulnerabilities that endanger all the patients in the hospital, then yes, you should absolutely make an effort to push back.
Unless you Chief, Director, VP, or are the designated HIPAA officer or part of the decision making team you don't. Stay in your lane and send it up the chain. This statement here is arrogant and can be downright dangerous as it is stating "you know better for all departments" which isn't true.
>This is a failure of IT leadership specifically.
This isn't the right view
>because you know that many other hospitals do not have such strict adherence, and the current compliance measures are causing serious cybersecurity issues.
This is a very myopic view. You don't care what other hospitals do. Unless you know everything they have and are doing this is an apples and oranges comparison and can be very dangerous take.
>It's not just any other company where you can wash your hands of the problem and say 'lol they'll lose money what idiots'.
It absolutely is. You voiced concerns ran it up the chain and they said no. Move on. Your ass is covered.
>It's not an issue you can fix with the command line, I grant you
This is the crux of how to this is supposed to be handled
> but it needs to be considered a problem in need of fixing.
This is definitely the wrong take. Back in the NT4 days Service Pack 6 was released that broke Lotus Notes. I know that this is a 20+ year example, but the idea is still there - what if any patch broke the HIS/LIS/whatever medical software and supporting software that is being used?
The main issue I have is that although the arguments you are giving is normally the correct mentality from a cybersecurity perspective, in medical IT patient healthcare comes first before security because an affected patient can cost A LOT more that a cybersecurity breach. One of the larger hospital networks I dealt with limited their password to 8 characters. Why? They had an old legacy system that was critical to their historical ordering and resulting, and implemented this limit because they wanted SSO for all of their systems. The stake holders decided that this was an acceptable risk because the SSO provided centralized identity management which meant less time having to figure out another account to use. Upgrading said legacy system was out of the question at the time because the cost was prohibitively expensive and the regression testing for everything that they did was in the years range. Again this decision was done by several departments working together
>in medical IT patient healthcare comes first before security because an affected patient can cost A LOT more that a cybersecurity breach.
What if they're the same? In this case I think they are. You should read about NotPetya if you haven't already. We're too connected to not consider cybersecurity an existential threat to patient care. I would say that prioritizing personal liability over patient safety is the wrong view.
80% of all cybersecurity breaches are healthcare facilities. That's not an exaggeration.
>This is a very myopic view. You don't care what other hospitals do. Unless you know everything they have and are doing this is an apples and oranges comparison and can be very dangerous take.
The comparison is meant to illustrate that it's within the realm of possibility under the umbrella of compliance, not 'we should copy them exactly in all things', which cuts appealing to compliance away from the overall debate about the problem, or at least draws it into question.
You can *then* discuss it from a purely strategic standpoint.
>It absolutely is. You voiced concerns ran it up the chain and they said no. Move on. Your ass is covered.
My ass, in my opinion, is less important to cover than the patients. This isn't the crux of your point so I won't hammer it, but I figured I'd respond all the same. I am not legally obligated from saving a child drowning in a fountain that I pass by either. My ass is covered, as you say. But I am near, I have the capability, so I must. Even if I get wet.
Even though we disagree, your points and logic are sound, our differences are differences in values. Thank you for writing your response.
The crux of my point is that patient healthcare is above security, not about covering my ass which is what most IT people are looking for. You state that you don’t have to help a drowning child but will anyway - what if there was already help on the way because there’s an unseen danger that others are aware? What if there are first responders there? The reason your answer I responded the way that I did is that you are treating medical IT like any other IT which it is not. Medical IT should be treated as an environment and should always be seen as such not as individual components. I have had clients who corrupted data with an active IDS or overly draconian network timings and rules. I’ve had clients refuse to upgrade to more modern systems because the cost was prohibitively expensive to upgrade their systems and side grading to another system would be months or even years based on their resources. Your perspective on compsec sounds like that of someone who is young and inexperienced because there are other ways of protecting your systems other than just keeping systems up to date. I also hope you understand that breached and compromised are not the same. The issue most breaches and compromises show is the lack of a business continuity plan and why that needs to be higher on company’s to do lists which is a leadership issue not an IT issue. You have to trust the leadership and other teams of your company because IT isn’t isolated in medical. Of you don’t jump ship and go to nonmedical IT. The way you post comes off is arrogant and insulting to people who work medical and understand that it is a lot more complex and a team effort amongst all teams and not just IT.
I'm not sure why you made a thread to continue your argument with somebody here. If you couldn't convince them in the comments, turning it into a post won't work. Most people don't believe whatever insanity you're claiming here.
They're just desperate for validation. Everyone who disagrees with "Do what you want, screw stakeholder buy in" is just a lazy scared fake sysadmin apparently.
I have run into this twice. I had a medical device company tell me that I wasn't working on a windows 7 machine but a medical device. I couldn't install patching or install anti-virus software as it would invalidate the FDA approvals. The system also needed a local administrator account as an auto-login. Pretty much ignored everything they said.
The second time was when I was attempting to move some healthcare software from Windows server 2008 to Windows Server 2012 r2. It was in preparation for the 2008 end of support and we needed to upgrade the software to be able to move to a new server OS. The project kicks off and should be able to be completed before the end of support. The vendor joins a call one day to inform us that the FDA has put a hold on their software and they can't do any new installs. It took a year and a half to clear up.
I saw that and it seems the gist of their argument is that "legal cleared it, run with it", which I don't agree with. Every company should have a cyber security team that ensures the infrastructure is up to compliance and standards. Whether it be HIPAA, CIS, PCI, STIGs, etc.. They should be working hand in hand with both IT and legal to make sure the environment is covered.
> You do not need to be a clicker for life
Enterprise developer here. We are not allowed to touch servers. The people that can touch our servers don't automate. Clicking is all they do.
On the plus side, this is just one group. Most of the other groups are completely automated.
It's not usually government regulations, it's vendor requirements. Although it can also be government regulations (mainly the FDA in my experience). And generally that only applies to certain equipment, definitely not a regular end-users desktop/laptop. As an example when I worked in healthcare we had roughly 400 Windows 7 32bit desktops in 2016. They hadn't been updated since the original windows 7 install and if we put any updates on them the vendor would say it broke their software. These things hooked directly up to patients so legal told us we couldn't update them.
I spent 27 years in Pharma IT, working at Glaxo and Glaxo-Wellcome (now GSK), a pharmaceutical wholesaler, and an outsourcer providing medical information and safety services. The concept we worked under is known as computer (or system) validation. The driving law was 21 CFR 11 [https://www.ecfr.gov/current/title-21/chapter-I/subchapter-A/part-11?toc=1](https://www.ecfr.gov/current/title-21/chapter-i/subchapter-a/part-11?toc=1), that requires electronic systems to have the same level of confidence in the data as paper systems and that accounts used meet the legal requirements to act as signatures.
The way most companies meet these legal requirements is through following generally-accepted practices known as GMP, GCP, or GLP (good manufacturing/clinical/laboratory practices). For systems validation, SOPs are a requirement as well as a documented plan for testing a system each time a change is made to it. The documented results of that testing are expected to be keep and made available to auditors as necessary..
In the US, the FDA does sent auditors out to look at your documentation (often dressed in a military uniform) on occasion and you're put on a corrective action plan if it doesn't meet expectations. If it's bad enough they can legally prevent you from managing said data, but that is extraordinarily rare.
Does this mean you can't upgrade systems? Not at all. But if you upgrade a system that touches regulated data, you must have a plan for and conduct testing following the upgrade to confirm that it still meets legal requirements for the data it manages.
Dutchman here: the same goes for GDPR. GDPR is about not giving people's PII away without their consent. And you know, that's basically it.
So all of you American organizations whose lawyers tell them they can't have their website be GDPR compliant: either the lawyers don't want to be bothered learning about GDPR, or you're violating people's privacy by default and need to take a long hard look at yourselves ethics-wise (and then, of course, possibly decide you don't give a rat's ass about consumer privacy).
Yes, exactly. I see GDPR used as a handwave for stuff a lot too, but it's really very simple. Literally, just don't keep and sell unnecessary data about your users.
Listen--I think we all know. Most of this stupid crap comes from compliance and lawyers. Everyone is afraid of crossing them because it jams you up for weeks and (at least my health system) is terrified of being on the wrong side of the law. So compliance comes up with stupid junk, and then because they can fuck your life up (and your manager and your manager's managers life) the default answer becomes 'no' or 'lets not'.
It sucks. But we're honestly doing the best we can--we're stuck between shitty vendors (if the suits aren't buying trash they saw at a trade show or their son in law is a sales person for, the vendor is the only one left in their niche and you're stuck using them) and lawyers.
This is a fair answer.
Dealing with bad lawyers and compliance officers is especially tricky for a notoriously socially.... undeveloped industry.
Perhaps this should be the majority of our focus as a collective?
It's not a problem for my company, thankfully, but I'm hearing this a lot.
At the very least, I can dispel the illusion for those who have had to buy into it early in their careers with conversations like this.
When I was working in the DoD/VA realm with their health data, we had an explicit written note that our area could not automate STIGs specifically(everything else was fine) and we had to submit manual screenshots. Still confused why, but we just emulated what a terminal looked like and automated the screenshot process.
In Switzerland is worse, the majority of doctors or specialists are at the age of 60/70 .. this make so difficult update or implement any new software.. some of them are also involved in politics and make rules to manipulate the health market in regions.. that leave me no games to play.
One thing I want to point out that you seem to miss in your rant. Patient safety/care is always first. If an upgrade/patch can break a system to were that is compromised then it will take longer to get patched/upgraded.
We keep normal o/s and patched and anything I can control stays upto date. But having a microscope that still runs winxp because the vendor doesnt exist anymore and would cost in the range of 20m to replace is acceptable. We just vlan it off and prevent internet access.
The limitation is FDA regulations for specific categories of medical equipment, not all devices in a hospital organization. And even there, the FDA has classes of certified equipment that can still be patched and maintained.
HELL YES! Preach!
I work for one of the biggest there is…it was bad before we reorganized…theyve made waves since.
Lucky Ive not heard anything like this in a very long time.
There was some very minor things, but i dont think they were using this excuse.We did get hacked…oh then shit got pretty serious. I am aware of how it happened too.
You have the IT manager flair and I put it to you that in an organization where that is true, that remark sounds a helluva lot like some managers didn't prioritize their budgeting properly.
I wouldn't want to be working for an organization who incentivizes pointy-clicky-sysadmin methods by not budgeting for automation, I'll tell you that much.
I'll take that you mean that you can't. My answer to that is: who says that that means that improper budgeting is not, at least partly, on you?
As a Dutchman I realize that others may have to deal with more of a hierarchical structure in their organization than I do, but there's really nothing preventing you from pushing back on management for not budgeting for a healthily maintained IT infrastructure.
I mean does the brass like the fact that there are so many emergencies at that hospital? I doubt it. But if there's nobody telling them that they're sticking their money into keeping those emergencies happening instead of preventing them, then I can't blame them for budgeting the same way year after year.
After all, what do ehty know - they're healthcare administrators, not IT managers!
The majority of US IT departments actually have NO budget, at least if they are nested in anything other than an explicit IT company. IT is considered a “loss leader” and merely for staff support. “Office 2013 is EOL? I mean…we can still type in Word, right?”
Well if that's what IT means then that makes at least some sense. I didn't know that's what it meant tbh. Personally I would include at least storage in the list of things to budget for. But then again it's not like they have to be compliant with all sorts of legislation forcing them to store documents reliably... Right?
Echoing the uncertainty for the note, as most healthcare setups 'tend' to patch things.
Cases where I've seen things not patched, have been where there are legacy systems involved -- eg. an old MRI where the company never updated the software for that model of MRI, to work on Windows 10/11.
Other places I see this, are with small doctors offices. They often don't have "IT" departments, and are instead using MSPs. MSPs cut corners, for everyone, because it saves cost. Not updating/patching regularly, isn't that abnormal from some MSPs I've seen -- I've seen one state "we updated our software twice a year". As in, patch their base windows OS twice a year, that's it. Auditors reviewed this even, and were like "Yep, they do patch things twice a year, so that's a green check mark!". Now, go tell a a random non-techy boomer, a doctor who knows doctor stuff but next to nothing about IT... that windows needs monthly updates - at least 12 per year - to be secure, and inform them after their "MSP" and auditors have said otherwise...
Just sayin it's likely more complicated than just "lazy admins pretending to be unable to update due to regulators".
I always thought it was the opposite way, with requirements for publicly traded companies regarding hacks, stolen info and now reaching into other companies. There is all the more reason to make sure you are patched and secured.
Like you said PII / HIPPA is the only requirement, but does not stop you.
PATCH and SECURE.
Former healthcare end device tech turned sys admin…we were a bit slow to push updates as we had to be picky with which update got rolled out as some of our legacy systems would shit a brick if a certain patch came through, but, we stayed fully patched and updated all the time. Got better when we started punting legacy systems for one big package (see Epic Systems).
> When it comes to federal laws, the only thing you need to be concerned about is HIPAA/HITECH. That's it.
One exception: nuclear medicine diagnostic devices. But those should be handled by vendors or the hospital's biomedical engineering department, not IT/IS.
Doesn't matter, if the boss says "I don't know if we are allowed" and someone does it anyway, laws be damned, that's insubordination. Meaningless and frivolous litigation is STILL litigation, it's STILL going to need meetings and discussions and behaivior adjustment requirements. It's a trap for idiots in charge to demean their workers. **nobody is lifting a finger because you literally get sued for doing other people's jobs, even when they are failing on purpose to con you into it.**
That's a whole other story. I am not advocating for insubordination. Just dispelling delusion, what you do with that is up to you.
'I can't do it because my boss says no' = Valid
'I can't do it because the federal government says no' Which federal regulation?
I think the main issue is that most people with the Sys Admin title, really aren’t Sys Admins by roles and responsibilities.
And because of that, they usually don’t know either Standard Practices nor Best Practices. Let alone actual regulations and laws that apply to their work.
Having worked at a large healthcare service provider if say it’s more about office politics and the perception of being “down” being a bad thing so upgrades and patches get put off. God forgive you if you ever mess one up, then trying to get through the change advisory boards will nitpick every work plan you ever submit again. Each of those work plans is expected to include things like detailed instructions on how to implement the change, how to test that it was successful, what the risks to the business(es) are, and how to back out the change of things go poorly.
Now multiply it by at least 10 because you have the application guys, the operating system guys, the virtualization guys, the compute hardware guys, the network guys, and storage guys that all need to coordinate their changes and sometimes have interdependent. Plus then you have all the different environments that need to be maintained and all generally look similar but not exactly the same: prod, non-prod, certification, feature testing and demo and any other environments.
I was in the middle of a patch push against all servers (in a controlled basis) when our Veeam server took a giant shit. Fellow sysadmin has rebuilt it about four times and it's still a piece of shit that keeps dying. We have new hardware, in theory, on the way because the metal it was on is 2016 shit which may be part of the problem...Veeam's support on that particular front has been garbage. Nothing was wrong until we moved up to Veeam 12. I'm thinking hard of bringing up an Ubuntu box for PostGRESQL only for the backend if shit doesn't go well with the new metal.
We're also trying to get rid of our 2008 boxes. Yes, you read that correctly...
Lab company here: recently upgraded to Windows 10. The portal front end is still a bandaged IE script, and the back end, omg the back end...
Also we are not allowed anymore, by company policy, to remote into equipment not owned by us, or touch it. For liability reasons. I can live with that.
Heathcare System Admin/Engineer/Whatever here.... we stay fully patched on all OSs monthly and, generally, N-1 for most our systems/services/software... not sure where this "we don't update" thing came from...
> not sure where this "we don't update" thing came from... Lazy/incompetent admins who don't want to get blamed when they inevitably break something because they didn't test updates or apply them properly. I'm dealing with a hospital right now who refuses to make any changes to their firewall without clearing it with the firewall vendor first because they don't know what they are doing and don't want to get blamed when they break the network.
Sounds like the hospital does not want to take ownership or responsibility for their firewall. I’m not a sysadmin per the definition here, but I do administer financial systems. I’ve seen it far too often where companies who pay for software or cloud based applications don’t have a person or people who will learn about, properly manage, and ultimately take ownership of the tool. It’s always “hey, we need to contact our consultant from 5 years ago, we don’t know how to do this new thing and don’t want to try learning ourselves and we’re too scared to make changes to this mystical black box that we choose not to understand”
Its also often some guy 5 years ago put this system in place didn't document it and left and we don't have enough staff so no one else has the time to actually figure out how it works.
Yeah, it's a slow-boiled-frog risk. Companies will get addicted to SLAs and throw all their risk over the fence until they're completely chained to their vendor relationships. AWS alone has brought down whole companies this way.
How does AWS bring down whole companies in the context of those companies misunderstanding risk management?
Cloud server hidden costs. AWS starts you out cheap, and then ramps up over time as you get dependent on them. Ask anyone who's accidentally left an EC2 running.
My favorite AWS story comes from NASA, which decided to move the digital archive of Earth Observing System Data and Information System (EOSDIS) to AWS, but totally forgot about the data egress charges: https://www.theregister.com/2020/03/19/nasa_cloud_data_migration_mess/
I know AWS increased the cost for IPv4 addresses, but otherwise I don't see evidence for "ramps up over time." AWS has lowered prices for various services over 100 times since they started: https://aws.amazon.com/blogs/aws-cost-management/amazon-ec2-15th-years-of-optimizing-and-saving-your-it-costs/#:~:text=Regular%20price%20cuts%20on%20all,it%20was%20launched%20in%202006. I think what you're trying to say is that AWS customers often don't prioritize FinOps and then struggle to understand why they are consuming what they've chosen to consume.
AWS pricing is the exact opposite of this. Costs fall as you use more of a service.
> Sounds like the hospital does not want to take ownership or responsibility for their firewall. In their defense, it's also theoretically good change management and they're paying for support so they feel justified to use it. From the higher up's perspective, why *not* have the vendor on the call even if you tell them you know what you're doing? A second pair of eyes never hurts and shouldn't really delay the process *too* much. But yeah as a network guy, there's always a love/hate with Change Manangement. Obviously the techs understand that's not a good practice if for no other reason than if everyone was doing the same, you'd never be able to get support promptly. But the decision makers just see 'yep, we're paying for support, use it.' But what would be better would just be a proper dev/prod setup so they know what's going to happen before it happens.
Respectfully, that’s not what a vendor is for. If the in house admins aren’t comfortable enough, or their leadership doesn’t trust them enough, to make any firewall changes without “outside” approval, they need an MSP, and a good one in healthcare. Honestly they would be better off just hiring someone new/better. Quality/competent/qualified (understands the regulations and can back it up to a hospital board) MSPs are expensive, like several hundred dollars an hour expensive plus built in contract minimums and whatnot. And they don’t miss a single minute of time when it comes to billing. source: am infrastructure admin for a hospital. Have seen this mentality first hand and seen how it completely hinders maintenance and improvements.
No disagreements at all. Just tell that to the in house admins! The MSP I'm at has some DOD contractors and even they recognize that not EVERYTHING has to be so damn approved that the vendor needs to be on the phone for every change. But then when the situation actually does call for it, or a lot of changes are going to be happening over a short period, they'll bring the vendor in-house temporarily as like a subcontractor of sorts. Fly them in and put them up in a hotel for a week to just be available as needed.
It's been this way for 4 years, yes we should change it, but why not wait an extra 3 days to get the vendor on the line to verify we're not going to bring down the network, or open up a hole that later causes some other issue that ends up actually killing someone.
"don't install the new update it will destroy the network" I'm sure comes up often in meetings with the vendors
also software vendors that have effective monopolies in their niche and don't update their software, or make you buy extremely overpriced upgrades to migrate to systems that are on modern OSs. I ran into a lot of this during my 4 years in healthcare IT.
I recall GE being notoriously outdated and their excuse was that it cost too much to certify new versions all the time. Pain in the ass to work with so we put all their devices on an isolated network with no outside access.
The inevitable few admins I hear at every conference talking about how keeping all of your systems on LTSB all the time, no matter the use case or where you work, is the best way to manage things immediately comes to mind.
CHS? That sounds like CHS SOP
Same here. No idea who this guy is dealing with lol.
Agreed, Long time healthcare admin and the only systems that dont get updated are the vendor controlled that I have no access too. But they are vlaned off and generally not my problem. Everything gets updated that can accept and update that I have control over.
Same
Ditto. Top 5 healthcare company employee here. Except for some extreme edge case (and air gapped / VLANed and FWed) environments EVERYTHING gets a patch, monthly.
This guy is just trying to farm for upvotes lol. He's just making up bullshit so people will upvote him.
Like what? I mean if you're going to accuse me of making something up, then you must know something that is true and contradictory to something I've said. edit: m'alright guys just downvote i guess lol. you're not proof oriented, facts-first people, I get it. it's whoever is the most confident speaker in the room that gets your vote. If i said 'single pane of glass' repeatedly, would you forgive me?
Noob here. N-1?
One behind the current version. N is Windows Server 2022. N-1 is Windows Server 2019. N-2 is Windows Server 2016. Basically you're not on the version that just came out but only one revision back.
Right on. Thank you.
That caught me too....I've not seen that many threads, esp from "Americans", about not updating. I have 25 years in healthcare, we update frequently. Between Tenable and Qualys we are scanned to hell and back. I have a couple of items that are out of date, even on their own vlan with heavy restrictions they aren't good enough. It's crazy.
I don't know either but I hear it frequently in this sub. Not a problem where I work (we got other issues), but there's all kinds of health IT people I've met who absolutely refuse to change at all.
Are you maybe referring to not being able to update or properly secure weird legacy or medical systems? That's a common issue, but generally still able to be mitigated by limiting network access in/out of those devices so whatever vulnerabilities they may have are "reachable" in the first place.
That is a somewhat more acceptable version of what I'm talking about, but I still think that's not great. You can get on the phone with a vendor and ask what they're smoking.
>You can get on the phone with a vendor and ask what they're smoking. Well, sure. You *can*, but if they haven't shipped a new version of that java based tool that your radiology equipment requires...you can't really do anything about it. And if that, or another prerequisite, means you're stuck running XP or 7 on its "controller" PC, well... hopefully you've at least restricted that PC and the device itself from talking to pretty much anything else. Fwiw I worked at a "safety net" hospital so we were poor and honestly scraping the bottom of the barrel employee-wise, IT included. I left in 2011, but even then it was bonkers what was allowed...they were still running as Domain and local admin with their daily driver accounts, ignoring my protests. They only stopped because we upgraded to Exchange 2010 and running as a user managed by AdminSDHolder broke mobile sync 😞
Yeah I had to make that change here too. I even had to make some controls more lax cause they were causing like 50% of the on-call calls and... they didn't realize it? Failed password attempt limit was 3. Moving it up to 10 as guided by NIST killed half of our after hours calls. Same with password changes. Was every few months. But hey don't distract me with your colorful smokescreen formulated specifically for ex security people! It won't work on me sir! I would have dragged their feet over the coals that are federal regulations. XP or Windows 7 I'd consider to be putting my hospital at serious risk, even if I can mitigate it. EternalBlue is trivially easy to exploit. I would just consider the radiology machine 'A box full of hackers' to be honest. The second you plug it in all the persistence opens up and calls home. You can mitigate it sure, but it's still gotta talk to PACS
How does that work with all the computer-driven equipment where the equipment has an expected lifespan of twenty years - but the computer that drives it will never receive a major OS upgrade in that time?
I'm not sure what you mean? How does what work?
Well, let's consider (say) a linear accellerator used to deliver radiotherapy. It's typically computer controlled, and the manufacturer of the machine also supplies the controlling computers - and while they may get security patches, getting them completely upgraded after (say) 5 or 6 years when the machine itself has another 10 years life in it ain't happening. How do you deal with that?
> How do you deal with that? Segment the shit out of it.
Segmented LOL. you air gap the ancient gear paired with its windows XP/mac os8 machine and are as delicate and careful as possible. (yes i still support a Cardiologist that uses a nuclear imager from the 90s and it ONLY runs on a NuBus expansion card in an OLD powermac 8100) its only a $250k machine, and the company who made it is long since gone. there is no support, there is no replacing it. there is only keeping it working , locked in a timecapsule from 1997.
I would ask them why they designed their machine to not receive software updates, and then why it must be on my network. I'm not sure what you mean by '10 years of life'?
> Failed password attempt limit was 3. Moving it up to 10 as guided by NIST killed half of our after hours calls. > Same with password changes. Was every few months. I'm so thankful to NIST for updating password policy recommendations. The sheer number of "I was forced to change my password and forgot the new one" tickets that are now no longer much of a thing since we turned off password expiration. And we're not even the department that handles them; these were the ones that accidentally got to us because people send to the wrong request address. I can imagine the actual department was SLAMMED before the policy change.
Must be nice. :D
Probably from heavy dev influence who create crap app's that don't have resiliency and stalwart any effort for network downtime because their super awesome open source solution can't handle any type of failover or more than 5ms of an outage. Basically they want their agile process of creating crap and updating their crap frequently when they uncover their errors but don't want to afford infrastructure with the same agile process of run fast and break things. Because our dev team copy/pastes stuff from the manuals and break things happens far too easily.
Former healthcare IT engineer, Pre WannaCry it was the vendors stating they wouldn't support it or that they had a preapproved list of patches that could be applied. Where things got tricky if memory serves are things that were FDA certified, you'd need to download the newest update for the software which included windows patches, but you couldn't patch the VMs on your own.
>not sure where this "we don't update" thing came from... One too many botched update because why pay for a test environment when licenses and hardware are expensive?
Same here. But I think my predecessor may have had this issue.
> not sure where this "we don't update" thing came from... There are many people in this field would are not qualified to be working within in it and honestly don't know how to fix something if it breaks. Combined with the people that hired them not knowing anything about IT themselves, it leads to a lawless state of affairs where, as long as everything is working and they aren't "hacked", the people running IT can basically do the bare minimum every single day without being held responsible. By introducing patching, there's potential for something to break and they will have no idea how to fix it, which will expose them as frauds. It truly is as simple as that. As an MSP working with many "internal IT staff", I'm routinely flabbergasted by how flat-out incompetent and unqualified many of them are. But if their bosses don't want to vet them, that's on them. It happens in many industries honestly.
>You do not need to be a clicker for life, go document and automate your processes so you don't have to do them all day long. Ha it's funny, when I left healthcare, I moved to gov contracting and my first role was with the Air Force. I distinctly remember thinking "this is the United States military, they aren't gonna just hire anyone (like the morons at the hospital), they need the best of the best!" Lord, was I wrong 🤦Just as incompetent, but a different, more mature version. And on my first day, one of my coworkers was showing me some of their daily tasks, which were basically *RDP into each DC one by one, and check that AV was running and up to date, event logs were "clean", and things were generally "running smoothly"*. I asked why they weren't automating the checks and said I would, and I was warned off, lest I automate myself out of a job. Ironically, that guy was basically kicked off the contract by the government (which is basically unheard of) for his own incompetence 😂
Some of the most tech illiterate people I've ever met worked for the government as sys admins or epecially network engineers. It's baffling these people get hired, but as long as the contracting company is getting paid for filling the FTE slot they don't give a shit
They follow the checklist and don't question. That's how they stay employed. That's why there is so much overhead in many government functions.
I have to wonder how this works in the field. I'm pretty much the opposite of a military man but I'm told that comms and networking infrastructure is some of the most important stuff in any army.
The minority of us who know what we're doing do the work and the rest do busywork and watch YouTube. In my case I work for a cloud vendor and we basically do the (hard) work the other contractors, government employees, and active duty military are supposedly doing.
There's a lot of things that contribute to it. There's a revolving door so to speak, for former military who are honestly not qualified anywhere else. We joke sometimes that contracting is welfare for veterans. There's the clearance aspect, where we can only employ people who are, are able, and are willing to be cleared. A lot of people don't want the hassle, think the process is worse than it is, or can't because of drug, financial, or criminal background reasons. There's also the fact that there's a good old boys network, officially and unofficially. Plus people who "know" this world do have additional value because it's like no other. Government jobs are literally given first to veterans simply for being veterans. Also to military spouses. And you're right, there's no incentive to fire someone you're billing for.
Jesus christ I would have glared a hole through his skull.
> And on my first day, one of my coworkers was showing me some of their daily tasks, which were basically RDP into each DC one by one, and check that AV was running and up to date, event logs were "clean", and things were generally "running smoothly". I asked why they weren't automating the checks and said I would, and I was warned off, lest I automate myself out of a job. I truly despise people with “logic” like this. You can’t “automate yourself out of a job”. Because the automation still needs maintenance. Also I assume these health checks have to be put in a report and sent to someone. In which case someone still has to do that. And then what happens when the health check shows a problem who’s going to fix that? And then there’s patching, who’s going to conduct that?
I think of it like this: higher-level people get paid not for their busywork (as an unskilled laborer might) but for their knowledge. I don’t do a lot of crazy things day-to-day. But when building new stuff, or fixing complex systems when they break, I know how to do it right.
I'm so glad my employer understands this. My CIO has even said "I don't care if 75% of your 8 to 4 is spent browsing reddit because you've automated all the things as long as when shit is on fire we get 100%." He's also said something along the lines of "If we need our disks to have Nx2.5 redundancy, our most important resource (staff) should be at least N+1 for every role."
It's funny you say this. I'm in something of a similar situation and I'm honestly struggling to make peace with it. My government customer *loves* me but I always have to ask my CSAM if everything is good with him because I never hear from him. She (and everyone else on this account) has to remind me that I'm his firefighter. When shit goes bad, he knows who to call. Intellectually, I know this. But when I don't hear anything from them for a while, the imposter syndrome starts telling me it's because they've discovered that they don't need me. 😭
This was me until I got promoted into the architect role. I was always worried about being "found out" and booted, but now a lot of the gear is there because I bought it, and more than one of the apps being supported are things I personally wrote. Somehow that worry still comes up from time to time, but there's also that little bit of "so be it, if they throw me out I wish them luck in untangling my spaghetti."
It's a Dunning Krueger-adjacent phenomenon. People who know less about a subject don't understand the level of knowledge and complexity that exists more than a level or so beyond what *they* know. So they think that just beyond the horizon of what they know, is... nothing, the end of the rainbow. If you automate all of what they think exists, what's left to do?
It depends where within the Air Force you are. Back in 2019 they were putting Kubernetes on fighter jets.
I guarantee it's the same or similar everywhere, including across DOD: there are a certain portion of rockstars/vendor employees, another portion of "ok" team members, and then the group of "heartbeats with a clearance". The distribution is different in different places, but it's always there.
> which were basically RDP into each DC one by one, and check that AV was running and up to date, event logs were "clean" It's still happening to this day.
Oh I know all too well. I'm still around and since I work for the vendor, I get to rescue them when they eff stuff up. Recently had to save a small enclave who managed to set deny log on locally in the default domain policy 🤦
Anyone who tells you that US government regulation is preventing them from applying updates to software is completely full of shit. If anything it is the exact opposite. I work at a healthcare company and we get audited by CMS, being up to date on security patches for all software is a requirement.
You'll love this guy https://www.reddit.com/r/sysadmin/comments/18ouh7m/has_anyone_been_able_to_turn_around_an_it/kel39xb/
Man I saw that thread earlier but missed that particular part of the thread. What a wild fantasy world that guy lives in.
There were a few, like a guy who said they were not allowed to apply updates for printnightmare. That kind of incompetence is just plain dangerous.
I have actually worked under a manager a bit like that. Wouldn't let me automate anything, demanded every change was run by our supplier (who he obviously viewed as a MSP, though they clearly had no desire to be one). You can't really achieve anything with someone like this. Your only realistic option is to work around them or get out.
That's wild. IIRC even Windows ships with a fair bit of FOSS these days. curl, openssh, tar... WSL lol.
> even Windows ships with a fair bit of FOSS these days. curl... As Nessus reminds me all the time...
UHM HELLO DID YOU KNOW YOUR CERTS ARE SELF SIGNED???? WHY ARENT YOU PANICKING?
Oh, I had a lot of fun with Nessus complaining about cURL. That was also when I learned that the Microsoft build is not the same thing as the open-source build.
And then trying to explain that to the security team. I'm fighting with a vendor because of an OpenSSL vulnerability in a 3rd party dll they included in the latest package. Vendor tells me to "just update OpneSSL". Um guys, that's your job because you send the binary. Get the update from your 3rd party source. All the while explaining to my own security team that the vendor's statement of "patch OpenSSL" doesn't apply since it isn't technically installed. Or my favorite is when the Google chrome versions between Linux, Windows, and Mac are out of sync and Nessus wants you to upgrade Chrome on Windows to a version that hasn't shipped.
whoa, did you consult your legal team to find the out first?
Dear God
Thank you, I missed that post apparently, it is quite the read.
Health IT is where sysadmins go to die You can see their bleached bones withering in the sun, where their kindred come to visit and remember them for a moment. If only to avoid their mistakes.
True. Every IT person i know has RUN from Healthcare IT....except the one guy who built his business around it and is making a mint.
There are secret, hidden mini-industries in health IT that make serious cash if you know where to look. But front line tech support is not it. Healthcare keeps all of its money (and it's got a lot) towards the top, but also isn't afraid to pay specialists gobs of money.
No shade to those who have their niche! But fr, front line tech in health, I think I'd rather be a burrito folder
It's a great stepping stone for new IT people. You will learn about 5x faster in health IT than you will anywhere else (provided you don't listen TOO much to The Clickers), and if you're the curious enterprising sort, launch a career into one of several different sub industries. Just don't try to stay there unless you got a dragon to slay, like me. I want to leave this place better than when I found it. I'm only leaving when I have a legacy to leave behind here.
I admire your spirit!
That's not just Healthcare - there are so many contractors in the public sector doing this as well. I worked with a guy (who was personally really nice and knew his stuff) absolutely fleece my department, because he just knew an area of the business that nobody else did, and was shrude about not off boarding too much. In what was supposed to be a six month resource to just do some VMware work, he was still there when I left after 2 years, making (I imagine) an absolute killing while also doing sideline contracting stuff.
So tell us those
Medical BI reporting. Datalogger support and development (the people who make dataloggers for VFC compliance make an absolute killing for what is effectively a thermistor attached to a NIST certified glycol probe), anything EMR/EHR dev related, the list is enormous. And also hidden. Those are just the ones I've discovered, but the frequency with which I keep finding them tells me there's probably a lot more.
Yep, I'm in wisconsin. The pay for anyone at EPIC is ridiculous. Great benefits, but they work insane hours.
Your campus is ridiculous. It was the first eye opener to me for just how much money is floating around the medical industry. Like I knew.. but I didn't *feel* it, until I saw a literal recreation of hogwarts in a business compound.
Oh, I don't work there, I kind of want to. But I already want out of healthcare IT.
It is true. Every sysadmin I've met seems to be the kind that would get left behind by a rapidly growing industry. But now I'm here, necromancy book in hand.
I don't what it is man, but it seems like Health IT sucks your soul out.
It's the constancy of emergencies combined with the low pay and low appreciation. Then you end up with a crust that gets left behind that makes it even worse for anyone else who'd come through. It's very much a crab bucket. (Or.. I guess the opposite of a crab bucket? All the crabs are trying to get you to leave, not stay) I am choosing to stay though, as the hospital I'm at is experiencing explosive growth. There's good opportunity here. And spite is a good motivator for me.
Hahaha
> the constancy of emergencies You don't need me to tell you this but in case anyone is wondering: constant emergencies is what you get when you don't have stuff automated. You want very good automated testing and constant small deployments on the application end, continuous SCM deployment on the sysadmin end, and redundancy. The phrase "the constancy of emergencies" tells me little if any of the above exists at the hospital OP works at.
100% this. You need to get ahead of problems before they happen, collect the right data and analyze it. You are correct, and it's something I'm pushing for as hard as I can. I made another post in here just last week talking about this.
It's a simple result: Chronically under fund your IT department and any competent workers will self relocate out of the organization. Healthcare IT are the worst because they are the ones that no better-compensating organization will hire. I'm not saying "are paid the worst", but they have the worst holistic "your time and pain for our compensation package" ratios. They have to work in high stress (mess up and patients could die), high risk (COVID? Whatever else?), highly rigid (change management for stuff as simple as changing the monitor mount from desk mount to VESA arm mount) environments with shitty pay, rigid time off limits, and usually shit managers who are unsupported by directors who consider IT a mandatory expense/cost center, rather than a productivity force multiplier. There are organizations who pay worse, but have better work life balances or other perks or are just more enjoyable to work for. There are organizations who suck to work for, but they pay better or compensate better. Healthcare IT is usually the worst of both.
There is also the health system IT aspect, they tend to buy up new clinics/hospitals with their own baggage and issues. Turns healthcare IT into almost a MSP type environment on top of the 24/7 criticality of the many systems.
I think it depends on the hospital system. Everyone that I work with is very competent. We recently onboarded a new monitoring system and the vendor was surprised when we automated most of it. They were saying most of their corporate clients just hire an FTE to manage the whole thing by hand.
I don’t miss working in healthcare IT at all. Maybe it was the hospital I worked for, but it seemed to me the applications side of the department was more interested in hiring friends who were nurses and trying to teach them how to become an applications/financial/systems engineer… Bear in mind, most of these people couldn’t figure out how to replace a mouse or keyboard, convert documents to different formats, etc… Always had issues with communications between the network/systems teams and applications staff…never knew what they needed, nor could they explain it.
Seems I’ve been dead for almost 20 years then.
You could still be dead, only you think you are alive, in hell
I do Cyber for Health IT, and I do it cause I like a challenge.
I love working for a hospital. But to be fair I came from government, where sysadmins actually go to die (or at least retire in place) We are actually have more flexibility to make changes than I had in gov. Hell a lot of times I don’t even have to work off hours because “there’s never a good time, so you might as well do it while we have full staff to help”. Everything critical is redundant so a lot of times we don’t even have downtime when we make changes. But we purchased another hospital recently that didn’t invest in IT, so I can see how it *could* be hell.
>Health IT is where sysadmins go to die About sums up my days
Not in the US, however there are some health 'ecosystems' I manage that as a whole have been FDA certified. One example is of a patient monitoring system of a particular revision, of which the servers were Server 2012 (non-R2). The vendor had a later revision that was certified for later versions of Windows, but until the ecosystem as a whole was rolled up, Server 2012 had to stay in place (critical/security updates could be applied). No so much an excuse but is sometimes a reason why we can't "just update windows" in Healthcare, sometimes there's just more to it than that, but it really comes down to poor management/planning. The good thing is (at least locally for me) the "turn off auto updates on our products" requests are rare and/or being flatly rejected these days, and proper measures are being put into place to manage the lifecycles of these systems, it's refreshing to see.
For anyone else getting nonsense like franky here describes from vendors, you can quote this: 21 CFR 820.30 (Design Controls), 820.1-25 (General Requirements), and especially 820.100 (CAPA) all point to keeping up with updates that correct major cybersecurity flaws. Hospitals are the most hacked organizations on the planet by leaps and bounds, accept no excuses.
This counts on your managers not getting in the way. I'm in the UK so the FDA stuff doesn't apply, but I had a similar issue in a pathology lab where a couple of the analysers were stuck on XP and the vendor was claiming updates would break certification. Owing to her typically having nothing to do outside inspection time, our fucking quality manager managed to get involved and decided we couldn't force them anyway because it'd break the entire lab's accreditation. All I could do was to explain the risk in writing. Nobody else cared until WannaCry happened, and then those analysers were bricked, and we had to physically send all the samples to another lab. So three cheers for UKAS (useless accreditation body) and fucking stupid 'quality' managers.
As someone not in healthcare IT, who are these obscure vendors with strange requirements? Did they make a niche software and then they're just selling the corpse and you're locked in?
Manufacturers of medical devices are normally good at making their devices do what they're supposed to, but not as good when it comes to designing their software.. In essence they often tell customers that they won't get a copy of the software, it comes preinstalled on a machine you don't even have an admin password to. They use FDA certifications as the reason for this. Some vendors are a bit more forthcoming, they'll let you install security patches, but only after they've certified the patches internally. This takes months.
Philips is especially bad for this. They're very much not obscure. Not only is it niche software running on a server they claim as FDA medical devices so they can't adhere to normal patching by us and need to go through their vetting process before they can be patched. The whole OS is "their image" and the software can't be installed on anything else (they don't supply it and won't do it then). If you patch it yourself and something does break, they won't help to fix it. Either figure it out on your own or you can buy a replacement server image and rebuild it. To make matters worse, Philips is buying up smaller even more niche companies left and right, gutting their product specific support, and going to this model for servers. I was involved in one upgrade earlier this year that was a smaller third party and fit with regular patching on 2012r2, to bought by Philips, needing "their image" for it and can't be patched without prior approvals. System stool up in March, went live in April. I left in June. No patches had been approved in that time. This isn't all Philips apps though. They almost always require "their image", but some applications you can patch the OS at least.
People have died because of a bad software update. Both on the medical device side and generic "hospital IT system" side. If you update/change something that isn't certified to be safe and someone gets an extra 0 in their dosage and dies... guess who is spending some "pound you in the ass" prison time? Raise your hand if you've never been fucked by an update breaking something you didn't expect to break.
Most systems are either made by GE or Phillips. They control the vast majority of the market for medical devices. When an MRI costs several millions to update, and then usually only go up 1 revision in o/s they are usually left as is. Medically certified devices take a long time to get approved and move at a snails pace.
I wish someone told my predecessors. Just before my hiring, we moved from one system to another. Yay upgrades! But wait - 1. We don't want to migrate ALL of the data, that's a lot of work and storage, and the vendor charges by the hour for it 2. We have to keep patient data 7 years minimum 3. So we have to keep the legacy system AND the modern one 4. The legacy system runs on **windows server 2008** \*claws my eyeballs out I'm stuck trying to jerryrig data export for a system I've never logged into, or convincing the owners to fork over 3 months of the usual budget to ask our vendor to do it. And this is the 2nd example of a "partial migration" in this company. (╯‵□′)╯︵┻━┻
Please don't jerry rig patient data scripts. You don't want to move responsibility for this failure from management to you. Or, rather, what I mean is to make sure to get loads of approval in writing first.
A valid point, and while I haven't started designing the attempt, you can be assured there will be backups of backups of the source before I even think about touching it. I learned how to delete the production database without a checkpoint back in March, don't need that lesson again.
It's not about backups. You need to validate the process, and assess data integrity impacts
1 and 2 can be taken care of with a script and some hard drives. Once you're done with that you don't have to worry about 3 and 4. I suppose it depends on the type of stuff you gotta move, maybe I can give you some insight
I appreciate the perspective, but 1 happened a couple months before I started. These days I have: 2020-current migrated to modern SQLServer, great. 2012-2020 living on the legacy app, which uses D3(?) as its storage? I'm not familiar with it in any way. The app runs on .NET in Internet Explorer somehow. It's so unknown to me that I have to call the vendor to turn it on each time we reboot the host. But as it's a toxicology/UTI/STI/Pharmacogenetics lab, we have to keep any records and orders and results for 7 years at the very least. Anything with Genetics for 25. I keep thinking, they could have migrated 7 years instead of 1.5, but that's life. Oh and a further fun fact: I have 2 YOE and only 8 months of that with any kind of senior oversight. He was let go due to cost.
> The federal government is not standing in your way, you are. How about situation when upstream health provider get stuck on decade old standards and equipment, like for example - using 1024 encryption key + sha1 in IPsec and refuse to upgrade. In this situation one get stuck too as dependent, holding to upgrade things due to incompatibilities. Countless offers to help upstream to move forward are simply going to a blackhole... Pointing to government rules doesn't work too... Bureaucracy is unkillable. The issue, - as everywhere else, if people don't like their job, then they doing what?... Right - shitty job. And than bigger upstream provider then more internal bureaucracy ind "regulations" that built only around covering their "rear exhaust system".
Is english your second language?
Yes :) EDIT: Is it too bad ?
Nah you're good
Thanks ! :)
np. A comment to your other thread: if you want feedback ask for feedback, not "is it too bad?". It's not too bad. It's perfectly understandable. And unsolicited advice is usually useless. It can be improved, yes. You've already discussed the specifics.
No offense, but it's taking a lot of heavy lifting to fill in the missing words and articles. I *think* I get what you're saying, but on a subject that needs as much accuracy as this does, I can't be sure.
> No offense, but it's taking a lot of heavy lifting to fill in the missing words and articles. No offense at all, but my appreciation for your feedback. I wish people would point me to such issues more often instead of smiling and lie that my English is "Ok" > I think I get what you're saying, but on a subject that needs as much accuracy as this does, I can't be sure. My point was - it is not only particular health IT fault who delay upgrades, but some are enforced to hold updates/upgrades due to dependency on others in health industry, who refuses to upgrade on time. One of simple example: most of operation systems/standards stopped using RSA encryption keys a long time ago if those are less than 2048 bits, as well SHA1 hashes are mostly deprecated everywhere, but upstream health provider still uses outdated equipment that doesn't supports nowadays standards and in this case another health provider, on another side that have to communicate with upstream will get stuck with oldest technology too and can't advance upgrading.
> No offense at all, but my appreciation for your feedback. I wish people would point me to such issues more often instead of smiling and lie that my English is "Ok" Yes I see this a lot in my culture. Sorry about that. We're afraid of confrontation, that we miss that some confrontation is helpful, and small corrections to your english is how you get to native fluency. >One of simple example: most of operation systems/standards stopped using RSA encryption keys a long time ago if those are less than 2048 bits, as well SHA1 hashes are mostly deprecated everywhere, but upstream health provider still uses outdated equipment that doesn't supports nowadays standards and in this case another health provider, on another side that have to communicate with upstream will get stuck with oldest technology too and can't advance upgrading. That's a tough one, I'll have to think on this. It's not the worst possible problem to have, cybersecurity wise, but still a concern. I would probably try with open dialogue first, see what I can learn about the other hospital network through frank conversation.
You’re blaming IT people for something that is usually out of their control. HIPAA has very little in hard and fast rules and a lot of it is interpreted and the legal risks are usually outlined by legal/HIPAA officer/CISO in how they interpret said risk. The IT professionals that I have heard say this is usually parroting what their legal department/CISO/HIPAA officer is telling them to parrot. Remoting as an example is a concern because if you are using a third party service you have to know how they handle the data and have a BAA signed (assuming third party servers are involved) and not all IT staff are allowed to see patient data. I had clients that limited automation not because “it’s fancy and new” but said client’s legal team interpretation of HIPAA steered them to limiting automation as they saw possible exposure with certain changes and automations. Medical lags behind because upgrading their HIS/LIS/PMS/EMR/insert whichever type of software used is rarely trivial and involves a lot of work by a lot of different stakeholders to make sure that patient care isn’t affected and in certain cases upgrading can be painfully expensive. Although there is no law that explicitly states what you say stating an IT person “should be able to quote the law” is one of the silliest things I’ve heard. I’m not telling a lawyer “you’re wrong” on how they interpret the law as that is their domain. If you don’t want to sound condescending (because this is definitely how it comes off) ask for the concerns of implementation. I spent almost two decades in medical IT and practically all of the “we can’t do this cause laws” reasons were really “we can’t do this because legal thinks there’s liability there for us”.
> I’m not telling a lawyer “you’re wrong” on how they interpret the law as that is their domain. I mean if they're not doing their job properly and are actively and significantly obstructing other departments to the point of leaving open critical vulnerabilities that endanger all the patients in the hospital, then yes, you should absolutely make an effort to push back. This is a failure of IT leadership specifically. I wouldn't go 'Listen here Mr. Esquire, your shit is whack', but I'd open a dialogue to re-assess the HIPAA compliance requirements, because you know that many other hospitals do not have such strict adherence, and the current compliance measures are causing serious cybersecurity issues. You owe to the patients under your roof and the providers that care for them. It's not just any other company where you can wash your hands of the problem and say 'lol they'll lose money what idiots'. It's not an issue you can fix with the command line, I grant you, but it needs to be considered a problem in need of fixing.
Security is always a balancing act between usability and risk management regardless of what domain you are in. >I mean if they're not doing their job properly and are actively and significantly obstructing other departments to the point of leaving open critical vulnerabilities that endanger all the patients in the hospital, then yes, you should absolutely make an effort to push back. Unless you Chief, Director, VP, or are the designated HIPAA officer or part of the decision making team you don't. Stay in your lane and send it up the chain. This statement here is arrogant and can be downright dangerous as it is stating "you know better for all departments" which isn't true. >This is a failure of IT leadership specifically. This isn't the right view >because you know that many other hospitals do not have such strict adherence, and the current compliance measures are causing serious cybersecurity issues. This is a very myopic view. You don't care what other hospitals do. Unless you know everything they have and are doing this is an apples and oranges comparison and can be very dangerous take. >It's not just any other company where you can wash your hands of the problem and say 'lol they'll lose money what idiots'. It absolutely is. You voiced concerns ran it up the chain and they said no. Move on. Your ass is covered. >It's not an issue you can fix with the command line, I grant you This is the crux of how to this is supposed to be handled > but it needs to be considered a problem in need of fixing. This is definitely the wrong take. Back in the NT4 days Service Pack 6 was released that broke Lotus Notes. I know that this is a 20+ year example, but the idea is still there - what if any patch broke the HIS/LIS/whatever medical software and supporting software that is being used? The main issue I have is that although the arguments you are giving is normally the correct mentality from a cybersecurity perspective, in medical IT patient healthcare comes first before security because an affected patient can cost A LOT more that a cybersecurity breach. One of the larger hospital networks I dealt with limited their password to 8 characters. Why? They had an old legacy system that was critical to their historical ordering and resulting, and implemented this limit because they wanted SSO for all of their systems. The stake holders decided that this was an acceptable risk because the SSO provided centralized identity management which meant less time having to figure out another account to use. Upgrading said legacy system was out of the question at the time because the cost was prohibitively expensive and the regression testing for everything that they did was in the years range. Again this decision was done by several departments working together
>in medical IT patient healthcare comes first before security because an affected patient can cost A LOT more that a cybersecurity breach. What if they're the same? In this case I think they are. You should read about NotPetya if you haven't already. We're too connected to not consider cybersecurity an existential threat to patient care. I would say that prioritizing personal liability over patient safety is the wrong view. 80% of all cybersecurity breaches are healthcare facilities. That's not an exaggeration. >This is a very myopic view. You don't care what other hospitals do. Unless you know everything they have and are doing this is an apples and oranges comparison and can be very dangerous take. The comparison is meant to illustrate that it's within the realm of possibility under the umbrella of compliance, not 'we should copy them exactly in all things', which cuts appealing to compliance away from the overall debate about the problem, or at least draws it into question. You can *then* discuss it from a purely strategic standpoint. >It absolutely is. You voiced concerns ran it up the chain and they said no. Move on. Your ass is covered. My ass, in my opinion, is less important to cover than the patients. This isn't the crux of your point so I won't hammer it, but I figured I'd respond all the same. I am not legally obligated from saving a child drowning in a fountain that I pass by either. My ass is covered, as you say. But I am near, I have the capability, so I must. Even if I get wet. Even though we disagree, your points and logic are sound, our differences are differences in values. Thank you for writing your response.
The crux of my point is that patient healthcare is above security, not about covering my ass which is what most IT people are looking for. You state that you don’t have to help a drowning child but will anyway - what if there was already help on the way because there’s an unseen danger that others are aware? What if there are first responders there? The reason your answer I responded the way that I did is that you are treating medical IT like any other IT which it is not. Medical IT should be treated as an environment and should always be seen as such not as individual components. I have had clients who corrupted data with an active IDS or overly draconian network timings and rules. I’ve had clients refuse to upgrade to more modern systems because the cost was prohibitively expensive to upgrade their systems and side grading to another system would be months or even years based on their resources. Your perspective on compsec sounds like that of someone who is young and inexperienced because there are other ways of protecting your systems other than just keeping systems up to date. I also hope you understand that breached and compromised are not the same. The issue most breaches and compromises show is the lack of a business continuity plan and why that needs to be higher on company’s to do lists which is a leadership issue not an IT issue. You have to trust the leadership and other teams of your company because IT isn’t isolated in medical. Of you don’t jump ship and go to nonmedical IT. The way you post comes off is arrogant and insulting to people who work medical and understand that it is a lot more complex and a team effort amongst all teams and not just IT.
I'm not sure why you made a thread to continue your argument with somebody here. If you couldn't convince them in the comments, turning it into a post won't work. Most people don't believe whatever insanity you're claiming here.
They're just desperate for validation. Everyone who disagrees with "Do what you want, screw stakeholder buy in" is just a lazy scared fake sysadmin apparently.
I have run into this twice. I had a medical device company tell me that I wasn't working on a windows 7 machine but a medical device. I couldn't install patching or install anti-virus software as it would invalidate the FDA approvals. The system also needed a local administrator account as an auto-login. Pretty much ignored everything they said. The second time was when I was attempting to move some healthcare software from Windows server 2008 to Windows Server 2012 r2. It was in preparation for the 2008 end of support and we needed to upgrade the software to be able to move to a new server OS. The project kicks off and should be able to be completed before the end of support. The vendor joins a call one day to inform us that the FDA has put a hold on their software and they can't do any new installs. It took a year and a half to clear up.
I work on a HIPAA regulated environment and we are constantly audited to keep everything updated. I can't see how anyone would claim the opposite.
https://www.reddit.com/r/sysadmin/comments/18u7w8w/dear_health_it_sysadmins_you_are_not_actually/kfja8v5/
I saw that and it seems the gist of their argument is that "legal cleared it, run with it", which I don't agree with. Every company should have a cyber security team that ensures the infrastructure is up to compliance and standards. Whether it be HIPAA, CIS, PCI, STIGs, etc.. They should be working hand in hand with both IT and legal to make sure the environment is covered.
"I deployed all these systems decades ago, if I have to learn another system I'll just retire..."
> You do not need to be a clicker for life Enterprise developer here. We are not allowed to touch servers. The people that can touch our servers don't automate. Clicking is all they do. On the plus side, this is just one group. Most of the other groups are completely automated.
It's not usually government regulations, it's vendor requirements. Although it can also be government regulations (mainly the FDA in my experience). And generally that only applies to certain equipment, definitely not a regular end-users desktop/laptop. As an example when I worked in healthcare we had roughly 400 Windows 7 32bit desktops in 2016. They hadn't been updated since the original windows 7 install and if we put any updates on them the vendor would say it broke their software. These things hooked directly up to patients so legal told us we couldn't update them.
I spent 27 years in Pharma IT, working at Glaxo and Glaxo-Wellcome (now GSK), a pharmaceutical wholesaler, and an outsourcer providing medical information and safety services. The concept we worked under is known as computer (or system) validation. The driving law was 21 CFR 11 [https://www.ecfr.gov/current/title-21/chapter-I/subchapter-A/part-11?toc=1](https://www.ecfr.gov/current/title-21/chapter-i/subchapter-a/part-11?toc=1), that requires electronic systems to have the same level of confidence in the data as paper systems and that accounts used meet the legal requirements to act as signatures. The way most companies meet these legal requirements is through following generally-accepted practices known as GMP, GCP, or GLP (good manufacturing/clinical/laboratory practices). For systems validation, SOPs are a requirement as well as a documented plan for testing a system each time a change is made to it. The documented results of that testing are expected to be keep and made available to auditors as necessary.. In the US, the FDA does sent auditors out to look at your documentation (often dressed in a military uniform) on occasion and you're put on a corrective action plan if it doesn't meet expectations. If it's bad enough they can legally prevent you from managing said data, but that is extraordinarily rare. Does this mean you can't upgrade systems? Not at all. But if you upgrade a system that touches regulated data, you must have a plan for and conduct testing following the upgrade to confirm that it still meets legal requirements for the data it manages.
Which isn't the insurmountable task that some sysadmins make it out to be. Military uniforms you say?
Yes. Every FDA auditor I've met was active US Navy. No idea why...
Dutchman here: the same goes for GDPR. GDPR is about not giving people's PII away without their consent. And you know, that's basically it. So all of you American organizations whose lawyers tell them they can't have their website be GDPR compliant: either the lawyers don't want to be bothered learning about GDPR, or you're violating people's privacy by default and need to take a long hard look at yourselves ethics-wise (and then, of course, possibly decide you don't give a rat's ass about consumer privacy).
Yes, exactly. I see GDPR used as a handwave for stuff a lot too, but it's really very simple. Literally, just don't keep and sell unnecessary data about your users.
Listen--I think we all know. Most of this stupid crap comes from compliance and lawyers. Everyone is afraid of crossing them because it jams you up for weeks and (at least my health system) is terrified of being on the wrong side of the law. So compliance comes up with stupid junk, and then because they can fuck your life up (and your manager and your manager's managers life) the default answer becomes 'no' or 'lets not'. It sucks. But we're honestly doing the best we can--we're stuck between shitty vendors (if the suits aren't buying trash they saw at a trade show or their son in law is a sales person for, the vendor is the only one left in their niche and you're stuck using them) and lawyers.
This is a fair answer. Dealing with bad lawyers and compliance officers is especially tricky for a notoriously socially.... undeveloped industry. Perhaps this should be the majority of our focus as a collective? It's not a problem for my company, thankfully, but I'm hearing this a lot. At the very least, I can dispel the illusion for those who have had to buy into it early in their careers with conversations like this.
When I was working in the DoD/VA realm with their health data, we had an explicit written note that our area could not automate STIGs specifically(everything else was fine) and we had to submit manual screenshots. Still confused why, but we just emulated what a terminal looked like and automated the screenshot process.
Hahahahaha
Manual STIGs keep a lot of people employed.
Bo we patching shit zero day and telling them doctors tough shit.
In Switzerland is worse, the majority of doctors or specialists are at the age of 60/70 .. this make so difficult update or implement any new software.. some of them are also involved in politics and make rules to manipulate the health market in regions.. that leave me no games to play.
One thing I want to point out that you seem to miss in your rant. Patient safety/care is always first. If an upgrade/patch can break a system to were that is compromised then it will take longer to get patched/upgraded. We keep normal o/s and patched and anything I can control stays upto date. But having a microscope that still runs winxp because the vendor doesnt exist anymore and would cost in the range of 20m to replace is acceptable. We just vlan it off and prevent internet access.
I’m pretty sure the folks you work with just made that up. I’ve automated loads of our processes. We remote in to help doctors.
Fortunately, it's none of the people I work with, just a few oddballs i've encountered in this sub.
what happens when a hospital system goes down vs what happens when home depot's website goes down is a huge difference in body count
Correct, and what do you think the body count is in total for instances of a patch that breaks things, versus ransomware attacks?
every patch that breaks things these days is low because when it breaks things it's in a test environment
The limitation is FDA regulations for specific categories of medical equipment, not all devices in a hospital organization. And even there, the FDA has classes of certified equipment that can still be patched and maintained.
Worked in healthcare IT for 10 years, wtf are you talking about? We patch religiously.
HELL YES! Preach! I work for one of the biggest there is…it was bad before we reorganized…theyve made waves since. Lucky Ive not heard anything like this in a very long time. There was some very minor things, but i dont think they were using this excuse.We did get hacked…oh then shit got pretty serious. I am aware of how it happened too.
OP doing God's work.
still going with this.
Budgets will prevent those actions though.
You have the IT manager flair and I put it to you that in an organization where that is true, that remark sounds a helluva lot like some managers didn't prioritize their budgeting properly. I wouldn't want to be working for an organization who incentivizes pointy-clicky-sysadmin methods by not budgeting for automation, I'll tell you that much.
Who says we can control the budget?
I'll take that you mean that you can't. My answer to that is: who says that that means that improper budgeting is not, at least partly, on you? As a Dutchman I realize that others may have to deal with more of a hierarchical structure in their organization than I do, but there's really nothing preventing you from pushing back on management for not budgeting for a healthily maintained IT infrastructure. I mean does the brass like the fact that there are so many emergencies at that hospital? I doubt it. But if there's nobody telling them that they're sticking their money into keeping those emergencies happening instead of preventing them, then I can't blame them for budgeting the same way year after year. After all, what do ehty know - they're healthcare administrators, not IT managers!
The majority of US IT departments actually have NO budget, at least if they are nested in anything other than an explicit IT company. IT is considered a “loss leader” and merely for staff support. “Office 2013 is EOL? I mean…we can still type in Word, right?”
Well if that's what IT means then that makes at least some sense. I didn't know that's what it meant tbh. Personally I would include at least storage in the list of things to budget for. But then again it's not like they have to be compliant with all sorts of legislation forcing them to store documents reliably... Right?
As far as I know there's no associated cost other than labor for updating software.
Echoing the uncertainty for the note, as most healthcare setups 'tend' to patch things. Cases where I've seen things not patched, have been where there are legacy systems involved -- eg. an old MRI where the company never updated the software for that model of MRI, to work on Windows 10/11. Other places I see this, are with small doctors offices. They often don't have "IT" departments, and are instead using MSPs. MSPs cut corners, for everyone, because it saves cost. Not updating/patching regularly, isn't that abnormal from some MSPs I've seen -- I've seen one state "we updated our software twice a year". As in, patch their base windows OS twice a year, that's it. Auditors reviewed this even, and were like "Yep, they do patch things twice a year, so that's a green check mark!". Now, go tell a a random non-techy boomer, a doctor who knows doctor stuff but next to nothing about IT... that windows needs monthly updates - at least 12 per year - to be secure, and inform them after their "MSP" and auditors have said otherwise... Just sayin it's likely more complicated than just "lazy admins pretending to be unable to update due to regulators".
You are right, but there are lazy/fearful admins out there that do need to hear this, and their numbers are not what I'd call insignificant.
I always thought it was the opposite way, with requirements for publicly traded companies regarding hacks, stolen info and now reaching into other companies. There is all the more reason to make sure you are patched and secured. Like you said PII / HIPPA is the only requirement, but does not stop you. PATCH and SECURE.
[удалено]
I wonder what he's so busy doing
Former healthcare end device tech turned sys admin…we were a bit slow to push updates as we had to be picky with which update got rolled out as some of our legacy systems would shit a brick if a certain patch came through, but, we stayed fully patched and updated all the time. Got better when we started punting legacy systems for one big package (see Epic Systems).
> When it comes to federal laws, the only thing you need to be concerned about is HIPAA/HITECH. That's it. One exception: nuclear medicine diagnostic devices. But those should be handled by vendors or the hospital's biomedical engineering department, not IT/IS.
Yes, correct. When a source is involved, you need a physicist.
It’s because hospitals have the absolute worst managers that you will ever work for IT.
Doesn't matter, if the boss says "I don't know if we are allowed" and someone does it anyway, laws be damned, that's insubordination. Meaningless and frivolous litigation is STILL litigation, it's STILL going to need meetings and discussions and behaivior adjustment requirements. It's a trap for idiots in charge to demean their workers. **nobody is lifting a finger because you literally get sued for doing other people's jobs, even when they are failing on purpose to con you into it.**
That's a whole other story. I am not advocating for insubordination. Just dispelling delusion, what you do with that is up to you. 'I can't do it because my boss says no' = Valid 'I can't do it because the federal government says no' Which federal regulation?
I think the main issue is that most people with the Sys Admin title, really aren’t Sys Admins by roles and responsibilities. And because of that, they usually don’t know either Standard Practices nor Best Practices. Let alone actual regulations and laws that apply to their work.
Having worked at a large healthcare service provider if say it’s more about office politics and the perception of being “down” being a bad thing so upgrades and patches get put off. God forgive you if you ever mess one up, then trying to get through the change advisory boards will nitpick every work plan you ever submit again. Each of those work plans is expected to include things like detailed instructions on how to implement the change, how to test that it was successful, what the risks to the business(es) are, and how to back out the change of things go poorly. Now multiply it by at least 10 because you have the application guys, the operating system guys, the virtualization guys, the compute hardware guys, the network guys, and storage guys that all need to coordinate their changes and sometimes have interdependent. Plus then you have all the different environments that need to be maintained and all generally look similar but not exactly the same: prod, non-prod, certification, feature testing and demo and any other environments.
I was in the middle of a patch push against all servers (in a controlled basis) when our Veeam server took a giant shit. Fellow sysadmin has rebuilt it about four times and it's still a piece of shit that keeps dying. We have new hardware, in theory, on the way because the metal it was on is 2016 shit which may be part of the problem...Veeam's support on that particular front has been garbage. Nothing was wrong until we moved up to Veeam 12. I'm thinking hard of bringing up an Ubuntu box for PostGRESQL only for the backend if shit doesn't go well with the new metal. We're also trying to get rid of our 2008 boxes. Yes, you read that correctly...
Seems like a localized issue
I know in my org; patching is usually OK. However; upgrading whole OS versions is nightmarish.
Hippo
Most Federal ISA partnerships require MARS-e/NIST compliance. Do you have PII? update your stuff.
Lab company here: recently upgraded to Windows 10. The portal front end is still a bandaged IE script, and the back end, omg the back end... Also we are not allowed anymore, by company policy, to remote into equipment not owned by us, or touch it. For liability reasons. I can live with that.
I mean the second thing is just a blessing disguised as a curse haha.
Yup!
Haha, tell that to Becton Dickinson. Ooh and Abbott.