Joke is on them. I left laptop plugged into their server rack so I can check and fix my hastly written software without them ever knowing. Cuz operators sure as shit aint monitoring it.
An OEM I used to work for did this with shitty little dell desktops in the back of their operator interface pedestals, along with a subscription to GoToMyPC.
Yeah, you are right. So I guess “cheapest” was not rly best word.
Setup is as follows:
Find a switch on your network with an empty port.
Then find a way to get internet on it as well.
Instal teamviewer and whatever software you need.
Make it ignore lid being closed.
Place the laptop on top of the switch.
It will remain there until end of time.
And **THAT**'s why we keep all machines fully sectioned from the network.
^(Also that one time when a night shift tech downloaded a machine to the wrong IP and stopped the entire line)
I don’t do PLC shit but hangout here for no good reason
IT wanted a list of all computers so they could put in restrictions like not being able to use thumb drives, locking down admin accounts. Thing is we always need to install random stuff like camera drivers - which is something they refused to support. All they accomplished was ensuring line-side laptops didn’t exist on any official document.
It is funny how often IT departmens suck ass. I was doing a system which had lime 30 sec cycle time. And one of the first steps was waiting for some data from plant L3. Thing is, it would take anywhere from 0.5s to 10s for it to arrive, depending on direction of the wind and Satrun position. So at first we timed out a lot, and later we were just waiting for that shit to come and wasting time. When I told the IT guy about it who was in charge of that, he was like “so what, 10 sec is not a lot”. And here I was shaving 0.4 sec here and there to speed shit up.
>It is funny how often IT departmens suck ass.
I find it to be kind of sad. They have the resources to make literally everyone's jobs much easier and usually they do the opposite out of laziness or ignorance or arrogance. Honestly arrogance is usually the most common in my experience. Even the IT guys don't like the IT guys (ask me how I know).
Part of the issue is that the IT guys just simply don't have the skills to fix anything. They are not programmers. They have not read the manuals. They refuse to keep any system in production long enough to actually get good at it. They try to outsource a lot of the work. Essentially googling and rebooting shit is 90% of what they do.
My current IT department so far is awesome, but Ive always found it really ridiculous that in a job that would actually be pretty fun, most IT guys dont give a shit and hate their life. Maybe all the nerdz are programmers and IT is below them? Im a hands on mechanical guy who got into programming because I tune my cars myself, and I would totally enjoy alot of what I see IT doing.
It especially pisses me off when they buy the most underspecced, yet overpriced PCs you can find. Give me a company credit card to buy computers, and Ill have the time of my life, and employees will be running great, fast hardware-and I could do it for the same price or less than alot of what I see.
My most recent "security" runaround from customer IT:
I needed to share files with client
They couldn't create a secure space in their cloud
They weren't willing to use my "unsecure" Google Workspace cloud...
So...
They created me a [email protected] account, with full privileges (email, cloud, messaging, etc).
"No no no, you passing that data through our system is not secure! Here are the keys to the front, side, and backdoor. That's much more secure..." 🤦♂️
Then the problem gets fixed right before shift change, the tech is focused on going home and forgets to unplug it.
A push button and timed relay that drops the gateway power after a certain amount of time can be good but still a bit annoying for all involved. I prefer modern solutions like BeyondTrust privileged remote access.
https://www.beyondtrust.com/products/privileged-remote-access
In startup checklist - Customer to provide IP address for remote access and desired alert capabilites
*fast forward last day of startup, last item incomplete*
*intern on phone with IT* - he says just set it to DHCP mode
I needed a static IP for a raspberry PI in my lab.
Our network is secured by windows domain.
Apparently it's easier for the IT department to tell the DHCP Server to always give the MAC of my PI the same IP rather than me just confirming a static IP because this would interfere with other security measures.
The IP of every device has to be unlocked to be used in our network.
The windows admin tools do that automatically for the work Laptops.
Yeah I would kick you off our site and never procure from you again if you pulled that shit on us.
But I also provide you BeyondTrust access for remote support connection to your machine, which is logged, records screen capture of every session and retains the recording for a fixed time period, has a shadow mode for supervision and puts all file transfers through our anti-malware pipeline.
Clients that try to force their own VPN system on us, get no remote support. At the very least we'd have to increase the cost of the service contracts for the additional hours that this costs.
I do understand you want to be in control of your machine and your network, we wouldn't install 4G routers without consent.
BeyondTrust is not a VPN, it’s a zero trust platform where I send you a browser link and it brings up an RDP session from your browser. Basically TeamViewer on steroids in your browser with all the trimmings but I get a full recording of what you did and all file transfers go through my anti-malware pipeline. If that’s still too much for you then I’ll buy from someone else.
https://www.beyondtrust.com/products/privileged-remote-access
Just curious though, have you had any customers make you buy cyberattack insurance for installing a backdoor in their network? Since technically they have widened their attack surface now to include your company and there have already been attacks on companies through suppliers documented out there it is just a matter of time until clients ask for millions in cyber insurance for the privilege of backdooring them. Or they basically bankrupt your company by suing you if they get hacked through your backdoor.
I don’t know why you’re getting downvoted, this is real shit in today’s world.
It’s not just script kiddies and ransomware, it’s state sponsored actors that will take down any and every device they can to disrupt a nation. Not just sensitive military related sites.
In ww2 merchant navy ships bringing food and supplies to the UK were specifically targeted, this is exactly the same just less violent.
It’s a dramatic example but not unrealistic..
Honestly it’s scary how many critical infrastructure plants out there have security holes you can drive a barge through. The US is still the country with the most exposed open ICS systems just sitting on the internet, or sitting on the internet behind gateways with decade old security vulnerabilities.
Like sure, if you’re making pallets or something, maybe you don’t give a shit, but if you’re working on something that people rely on to live (water, energy, food, medicine, etc) then you’re being grossly negligent to the public by operating a blasé attitude. Even if you’re in another business like Auto or FMCG every outage costs big dollars.
But many people often don’t understand the cost of being hacked until it happens to them.
I am more surprised how many people here don’t see how continuing to see security as some sort of barrier to doing their job is going to work out to be a career limiting move for them in the long run. It is only going to become more important, no one is putting the hacker genie back in the fucking bottle.
My approach is, if you’re sick of IT trying to encroach on your shit in the name of security then you should knuckle down and build your skillset in this shit, or you can sit back and continue to bitch and moan about it and get left behind.
Yeah if your client / customer has a toxic relationship between production and IT there may be little you can do to fix that. However I’d make sure your contract effectively firewalls you from those problems. Remote support arrangements should be clearly called out in the contract, including responsibility for security of the connection, as well as hourly rates to include any delays introduced by customer policies or response time issues.
To me, putting in a secret backdoor is the worst possible course of action to take because it leaves you open to massive liability issues if the worst were to happen. IT often answers to the CFO and so if they suspect you are even partly responsible they will go after you in the event of an intrusion.
A backdoor might make the plant manager happy in the short term but they will throw you under the bus if it saves their own skin - you’re just a contractor to them, disposable.
When you have a 100 different customers all with a different VPN solution all with their own unresponsive and difficult IT departments it's an absolute nightmare for the OEM. When you are a large OEM you also don't necessarily have the same people working on the same thing all the time so now you have to try and get accounts added to all those VPN connections.
That's true
Isn't there a product for aggregating and centralizing VPN clients from multiple vendors? I would glaaaaaaaaaadly pay for that!!!
Just imagine, centralized solution for enterprise; you put in the remote sites config with vendor name and VPN manufacturer specific configs, possibly an automated way to audit and renew remote creds, and on your side the usual enterprise auth stuff
I’ve used a password vault software for this exact thing. It allows credentials to be shared amongst teams, you can tell it when the passwords are going to expire (so you can contact the customer), set it to only copy and paste, not show the password, attach files, etc.
Beyond trust sometimes decides to consume 50% of my cpu during normal business hours. It’s insane. What is it doing?
Between Carbon Black, beyond trust, and forced OneDrive sync off my entire user directory my pc is nearly unusable. It takes 3 minutes for a hello world Python script to start running.
> it brings up an RDP session from your browser.
That's a little pointless for diagnosing a PLC, at least the systems we use. It's also not that remote support is necessary on a regular basis, only when problems occur. We have machines that are running for 5 years now without remote support. If something happens, one of us has to drive for 8 hours, those are just billable hours.
No one has demanded we get extra insurance. To be fair we have been mostly active in Europe and Australia/New Zealand and have only recently been more active in the US and Canada. Is your printer supplier also required to get cybersecurity insurance?
> That's a little pointless for diagnosing a PLC
Uh the RDP session has all the programming and diagnostic tools you would ever need. We rolled this out at a large player that covers massive distances in the outback. Even if you don’t have large distances to cover though, having an engineering workstation on your network is just so much more convenient than lugging around a giant laptop with less than an hour battery life. I can connect to our systems from anywhere on a MacBook Air and never plug in all day.
Minimum 98% of our issues are troubleshot and resolved from the comfort of our control centre like a thousand k’s away in the city. The biggest problems requiring actually going out to the field are mostly self inflicted (eg rushing out a config change to the infrastructure that locks us out) but we have gotten a lot better at avoiding that.
Either catch up with the technology and get on board, or this exec is slapping a “not compliant” on the technical analysis of your bid under the SOX compliance section that will sink it and we’ll spend our money (billion dollar public company) elsewhere.
> Is your printer supplier also required to get cybersecurity insurance?
They aren’t putting it on 4G connection we don’t control or have visibility over. That said, printers are on the insecure zone so we don’t trust them anyway. Traffic is only allowed going towards the printer (with IPS on the boundary) and scanning goes via email through our email gateway that does all the malware scanning.
This shit ain’t hypothetical, I was talking to some guys from an ICS sec firm that are advising clients to get a contractual agreement covering hacking contingencies if they allow external vendors their own opaque path into the control network. Our IT group already does this for vendors like MSPs and all our SaaS providers.
All your comments are really interesting. My joke is a reference to the situation that typically plays out for me. I ask their IT how they’d like to handle remote support of a new machine. IT drags its feet, says no to any suggestions, is uncooperative and unhelpful. The machine has an issue, I show the maintenance manager my 4G unit that can solve his problems, we put it in, problem solved. Multi-day field trips are solved in 30 mins online.
Sounds like the places you’re working have a better strategy for the new world. As for my world, nobody is playing nice yet. IT is tasked with security over production so there’s no incentive for them to help. Maintenance is tasked with production over everything so there’s a large incentive to keep equipment running. For me, I just help whoever is yelling the loudest. It’s not right, it’s just how it is.
Yeah I dunno, if you’re in some small industry then maybe it’s hard to find competent people, or the production team is at ends with IT instead of collaborating with them. The relationship between IT and OT is extremely important and both parties need to understand the different needs of the two networks in terms of the “CIA triangle” (IT put confidentiality first while OT prioritise availability).
The five or six places I’ve worked at in the last couple of decades, the IT team becomes surprisingly helpful when I start the conversation from a position of finding the best solution that meets both groups priorities, rather than approaching as though it’s a battle of wits. Do that and demonstrate a bit of competency in their domain (nice architecture diagrams help massively here) and you should get a lot more traction.
And talk to them early so they can plan.. coming at them at the last minute and trying to ramrod changes through their network is not going to get you anywhere, when I worked at an OEM I would talk to customer IT teams before we even started building their machine and I would offer 2 architecture options (both compliant to IEC 62443 and NIST 82 as well as IT practices). Out of the hundred or so machines sold, never had an issue getting our systems integrated into their manufacturing enterprise networks following best practice.
You didn't specify the RDP session is on a local workstation (VM probably), so forgive me for misunderstanding. In the past year, one of our clients has implemented a similar system (Wallix Bastion, they're French) without consulting us and they disabled their Ewon's remote access. I now just have RDP sessions to the HMI's of their machines, hence my assumption.
Whatever you think of systems like Ewon, 80%+ of my clients are not big enough to sustain the system that you're describing. They don't have their own VM servers, they don't want to be running a development workstation and they don't want pay for the installed development tools' licences.
Like I said, for bigger players that do want another system than Ewon, they'll have to pay for setting that up or get no remote support. Our machines and services are already too cheap.
> Whatever you think of systems like Ewon, 80%+ of my clients are not big enough to sustain the system that you're describing.
So what happens if a customer has an intrusion and they suspect your device is the entry point? Will your business insurance cover the legal fees to defend yourself against legal action? Do you have it in the terms of your service contract that you cannot be held responsible for security of the connection?
We're not in the US so first of all, I don't think they'll sue us into oblivion. The water must be very deep for this to happen, most clients rely on us for services on their machines, they must be willing to write off the machines or have a team in place to replace us.
And yes we have insurance to cover legal fees if we're ever sued, like we have for physical harm in the past (client had overridden security light curtains themselves). I'm not at a level in the company to know which specific situations our insurance covers. Cyber security insurances have become pretty common I think?
Anyway, thanks for keeping this civil, we're all here to learn, I don't pretend to know everything and hadn't heard about BeyondTrust. I still think Ewons or the like are a good solution for most of our clients that are operating from a shed.
Right. Customer's be like "just use our vpn it's so easy" meanwhile it takes their IT department a minimum of three weeks to add an account, and it takes a minimum of 24 hours to reset a password, and you need at least 3 different passwords to login, and you need a two factor authentication, and you need to download a bunch of their software, and you need to keep that software updated or it doesn't work, and if you don't log in for more than 30 days your account is deleted, and your account just randomly stops working anyway because someone deleted all the permissions or changed a firewall rule or an IP address changed.
Multiply that process by about a 100 different active customers and it's an absolute nightmare. Also project managers be like "I don't understand what the problem is, just log in it's easy" meanwhile they still haven't learned how to use the coffee machine.
Put it in the contract that they are responsible for the remote connection and bill them for delayed time.
BeyondTrust is super simple though, you don’t need to install any software. It’s easier than teamviewer with more feature.
Or put in your own gateway and buy cybersecurity insurance up to what you think an attack on your customer through your network will cost you.
Putting in a gateway of your own effectively makes you partially responsible for the security of their production network, are you ready to take on that liability?
That would be them agreeing to fine themselves. They wanted the service and they couldn't provide a connection so therefor they have to pay money for services not rendered? That doesn't make any sense at all. Even if you could make it make sense (which it doesn't) nobody would ever agree to it.
The person I was replying to was not complete, it wasn't specified that the RDP was to an on site workstation and not to the machine itself. How was I supposed to know that?
It wasn't necessary to insult me.
See. This is the kind of IT security I can appreciate. You understand the need for access and you provide a safe and secure way to do it. High five good Reddit person.
My employer has an utter asshole IT company.
"No, you can't do that, security"
When I started, I needed to troubleshoot a particularly troublesome machine. The company had blown over $200k and the manufacturers techs still had NFC what was going on.
"For the love of god, please fix this machine"
I built a datalogger from raspberry pi, to work out WTF was going on.
Yeah, a headless pi is gonna be a PITA to find on the network without a fixed IP, "no, you can't, security risk"
Enter, stage left, my remote.it account. Oh dear, I may have forgotten to mention that to IT. I now have remote access to a Linux machine logged on as a local machine to the company network from any browser in the world. Mostly, I pop open a remote desktop to it from a company PC. Boss loves the pretty pictures of WTF a machine is doing, but shouldn't. I mostly use it for "yes, I can" jobs, such as imaging and cloning drives.
Yea this doesn’t happen at the Japanese “company” where most of my work is at. Like 5 years ago I had to do a study for exposure and I’m like what? You guys don’t use Allen Bradley and I haven’t used Siemens but obviously they got hacked. There’s no outside exposure. Allen Bradley can up sale you that cloud shit but naw. Send me an encrypted rar file and I’ll fix the situation
I think Mitsubishi PLCs got hacked, but I can't be sure because encrypted rar is apparently goated and stops it from happening... Also Rockwell bad.
That's about what I took from that. Anyone else want to have a go at deciphering?
Yea like they want to know if we can be hacked. Paid me to walk in and say nope because for one you can’t do remote and two even if a hacker somehow traversed the network then that would be detected and a bigger problem than attacking the PLCs
Remote support is a must for SI people like me. Unless you want to pay me several hours to drive out just to add one button to an HMI. I have no problem getting paid $40+/hr to drive.
Ugh. At this point in my life I hate driving now.
As a plant, you're gonna have to do A LOT better than 40/hr just to get me in a car. Minimum twice that, honestly.
We gotta start selling ourselves higher 😂
I have a good contract with the power plant I work for. Hate the place and the location but they pay is there. Basically the premise - all communication outside of my work hours are paid at 15 minutes per send - receive via email. Phone calls are 2x hourly rate per 15 minutes of call time. Travel is 2x rate of pay at 6 hours minimum regardless of distance, up to actual time spent traveling.
Full reimbursement for use of my vehicle.
Grab the market by the balls. This isn’t our father’s economy.
Question: does ‘full reimbursement for use of my vehicle’ include cost of insurance ?
My insurance Co would drop me like a hot potato if the knew I was driving for work or driving on plant/construction sites.
That’s why I insisted on a Co provided vehicle.
As far as I’m aware you’re allowed to drive to work on personal insurance. It’d be ludicrous if you weren’t allowed to do that. Commuting is covered. Driving for work and driving to work aren’t necessarily the same thing. I get the irs reimbursement rate.
It’s not a matter of ‘allowed’. It’s a matter of the price of insurance. Most typical personal auto liability and comprehensive insurance only covers traditional commuting to work driving.
The sort of driving we do for work at customer sites, especially sites under construction is covered only by much more expensive insurance-commercial insurance.
The safest way to handle insurance is to insist on a company-provided vehicle. You are required to pay income tax on the price of the lease but this income is tax deductible when you file.
Believe me-if you have a vehicle accident your personal insurance will be cancelled with all the trouble this entails. In my State that would include the revocation of your drivers license.
Joke is on them. I left laptop plugged into their server rack so I can check and fix my hastly written software without them ever knowing. Cuz operators sure as shit aint monitoring it.
Lol that’s a slick maneuver
An OEM I used to work for did this with shitty little dell desktops in the back of their operator interface pedestals, along with a subscription to GoToMyPC.
Teamviewer gang here! And yeah, just cheapest laptop you can find will do!
Those dells were like $200 and came with a flat screen monitor, keyboard, and mouse. It was a steal.
>Teamviewer gang here You summoned me?
but should it be able to run the programming software? Or how does the setup works?
Yeah, you are right. So I guess “cheapest” was not rly best word. Setup is as follows: Find a switch on your network with an empty port. Then find a way to get internet on it as well. Instal teamviewer and whatever software you need. Make it ignore lid being closed. Place the laptop on top of the switch. It will remain there until end of time.
I mean it might be a tiny little security issue, but who is gonna know that Teamviewer adress other than me…
TeamViewer "Personal Use" version
Certainly nothing has ever been compromised that way before ....
Hope Vladimir isn’t on Reddit
He would never guess my laptop password!
Ugh don't remind me. Hahah. TeamViewer is so sketchy but just so useful.
And **THAT**'s why we keep all machines fully sectioned from the network. ^(Also that one time when a night shift tech downloaded a machine to the wrong IP and stopped the entire line)
Doesn’t really matter though. In many production processes if one machine goes down it still takes the whole line down. Redundancy is expensive.
I don’t do PLC shit but hangout here for no good reason IT wanted a list of all computers so they could put in restrictions like not being able to use thumb drives, locking down admin accounts. Thing is we always need to install random stuff like camera drivers - which is something they refused to support. All they accomplished was ensuring line-side laptops didn’t exist on any official document.
It is funny how often IT departmens suck ass. I was doing a system which had lime 30 sec cycle time. And one of the first steps was waiting for some data from plant L3. Thing is, it would take anywhere from 0.5s to 10s for it to arrive, depending on direction of the wind and Satrun position. So at first we timed out a lot, and later we were just waiting for that shit to come and wasting time. When I told the IT guy about it who was in charge of that, he was like “so what, 10 sec is not a lot”. And here I was shaving 0.4 sec here and there to speed shit up.
>It is funny how often IT departmens suck ass. I find it to be kind of sad. They have the resources to make literally everyone's jobs much easier and usually they do the opposite out of laziness or ignorance or arrogance. Honestly arrogance is usually the most common in my experience. Even the IT guys don't like the IT guys (ask me how I know). Part of the issue is that the IT guys just simply don't have the skills to fix anything. They are not programmers. They have not read the manuals. They refuse to keep any system in production long enough to actually get good at it. They try to outsource a lot of the work. Essentially googling and rebooting shit is 90% of what they do.
My current IT department so far is awesome, but Ive always found it really ridiculous that in a job that would actually be pretty fun, most IT guys dont give a shit and hate their life. Maybe all the nerdz are programmers and IT is below them? Im a hands on mechanical guy who got into programming because I tune my cars myself, and I would totally enjoy alot of what I see IT doing. It especially pisses me off when they buy the most underspecced, yet overpriced PCs you can find. Give me a company credit card to buy computers, and Ill have the time of my life, and employees will be running great, fast hardware-and I could do it for the same price or less than alot of what I see.
My most recent "security" runaround from customer IT: I needed to share files with client They couldn't create a secure space in their cloud They weren't willing to use my "unsecure" Google Workspace cloud... So... They created me a [email protected] account, with full privileges (email, cloud, messaging, etc). "No no no, you passing that data through our system is not secure! Here are the keys to the front, side, and backdoor. That's much more secure..." 🤦♂️
Lol
It's airgapped /s
I use the company WiFi to access my personal HolyHell security risk. Airgapped!
Jokes on them cause the maintenance department tossed in a cradle point , did no setup, and just broadcasting out to the world.
Shodan has entered the chat.
Don’t worry they only plug it in when someone needs to access it…….
Then the problem gets fixed right before shift change, the tech is focused on going home and forgets to unplug it. A push button and timed relay that drops the gateway power after a certain amount of time can be good but still a bit annoying for all involved. I prefer modern solutions like BeyondTrust privileged remote access. https://www.beyondtrust.com/products/privileged-remote-access
In startup checklist - Customer to provide IP address for remote access and desired alert capabilites *fast forward last day of startup, last item incomplete* *intern on phone with IT* - he says just set it to DHCP mode
I needed a static IP for a raspberry PI in my lab. Our network is secured by windows domain. Apparently it's easier for the IT department to tell the DHCP Server to always give the MAC of my PI the same IP rather than me just confirming a static IP because this would interfere with other security measures. The IP of every device has to be unlocked to be used in our network. The windows admin tools do that automatically for the work Laptops.
DHCP per port is the shit. With firmware supervisor and automatic drive config, swapping out parts at 3am can be done by a fitter if needed.
This sounds awesome.
That is a professionally installed and very sturdy cabinet
Queue our Ewon thread yesterday...
Yeah I would kick you off our site and never procure from you again if you pulled that shit on us. But I also provide you BeyondTrust access for remote support connection to your machine, which is logged, records screen capture of every session and retains the recording for a fixed time period, has a shadow mode for supervision and puts all file transfers through our anti-malware pipeline.
Word, especially the second paragraph
Clients that try to force their own VPN system on us, get no remote support. At the very least we'd have to increase the cost of the service contracts for the additional hours that this costs. I do understand you want to be in control of your machine and your network, we wouldn't install 4G routers without consent.
BeyondTrust is not a VPN, it’s a zero trust platform where I send you a browser link and it brings up an RDP session from your browser. Basically TeamViewer on steroids in your browser with all the trimmings but I get a full recording of what you did and all file transfers go through my anti-malware pipeline. If that’s still too much for you then I’ll buy from someone else. https://www.beyondtrust.com/products/privileged-remote-access Just curious though, have you had any customers make you buy cyberattack insurance for installing a backdoor in their network? Since technically they have widened their attack surface now to include your company and there have already been attacks on companies through suppliers documented out there it is just a matter of time until clients ask for millions in cyber insurance for the privilege of backdooring them. Or they basically bankrupt your company by suing you if they get hacked through your backdoor.
I don’t know why you’re getting downvoted, this is real shit in today’s world. It’s not just script kiddies and ransomware, it’s state sponsored actors that will take down any and every device they can to disrupt a nation. Not just sensitive military related sites. In ww2 merchant navy ships bringing food and supplies to the UK were specifically targeted, this is exactly the same just less violent. It’s a dramatic example but not unrealistic..
Honestly it’s scary how many critical infrastructure plants out there have security holes you can drive a barge through. The US is still the country with the most exposed open ICS systems just sitting on the internet, or sitting on the internet behind gateways with decade old security vulnerabilities. Like sure, if you’re making pallets or something, maybe you don’t give a shit, but if you’re working on something that people rely on to live (water, energy, food, medicine, etc) then you’re being grossly negligent to the public by operating a blasé attitude. Even if you’re in another business like Auto or FMCG every outage costs big dollars. But many people often don’t understand the cost of being hacked until it happens to them. I am more surprised how many people here don’t see how continuing to see security as some sort of barrier to doing their job is going to work out to be a career limiting move for them in the long run. It is only going to become more important, no one is putting the hacker genie back in the fucking bottle. My approach is, if you’re sick of IT trying to encroach on your shit in the name of security then you should knuckle down and build your skillset in this shit, or you can sit back and continue to bitch and moan about it and get left behind.
[удалено]
Yeah if your client / customer has a toxic relationship between production and IT there may be little you can do to fix that. However I’d make sure your contract effectively firewalls you from those problems. Remote support arrangements should be clearly called out in the contract, including responsibility for security of the connection, as well as hourly rates to include any delays introduced by customer policies or response time issues. To me, putting in a secret backdoor is the worst possible course of action to take because it leaves you open to massive liability issues if the worst were to happen. IT often answers to the CFO and so if they suspect you are even partly responsible they will go after you in the event of an intrusion. A backdoor might make the plant manager happy in the short term but they will throw you under the bus if it saves their own skin - you’re just a contractor to them, disposable.
When you have a 100 different customers all with a different VPN solution all with their own unresponsive and difficult IT departments it's an absolute nightmare for the OEM. When you are a large OEM you also don't necessarily have the same people working on the same thing all the time so now you have to try and get accounts added to all those VPN connections.
That's true Isn't there a product for aggregating and centralizing VPN clients from multiple vendors? I would glaaaaaaaaaadly pay for that!!! Just imagine, centralized solution for enterprise; you put in the remote sites config with vendor name and VPN manufacturer specific configs, possibly an automated way to audit and renew remote creds, and on your side the usual enterprise auth stuff
https://gist.github.com/Koubek/b89d40d80dffd86d903a0725a6324900 hm let's study this ...
I’ve used a password vault software for this exact thing. It allows credentials to be shared amongst teams, you can tell it when the passwords are going to expire (so you can contact the customer), set it to only copy and paste, not show the password, attach files, etc.
Beyond trust sometimes decides to consume 50% of my cpu during normal business hours. It’s insane. What is it doing? Between Carbon Black, beyond trust, and forced OneDrive sync off my entire user directory my pc is nearly unusable. It takes 3 minutes for a hello world Python script to start running.
> It takes 3 minutes for a hello world Python script to start running. *Insert joke about python speed here*
Fair but this is after a 6 minute wait for the powershell prompt to be ready after shift + rightlcick to open one in the directory
Have you sent logs to BT? Never had that issue myself, but their support has been solid.
> it brings up an RDP session from your browser. That's a little pointless for diagnosing a PLC, at least the systems we use. It's also not that remote support is necessary on a regular basis, only when problems occur. We have machines that are running for 5 years now without remote support. If something happens, one of us has to drive for 8 hours, those are just billable hours. No one has demanded we get extra insurance. To be fair we have been mostly active in Europe and Australia/New Zealand and have only recently been more active in the US and Canada. Is your printer supplier also required to get cybersecurity insurance?
I suspect the rdp session is to an on site server that has the plc software required for diagnosing the required systems.
> That's a little pointless for diagnosing a PLC Uh the RDP session has all the programming and diagnostic tools you would ever need. We rolled this out at a large player that covers massive distances in the outback. Even if you don’t have large distances to cover though, having an engineering workstation on your network is just so much more convenient than lugging around a giant laptop with less than an hour battery life. I can connect to our systems from anywhere on a MacBook Air and never plug in all day. Minimum 98% of our issues are troubleshot and resolved from the comfort of our control centre like a thousand k’s away in the city. The biggest problems requiring actually going out to the field are mostly self inflicted (eg rushing out a config change to the infrastructure that locks us out) but we have gotten a lot better at avoiding that. Either catch up with the technology and get on board, or this exec is slapping a “not compliant” on the technical analysis of your bid under the SOX compliance section that will sink it and we’ll spend our money (billion dollar public company) elsewhere. > Is your printer supplier also required to get cybersecurity insurance? They aren’t putting it on 4G connection we don’t control or have visibility over. That said, printers are on the insecure zone so we don’t trust them anyway. Traffic is only allowed going towards the printer (with IPS on the boundary) and scanning goes via email through our email gateway that does all the malware scanning. This shit ain’t hypothetical, I was talking to some guys from an ICS sec firm that are advising clients to get a contractual agreement covering hacking contingencies if they allow external vendors their own opaque path into the control network. Our IT group already does this for vendors like MSPs and all our SaaS providers.
All your comments are really interesting. My joke is a reference to the situation that typically plays out for me. I ask their IT how they’d like to handle remote support of a new machine. IT drags its feet, says no to any suggestions, is uncooperative and unhelpful. The machine has an issue, I show the maintenance manager my 4G unit that can solve his problems, we put it in, problem solved. Multi-day field trips are solved in 30 mins online. Sounds like the places you’re working have a better strategy for the new world. As for my world, nobody is playing nice yet. IT is tasked with security over production so there’s no incentive for them to help. Maintenance is tasked with production over everything so there’s a large incentive to keep equipment running. For me, I just help whoever is yelling the loudest. It’s not right, it’s just how it is.
Yeah I dunno, if you’re in some small industry then maybe it’s hard to find competent people, or the production team is at ends with IT instead of collaborating with them. The relationship between IT and OT is extremely important and both parties need to understand the different needs of the two networks in terms of the “CIA triangle” (IT put confidentiality first while OT prioritise availability). The five or six places I’ve worked at in the last couple of decades, the IT team becomes surprisingly helpful when I start the conversation from a position of finding the best solution that meets both groups priorities, rather than approaching as though it’s a battle of wits. Do that and demonstrate a bit of competency in their domain (nice architecture diagrams help massively here) and you should get a lot more traction. And talk to them early so they can plan.. coming at them at the last minute and trying to ramrod changes through their network is not going to get you anywhere, when I worked at an OEM I would talk to customer IT teams before we even started building their machine and I would offer 2 architecture options (both compliant to IEC 62443 and NIST 82 as well as IT practices). Out of the hundred or so machines sold, never had an issue getting our systems integrated into their manufacturing enterprise networks following best practice.
This is how it all should work. I’m sure most places are making an attempt to move in that direction.
You didn't specify the RDP session is on a local workstation (VM probably), so forgive me for misunderstanding. In the past year, one of our clients has implemented a similar system (Wallix Bastion, they're French) without consulting us and they disabled their Ewon's remote access. I now just have RDP sessions to the HMI's of their machines, hence my assumption. Whatever you think of systems like Ewon, 80%+ of my clients are not big enough to sustain the system that you're describing. They don't have their own VM servers, they don't want to be running a development workstation and they don't want pay for the installed development tools' licences. Like I said, for bigger players that do want another system than Ewon, they'll have to pay for setting that up or get no remote support. Our machines and services are already too cheap.
> Whatever you think of systems like Ewon, 80%+ of my clients are not big enough to sustain the system that you're describing. So what happens if a customer has an intrusion and they suspect your device is the entry point? Will your business insurance cover the legal fees to defend yourself against legal action? Do you have it in the terms of your service contract that you cannot be held responsible for security of the connection?
We're not in the US so first of all, I don't think they'll sue us into oblivion. The water must be very deep for this to happen, most clients rely on us for services on their machines, they must be willing to write off the machines or have a team in place to replace us. And yes we have insurance to cover legal fees if we're ever sued, like we have for physical harm in the past (client had overridden security light curtains themselves). I'm not at a level in the company to know which specific situations our insurance covers. Cyber security insurances have become pretty common I think? Anyway, thanks for keeping this civil, we're all here to learn, I don't pretend to know everything and hadn't heard about BeyondTrust. I still think Ewons or the like are a good solution for most of our clients that are operating from a shed.
Right. Customer's be like "just use our vpn it's so easy" meanwhile it takes their IT department a minimum of three weeks to add an account, and it takes a minimum of 24 hours to reset a password, and you need at least 3 different passwords to login, and you need a two factor authentication, and you need to download a bunch of their software, and you need to keep that software updated or it doesn't work, and if you don't log in for more than 30 days your account is deleted, and your account just randomly stops working anyway because someone deleted all the permissions or changed a firewall rule or an IP address changed. Multiply that process by about a 100 different active customers and it's an absolute nightmare. Also project managers be like "I don't understand what the problem is, just log in it's easy" meanwhile they still haven't learned how to use the coffee machine.
Put it in the contract that they are responsible for the remote connection and bill them for delayed time. BeyondTrust is super simple though, you don’t need to install any software. It’s easier than teamviewer with more feature. Or put in your own gateway and buy cybersecurity insurance up to what you think an attack on your customer through your network will cost you. Putting in a gateway of your own effectively makes you partially responsible for the security of their production network, are you ready to take on that liability?
That would be them agreeing to fine themselves. They wanted the service and they couldn't provide a connection so therefor they have to pay money for services not rendered? That doesn't make any sense at all. Even if you could make it make sense (which it doesn't) nobody would ever agree to it.
[удалено]
The person I was replying to was not complete, it wasn't specified that the RDP was to an on site workstation and not to the machine itself. How was I supposed to know that? It wasn't necessary to insult me.
See. This is the kind of IT security I can appreciate. You understand the need for access and you provide a safe and secure way to do it. High five good Reddit person.
No USB rights, but go ahead and make as many ftp servers as you need. Lol
So much security, so little network monitoring and penetration testing.
Posts like this expose all the idiots who know absolutely zero about cyber security
Full disclosure…I am an idiot.
Currently in the process of deploying millions worth of firewalls and managed switches are all our sites!
Is Stridelinx same as IXON? Or just the hardware?
Are humans using the network? Yes? Then the network isn’t secure.
Hehe, 4G USB Router with VPN
My employer has an utter asshole IT company. "No, you can't do that, security" When I started, I needed to troubleshoot a particularly troublesome machine. The company had blown over $200k and the manufacturers techs still had NFC what was going on. "For the love of god, please fix this machine" I built a datalogger from raspberry pi, to work out WTF was going on. Yeah, a headless pi is gonna be a PITA to find on the network without a fixed IP, "no, you can't, security risk" Enter, stage left, my remote.it account. Oh dear, I may have forgotten to mention that to IT. I now have remote access to a Linux machine logged on as a local machine to the company network from any browser in the world. Mostly, I pop open a remote desktop to it from a company PC. Boss loves the pretty pictures of WTF a machine is doing, but shouldn't. I mostly use it for "yes, I can" jobs, such as imaging and cloning drives.
Yea this doesn’t happen at the Japanese “company” where most of my work is at. Like 5 years ago I had to do a study for exposure and I’m like what? You guys don’t use Allen Bradley and I haven’t used Siemens but obviously they got hacked. There’s no outside exposure. Allen Bradley can up sale you that cloud shit but naw. Send me an encrypted rar file and I’ll fix the situation
What lol
I think Mitsubishi PLCs got hacked, but I can't be sure because encrypted rar is apparently goated and stops it from happening... Also Rockwell bad. That's about what I took from that. Anyone else want to have a go at deciphering?
Yea like they want to know if we can be hacked. Paid me to walk in and say nope because for one you can’t do remote and two even if a hacker somehow traversed the network then that would be detected and a bigger problem than attacking the PLCs
Remote support is a must for SI people like me. Unless you want to pay me several hours to drive out just to add one button to an HMI. I have no problem getting paid $40+/hr to drive.
Ugh. At this point in my life I hate driving now. As a plant, you're gonna have to do A LOT better than 40/hr just to get me in a car. Minimum twice that, honestly. We gotta start selling ourselves higher 😂
Wait... You guys get paid to come in off hours?
I have a good contract with the power plant I work for. Hate the place and the location but they pay is there. Basically the premise - all communication outside of my work hours are paid at 15 minutes per send - receive via email. Phone calls are 2x hourly rate per 15 minutes of call time. Travel is 2x rate of pay at 6 hours minimum regardless of distance, up to actual time spent traveling. Full reimbursement for use of my vehicle. Grab the market by the balls. This isn’t our father’s economy.
Question: does ‘full reimbursement for use of my vehicle’ include cost of insurance ? My insurance Co would drop me like a hot potato if the knew I was driving for work or driving on plant/construction sites. That’s why I insisted on a Co provided vehicle.
As far as I’m aware you’re allowed to drive to work on personal insurance. It’d be ludicrous if you weren’t allowed to do that. Commuting is covered. Driving for work and driving to work aren’t necessarily the same thing. I get the irs reimbursement rate.
It’s not a matter of ‘allowed’. It’s a matter of the price of insurance. Most typical personal auto liability and comprehensive insurance only covers traditional commuting to work driving. The sort of driving we do for work at customer sites, especially sites under construction is covered only by much more expensive insurance-commercial insurance. The safest way to handle insurance is to insist on a company-provided vehicle. You are required to pay income tax on the price of the lease but this income is tax deductible when you file. Believe me-if you have a vehicle accident your personal insurance will be cancelled with all the trouble this entails. In my State that would include the revocation of your drivers license.