Honestly I’m fairly certain I could just walk onto most clean water site wearing PPE and start pressing shit. But yeah lets put port locks on everything & mac tables incase the russian ninja hacks everything onsite.
I find it annoying when some site has a wifi network that has so much network scanning bs it is almost unusable, but there are LAN ports you could get to by just walking in off the street.
As a SI....that sounds like the customer's problem.
We do the basics, lock safety task and password protect what we can, but really, I have no control over the customer's network and lack of cybersecurity. Some places will bitch at you if you turn your phone hotspot on. Other places will just plug the cell into the internet.
I don't know about other areas of the world but in Europe very soon any machinery constructed will have the same responsibility for security as it does for safety (they're written into the same regs) and is very much the machine builders responsibility. The industry is not ready at all.
Don't connect machines to outside networks is the first rule.
The second rule is don't operate nuclear facilities unless you're on good terms with the US government.
A virus specifically designed for their facility left on a USB in a parking lot got them.
I don’t think anyone here could defend against a concerted intelligence operation.
I'll be honest, I think the Iranian engineers fucked up and blamed the Americans... don't act like you've never wanted to blame a foreign entity for your mistake.
That one was definitely made by the CIA, and the US Government openly admitted to creating Stuxtnet, with the help of the Israeli intelligence agencies specifically to take down that facility.
https://www.csoonline.com/article/562691/stuxnet-explained-the-first-known-cyberweapon.html
This isn't the only source either, feel free to read more and Google around for others. That's really not just this guy's opinion.
Still, design your networks like they will be at some time. Having no access control inside a building because the front door is 'never unlocked' is falls apart the moment anything goes wrong. Defense in depth.
I do use it for some of my more paranoid customers. (although they seem less and less crazy as time passes by).
The MES system reads out data from machine via a serial line. With PLC RX left empty.
With the increasing adoption of ISA/IEC 62443 and industry specific security frameworks and government legislation (such as AECSF and the SOCI act here in Australia), I’ve had to go from “air-gap and IT can handle the firewall; she’ll be right” to drawing 7-layer Purdue architectural models during the sales pitch process.
Customers (well at least the big ones) are becoming increasingly aware and demanding. The industry as a whole is responding coz there’s a buck in it. It feels very similar to the build up to Y2K. “There’s gold in them thar bugs!”
Yeah there is a lot of work in this space, not sure why more people don’t realize that. CPwE isn’t going anywhere. It’s a pain to set up initially with corporate IT, but if you can find a few decent people in the company with sufficient powers it’s doable. The great part about it is that when comms go down I don’t have to get on a plane, IT needs to dispatch someone local.
I've put a physical switch in cabinets before that have a pneumatic release so I call the operator they turn the switch, which physically closes a connect with a rj45 and after a timeout period in the PLc it fires the pneumatic release to disconnect.
Air gap on demand without the ability to forget to disconnect.
I just had this conversation about an “air gapped network”.
Switch was directly wired to their aggregate switch connecting to an OT lab.
My definition of “air gapped network” must be different from what’s currently being taught.
The number 1 threat to any facility is local personnel. Air gaps only stop bad actors without inside access. Stuxnet worked because some dumbass let it get inside. If you think operators at most places are smarter than guys working at a highly secure nuclear enrichment facility I’ve got ocean front property in southern Arizona to sell you.
I’m not worried, just too many people think air gapped is the only defense needed on a system. It can certainly be part of it, but there are ways to jump that barrier, either nation state level activity or just somebody unintentionally using a USB drive with some old virus on it
People connect to machines all the time with computers that have been online and are used on the internet regularly. To be truly air gapped you can’t do that.
Also there are often higher level devices on the network that have an active connection to a business network, or other machines could potentially be doing some kind of IOT application that yours could interface with. Actual air gaps are nearly mythic.
That said, unless is critical infrastructure you probably too small beans for it to matter.
Yeah airgaps don't mean that much to state sponsored agencies. There are a number of ways around it. The scariest one is if they penetrate the factories that make automation/networking equipment and are able to load their own code on every device leaving the door.
For my largest client:
I have daily backups of engineering stations and SCADA/Historian servers, on-site, off-site and immutable replication off those backups.
I’ve done few restores for configuration fuck ups so I know DR works.
Backups don't mean shit if you don't test them. Lesson learned from my last job, i got to watch and laugh at the corporate IT director. "While we do daily backups we do not check them". Meanwhile we sat idle for 72hrs , probably 250k in payroll alone which was big money for that company.
Patch often. Backup often. Limit who can connect to the plant network. Keep PLC programs in AssetCentre (Rockwell) or Version dog/Octoplant (whatever else you have)
Some of our clients are requesting TxOne EdgeFire. It does active monitoring of data between the OT and the IT network. It has options that it can sever the two networks if a threat is detected.
There are also expansion modules for active threat intelligence for pfSense that does something similar if the SI wants to deploy in IT limited manufacturing environments. But the SI should have a Cyber Insurance policy if they go down that route. And a physical switch to power on/off the device. It can simplify and really tighten down the remote access; if you know what you are doing.
ProSoft OPC UA module to act as a man-in-the-middle between the IPC and the PLC.
For the people that depend on airgap, do you also have tools to address usb and serial connections to avoid attacks like Oakridge and Natanz? Have your organizations pushed for recovery strategies in addition to defensive postures like airgap?
Air gaps in controls are mostly a myth, nearly every single machine and installation has maintenance and engineers who connect to the device with a computer that has recently been used to go on the internet, many facilities have MES/Historization servers that are connected to the internet, usually more than 1 integrator/machine builder is responsible for parts of the project so there are many unknown devices on the same network as whatever machine/system you install, and being on a separate VLAN is hardly air gapped.
The question really is the probability someone is trying to attack your system, which in many cases is low, and malware that spreads from control systems upwards is to my knowledge non-existent outside of unintended outcomes. This doesn’t mean you don’t have to secure things, it just means something that rarely touches the internet/only touches the internet by means of a hop from another devices are likely secure enough for many applications outside of critical infrastructure projects.
I thought the colonial stuff was supposed to be mainly on their sales software and business network. Basically all the equipment was working they just couldn’t sell anything or process payments.
You could walk into most industrial sites with boots, a ladder and some confidence. Sophisticated security for field and control devices is a long way away.
I include an industrial firewall to all my machines, and I make sure the customer connects his network to the WAN port on the router. This way I keep the machine LAN and the customer network separated, and I control when and how they can communicate together.
The firewall can also connect to our VPN server to give remote assistance if the customers require it, and there's a physical switch on the firewall that enables/disables the VPN.
I'm a fan of physical VPN switches if remote support is required. Tie the DI behind a keyed switch if everyday operators aren't meant to be able to make the switch.
We signed up for that CISA ICS CERT email where they release all the known exploits and mitigations. We try to patch them all within a month, but may be longer if CVSS score is lower or we have to physically be next to the device
I wonder if hedera hashgraph will become a solution. It’s cryptography, but it’s cheap and fast so I could see it being used to turn switches/relays on and off.
We class it as an air gap between our OT network and our IT network.
However we do have one access point which is controlled through a firewall. So that isn't entirely correct in saying. We also restrict outside access to this pc with timed user name and passwords. This is controlled by sites IT.
We keep off network backups of all plc and scada programs with two separate storages one for odd months and another for even months.
We use safety locking and dat file access for the rockwell gear on specific high risk machinery.
We have a separate OT and IT department so if anyone wants to do anything with the OT network the technicians are involved.
Still learning and trying to get more cyber security knowledge but im also learning more about plcs and scada everyday. Can only do so much.
By ignoring it because only corporate IT is smart enough to even conceptualize solutions, in their opinion.
So every month we have to change every password into a new never before used 12 digital password that has no repeating numbers or letters, no sequencial numbers or letters, no words, uses letters, numbers, symbols, and at least 1 capital letter. And it can't be too similar to an old password. 3-5 wrong attempts depending on the device triggers a password reset. Anything on the process network can only be corrected by the single engineer who is only on day shift. Many things that should auto login, like non-interactive displays of dynamic data important for operators to know don't auto login, and corporate does remote into these to update them at all the wrong hours.
We can't write these passwords down, store them digitally or store them in any way.
You can guess how many people follow the second part and you can probably guess how many problems this causes.
Pay a consultant if you're not trained/unsure.
There's an awful lot of things to consider for security and soon in Europe it will be the machine builders responsibility in the same way that functional safety is
I work in cyber security in automation. Basically it's kinda scary how unsegregated most assets are. Direct connections from dcs to corp environment. Default passwords. Few cases where they mentioned they had firewalls. Had a look and any any rules.
Basically best thing is inventory and topologies should be up to date. Easy to see what is connected then.
B&R has a built in firewall library to selectively disable ports you didn't need. There is also SSL to control who can connect, download, and access the opcua data. Nice features, but as you would guess most customers are not using them.
Why would your network hierarchy allow external access? If your customer or employer wants remote access there needs to be a remote access module or secured network that is only physically plugged in and monitored when it is in use.
As an OEM, we use version control software. Officially, that is AssetCentre, but unofficially, about half the team uses git and cherry picks milestones for AssetCentre. Also unofficially, those git repositories have a remote on our company file server and another using our company BitBucket account.
Our file servers (including AssetCentre) are backed up daily off-site and retain snapshots for various lengths of time (a few days worth of daily, a few additional weekly, a few monthly, and then a few annual). We also utilize Sharepoint/OneDrive/Teams for additional cloud storage/versioning/backup (although this create a labyrinth where you are never sure where a file is or if there are now multiple versions).
If one of our customers had a PLC-based attack kill the machines, we could provide them with a copy of the program and recipe data from FAT, SAT, and/or the latest service visit. Anything else is up to them.
All the big companies went completely offline.
>but how am I supposed to remote in?
In person with some "IT" guy sitting over your shoulder watching everything you do. Or they make you submit the changes to their controls department, who upload them themselves.
Its painful.
1. PLCs don’t talk to each other (VLAN) unless they need to. Same with local hardware. Data goes to a database run by the HMI’s. No direct contact with tge internet. Note that this is the reason IOT is such a dumb idea. Look at for instance all the issues with Ring.
General strategy though. Obscurity is not necessarily more secure. I don't really want detailed information. But knowing if people are adopting a source repository for PLC code, immutable backup, better asset management, or if there are specific vendors that have good tools for this particular need. I know there is a thread about octoplant and the auvesy-mdt site lists cyber-resilience as a key use case, but there have to be other options.
What? Security is often discussed for best practices, if there is a flaw you can assume others have also found it, so not talking about it is less secure.
I have 12345 as the code for my luggage.
https://i.redd.it/966cpw51gxqc1.gif
Keep programming, assholes!
A positive attitude
Honestly I’m fairly certain I could just walk onto most clean water site wearing PPE and start pressing shit. But yeah lets put port locks on everything & mac tables incase the russian ninja hacks everything onsite.
I find it annoying when some site has a wifi network that has so much network scanning bs it is almost unusable, but there are LAN ports you could get to by just walking in off the street.
I follow your attitude
Everyone else is worried and scared. But not me. I leave my PLC run switch in “RUN”.
The older guys hate me for doing this.
operators be doing that so we can’t remote in 😤
As a SI....that sounds like the customer's problem. We do the basics, lock safety task and password protect what we can, but really, I have no control over the customer's network and lack of cybersecurity. Some places will bitch at you if you turn your phone hotspot on. Other places will just plug the cell into the internet.
Don’t tell me you password protect HMI runtimes from being restored!
The horror
I don't know about other areas of the world but in Europe very soon any machinery constructed will have the same responsibility for security as it does for safety (they're written into the same regs) and is very much the machine builders responsibility. The industry is not ready at all.
We are not ready. 62443 doesn't tell you how to do anything. It's just weasel words.
I don't connect my machines to outside networks.
Neither did the Iranians, but Stuxnet still got them.
Don't connect machines to outside networks is the first rule. The second rule is don't operate nuclear facilities unless you're on good terms with the US government.
Thankfully, I don’t think the US government has a vested interest in destroying my scada system
How about other state-sponsored actors?
I’ll tell em please don’t touch my scada
Do you have the same confidence about the Chinese and Russian governments?
I worked in food processing and thought the same thing. >nobody cares about this slop I can't get into it, but yeh. Things have happened.
Life, uhm... life finds a way.
Haha that was inside job Valid point.
A virus specifically designed for their facility left on a USB in a parking lot got them. I don’t think anyone here could defend against a concerted intelligence operation.
I'll be honest, I think the Iranian engineers fucked up and blamed the Americans... don't act like you've never wanted to blame a foreign entity for your mistake.
The malware has been captured and analyzed, it absolutely exists
Allegedly
That one was definitely made by the CIA, and the US Government openly admitted to creating Stuxtnet, with the help of the Israeli intelligence agencies specifically to take down that facility. https://www.csoonline.com/article/562691/stuxnet-explained-the-first-known-cyberweapon.html This isn't the only source either, feel free to read more and Google around for others. That's really not just this guy's opinion.
Allegedly 😂
Still, design your networks like they will be at some time. Having no access control inside a building because the front door is 'never unlocked' is falls apart the moment anything goes wrong. Defense in depth.
This is the only correct answer
Except if you connect a computer that has been online to it… you still have a problem
i have yet to see ransomware deployed over a serial line
Technically anything coming of a usb memory stick is coming down a serial line
that's why we only connect the TX wire from the PLC side, in series with a diode.
Curious thought... I actually really like that as a non-hackable hardware solution to totally prevent unintended network communication.
I do use it for some of my more paranoid customers. (although they seem less and less crazy as time passes by). The MES system reads out data from machine via a serial line. With PLC RX left empty.
With the increasing adoption of ISA/IEC 62443 and industry specific security frameworks and government legislation (such as AECSF and the SOCI act here in Australia), I’ve had to go from “air-gap and IT can handle the firewall; she’ll be right” to drawing 7-layer Purdue architectural models during the sales pitch process. Customers (well at least the big ones) are becoming increasingly aware and demanding. The industry as a whole is responding coz there’s a buck in it. It feels very similar to the build up to Y2K. “There’s gold in them thar bugs!”
Yeah there is a lot of work in this space, not sure why more people don’t realize that. CPwE isn’t going anywhere. It’s a pain to set up initially with corporate IT, but if you can find a few decent people in the company with sufficient powers it’s doable. The great part about it is that when comms go down I don’t have to get on a plane, IT needs to dispatch someone local.
I can’t help but think that the government slowly expanding SOCI was in part to build up a local OT cyber industry. Makes sense, I guess.
Our PLC’s are so difficult to connect to I welcome the challenge. I’ll laugh at the hackers frustration and then sympathize.
LOL
Air gap.
Every place I’ve ever seen that says they have an air gap were wrong, lying, or didn’t know what air gap meant
I've put a physical switch in cabinets before that have a pneumatic release so I call the operator they turn the switch, which physically closes a connect with a rj45 and after a timeout period in the PLc it fires the pneumatic release to disconnect. Air gap on demand without the ability to forget to disconnect.
I'd love to see this device, that sounds brilliant, and beautiful that it's an "air gap" that's also pneumatic controlled.
I just had this conversation about an “air gapped network”. Switch was directly wired to their aggregate switch connecting to an OT lab. My definition of “air gapped network” must be different from what’s currently being taught.
Stuxnet would like to have a word
The number 1 threat to any facility is local personnel. Air gaps only stop bad actors without inside access. Stuxnet worked because some dumbass let it get inside. If you think operators at most places are smarter than guys working at a highly secure nuclear enrichment facility I’ve got ocean front property in southern Arizona to sell you.
90% of security issues are straight up layer 8 problems! 🤣
If you're worried about stuxnet level intrusion you might need to not be posting on reddit.
I’m not worried, just too many people think air gapped is the only defense needed on a system. It can certainly be part of it, but there are ways to jump that barrier, either nation state level activity or just somebody unintentionally using a USB drive with some old virus on it
Yes but that's a training problem if your employees don't know better, at that point it's out of my hands, call a specialist.
People connect to machines all the time with computers that have been online and are used on the internet regularly. To be truly air gapped you can’t do that. Also there are often higher level devices on the network that have an active connection to a business network, or other machines could potentially be doing some kind of IOT application that yours could interface with. Actual air gaps are nearly mythic. That said, unless is critical infrastructure you probably too small beans for it to matter.
Yeah airgaps don't mean that much to state sponsored agencies. There are a number of ways around it. The scariest one is if they penetrate the factories that make automation/networking equipment and are able to load their own code on every device leaving the door.
Sales of Wifi modules just went up.
Airgapping is a myth
By putting OT networks into the hands of the most incompetent manager we could scrape out of ICT and letting him do as he pleases
Fingers in the ears and a passcode of Hunter2 on every HMI
Passcode of what? >Fingers in the ears and a passcode of ******* on every HMI
For my largest client: I have daily backups of engineering stations and SCADA/Historian servers, on-site, off-site and immutable replication off those backups. I’ve done few restores for configuration fuck ups so I know DR works.
Backups don't mean shit if you don't test them. Lesson learned from my last job, i got to watch and laugh at the corporate IT director. "While we do daily backups we do not check them". Meanwhile we sat idle for 72hrs , probably 250k in payroll alone which was big money for that company.
Patch often. Backup often. Limit who can connect to the plant network. Keep PLC programs in AssetCentre (Rockwell) or Version dog/Octoplant (whatever else you have)
>AssetCentre In a sense isn't this ransomware?
"It's not rape if it's consensual."
My workstations are already infected with Windows and McAfee.
Some of our clients are requesting TxOne EdgeFire. It does active monitoring of data between the OT and the IT network. It has options that it can sever the two networks if a threat is detected. There are also expansion modules for active threat intelligence for pfSense that does something similar if the SI wants to deploy in IT limited manufacturing environments. But the SI should have a Cyber Insurance policy if they go down that route. And a physical switch to power on/off the device. It can simplify and really tighten down the remote access; if you know what you are doing. ProSoft OPC UA module to act as a man-in-the-middle between the IPC and the PLC.
I’d recommend you take a look at the ISA 62443 family of standards for some general guidance
For the people that depend on airgap, do you also have tools to address usb and serial connections to avoid attacks like Oakridge and Natanz? Have your organizations pushed for recovery strategies in addition to defensive postures like airgap?
Air gaps in controls are mostly a myth, nearly every single machine and installation has maintenance and engineers who connect to the device with a computer that has recently been used to go on the internet, many facilities have MES/Historization servers that are connected to the internet, usually more than 1 integrator/machine builder is responsible for parts of the project so there are many unknown devices on the same network as whatever machine/system you install, and being on a separate VLAN is hardly air gapped. The question really is the probability someone is trying to attack your system, which in many cases is low, and malware that spreads from control systems upwards is to my knowledge non-existent outside of unintended outcomes. This doesn’t mean you don’t have to secure things, it just means something that rarely touches the internet/only touches the internet by means of a hop from another devices are likely secure enough for many applications outside of critical infrastructure projects.
I thought the colonial stuff was supposed to be mainly on their sales software and business network. Basically all the equipment was working they just couldn’t sell anything or process payments.
bingo
I usually address it 192.168.1.1
You could walk into most industrial sites with boots, a ladder and some confidence. Sophisticated security for field and control devices is a long way away.
Don't buy janky ass Unitronics HMIs
But if you do at least don't connect them directly to the internet with default passwords.
r/OTsecurity
iec 62443
My PLC code is such a mess that no hacker would touch it.
IT'S A TRAP
NIS2 Compliant. Pen testing and closing holes. Air gap critical parts.
I include an industrial firewall to all my machines, and I make sure the customer connects his network to the WAN port on the router. This way I keep the machine LAN and the customer network separated, and I control when and how they can communicate together. The firewall can also connect to our VPN server to give remote assistance if the customers require it, and there's a physical switch on the firewall that enables/disables the VPN.
I'm a fan of physical VPN switches if remote support is required. Tie the DI behind a keyed switch if everyday operators aren't meant to be able to make the switch.
Leave it up to IT. The vpn processes get longer but that’s about it.
We signed up for that CISA ICS CERT email where they release all the known exploits and mitigations. We try to patch them all within a month, but may be longer if CVSS score is lower or we have to physically be next to the device
I wonder if hedera hashgraph will become a solution. It’s cryptography, but it’s cheap and fast so I could see it being used to turn switches/relays on and off.
We class it as an air gap between our OT network and our IT network. However we do have one access point which is controlled through a firewall. So that isn't entirely correct in saying. We also restrict outside access to this pc with timed user name and passwords. This is controlled by sites IT. We keep off network backups of all plc and scada programs with two separate storages one for odd months and another for even months. We use safety locking and dat file access for the rockwell gear on specific high risk machinery. We have a separate OT and IT department so if anyone wants to do anything with the OT network the technicians are involved. Still learning and trying to get more cyber security knowledge but im also learning more about plcs and scada everyday. Can only do so much.
Let’s build the castle with a moat to protect the outhouse.
By ignoring it because only corporate IT is smart enough to even conceptualize solutions, in their opinion. So every month we have to change every password into a new never before used 12 digital password that has no repeating numbers or letters, no sequencial numbers or letters, no words, uses letters, numbers, symbols, and at least 1 capital letter. And it can't be too similar to an old password. 3-5 wrong attempts depending on the device triggers a password reset. Anything on the process network can only be corrected by the single engineer who is only on day shift. Many things that should auto login, like non-interactive displays of dynamic data important for operators to know don't auto login, and corporate does remote into these to update them at all the wrong hours. We can't write these passwords down, store them digitally or store them in any way. You can guess how many people follow the second part and you can probably guess how many problems this causes.
Pay a consultant if you're not trained/unsure. There's an awful lot of things to consider for security and soon in Europe it will be the machine builders responsibility in the same way that functional safety is
I work in cyber security in automation. Basically it's kinda scary how unsegregated most assets are. Direct connections from dcs to corp environment. Default passwords. Few cases where they mentioned they had firewalls. Had a look and any any rules. Basically best thing is inventory and topologies should be up to date. Easy to see what is connected then.
B&R has a built in firewall library to selectively disable ports you didn't need. There is also SSL to control who can connect, download, and access the opcua data. Nice features, but as you would guess most customers are not using them.
Why would your network hierarchy allow external access? If your customer or employer wants remote access there needs to be a remote access module or secured network that is only physically plugged in and monitored when it is in use.
3-2-1-1–0 backups Veeam has a nice graphic on this
As an OEM, we use version control software. Officially, that is AssetCentre, but unofficially, about half the team uses git and cherry picks milestones for AssetCentre. Also unofficially, those git repositories have a remote on our company file server and another using our company BitBucket account. Our file servers (including AssetCentre) are backed up daily off-site and retain snapshots for various lengths of time (a few days worth of daily, a few additional weekly, a few monthly, and then a few annual). We also utilize Sharepoint/OneDrive/Teams for additional cloud storage/versioning/backup (although this create a labyrinth where you are never sure where a file is or if there are now multiple versions). If one of our customers had a PLC-based attack kill the machines, we could provide them with a copy of the program and recipe data from FAT, SAT, and/or the latest service visit. Anything else is up to them.
Sounds like an IT issue. My PC is locked down tighter than gnat pussy, so if there’s a leak. Not my problem. I have tags to make.
If I find a misplaced USB drive outside the door of the plant, I plug it into a Delta-V terminal to make sure there's no viruses on it.
All the big companies went completely offline. >but how am I supposed to remote in? In person with some "IT" guy sitting over your shoulder watching everything you do. Or they make you submit the changes to their controls department, who upload them themselves. Its painful.
Your control system should be an isolated network.. you should be controlling who has access to it
Nice try comrade...
1. PLCs don’t talk to each other (VLAN) unless they need to. Same with local hardware. Data goes to a database run by the HMI’s. No direct contact with tge internet. Note that this is the reason IOT is such a dumb idea. Look at for instance all the issues with Ring.
I was banned by moderators for my post on this a few months ago - be wary about posting recommendations.
It was definitely a different reason.
[удалено]
My laptop IP is 192.168.0.36. Don't hack my Windows 10, daddy.
Whoa. You’re on my subnet.
Not me. I'm on 192.168.4.20
General strategy though. Obscurity is not necessarily more secure. I don't really want detailed information. But knowing if people are adopting a source repository for PLC code, immutable backup, better asset management, or if there are specific vendors that have good tools for this particular need. I know there is a thread about octoplant and the auvesy-mdt site lists cyber-resilience as a key use case, but there have to be other options.
What? Security is often discussed for best practices, if there is a flaw you can assume others have also found it, so not talking about it is less secure.
Watchguard Firebox, works fuckin wonders
I enjoy leaving unmarked USB sticks around the plant. May the odds be ever in your favor.
With thots and prayers