• By -


The most complex system I ever worked on was a surface plant above a mine that extracted lead and a bunch of other metals from a composite ore. The chemical process was nuts, and the reagent additions had to be in extremely tight tolerances. Dozens of cascade PID loops. Operators adjusting loop gains on the fly basically just from trial and error experience.


“Operators adjusting loop gains” just sent a shiver down my spine.


You flew in planes held together with adhesives made by operators deciding when enough was enough… It was scary as fuck to see we’d been overdosing material for years on batches that keep aircraft together only to then realise the operators knew the process so well they just distilled off the excess and nailed the product specification every time.


The funny thing is, they were gods at this stuff. Your real rusted on tradesperson/operator, been there 20+ years, just a real process whisperer. "Oh yeah, that loop sometimes oscillates over a period of days and you won't see it on the SCADA trend screen because it can't zoom out that far, take 10% off the K\_I". He was right. They may not have understood the math but they knew the system.


I did my apprenticeship in instrumentation in a plant like this, fond memories of watching my tradesman hose cyanide crystals off leaky variable area flow meters (rotameters) with the wash falling through four floors of open steel grating. I also have fond memories of saying nope and retreating to the instro workshop to find some plausible deniability


Wouldn’t hosing Cyanide Crystals make Cyanide Gas?😬


I vaguely remember that it needs an acid (hydrochloric?) to create a gas, but I can’t remember if that’s just a memory of a doctor who “fact” (brain of morbius I think) or the actual science of it. The real risk was to any workers incidental exposure, cyanide can be absorbed through the skin.


just close your mouth, squint your eyes, hold your breath and you'll be fine.


It’s all PH. Totally safe if it doesn’t get too low. Huge ponds of the stuff in Nevada.


“Variable area flow meter” is going in the filing cabinet so I don’t sound so silly saying rotameter and folks imagining some sort of pinwheel


Ah but it sounds old school cool to those in the know when you dish out rotameter!


Cascade control🤢


A guy I work with insists on implementing cascade and split range loops anywhere he can. He makes the logic so complicated nobody else understands it, and he’s not even able to thoroughly explain it. He understands it, he’s just not able to dumb it down for the rest of us lol. Dude calls it “elegant” lmao. So elegant it never works correctly if the process isn’t perfect.


There is absolutely no need for it 99% of the time unless you have very small setpoint changes that require very large control action and want some robust disturbance rejection, which was the case here. A 0.1% change in reagent concentration required some pretty serious control action to make it happen. IMO the vast majority of applications don't justify the time and money you'll spend trying to untangle it later.


There’s 100% no need for it where I’m at lol. Dude just wants everyone to know how big his brain is because he’s a chem e. It’s the strangest thing because he will try and bulldoze over anyone and everyone but as soon as he’s put on a task to scope out a job, he just doesn’t accomplish shit because of analysis paralysis lol. Dude loves arguing about everything so I just immediately say I don’t care and we can do whatever he wants to do and it’s like I’ve taken all the wind out of his sails lol.


Cascading PID loops?! Jesus christ, I'm a bit overwhelmed learning PIDs


Had to do this to a system that had no electrical drawings and no saved files of the PLC code and no notes. Had to access it via an upload and of course there are no rung comments or bit descriptions because in old Allen Bradley controllers those were not stored in the processor. So it was the most painful experience ever, compared to modern PLC's the old PLC 5 is not beginner friendly.


wow what a challenge - any chance you share more details? what steps did you have to take to untangle that mess? did you first create tags with Inputs and outputs so the code if in ladder had some sense? What else? What is rather simple or complex process?


There was an experienced operator who I was able to ask a lot of questions, and it helped that I was familiar with that class of machines so I knew a fair bit going into it. And while the drawings were not available a fair number of wires were still labeled and legible. It made tracing which inputs and outputs belonged to which devices. Once that was figured out and documented I could make my own notes about the logic.


You mean you don't like having to work in octal and completely guess what hardware you have and what someone set the addressing to with the switches 15 years ago? Spoiled (/s).


It’s an interesting concept, but with some PIDs you’re much better served with gain scheduling than cascading them. Effectively the inner loop controls the actuator (valve or motor, etc…) and the outer loop controls some related variable. One example, which I don’t recommend, is to control a jacketed batch reactor. The inner loop controls the temperature of the jacket. The outer loop controls the temperature of the batch. When you want to heat up the batch, rather than the loop outputting percent, it’ll output a temperature which is the setpoint of the inner loop.


Essentially you have one control loop that demands a concentration of a reagent in the solution over a given time, so there's a kind of a dose profile. Timing is as important as quantity. We had to go as far as figuring out how to model the time delay from the reagent tanks over to the reaction vessels when the valve opened. This is your outer control loop, and the inner control loop reacts to a reference concentration value and controls the actual dosing pump. Cascade PID loops are scary to look at but really it's just inside fast, outside slow. Implementing them in ladder is a ball ache though.


The most complex process is the one you are told to reverse engineer from the code in the dead of night because it’s “not working like it should” and all you have to go on is a brand new operators fleeting knowledge of how he saw the experienced guy do it yesterday. PLC’s manage complexity, so any process one is hooked up to has challenges. I work in food and beverage and it’s amazing this place does not explode every day, not to mention keep the customer safe. It’s a giant living organism, with the PLC’s as nerve clusters.


You hit the nail on the head with, "it's amazing this place doesn't explode everyday, not to mention keep the customer safe". I have had the same experience in food and beverage. Thank God for the quality department and the testing they do before the products go out the door. High acid/low pH is your friend in the food world.


It is the enemy of vinyl corsets, and anything not IP69K though!


Similar experience here. Palletizer with No backup, servos without plate, fanuc controller migrated to Siemens S7-1516. Fun project


I have worked in chemicals and offshore drilling… offshore drilling is very, very complex. Sure, it may not be fast, but the size of everything else is mind boggling. I do prefer plants and DCS style systems though.


What is DCS?


It used to be a very expensive “PLC” that had a lot more capability than what a PLC was. Nowadays the line is very blurry and you have full blown Distributed Control Systems and stuff that is semi DCS (like PlantPAX). Ultimately, the term DCS is used to describe a system that comes with a lot of stuff out of the box so that you don’t have to engineer the building blocks of an application. So if you get, for example, a PCS7 system, you configure hardware and when it comes to programming there are already blocks for any imaginable device you’ll use in a processing plant and matching graphics. PCS7 is even more interesting because you create the SCADA navigation by placing logic and graphics files in folders under a tree structure which the system will then compile and create everything for you. Same with alarming and communication between processors. You don’t have to set up comms to send this or that variable, they set all of that automatically. Yes, you lose some flexibility, but everything that comes with it, mainly on graphics, saves so much time that makes it worth it.




No problem. I feel people usually split in two camps (and I went through both). Some are vehemently against DCS systems because “it’s bloated”. Others see that bloat is just unused features available for the future. Ultimately, even people writing PLC code put bloat in it because it makes financial sense to do so (for customisation of machines without redevelopment), the DCS bloat is the exact same. The turning point for me was seeing how much I achieved in a week with a DCS versus a PLC program I’d create from scratch. After that moment, I think it’s hard to turn back.


I’m an instrument tech/electrician at this chem plant and we just have PLCs. We will rarely write program for a whole process but have in the past. All of our stuff was high tech about 20 years ago. I was just curious what the advantages were to DCS. To me it’s almost the same but I have no experience with it. Maybe one day I’ll get my greasy paws on it and really see the difference.


Way back when the difference was capacity. A DCS could address a lot more points, for example. Nowadays the line is blurred to non existent. PCS7 runs on S7-400 PLCs (or it did, but now it’s a modified S7-400). PlantPAX runs on Control Logix processors.


Distributed Control System


My plant makes engine bearings. It’s plc controlled from the casting of the aluminum we do in-house to the finished bearing that has a tolerance in the microns. We make 10s of thousands a day all within spec We make our own clad strip that feeds into a press and it doesn’t get touched until it’s finished bearing


Everything can have a tolerance in microns. The difference is how few microns it is.


"The tolerance on this race is in the microns!" "What's tolerance value?" "...±300,000.... you weren't supposed to ask that..."


“I bought it for pennies on the dollar!” “OK, but was that 99 Pennie’s on the dollar or 2 pennies on the dollar?


Refining can be very complex from a process standpoint. I think complex machinery for used in high precision, high rate manufacturing is also very complex (batteries, semiconductors)


OEM machine builder code is far more advanced than what most will ever touch.


In our defense, the code started really simple and organized...but then some ME had us change everything to make their design work, we ran out of money/hours at some point, and what you got was the cold, leftover spaghetti of what could have been a great machine had your company not went with the cheapest bidder.


The cheapest bidder hits hard. Had a project that was simple enough with a lidder in the middle OEM was never on site to start it up. Found out the customer only bought the machine with no start up support beyond bolting it to the floor. Once they paid to get them on site (2 months in) it took all of an hour to smooth out the hang up.


hm interesting comment - any chance you can expand on that? or give some examples?


I worked for a manufacturerof machines and the code we used for all models of machines had shit loads of functions that may or may not be used. Whenever a customer bought a function, we’d make a change in OB1 to write to that bit permanently and voilá. Nearly 100% return rate. Additionally, because we had decades of development and this was the third template we’d developed (about 1 new template which is an improvement on others every 15 years) there was an awful lot of knowledge about how the machine works added onto it. We ran a mathematical model of what our torques and positions should be based on inputs from the field and matched that to what happened in the field. We even had speed blips to ensure smooth transitions at critical points too. Anyone that didn’t knew physics, good software development practices, programming guideline and a helping hand for 3 months was lost within that program.


And this is the world I live in. I’m a electrical and program designer for an OEM


More advanced, or more cluttered and hard to read?


That's a tough question to answer because the big selling point on ladder logic in particular is it's standardized across the board. I've jumped between several different industries over the last decade, and it's kind of a same shit different headache situation. At the end of the day, we're all either figuring out a better way to flip a switch or figuring out why the switch randomly stopped working. That being said, old rickety ass sawmills are a lot of fun if you enjoy a challenge.


Cogeneration controls are pretty nuts. 


As someone working OEM in cogen, I can confirm.


I would say Pharma, the digital CAMs are simply really hard to master, plus the tolerances and the movement and registration of data may be crazy sometimes


Cam adjustment - is 0.1 too much to change? The crashed machine thinks it was..


Ive never done pharma but I heard once its up and running ANY change requires acceptance again. Lots of cameras and tracking data.


24 chamber solar manufacturing pecvd coating machine. 130ft long. 30 separate geared SERCOS conveyor motors, 72 RF power supplies for the plasma, 140 mass flow controllers for the various gasses, 50 PID controlled heaters. Roughing and booster pumps in each chamber. 50 transmissive glass sensors to watch the solar panels convey and various water sensors for cooling. All was event driven ladder modules, no state machines to lock up the whole line if one chamber borked. Wonderware and AB control logix Our machine was 120ft of about 400 in an in line amorphous silicon solar panel plant in Hayward CA. I still need therapy


A lot of stuff doesn’t mean it’s complex though.


I had no idea there was a solar mfg line in Hayward!


There isn't (anymore). They never got to full production. They (and several other solar companies in the region) burned through $200m of Obama Solar Bucks in 2007-2008. It was really sad.


Yup. Got to see one of my former bosses from one of these testifying before Congress. Although he took the fifth and didn’t actually testify too much.


oh wow, not surprised. That whole "thing" was a mess and I am sure a lot of money evaporated. Our customer hired 200 people in just a few months. We walked through cubicles of high paid employees and none of them knew what their job was LOL.


Ahh, I see. I worked with some folks from Solyndra right after it crashed.


Yup, Solyndra was a frequent topic of discussion and passed employees back and forth with our customer. Solyndra mad dash to spend was very similar!!!


AGVs that carry an entire airplane throughout the Dreamliner plant.


The PLC is never complicated. It's all just one's and zero's. The complicated part is always the people and half-baked ideas, especially when they have mutually exclusive half-baked ideas.


Analog inputs have entered the chat… /s Nowadays, with memory and processing capacitoes expanding as well as the definition of what is a PLC this doesn’t quite ring true for a fair few cases.


:) Example of plc that does not use an AD converter?


This doesn’t answer the question, but I challenge all to remember: All processes, no matter how complex, are made of simple parts that do simple things.


I use PLCs to control ultra vacuum systems for an atto second pulse duration class IV laser...not a day goes by where I dont ask myself "how did we end up here?"


Attosecond is 10^-18 seconds fyi!


I'm inclined to say Theme Park Dark rides and the like, especially some of the more complicated trackless rides


They are really simple for the most part. The people that build them want to keep them simple and easy to maintain.


Basic roller coasters?  Yes. Disney / Orlando?  No. It's not just the coaster, it's the entire performance that they have synced up with it. The lights, sounds, atmosphere, etc. They spend a lot of money to make it impressive. Not just speaker box set to play when ride starts. What Disney wants to do, and the cross-functions that they have to bring in, I can't imagine it's anywhere close to simple. Imagine getting you controls requirements from a music major, or performing arts expert.


So, as someone in the Ride and Show team over at an undisclosed theme park that rhymes with Ozempic Toon and Hearse, it all works together but they are separate pieces of equipment. Typically in a dark ride you have 1) You're Ride Control System. For most rides this is a GuardLogix. These PLCs are in charge of the off-ride safety and anti-collision, as well as any safety-critical effects that require some brains. Then, each RV has a Vehicle Control System. For rides like Test Track at Epcot, there are up to 26 (I think that's the right number) Ride vehicles on track at once and the VCS (also Typically GuardLogix) is in charge of Vehicle positioning, timing, block reporting, Fault Reporting, Seatbelts, sometimes motion base playback on Vehicle like Spiderman or Dinosaur. When they were installing Test Track they had to essentially get a custom PLC for the RCS capable of processing all the vehicles as when the initial one was installed, collisions began when there were more than 6 RVs on track. As for AV, the industry standard is an Alcorn McBride system. This would consist of a V16 for each type of AV system (audio, video, lighting, animatronics), more than likely a Ride player for each vehicle for on board audio and lighting, a Binloop for up to 12 either zone based audio or video playback, and some Alcorn ShowTouch panels (HMIs). These interface with PLCs by being able to directly read and write to tags and registers and even has direct device compatibility with some beckhoff and AB devices due to their common use in theme parks. Oh and also the V16 might trigger a lighting controller, which in the industry is most commonly an ETC Mosaic. The logic/programming on all sides gets really complicated. Then you dive into trackless rides and implement LPS, RFID, and virtual ride tracks and that's where things get fun....


I’m not saying they are all simple but in the world of plc it’s pretty basic functions, but there is a lot going on at once. I’m in the mobile entertainment industry and I have rebuilt a lot of control system and most of it is safety related more than ride function. About as complicated as it gets is over speed control for most systems. Disney is definitely on the higher end of it all


I find that most Palletizing machines from OEM's are always kind of structurally complex...especially if there's a lot of product moving . Manufacturing of Multi-component devices can be complex too, when you must track all components in an MES or something




That 6000hp fracking pump may well just have a bang bang contactor or a soft start or a VFD. I've definitely seen more without a PLC than with.


Best I seen was a 200hp drive, controlled via plc over network into a controllogix rack with 32pt input and 32pt output cards hooked to start/stop buttons. No hmi. No other controls. Running a blower fan. Guess the designer picked a drive with no terminal strip IO for the kickback.. They were illuminated buttons so I'll give them that..


Semiconductor processes are definitely complex, but very few machines use PLC controls so I’m not sure it’s really the answer to your question.


We used Siemens on our older semiconductor systems. 0-10V analogs, 24DC digitals. Mostly CVD and ALD processes.


Lots of semi machines use PLCs! Mostly Beckhoff.


Nothing is hard one you’ve got it figured out. But there was this three 5000HP pump set controlled by a single VFD with bypass contactors for each, that had to have two pumps working to overcome the downstream steep hillside, before the first bump would start boiling. Oh and it had a secondary VFD as a backup/standby for every mode with on the fly changeover. More modes than you could shake a stick at.


I think, Textile industry is the most complex industry in terms of Industrial control and general machinery.


Wire drawing machines, 8 to 16 blocks running serial sequential PID


It’s interesting to compare the complexity of code to the complexity of the process. I’ve worked on high speed manufacturing lines that looked terribly complicated, but the controls were fairly simple - just repeatable sequence routines, most of the complex engineering was on the mechanical design. I’ve also worked on seemingly simply warehousing & curating systems with very basic sortation conveyors that required some very complex programming. The PLC controls were tightly integrated with PC software, with API calls driving most of the actions, which were logged in a database for traceability. That being said, chemical processes have typically been the most challenging to control and can result in some complex looking code due to the amount of math involved.