The Path to a Pain-Free Control System Upgrade

Speakers

Travis Cox

Co-Director of Software Engineering

Inductive Automation

Henry Palechek

Information & Process Control Supervisor

Water District in California

Sam Matson

Director of Energy Technology

Commercial Metals Company

About this Webinar

To upgrade or not to upgrade: for many industrial organizations, that is the question. Operating-system issues, security concerns, and maintenance costs all provide strong motivations for upgrading SCADA or ICS systems, yet the associated expenses and headaches cause many companies to hesitate.

But what if this "problem" is really an opportunity in disguise? What if upgrades could be less painful than you thought and more beneficial than you ever imagined? In this webinar, Travis Cox from Inductive Automation and a panel of experienced industrial professionals discuss how to bring your system up to date in a way that maximizes your investment and minimizes losses.

Learn about:

• The most compelling reasons to upgrade now
• How to get around common upgrade pain points
• Real-world upgrade success stories
• The benefits of switching to a unified industrial application platform
• And more!

Webinar Transcript

0:00-3:37
(The presenter, Travis Cox, briefly introduces Inductive Automation, Ignition software, and the panelists, Henry Palechek and Sam Matson.)
 

Upgrade Success Stories
3:38
Travis: All right, well thank you Henry and Sam for being here today. So let's now get into your upgrade success stories. Now that we've met you guys. First, we'll start with Henry. He works at a large water district as he mentioned, in the Greater San Diego area of Southern California. The district covers a 50 square mile area with 722 miles of pipeline and a population of over 268,000 people. They have a treatment capacity of 106 million gallons per day. And other interesting detail is that they use ozone as the primary disinfectant. So Henry, what were the biggest SCADA challenges that you guys had prior to doing the upgrade?

4:16
Henry: Yeah. Prior to 2007, we had had some issues trying to meet the requirements of the surface water treatment rule. Logging requirements. You have to provide water quality parameters to the state on a timestamp-type thing. And our legacy software just simply couldn't do that. And so what we were able to do was set up an OPC server to a SQL database to generate those reports. And then at the same time we did some other reports for our distribution manager, giving him daily water consumptions for the day, the week, the month and the year.

4:50
Travis: So we know that not all the problems were just trying to get some features implemented. You also had some hardware issues, specifically operating system issues as well, is that correct?

5:03
Henry: Yeah, that's correct. So the legacy software was having some problems, and I remember they showed up and, "Here's your new support contract. It's going up to $2,000, let me know when I can get a PO" And the software was actually crashing so often that I had the system generate an alarm once an hour just to show that the system was still running. And so we dropped the support contract because we weren't getting any support. Problem dragged on for a year or more.

5:25
Henry: And so, once you drop the support contract then you're no longer upgrading to the latest version. And so the software couldn't be made to run on Vista. And so we needed to find a solution that would run on a more modern operating system. The windows end of life of the operating system was kind of the death nail to the legacy SCADA software.

5:49
Travis: So what is the new system like? How does it solve some of those older problems?

5:53
Henry: The new system's actually quite nice. In the past, I would spend probably 80% of my time administering the central hardware and software of the legacy system and maybe 20% doing radio or PLC work. But now that's kind of flipped, where I'm spending only 20% of my time working on the central hardware and software, and the reality is the Ignition component of that is very small. It's more just changing out old servers, keeping Windows up to date, keeping the virus software up to date.

6:25
Henry: And so that 80%, it's freed me up. I'm now able to work on some of the legacy systems that we have, our radio network and the PLCs, making those modernizations on Ignition. That's freed up a lot of time to address other issues and the SCADA system as a whole.

6:45
Travis: So that sounds like some good benefits from doing the upgrade. Did you have any other results or ROIs that you gained from the new system?

6:55
Henry: Yeah. Stability is probably the other issue, and the software's really super stable. And I show up every morning and ask, "How's SCADA working?" "Fine." "Okay. So I guess I'll go find something else to work on today."

7:12
Travis: So if you can go back in time, what would you have change, if any?

7:18
Henry: Yeah. If I had a time machine, could go back in time, now realizing how much it's decreased my workload and how well the software works, I really wish that I had jumped on it a little bit sooner. We were kind of an early adopter back in 2007, at least on the water side. But yeah, I wish we had done it sooner. If I could go back I would do it sooner.

7:39
Travis: Okay. Well thank you for that. Just kind of get a sense of the challenges you had and what you've done to solve it. We'll be coming back around to some of these other pain points. We'll bring into that conversation again. So over here to Sam, from Commercial Metals Company (CMC) is based in Irving, Texas, has a strong presence throughout the United States. They are a market leader in metal recycling, mill products, fabrication, construction services, and heat-treated specialty steel products. So Sam, just like we discussed with Henry, what were the biggest SCADA challenges that you guys have?

8:17
Sam: Our first challenges that I faced in my career at CMC were in starting up a new facility. We had a system that came from the provider with very limited data history capabilities and with really no visibility of the process outside of the control room. So whenever we needed to see what was taking place, we'd have to go to the control room to find out. And any analysis of what happened in the past was very challenging because it was cumbersome to get any data out of it to do trending analysis and things like that.

8:51
Travis: Right. So what are some of the problems that you had with the old system? I know you mentioned that in your success story that you are now bringing data to corporate and be able to share the data with more people. Did the old system, was it proprietary or hard to get the information out?

9:09
Sam: Yeah. So the history, it was always stored local on the PC, is running HMI software. And so we would have to dump data into a CSV file, bring it into excel and analyze it. And if we wanted long time periods we had to dump segments out and bring them together in excel and build the trends and analysis after that. So there was a great deal of time spent collecting the data, organizing the data, getting it all together before we could even start looking at it to make decisions.

9:35
Travis: Okay. So in contrast, what does the new system look like and how did it solve some of those problems?

9:42
Sam: So now with our upgrade historian, we can look at data for any time period we want to. And at this point we can look at it from several facilities altogether, and very quick and easy. In some cases we built some analysis into the display screens so that there's actually some python running in the background, running aggregate and min and max averages and grouping data sets to help us really understand what's happening very quickly and take a decision shortly after that. So it's a matter of selecting a date and time that you want to look at the data for, and all of it's right there in front of the operator to take action.

10:19
Travis: Okay. Can you quantify any ROI and return on investment here?

10:24
Sam: So that was one of the challenges we had getting started, was helping our management group including IT to understand that although it's not the biggest name in automation, there's a lot of capabilities there. And so we took a leap of faith to get started, but we had a lot of confidence that we would be successful.

10:44
Sam: As far as return on investment goes. I can give you an accounting for one of our other facilities that started Ignition a few years after our first installation. And during that first month they found over 22 problems that they never knew existed even by looking at the trends and historical data.

11:00
Sam: And I think they paid that back within the first month after they'd installed it. And what's funny is sometimes now as we have some operators moving from facility to facility, and one of the things that they asked me as they're planning the move is, "Well, am I going to have Ignition available to me at my new facility? Because I can't run without it."

11:20
Travis: Right. Okay. Well, thank you both. Again, we're going to talk about the different pain points and some more detail about reasons to need to do an upgrade and then, we'll start discussing some of the different ways that we can approach the best practices for doing an upgrade. So again, thank you. We'll hear more of your perspectives as we go along.

11:45
Travis: Let's first start by defining our terms. “Upgrade” – what does that mean? Really it can mean upgrading to a newer version of the current software that you have or hardware or it can mean upgrading to an entirely new software platform. And so, when you hear that word “upgrade,” you’ve got to ask the question as to what we're really getting in?

12:07
Travis: A lot of times upgrading really means that I have new pain points, new problems that I need to solve and I'm upgrading to something, I'm making an action to get to either a newer version or a completely different platform to solve those pain points. Really upgrades are more about pain points than they are about specifically just having to keep something up to date.

12:29
Travis: Now keeping these up to date is very important when it comes to cybersecurity where IT is why they're patching servers and all of that. It's always important to keep your software up to date, but a lot of the reasons that we talk about why people had to do upgrades were because of the different pain points.
 

Reason to Upgrade: The Operating System
12:44
Travis: So in this webinar we're going to focus on the reasons of why you need to upgrade, whether it's a newer version or a whole new software platform, and especially how to solve the pain points associated with that upgrade. There are many reasons to do an upgrade, and I'm just going to focus on a few of the big reasons out there.

13:04
Travis: The first major reason is your operating system, specifically the operating system that is going obsolete. This is probably the most common reason that companies are operating their SCADA systems. Especially with Windows XP being stuck or tied to that specific operating system or a specific version of something there. And that it could be very problematic to potentially upgrade, because it can ensure a lot of new costs and a lot of resources to be able to get to a version that is compatible with newer operating systems.

13:36
Travis: And we all know that IT, they want to standardize, they want to upgrade to keep the whole infrastructure current. And don't want to have that one workstation sitting where that is on an older version, praying that it doesn't crash because we don't have any backups there to take over. So this is probably one of the biggest reasons to actually do the upgrade. And I know Henry, we talked about something earlier with you, this is something that you guys faced as well. Can you comment on the operations obsolescence?

14:14
Henry: Yeah. That's a huge issue. With our legacy system something as simple as moving from service pack three to service pack four would break the SCADA software and that I'd have to be patching it. And then, certainly a version change from XP to Vista, there wasn't a patch available. And so being forced to do an upgrade because Microsoft changes kind of the way their software works is not a good solution, not a good long-term solution.

14:44
Henry: But with Ignition, it's cross-platform. So we were able to migrate from XP to modern 64-bit operating systems, completely independent of the operating system. The software doesn't really care what operating systems it's on. And so that pain point of trying to manage windows updates to the SCADA software is gone.

15:09
Travis: Right. Sam, any comments to add to that?

15:13
Sam: We're in the same boat as far as IT wanting to get rid of those legacy operating systems. I mean, I guess I have one that maybe piggybacks onto a future point, but we had some proprietary software running on an old operating system and we didn't want to touch it because it was working, and we would break it if we moved. But we actually were able to develop the same functionality in an Ignition project and kind of get two for one, where we got rid of a piece of proprietary software and we were able to migrate to the new operating system with Ignition.

Reason to Upgrade: Yearly Software Costs
15:42
Travis: Okay. Thank you. So another big reason to upgrade is the expensive yearly software costs for maintenance and support. Especially when the costs are just associated with continuing to use the software that you have, and or getting support with that going forward. It's a big expense every year. And if you're not being able to take advantage of the latest and greatest technology or being able to solve the previous problem of operating system obsolescence, then you're just basically paying for it to just keep the system running.

16:16
Travis: And it is a cost that really is not allowing you to future-proof your system. So in many cases that money that you spend on maintenance costs could actually be used to purchase and deliver an entirely new system. To not only have potential lower maintenance costs, but to give you newer technologies to be able to solve some of these other issues like operating system obsolescence.

16:46
Travis: So it's really important to take a look at the maintenance cost that are associated with a different software and hardware versions you've got, to make sure that we're looking at future-proofing our systems. This is a pretty common thread as we go along. We're going to talk about future-proofing and the different things we have to look out for when doing the upgrades. So Henry and Sam, any comments or experiences on high maintenance costs or need to upgrade because of these costs?

17:16
Sam: Sure. For us, yeah, certainly we can see that our costs are quite high for our legacy SCADA systems and we have that as an objective longer term, is to migrate out of these higher cost systems that we're getting only marginal benefit out of.

17:31
Henry: So our support contract was kind of high in that we had every place where you put a workstation or a large monitor display, a trend or something was a software license. So we've got like 18 different computers displaying SCADA data throughout the plant. So I would have to be paying support contract on all that with my servers. I've just got a high availability combination. I've got the primary and the hot stand, but I'm really only licensing two pieces of software. So I'm getting a much, much better deal in the terms of the support contract because of the client server architecture.
 

Reason to Upgrade: Cybersecurity Concerns
18:05
Travis: So there are also some big raises to upgrade that relate OT-IT convergence. The push to make OT and IT work together is making us rethink our systems. Security is a major issue that needs to be tackled here. Cybersecurity threats and attack vectors are increasing every day, and many controls systems lack fundamental security measures like encryption, authentication, auditing, and more.

18:30
Travis: You may need to upgrade to a newer software just to be able to get those newer security features. Your security is only as strong as the weakest link, which is usually the legacy poll response PLC or an older version of a software platform, especially one that's on an older operating system that may have these potential vulnerabilities.

18:53
Travis: This is why in the IT world there's a constant effort to keep all of our systems to pass, keep everything up to date. While I know when working with IT, it's in a constant battle to plan when those upgrades and those patches actually happen. But the reason that those patches are being made is to take care of some of these vulnerabilities.

19:17
Travis: And that is something that in the OT world, we're just not so accustomed with. We want to keep things running. We don't want to touch them. And we sometimes neglect the cybersecurity or we try to air-gap the systems. And an unfortunate part now is that you can't do that. The worlds of OT-IT are colliding. The demand for that data on the business side to do more with that information, is just simply increasing.

19:44
Travis: And so it's now time to take a look at how we can protect our legacy devices and PLCs. A lot of times that involves using edge gateways in front of the PLCs that can deliver information through a secure channel to your SCADA system or to a central system, and to also explore software platforms that have modern IT security principles.

20:05
Travis: We're going to talk more about what those principles are as we go along, but these days we simply can't afford to overlook security. So Henry and Sam, security is always a concern, especially you Henry, with water districts there. So how has security impacted their decisions and what you're doing with these upgrades?

20:31
Henry: So, our IT group has been able to put in certification for security. They've been able to use active directory for the log on. They'd set up the virus software to automatically update the machines every day. They've set up Windows to update the clients every day and reboot. And then I will reboot the servers as needed. And so we've got a lot of those IT best practices that are in place, and because the IT guys know how to do that, they were able to help me out, because I don't necessarily do that type of stuff at a daily level.

21:04
Henry: I'm working more at the PLC levels. One thing that I've been able to do with the IA Modbus driver, is you can move it from the standard port 502 to some obscure port. And so I've done that. All my PLCs are on nonstandard port. So if a hacker comes in and starts looking for data on port 502, there is no data on the port 502. So I've done a little bit of security on my end at the PLC level as well.

21:30
Travis: Now, Sam I know you mentioned bringing the plant data and bringing that up to a corporate facility, you have that kind of experience as OT-IT convergence firsthand. What are your experiences here?

21:46
Sam: Excuse me. Our IT group reads the headlines just like everybody else, and certainly that gets everybody's awareness elevated from a cybersecurity standpoint. So one of their big concerns has always been the interconnectivity because our ERP system is very closely connected with our automation systems and then our process data historian also obviously is integrated very closely as well. So their main concern was when I wanted to bring more data out to our data consuming employees, how are we going to bring that out securely and protect our process?

22:20
Sam: So the architecture that we developed allowed for very neat openings across the firewalls to allow the data access in a one way fashion so that we were able to see what's happening in our process, able to look at historian data and have no concerns about impacting the process either from a real time standpoint or the data that's being stored by the historian.

Reason to Upgrade: Proprietary Software
22:45
Travis: Okay. Well thank you. All right. And probably one of the biggest reasons why companies have had to look at doing upgrades, have been from a proprietary software, and that a lot of these packages are proprietary. IT doesn't understand the proprietary software that we use in the OT world. Here we've got thousands of different pulling protocols that are out there.

23:13
Travis: We've got file formats that just aren't understood or software packages where they don't know how to maintain or how to go in there and make changes or look at configuration or diagnostics. And so that's why the worlds of OT and IT have been very much at a rift where they don't communicate too much today. They kind of just want to operate in their own, and really doesn't have their own people managing it.

23:33
Travis: But that really does not allow us to take advantage of what IT is trying to provide. And IT has to understand that we're trying to make operational technologies that we have continue to work and continue to run really well. So it's really because we speak completely different languages and that we're not communicating clearly across those lines. And this makes it really difficult to share data with other departments.

23:59
Travis: Makes it difficult on the business side to get data there or in the cloud, to integrate with third party solutions. And integrations can provide tremendous ROI where we can bring more context to the operators. But they also make it difficult to upgrade replaced system. Very difficult to scale. Again, because this mindset of being closed or locked into particular vendors there does not really allow you to win.

24:24
Travis: While it might provide some solutions to some problems. It doesn't start solving the newer problems we've got today of bringing information and converging these two worlds. So really it's all about speaking the same language, about communication, about using open standards. And we're going to talk more about this as we go along. So really got to evaluate control system closely to see if it could work.

24:47
Travis: What is proprietary? What is open? And if there is the ability to get that proprietary scope of format so that this integration can really happen. So, Sam and Henry, have you guys had to deal with proprietary software and had to figure out how to get that information to more places?

25:07
Henry: Sure. So I can remember when we were the Wonderware systems, the IT guys wouldn't touch it at all, but now that we're Ignition, I'm able to get a lot more support from the IT guys because they're not afraid to log into the webpage and do what they need to do. They're not afraid to take a look at the tables and the SQL database and help me get the data out to the users. And so, yeah, I think we're now much more speaking the same languages then we were in the past.

25:38
Sam: And this is Sam. From my standpoint, it's just really helped us to be able to connect our other in-house developed software that's more on the ERP side together with our manufacturing data so that we can see all in one what's happening each step of the process and the consequences or results altogether. So we can tie things together quickly inside of Ignition rather than having to bring it all out and match it up and then have something to look at and analyze.

26:08
Travis: Right. So as you guys can see, there are a lot of urgent reasons too to do upgrades. And we listed some major ones there, but there are a lot of smaller issues as well. But a lot of companies are very hesitant to do that. And the question is “Why? What is the elephant in the room?” It's because upgrades are scary and they're scary for a lot of different reasons.

26:26
Travis: They're scary from a financial standpoint. They're scary in terms of keeping your system running. They're scary in terms of the learning curve. You've got a lot of people who understand the old systems, who invested a lot of time in developing and have had training on those platforms and would have to learn some new things potentially. It's scary in terms of leaving the system you've invested so much in and facing potentially unknowns with newer technologies or newer arenas.

26:56
Travis: And it's also scary to think about potentially starting over. But at the end of the day there are some proven methods and it's really all about looking at the long-term benefits of the systems. The upfront, if you're too blinded by the upfront costs or some of the issues there, you lose out on the side of the overall long-term benefits where you're moving to a system that is really eliminating a lot of these major problems where we have to do an upgrade yet again. We would want the upgrades to be able to be done in a way that allow us to continue to evolve our system over time without having to replace the entire thing. That concept needs to go away. And so, we can make these less scary and less painful.
 

9 Steps to a Pain-Free Upgrade – Step 1: Upgrade to a Software Platform
27:39
Travis: So what I want to do is talk about the steps to a pain-free upgrade. The first overarching step is to upgrade to a platform. Let me explain what I mean by platform. A true platform is one that is developed with a unified approach, meaning that everything was thought of from the beginning to be simple in terms of getting it installed, getting it upgraded, adding new functionality, being able to deliver new technologies as they come out in the future, but have one environment to go to, one way to back it up, one simplistic view of that.

28:17
Travis: And when you have packages that have a lot of different functionality, that are different pieces of software that don't all work together or they're using different sorts of technologies, it can make it hard from that aspect to maintain that going forward. So a unified approach is really important, and that's just seeing a shift from applications to platforms and ecosystems that are being built up around platforms.

28:40
Travis: So offering that single development environment, working with different solutions, adding new functions easily, interoperability, integrations, plug-and-play, backwards-compatibility. And that's a huge one. Platforms are designed to be backwards-compatible because that is a platform that is meant to be growing as you go along. I mean, you can imagine, take a look at Google and Azure and AWS, they've cloud platforms that have to have backwards-compatibility.

29:09
Travis: They have to continue to make their systems that people have been using from day one to work going forward. And that's the same thing that's got to happen to the OT world. We've neglected this part in the OT world where IT is so used to these kinds of platforms, that are so used to being able to work with it and be able to keep it up to date and be able to take advantage of new technology. But that's not something we're used to in the OT world. So there are platforms there that are being brought in and that we can start taking advantage of.

Step 2: Choose Cross-Platform Software
29:37
The next step is to also choose software that's cross-platform compatible, so it can run really on anything. Especially now, we're being stuck to a particular windows operating system or being stuck to a particular hardware vendor, is not going to allow you to be agile as you go along, going forward.

29:57
Travis: You need to be able to use best-in-class hardware and use whatever operating system that's there. To be able to take advantage of fog computing environments, be able to take advantage of ARM devices, a little more constrained devices as well as high-power servers, embedded computing devices and more. So cross-platform compatibility, if the fact that we can run it on any of those as newer versions come out or as new things are there, we can easily work with that and run those platforms.

30:26
Travis: You'll avoid a lot of software obsolescence or forced upgrades, which as we talked about here can be very expensive. You won't be stuck to specific operating systems or versions of software, and which can be really limiting of course, what your system can do. So it's about that future-proofing. That is the number-one kind of theme that we're going to have throughout these different steps.
 

Step 3: Choose Software That Supports Modern Cybersecurity
30:50
Travis: To facilitate the convergence of OT and IT, you need to raise level of security in your control system.

30:55
Travis: All the communication over industrial networks should be encrypted following today's standards. Software should be pen-tested to identify any potential vulnerabilities and should be working with, especially here in the United States, with Department of Homeland Security with the industrial control system, cyber emergency response team, ICS-CERT, to make sure that where, especially with critical infrastructure, we're doing everything we possibly can to make sure that everything about the system is using those modern security paradigms there.

31:30
Travis: We should also be using two-factor authentication or federated identity providers that offer single sign-on as well as that multifactor to verify the person's getting into the system. But you can't just have that, you got to have auditing bills and systems so you know who's doing what, what's happening in there. Because if there was an issue, you need to be able to go back and logs and see what that problem was.

31:52
Travis: Having role-based or zone-based security permission models where we can limit what the functionality based on who a person is and based on where they're located. With the mobility that's out there today, there's people that'll be able to get data everywhere. Not only the plant floor, it's not just that dedicated station anymore. They'd be walking around with a mobile device. They can be at their home through a VPN connection. We've got to make sure that we protect the system, especially when it comes to safety, not allow certain things to happen.

32:19
Travis: And one of the biggest things is, again, the weakest link a lot of scenarios is legacy PLCs with poll-response protocols, like Modbus, like DF1, Ethan and IP, where honestly if you knew the IP address, if you knew the port, that protocol, you can do some real harm. And there's been seize attack vectors like looking at Stuxnet and others where they're able to get that malware in, that malware can come from a variety of different things. Imagine just plugging a USB drive into your control system and all of a sudden it takes down everything because it had a program that was able to discover and communicate to the PLCs.

32:57
Travis: So the PLC is something we really got to take a strong look at when it comes to, it's not just software, it's hardware as well. And a lot of networks are doing ... They're trying to do with firewalls, getting networks to protect it, which is really good. Another advantage is to be able to put an edge gateway that's dual nicked where one is talking to the PLC, and that's on a different subnet. Then the other side of the nick is going to the control network that's able to deliver that data through, say a protocol like OPC UA or MQTT so that we have security built into the transfer mechanism there for getting that information to SCADA or any other consumers of that data.

33:39
Travis: This technology exists with OPC UA and MQTT, especially these edge gateways that are there in the IoT world. It really allows us to get to a much better infrastructure and to also scale our systems as we go along to use best in class software and hardware. Honestly, MQTT is one of the big areas where we can future-proof our systems and have a plug-and-play architecture where we can use a lot more different systems and not just be stuck to one big monolithic system.

34:10
Travis: So I would encourage you guys to take a look at it, we've done a lot of webinars on MQTT. I'm not going to focus on that on this particular webinar, but it's one area that is a high interest, not just be able to have security and software, but look at it from a hardware perspective as well. To migrate from brownfield to a new architecture. How do we get there from where we are? That's an important step and one that everybody should be looking at as we look at upgrades. Upgrading could be more than just software. It can be the infrastructure as well.

Step 4: Choose Software That Supports Open Standards & Interoperability
34:41
Travis: Also, many organizations are shifting toward open standards. As we talked about, they can be understood by both OT and IT. It can allow both sides to really work together. As I was talking about with OPC UA and MQTT, those are two standards that can be easily understood by both parties, especially with MQTT these days. Is something IT is already using on their side. As well as SQL and time series databases and HTTP and HTPS or web services REST and SOAP.

35:12
Travis: And so, these are all things that they can understand if there was those integration points, IT can be a better partner with OT in that way. And it can really allow for a lot of plug-and-play systems, our interoperability, having secure platforms that are more flexible and easier to maintain. So it's much easier to upgrade or replace individual pieces than having to do the entire infrastructure. Those kinds of days need to be thought of as the past.

35:40
Travis: We've gotta be able to use best in class for all the different types of systems. The third-party integrations can be really important, as I was saying earlier, to add context to our applications. So not just having a software that provides a solution for one particular niche, but if that software could then integrate with other pieces really can share data, can allow us for doing a lot more in terms of business intelligence and machine learning or getting data to the business, providing more context to the operators.

36:15
Travis: So open standards really allow us to do all of this. And there's a big shift in industry. There is in the open process automation forum that is all about being able to develop the standards that all vendors are supporting. That's a big initiative by Inductive Automation to continue using open standards as we go along so that IT and OT can work together and so that the younger generation of engineers and operators that are coming in can also understand the new technology.

36:44
Travis: This is really important, it's something that we tend to forget about with our control systems. We have people who potentially, who've maintained and installed those systems who are getting to retirement age and when they are retired, what is the plan for who's going to support that going forward?

36:59
Travis: If you've got a system that's proprietary, that is built on specific versions of operating systems, just specific proprietary technologies. It's going to be very difficult for newer folks to bring that into the mix. So Henry, we've mentioned silver tsunami quite a bit with that. How are water utilities in particular being affected by the silver tsunami and how should that impact your software choices?

37:28
Henry: Yeah. So if you were to look at our plant 10 years ago, you would find that our operators had a joint combination of 135 years of experience. Then we had a couple of retirees and it dropped to 60. And we just recently had an operator with 27 years of experience retire. And so now we've got this younger staff and we've only got 24 years of operator experience now in our group. And so these younger operators are using the SCADA software a lot more. The older guys didn't have the technology and knew how to do stuff manually.

38:03
Henry: The newer guys are much more dependent on this SCADA software. Fortunately though, the SCADA techs that we hire, the younger guys, they seem to pick the software up much faster, because it's more straightforward modern software than the legacy stuff. So a little bit of a loss of operator experience, but Ignition kind of helps fill that gap in.

38:26
Travis: We’ll talk more about that, the younger generation, especially in terms of understanding the technologies and the training in just a moment.

Step 5: Find Tools with Complete Backwards-Compatibility
38:32
Travis: Another important step is to find tools with complete backwards-compatibility from vendors with the strong commitment to providing backwards-compatibility. This is really important. You have to have a proven track record. You have to have commitment to keeping things going.

38:44
Travis: And platforms. When you look at choosing platforms that is one of the core tenants of a platform. And we don't use this platform loosely with Ignition as a core tenant of what we do in Inductive Automation. So really when you put in systems, they should all have that thing kind of thing, because you want to be able to take advantage as you go forward. But if you put an architecture in place that does allow you to use the best in class in different or you can replace individual pieces you're not going to replace the entire thing.

39:15
Travis: It might be that if there was ever a need to switch to a new tool that you would be able to do that, you'd still be able to be agile in your environment. So this isn't really a standard practice in IT, to be backwards-compatible, but it's rare in OT. So it's important to really be looking at this as you choose different systems.

Step 6: Find Software with Simple & Flexible Licensing
39:35
Travis: Also make sure your new software has a simple and flexible licensing model. You look at long-term or total cost of ownership you've got ... When you add new things in place, when you add new devices, you add new areas of the plant. When you need more people to look at that data, you need to deliver to more types of systems. The size of the system can get bigger, as new technologies come out, there can be new things that you want to take advantage of. And the licensing model has to correspond with that, has to be simple, has to be able to allow you to grow with it and without putting serious financial limits on your compatibility.

40:12
Travis: And it's really about breaking barriers down. The whole OT-IT convergence and IoT, and all that we're talking about is breaking down more barriers so that we can get to larger solutions. So the licensing rule is one of the important steps and having a simple one and being able to scale in size and in functionality with that. So I know Henry and Sam, licensing, especially you Henry, licensing has been an important part. With what water districts, you can't just go and get a capex project to do a massive replacement, right? So how has that impacted both of you guys?

40:51
Henry: Yeah. So the licensing issue, like I'd mentioned earlier. So in our control room where we would have screens displaying data. That was a license for each one of them. And so real quick, you could end up with $60,000 worth of software. For Ignition, it doesn't work like that. It's just licensed by the server, and there's no licensing on the client. In fact, the clients are almost zero administration.

41:15
Henry: I remember when I did my first upgrade to Ignition, I got it to install and it copies the file and restarts the service in two minutes and all of the screens flash and I'm like, "Oh, that can't have worked. SCADA software doesn't upgrade that quickly." But it did. And so the issue of dealing with the licensing and stuff, that's all kind of gone away because it's centrally done on the server.

41:39
Travis: Yeah. I'm in the same boat with Henry in terms of, I would like everyone that wants access to data to be able to have that. And in the past we'd had to justify every single client installation and there was lots of discussions about it and it was a very slow process to get things in peoples' hands because of that cost. Now with unlimited tags, unlimited clients, everybody that wants it can have it and it becomes no barrier to helping people understand what's going on in the process.

Step 7: Choose Software with Strong Training & IT Support
42:09
Travis: Another real big concern here is training. Training is one of the big pain points of conversion. It can be expensive, difficult and very time-consuming. Especially getting people to commit to learn, learn all the platforms, to learn something new. When you really got accustomed to using a certain software package for a long time, the learning curve for a new software can be intimidating even if the old software is flawed.

42:34
Travis: However, technology keeps changing and so you eventually have to adapt. I mean, this is something in reality we've got to face. But too often the only way to learn about new software packages is spend a lot of money buying the software or taking really expensive training courses after you bought the software. When upgrading it’s essential that the new software platform provides a strong training program, IT support, and a user forum where you can find some quick help.

42:57
Travis: Software packages that rely heavily on proprietary technology will be harder to support and often get you locked into an expensive support contract. Software that offers web-based or free training options can save you a lot of money and are convenient for your staff and for future developers to come in and learn. So if you want a solution that your staff will buy into and that will be effective in the long-term, make sure the vendor's committed to real customer support and transparency and having free training information available out there on the Internet.

43:26
Travis: Especially the younger generation, the workforce, they are expecting to go to Google and find solutions to their problems and find areas that they're not accustomed to having to call into a support line and have to wait and do all of that. And there are plenty of stories. I know obviously I'm biased working at Inductive Automation, plenty of stories of our customers who have hired interns for the summer, who've been able to come in and use the free Inductive University training program that we provide to learn the software.

43:59
Travis: And they've had no experience in the industry and no experience with Ignition whatsoever, but in the summer have been able to build entire systems and the ability to provide a lot of benefit to the end-user. So it's the more transparency and more training the better it's going to be. And there's a lot of ... It's not just software, a lot of hardware vendors are also being very strongly committed to these kinds of free trainings.
 

Step 8: Salvage Assets from the Existing System

44:26
Travis: As we said, when you upgrade, there's often a fear of having to start over from scratch, but it doesn't necessarily have to be the case. It's likely that there are some assets that you can establish from the SCADA system and carry over to the new one. You develop the tags, you've had a lot of historical data. You've had a project that's been implemented, you can reuse a lot of that functionality. While it may not be a direct conversion, you still have those as assets. There was time well spent into architecting that. But it shouldn't be a straight conversion. Always think about it from how can we improve on what we've already built?

45:01
Travis: Can we take advantage of the new technologies, new opportunities that arise? And how you can really truly leverage that for a much better system. And now with the ability of getting data to the business, upgrading to a system that can give you the same function I had before, but it can also make it really easy to deliver data to more places. That's a huge win. There's many, many others that are out there.
 

Step 9: Focus on Long-Term Benefits
45:27
Travis: One of the things that makes upgrades seem scary is the idea of moving to a new system after you've invested so much into your current system. To overcome that fear you can focus on long-term benefits. It's all about the total cost of ownership. The future-proofing you'll receive by upgrading to an open, secure interoperable software platform, and really being committed in your infrastructure to that premise.

45:48
Travis: And there's options out there today. If you choose not to upgrade now, you can be forced to upgrade in the not-too-distant future, or you can simply be behind. You can make the most out of your upgrade by investing in a system that will work for the long term. If it has flexible licensing, open standards, a modular architecture, provides a platform that's unifying, gives you one simple way of installation upgrading, then you're going to get there.

46:12
Travis: And don't just look at the upfront price tag, look at the long-term ROI. If you're able to make your system more efficient and increase the amount of things you can accomplish with it, the upgrade will pay for itself, especially if it's easier to maintain. When we look beyond the short-term upgrades, it’s less of a scary proposition and more of an exciting opportunity. And you also have to look at what will the future maintenance cost and the maintenance costs of that going to be, for not only from the vendor, but for your staff to be able to work with it.
 

Recap & Closing Comments
46:39
Travis: And so these are all major considerations that we need to look at. So let's recap our discussion today. We've talked about the major reasons to upgrade, which are operating-system obsolescence or problems being stuck to a particular version of OS or hardware. Maintenance costs, security concerns and the need for OT-IT convergence. Even with those reasons, many organizations are afraid of the upgrade.

47:04
Travis: But there are ways to reduce the pain and turn this into an exciting opportunity. It's all about understanding the future-proofing, understanding long-term ROIs but getting buy-in from your organization to look at it for more than just providing the upgrades. There was a problem that we had to solve the upgrade, but at the end of the day the company is trying to move forward and trying to solve new problems that they're not even thinking about it.

47:27
Travis: So if you just look at the lens of replacing a system and in getting that functionality again, you're losing sight of all the other ways that we can leverage that system and that data for doing more within the organization. So we talked about those nine steps. Upgrading to a platform that has the characteristics we mentioned. Choosing a software that's cross-platform compatible. Have that supports modern cybersecurity solution.

47:56
Travis: Supports open standards and has interoperability built into it. Completely backwards-compatible. Simple and flexible licensing models, which is incredibly important. And it offers a strong training program with of course support. You can salvage a lot of what you've done before if you have to replace that with new systems. But when you replace it with new system make sure that these are the premises that you're looking at when replacing.

48:21
Travis: And of course, focus on the long-term benefits of upgrading not just the upfront cost. Even though that is what you're faced with it has to be justified, has to go through. That can pay for itself if you can think about it down the road. As a final point, I’d like to say that at Inductive Automation, we've always been very mindful about addressing all these problems that we put out here today. These things that we're telling you to mention, to look at, don't just apply to us, it applies to a lot of the hardware and software you're using.

48:57
Travis: Our founder was a control systems integrator. He had to go through many painful upgrades and we wanted to find a better way. Rather than developing traditional SCADA software, we focus on providing an industrial application platform where users can build the best solution for their needs. We made it modular, which is a more flexible and affordable way and allows you to future-proof your system.

49:16
Travis: We've also made our software cross-platform, works on any version of Windows, Linux or Mac OS, as well as it works in fog computing environments, constrained devices, especially ARM devices. And the upcoming release that's coming out here in a month with the Perspective module and the new cybersecurity standards we're putting into it is always an address to make sure that software's being relevant and staying current. That is completely backwards-compatible to our previous versions.

49:47
Travis: In relation to cybersecurity, we're doing more in the way of SSL. So we use SSL technology, but we're doing more in the way of a federated identity, two-factor authentication and single sign-on with better encryption models built in. And we also use very friendly standards as OPC UA, MQTT, SQL and much, much more. We've introduced an unlimited licensing model that allows you to have unlimited clients, screens, tags, connections in devices.

50:14
Travis: And we have a very extensive online training program called the Inductive University. They're very extensive online documentation. We have really good tech support and a great user forum to see what people are doing with the platform. So in wrapping here, Sam and Henry, can you just tell us about your experiences with Ignition. I know you've mentioned a bit of that as we've gone along here. Is there any final remarks about what Ignition has been able to do for you in terms of upgrades and about this concept as a whole? So Henry, we'll start with you.

50:48
Henry: Yes. My experience is I found Ignition to be very powerful open, a cost-effective solution. We actually did our system in a pay as you go. We didn't use any capital funding and we haven't used any capital funding for the past 11 years, keeping this system running. There's nothing really that I found that the software can't do. And it's not burdened with technologies like Microsoft terminal services or .Net.

51:14
Henry: And if I really need to do some heavy lifting, there is a scripting engine. But if I need to do heavy lifting I can do the python solution to do whatever work I needed to do. And we've also bridged our SCADA system into our laboratory management system where the chemists and stuff put their data in to get the SCADA data into other business systems as well. So yeah, I like it. It's great software.

51:39
Travis: All right, Sam over to you. I have a slide here that you want to show as well.

51:44
Sam: So yeah. Let's just help to illustrate the general architecture we have set up at Commercial Metals. You can see we've got Ignition running at facilities across our lines of business, and we really go through a mini upgrade at each of our facilities as we put these in and the justification of discussions around that.

52:03
Sam: And we've really caught a lot of momentum as we put in more and more systems because we can just say, "Well look, if you're not sure about this, let's go look at your sister’s shop right now, and we can go walk over to this individual's desk, pull it up, show him exactly what we're going to see if we hook his equipment up to it."

52:20
Sam: And it's been, like I said, building momentum like a snowball. So now there's almost no discussion, no to questions, "But I want to have that too. And when can you have it for me?" Just because it's so quick and easy once you have it installed, it's so quick and easy to install. But then once you have it, it just opens up the world to you.

52:37
Sam: The other big thing that has been good for us is that a lot of our processes, they're very energy intensive and as a result they're very cost-intense in terms of the conversion cost. So we put a lot of monitoring on and alerting on our specific tags to let us know when our processes maybe are outside of where we want to be operating. And that usually indicates there's a cost issue too.

53:00
Sam: So we can get alerts through our text messages and emails to let us know, "Hey, there's something that's not quite right." And we can take action immediately rather than just relying on the operator who's doing a lot of other things too to bring things back in control. So it's really brought a lot of process visibility, not only from trending but from the alerting side too.

53:18
Travis: All right, well thank you both Sam and Henry, that was a really great discussion here today. Now, that you've heard about Ignition, we invite you to try it for yourself. You can download the full version of Ignition for free at our website, inductiveautomation.com. Installs in about three minutes. You can use it in trial mode as long as you want.

53:34-58:48
(The speakers answer listeners’ questions for the remainder of the webinar; please go to the webinar recording to hear the Q&A.)