10 Steps to Architecting a Sustainable SCADA System

When trying to save money, organizations sometimes focus on short-term fixes that maintain the status quo. While these fixes may resolve the budget now, they can lead to much more expensive fixes later. Rather than getting caught in the endless loop of bolt-on solutions, save yourself stress and money by implementing a sustainable architecture.

Kent Melville and Annie Wise from Inductive Automation, and water/wastewater controls professionals Henry Palecheck and Jason Hamlin, discuss 10 steps for building a sustainable SCADA system that survives and even thrives using only your operational expenditure budget. 

You'll learn about:

• What type of hardware and operating systems to use
• Utilizing smart devices and MQTT
• The advantages of server-centric architecture and web-based deployment
• Rapid development with templates and UDTs
• Powerful alarming and reporting tools
• And more

For our blog post on sustainable SCADA for water districts, click here.

Transcript:

(The moderator, Annie Wise, briefly introduces the topic, Inductive Automation and Ignition software, and then introduces the presenters, Henry Palecheck, Jason Hamlin, and Kent Melville.)

Annie: I recently had the opportunity to attend ACE 2018, the annual Water Works Association trade show just a couple weeks ago in Las Vegas. And while I was there with the team members of Inductive, we've talked to a lot of end users and integrators in your industry, and many of them expressed a lot of concerns and challenges that they're facing. Some of those challenges are listed here, like insufficient resources, planning and phasing in new technology. We'll be discussing a lot of these issues today with our two panelists who are very experienced at building effective solutions.

Annie: So now, let's meet our panelists. We have Henry, is an Information and Process Control Supervisor for a large water district in California; and Jason, a Plant Instrument Technician for the city of Lynchburg, Virginia's Department of Water Resources. Henry and Jason, thank you so much for being here today. Let's start with Henry, followed by Jason; could you please tell us a little bit more about your organizations and what your role is there?

Henry: So I would be the person that oversees the SCADA system for our water treatment plant and distribution system. So we have a 106 MGD conventional water treatment plant with ozone as its primary disinfectant, and a 50-square-mile distribution system that serves water to five communities and three other water districts.

Annie: Thank you, Henry. Jason?

Jason: Yup, I'm the instrumentation tech at the city of Lynchburg. I work in the wastewater side of things. We serve a population of roughly 80,000, but we're BOD-loaded for roughly 250,000 because of our industry. We have a lot of industrial here. So anybody that's in water/wastewater, that's a big number. We're a regional wastewater plant. We bring stuff in from all the counties. We're permitted for 22 MGD, but we will receive up to 70 on rain events because we are a combined sewer overflow plant. And I'm in charge of everything from SCADA systems, computers, panels, all the way down to the raw instrument level out in the field itself.

Annie: Thank you, both; appreciate the background on a snapshot of your systems. Now, you've both been using Ignition for a while now in your organizations. So I'd like to take a few minutes to give the audience an overview of your systems and how they've evolved over the years. Henry, let's start by talking about your water district. About how long has your district been using Ignition, and why did you make the switch?

Henry: So we've been using it, August will be 11 years that we've been using Inductive Automation's software. And the switch was really triggered by two issues. Our legacy system simply couldn't produce reliable data for our monthly report. And then the second issue was due to Microsoft, the end of life of XP was really a problem for us in that the legacy software could not be made to run under Vista, so that was the nail on the coffin, if you will.

Annie: And can you give an overview of how you've used Ignition, and what the results have been?

Henry: So it oversees the treatment plant in the distribution system. The system's a pretty good size. We've got six servers and 23 plants and a high availability architecture. The plant operators and distribution operators have 230 screens that help them control the system, 9,600 PACs and 174 PLCs. And you can see that we're centralized there in Southern California, being the second largest water district in the community.

Annie: Excellent, thank you. Jason, now, let's talk a little bit about your water district and what you've been doing. Your department had two SCADA systems before Ignition. What motivated you to do it?

Jason: So our plant was divided between two systems: We had our dewatering process ran its own independent system, and the main plant had its side. The original prompting to migrate was the old system, and dewatering couldn't support newer processors that we were putting in on an upgrade, and we needed to look at something, and I had... Once I had gotten ahold of Ignition and started playing with it, so this... I really saw that potential, so this is where we wanna go. We backported it. We brought it in on that particular project; always had the idea that we could common the entire facility together, and we did. So we backported it down to the main plant, so we're running the entire facility, and it allowed us to have that single source of data for both parts of the facility, dewatering plus our main plant, everything could be aggregated together. A lot of good features in Ignition. I affectionately called my old systems Fisher-Price My First SCADA and SCADA 0.0, so much to the chagrin of my integrator.

Annie: So you started to elaborate on some of the ways that you're using Ignition. Could you talk about that a little bit more, and what the results have been from installing Ignition?

Jason: Yeah, so the biggest one for us was Ignition became more than just SCADA, and we saw that early on and really wanted to push that way. We're using it for SQL front-end, we're using it for data aggregation, data collection. We have... That unlimited project feature of it is great 'cause we use it for more than just SCADA. We have administrative dashboards, got some mobile clients. We definitely use this as our single source to pull all data in and distribute it out to the different ways people need to see it, as well as traditional SCADA roles of controlling things.

Annie: Wonderful. Well, thank you, both, Henry and Jason, especially for giving us that background on why you made the switch to Ignition. Many of our listeners may be finding themselves meeting some of those challenges that you described just recently. You both have a lot of experience to add to this discussion, and about building a sustainable SCADA system. With that, I'd like to turn it over to Kent to talk about the 10 steps you need to get you there. Over to you, Kent.

Kent: Great, thank you, Annie. And so, to start this discussion, let's talk about what not to do. So SCADA systems tend to lag behind other technology because most people approach it as a closed system with a specific lifespan. Inevitably, requirements change and vendors are hired to apply Band-Aids to make do until the whole system can be replaced. There's really a better way to manage your SCADA in the face of all these technological changes. And so, to do so, utilize open standards and a sustainable architecture. SCADA can adapt and grow with the requirements, as well as improve uptime and reliability. When approached correctly, this methodology will not only improve the technology adoption, but save time and money along the way.

Kent: So now that you know the right approach to take, where do you actually start? And so, the conventional thinking usually goes something like this: Obviously, if we want to move to a new, sustainable SCADA architecture, you need to rip everything out and start over again from the top down. And that's just wrong. Today, we're gonna start small and talk about the little changes that you can make that will go a long way. So let's look at each of these 10 steps and discuss them as we go along. So step one is hardware; look into your PLC, specifically. So it's easy to get to the point where your system includes hardware from many manufacturers that each talk in a variety of protocols. And there's a variety of reasons for this. And some of those are that well, there's a lot of PLC brands out there and they each have pretty long lifespans. It also can be contractors who are promoting their preferred brand of equipment, or maybe it's just a lack of planning on the part of the organization. Whatever the reason may be, maintaining support and connectivity to such a wide variety of devices and protocols is just not sustainable, which is why our first step is to set a standard protocol for your system.

Kent: And that doesn't necessarily mean that you have to immediately replace all of your old equipment. That's gonna be expensive and just not feasible. But for the new equipment you get, you can enforce the standard. It's an easier task to do so if you choose an open protocol such as Modbus, OPC or even MQTT.

Annie: Henry, I'd like you to jump in here for a bit and tell us how you've standardized your equipment with Modbus and OPC.

Henry: So we've had good success using the built-in Modbus driver. At our plant, we have a variety of PLCs from small to large, and we have Modbus power meters on all of the larger loads. And we're able to communicate directly using the Modbus protocol to our larger VFDs so we can read out power for energy efficiency and whatnot. We've also used the Allen-Bradley driver to talk to the SLC 5/04s in our ozone system. And that driver works really well, also.

Annie: Thank you for sharing that, Henry.

Kent: Yeah, and so with that, moving on to step two here; you've got that PLC and it's talking over a standard protocol so data collection should be a cinch. Well, what about remote sites? So if, for example, the new PLC is located in the lift station connected over radio, then pulling is gonna be limited by your bandwidth and latency. Also, if you lose a connection to that device, then you're probably losing data. To solve these problems, we need to have a device installed at the remote site to pull the device locally, and then just report by exception. This means data is only sent to the central location on change. If the network goes down, then the edge device can buffer the data, and then forward it up when the connection is restored. Some edge devices can even provide local HMI for visualization and control. And there are lots of hardwired options for that, but Ignition also... Or Inductive Automation, I should say, has a product called Ignition Edge, which can be installed on a small industrial PC, turn it into an edge device. And it contains multiple drivers, a storage for a data buffer, visualization, even email alarming. So it's a good option for adding more functionality at the edge of your network.

Kent: So now that we're collecting data and sending it to a central location, what should that central SCADA system look like? And that leads us to step three: The server-centric architecture, and focusing on redundancy. So a sustainable SCADA system should have a server-centric architecture. Rather than maintaining many installs on many machines, a server-centric system only requires that the software be installed on a central server. Because all data collection and visualization go through that server, it's a single point of failure, so redundancy is key to maintaining uptime in the event of server failure. A big feature of server-centric architectures is the licensing possibilities they present. Rather than charging per device, tag, or workstation, a true server-centric system would be licensed by the server and be unlimited for everything else. Inductive Automation is proud to say that our licensing includes unlimited tags, devices, and clients.

Annie: Jason, I know your water district has implemented server-centric architecture and redundancy. How has that benefited you?

Jason: Yeah, so a big one for us, one of our previous systems, we had a failure, which resulted in about an 18-hour outage of SCADA. Operators had to run everything manually while we were trying to restore the backups. That's a long story for a different day why that didn't work. But moving to the redundancy, that server-centric architecture with redundancy, at the same time, we also, when we moved to Ignition, we also moved to a virtualized environment, and with separate physical hosts. So we have unparalleled amounts of redundancy. I have the separate physical hosts; Ignition is always running. We very rarely have any type of downtime. Generally, when we do, it's caused by me. [chuckle] But I'm able to do those recoveries. When we've tested, I'm able to do full backup recoveries down to a matter of minutes. So coming from 18 hours to minutes is a huge thing for me.

Annie: It's definitely some compelling evidence, thank you.

Kent: Step four of this process is cross-platform. So traditionally, SCADA systems have been tied to specific versions of Windows. As those versions reach end of life, people are forced to upgrade their SCADA with the OS, and these upgrades can be very costly. By using a SCADA system that is fully cross-platform, not only could you run it on Linux, but any version of Windows. In a sustainable SCADA architecture, the OS and SCADA should both be able to be upgraded independently. Clients should also be able to run on any OS, meaning that if someone has a MacBook, that shouldn't prevent them from opening the dashboard on their laptop.

Annie: Jason, would you share your experience with upgrade issues, and how the cross-platform compatibility made a difference?

Jason: Sure. So the first and obvious one, not to bash Windows, but licensing fees; you gotta own a copy of it. When I was originally playing with the Ignition demo, I downloaded it to my laptop and set up a demo, and that worked fine for demo. But then, when I wanted to start really connecting it to some production PLCs and have a standalone machine, well, Linux is free; there's no licensing fee for that operating system. So it worked very well. In fact, when we moved into full-on production, we actually set up our primary server running with Windows, and a backup server running Linux. Not just because of the licensing fee savings, although that does exist, but it's a security best practice. There's a limited footprint for zero-day attack because we're actually running on two separate platforms. So there's just a general good reason to have this ability. Nobody here has a MacBook, unfortunately, but we do routinely run a mixture of Linux and Windows for a security standpoint and the cost savings.

Annie: Wonderful, that might be also a helpful scenario for some of our audience members listening in.

Kent: Yeah. And part of that server-centric architecture is web-launched clients, and they are so key that we decided to have stuff here all over themselves. So historically, when client screens were developed, they had to be exported, and then installed on each client machine or workstation. And the same process had to be completed each time a change was made. By having a server-centric architecture, clients can be web-launched from the server. That means no install at the client level, and changes can be pushed out automatically. This increases scalability, and allows for rapid rollout of necessary project updates.

Annie: Henry, what has been your experience with web-launched clients?

Henry: Well, the web-launched client is one of my favorite features of Ignition. In the past, when I would have the systems, I'd have to purchase a license file for each client computer. With Ignition, simply don't have to do that, that's why we have 23 clients in our system. Another feature that's nice is that the clients are virtually zero administration systems. There's no SCADA software to install, there's nothing to patch or keep up-to-date. You simply install the Java Runtime, get the computer network access, point it to the server, and you're done. So the web-launched clients have greatly reduced the amount of work taken to maintain my SCADA system, overall.

Annie: Glad you're happy with that feature.

Kent: Yeah, absolutely. So this leads us to step six. Now that we have an unlimited number of devices, with an unlimited number of tags, and an unlimited number of people wanting to see that data on screen, the problem is that we only have a finite number of people to configure those devices and create those screens. And it's not sustainable, with that finite number of people, to define each data point one at a time. So part of a sustainable SCADA architecture should be implementing an object-oriented approach so that data points can be grouped into types that can be defined once and used throughout the projects.

Kent: For example, take a motor. So each motor that you have will probably have similar values, and rather than defining each motor individually, you could create a user-defined type of a motor with all of those shared tags. And with that, you could also define history and alarms all directly on that UDT definition. So you can see here in the image that we've provided, this is a screenshot from Ignition directly, where you can create a data type of motor. And we added amps, HOA, run command, status; each of these tags can be placed in there. And each would have its own OPC item path, really, just a mapping to where those values exist on a PLC. But we want to use Indirection to make it so that we can pass in motor number.

Kent: You see the data-type parameter there, motor number, we want to pass that in to that OPC item path so that when we go to create instances of this data type of motor, all I have to do is pass in the motor number and it will automatically update the addresses for amps, HOA, run command, status, all that, so that they are indirectly pointed to the correct addressing. And with this, you don't have to create those motors one at a time now. You can even create 100 at a time just passing in those motor numbers and you're good to go. So with this, UDTs can rapidly increase your development productivity. And so, once you've got those instances created, you can now build out how they will be displayed on screen. So visualization templates are reusable graphics populated by the values from the UDT. And once the UDT and template are both defined, adding additional motors is simple.

Kent: So you can see here in this picture, this is, once again, the Ignition Designer, that in our tag browser, in the middle left-hand side, you can see in our motors folder, we have the instances of motor one, motor two, motor three, motor four. Those are all populated just by passing in the motor number parameter. And then, you can even just drag those onscreen, and we have some simple templates there that automatically get generated to show the status, show the amperage, and the name, all as part of that template. Now, templates support inheritance so you can get values from parent templates, and templates can be as big or as small as you want. And of course, you can override any of the values from the UDT definition to make them customized because we understand that as much as it'd be nice to say that all of the data points for all of your motors are exactly the same, all the PLC programming matches, we know that's not the reality. And so, you can customize each instance to match how a PLC was programmed there.

Annie: Jason, could you give us a little background on how templates have been useful in your work?

Jason: Oh, my gosh! Templates are one of those interesting features. I'm glad this got a slide. You use Ignition for years and you tend to forget about them. My previous system didn't have them, but I'm glad this was brought up 'cause the power here. Templates, UDTs are like water; you tend to not think or care about it until you don't have it. After using it for years, the quickness of my development time, I can't really compare that. And it's like every system should have this because it just makes everything much faster, much easier. Changes, you make one change in one spot, and it's replicated everywhere else, that template exists. It's been the only way that I've been able to support keeping my system running as a team of one person.

Annie: Thank you, Jason.

Kent: Yeah, and that's impressive; taking care as a one man team is always exciting. [chuckle] But step seven of this leads in to that concept of having to take advantage of what you have. And it's not always that you have all the people right there where you need them at all the time that you'd like them to be there. So in a perfect world, you have someone sitting, staring at your newly-created screens with their newly-created templates 24/7 to make sure the system is running smoothly. But that's just not realistic. This is where remote alarm notification comes into play. Part of a sustainable approach is taking care of what you have. When something goes wrong, as it inevitably will, you will have the ability to respond quickly. So for alarming, a flashing light on a screen is good, an email is better, but a text or phone call is best. The idea is to get as much information as possible to your operators as quickly as possible.

Annie: Jason, can you tell us what your water department has done with remote alarming?

Jason: Sure. Like you say, the flashing light on the screen. So our facility is manned 24 hours a day, 24/7. So that flashing light on the screen is actually... That still works well for us.

Kent: They actually recorded a Tarzan noise, and that Tarzan noise plays every time an alarm goes off, and it certainly gets everybody's attention. And with that, it's also fun. All the operators laugh every time they hear it, and then they go and they respond to the alarm. And so, it keeps a positive mood. It also goes ahead and allows people to respond to that alarm quickly. And I know that they're also taking advantage, over the City of Lynchburg, to have alarms get sent out via text to supervisors, to keep supervisors appraised of what's going on. So since they're always not at the facility, they don't always see those flashing lights on screen, but they need to know if they need to get involved when something serious happens. You wanna add anything to what I described with your system?

Jason: I think you covered most of it. But yeah, the remote alarming side of it, we do have escalation for our supervisors and admin, and we do push that out. We use text and email, and we actually drive the text through a cellular modem so it's a separate channel from our email. So again, that we always have that redundancy. And a big one for us, a new project, we recently have tied into a public notification system. We're using a third-party mass notification to push out alerts for the CSO project. It's part of our consent order to notify the public if there's an overflow. And that was something that we're able to drive and do through Ignition with automation, which was a very cool; fun project for me.

Kent: Yeah, absolutely.

Annie: Thank you for sharing that.

Kent: With this, in addition to remote alarming, another feature of a sustainable SCADA architecture would be automatic reporting. A key part of sustainability is efficiency; not just of digital resources, but of manual labor. And reporting is often the lowest-hanging fruit for reducing the manual effort required by your team. Time and time again, we see people with clipboards walking around, checking the status of equipment, writing their results on a whiteboard, and then someone else has a responsibility to come and take the data off that whiteboard and manually enter it into Excel, and format it and try to create a meaningful daily report. And for a decent-sized facility, that can easily be a full-time job.

Kent: And we've talked to people about this idea of automatic reporting a lot. And sometimes, there's some hesitation. Even though at this point, we have a server-centric architecture that you can connect to an unlimited number of devices, and you're connecting over standard protocol so it's easy to get all this data into the SCADA system, but still, people don't trust it. They don't trust the data. They wanna have a physical person going out, looking at that machine, and then manually writing down the value. And they think that if they do that, it'll be more accurate. And as people have gone and actually implemented automatic reporting, we have found that the opposite is actually true. The process of writing on a clipboard, transferring that with potentially questionable handwriting to a whiteboard, and then manual data entry into Excel is ripe with opportunities for human error.

Kent: And so, we found that as you automate that functionality, and you just let the machines collect that data, let your SCADA system collect that data, that accuracy increases dramatically. And that if manual checks are truly required, then what you can do is, rather than having a clipboard where they're writing it with their handwriting, you have a tablet that's running a client. And with that, it's got some form entry fields. And so, they can type in to those form entry fields as they walk around to each machine. And when they hit submit, it automatically puts that data into the SCADA system, and then creates those reports automatically. And so in that case, you get the best of both worlds. You get that direct human interaction where they're looking at the machines, but then, you eliminate the error of manual data entry. The whole idea is data should only be entered once, and then automatically entered into your system.

Annie: Jason, can you talk about how you implemented reporting in dashboards in your organization?

Jason: Sure. Kent, that basically just described my entire first project that was non-SCADA-related that we used Ignition for. So we had everything you were talking about, with the exception of instead of a whiteboard, it was our SCADA screens. But in the old system, the operators would look, they would write everything down on a clipboard, walk over to another computer, type it in manually. I think it's actually like an 802 dot standard, IP over human carrier, which is what actually we're doing. Computers will transfer data very efficiently to each other if they're programmed to do so, and they tend to do it without mistake. That was a big one for us. So that was, literally, our first non-SCADA project with Ignition, was to start aggregating and automating data collection.

Jason: There's always gonna be those things that the operators have to do that don't exist automatically in a manual grab sample, some field things that don't have sensors, that they go out and get that data before. And we're not, to the point you're talking about yet, we don't have the tablets yet, but it is something that we're working for. But using Ignition as a SQL front end has been a lifesaver for us because our operators, they're used to what they see on the screen for SCADA. They're already used to that interface. So when we're able to build a data front end that uses the same feel and function and features, it's a natural extension for them to use it. So they feel like it's a seamless system, even though it's two totally separate projects doing separate stuff.

Annie: Fantastic. Thank you, Jason. Henry, your organization has taken a different approach to reporting. Could you tell us about it?

Henry: Yeah, so reporting for us is key to meet our permit requirements with the State of California. Our legacy system had issue with logging data reliably. And so, that was actually why we first started using Inductive Automation software, was to get a reliable data logger, a reliable link into a SQL database just like Jason. But now, we use Hach WIMS. It's a package that's used in the water utility field. And so, what happens is Ignition drops the data into the WIMS server, and then the WIMS server takes and merges that data with the operational tests that the operators do, and then the lab data coming from the biologists and chemists, and that generates our monthly report.

Annie: Great. Thank you, Henry.

Kent: Step nine here is a little different than the other steps we've been talking about. So at this point, we have a pretty solid system. We have improved data collection and improved how we use that data by using SCADA best practices like remote alarming and automatic reporting. Well, let's expand beyond SCADA for a minute. And what we really wanna talk about is IIoT. The Industrial Internet of Things has been a huge buzzword lately. And contrary to popular belief, it can play a big role in the water/wastewater space. So IIoT is really all about how you get your data. And since water/wastewater has so many remote sites that are often connected over radio, satellite, or even cellular connections, getting large amounts of data from these sites requires a lightweight protocol. Enter, MQTT.

Kent: So earlier, we added edge devices to our architecture to pull data locally. These devices will then use standard protocols, whether it's Modbus or Ethernet IP, or whatever to pull these devices. But then, if you've got the right type of edge device, you can publish that data up using MQTT, rather than just OPC or some proprietary protocol. And to tell you a little bit about MQTT, MQTT is an ultra-light protocol developed through IBM 20 years ago, and it only has a two-byte overhead, and is very secure. It has a pub/sub protocol that publishes by exception. And by using the spark plug specification, you can have a store and forward, and auto discovery of tags, and a bunch of other great features for the industrial space.

Kent: With that, that data gets published up to an MQTT broker, centrally. And then any line of business application, including SCADA, could subscribe to it. With that, data could also be pushed securely into Microsoft Azure's IoT Hub or up into Amazon AWS through their Amazon Kinesis. And by that, you can take advantage of their analytics tools and functionality. We won't spend too much time on MQTT today, but you hear us talk about it a lot here at Inductive Automation. And we do highly recommend this architecture. And when you have data that is sitting at all these lift stations, we have people that are only getting poll rates of 15 minutes. And then, when the weather is bad, like in Lynchburg, apparently, today, it can be hard to get that data back reliably. And so, with that, if you implement this kind of thing, it can pull locally. And then whenever the network is good, however much bandwidth you have, it can publish that up effectively and efficiently. And so, we won't go any more into MQTT today, but we do have entire webinars dedicated to this topic. And so, feel free to go check out those other webinars, and you can also go on our website and read a lot of material about that. A little shout out here to our strategic partner, Cirrus Link. They have developed our MQTT modules, and their website is another great resource to learn about MQTT and its uses.

Kent: With that, we get to the final step here, step 10, which is really a discussion of how a sustainable SCADA architecture affects your CapEx versus your OpEx, capital expenditure versus operational expenditure. So really, when people are talking about a sustainable SCADA architecture, they say, "Well, all of this is great, but how does it affect my bottomline?" And so, let's talk about that. If you've implemented all the different steps we talked about today, you can now add devices and tags without worrying about getting knocked into next-tier pricing. You can also develop new screens, and easily push them out to all clients. And you can utilize templates and UDTs to build those quickly and scale effectively. And you can also upgrade the software at any time because you aren't tied to a specific OS, and redundancy allows you to do so with no downtime. And all of those new capabilities keep the software humming along for no additional cost. And so, by following our new standard, getting new devices will normally fit into an operational budget.

Kent: Also, all those people that were manually putting in data to Excel or walking around with the clipboards, their time is now free to train them on this new SCADA system, help them develop new screens. Then if they become really proficient, they could even replace some of those calls to a contractor every time you need to make a little change or create a new report. If that's all running smoothly now, and you're really able to do all these things with minimal cost, then what are you gonna use your CapEx for? And there are lots of options for what to do with that CapEx. You're never gonna run out of things you could spend money on. So let's talk about if you didn't have to spend it all on your SCADA system every few years, what could you spend it on? And that could be upgrading your hardware, it could be upgrading your network, fixing leaky pipes, getting water monitoring software, or any other items on your nice-to-have list. And so really, a sustainable SCADA architecture enables you to shift your focus from just keeping your SCADA system working to improving your overall system.

Annie: Henry, you told us before that your current system has allowed you to change the way you allocate your time. Can you share that with the audience a little more?

Henry: Yes, so I've seen my workload shift from 80% of the time maintaining the central hardware and software, and 20% of my time maintaining our radio system and PLCs. And it's now shifted the other way where I'm spending 20% of my time maintaining the central hardware and software. And it's not that Ignition is hard to maintain. So that 20% of my time on the central system is simply applying Windows patches, keeping the virus software up-to-date, and changing out old computer hardware, or maybe adding new clients to the system.

Annie: And Henry, could you also talk about how you're able to pay for your Ignition system?

Henry: So we deployed our system in-house, and we did it as a pay-as-you-go. We didn't use any capital funding. In fact, we haven't used any capital funding since deploying Ignition. We were able to reuse our existing serial radio network and PLC. The Ignition OPC driver was a key in interfacing to that older hardware. And then a few years back, we were able to phase in an Ethernet radio system that the driver was able to connect to without any issues. And now we're phasing out our legacy PLCs and modernizing those. Ignition has freed up resources to help me address the issues of my aging infrastructure, control my SCADA costs and create a sustainable SCADA solution.

Annie: Thank you, Henry. We really do appreciate those comments. Like Kent mentioned, it doesn't have to be a rip-and-replace kind of a transitioning. And that's very helpful for our audience. Jason, could you share your thoughts and experience about using operational instead of capital expenditures?

Jason: So we got our SCADA system under the traditional method. We, certainly, bought it under a capital expenditure project. However, Ignition is powerful enough to do anything, but it's still easy enough that I can understand how to do it, so even I can do stuff with it. We're able to leverage that. So I do a lot of work in-house, and then I use my operating expense money to get integrators to take that level that I can't do. So we're not using an integrator just to make small and minor changes or the stuff that I know how to do. So really, the end result is that with the combination of me and reduced costs from the integrators, we're able to get capital-ex level projects done on our operating budget.

Annie: Definitely, a very significant point. Thank you, Jason.

Kent: So with this, let's recap. Let's look at these 10 steps. So we started maybe at a remote lift station here. You've got a PLC. You want that PLC to be utilizing an open protocol that is easy to maintain, so you don't need your system to be able to have connectivity to 20 different protocols for PLCs. Also, since it's at a remote site, you should take advantage of having some kind of edge device there that can do that polling locally and then can just push out those changes on change and also can do some store and forward, so you don't lose data. Also potentially have an HMI screen there as well that you can maintain control and status and all that stuff even when the network connection is down. From that edge device we could just be publishing up to the central system via OPC or some other proprietary protocol like our gateway network. You can also take advantage of publishing that data up via MQTT to a broker, whether just on-premise or in the cloud, and then sharing that data with other applications or cloud services.

Kent: Once that data does get up to the central server, then we want that central server to have redundancy. We want it to be cross-platform. And we wanna be able to connect up to an unlimited number of devices, have an unlimited number of tags and even be able to launch an unlimited number of clients from there. And with that, we also wanna take advantage of rapid development practices, templates and UATs. And we also wanna take advantage of remote alarming, sending out those alarms via email, phone call, text message and do some automatic reporting so that we can eliminate the need for manually creating these Excel documents every day. With that automatic reporting isn't just something that gets saved onto a local drive, but it's something that could be automatically emailed out at the end of every day or start of a week or whatever you want to define in there or sent to a network, drive, FTP server, whatever you want to define there.

Kent: I already mentioned the IIoT and MQTT through that broker in the middle but publishing up using those reporting tools can be a big deal. In water wastewater, you can anticipate usage based on weather reports, integrating all these things there. And then finally, the CapEx and OpEx. If you're really doing this right, then it should alleviate a lot of costs. And so learning to work mostly on that OpEx budget, but being able to improve your system with that CapEx rather than just maintaining the status quo. That completes our sustainable SCADA architecture. And with that, I'll pass it back to you, Annie.

Annie: Thanks, Kent, for that great presentation. To wrap up, I'd like to get some closing thoughts from our panelists, who are implementing these steps in the real world. Jason and Henry in that order. Do you have any closing thoughts about the 10 steps or any other ways that Ignition has made it easier to build a sustainable SCADA system?

Jason: I think that really sums it up well. That's a great 10 steps. I mean, I hope I've shown it's definitely been solid for me. The only thing I'd really add, Ignition continues to innovate and it continues to put out changes. And I really appreciate that you guys are in tune with new technologies, emerging web technologies and security, which is kind of a really big one for me. And I appreciate that.

Annie: Thank you, Jason. I can speak for Kent and say he's very happy you've gotten rid of some of your clipboard usage.

Jason: Yes, absolutely.

Annie: Henry.

Henry: I was talking with a friend of mine, and he was asking why government agencies can't stay within a budget. And I told him, "They can if they make good choices." And Ignition is a good choice for SCADA in terms of reliability, system lifespan and sustainability.

Annie: Wonderful. All right. Again, gentlemen. We'll start with Jason and then Henry, how do you plan on expanding your SCADA system in the future?

Jason: We're gonna continue doing more system tie-ins, third-party expansions. We've got some projects that are going right now. We've... Even looking at some IIoT solutions that are out there, actually waiting on a hardware vendor to get a product in my hands right now for an upcoming project. That's kind of the growth that we're looking at.

Annie: Wonderful, Henry.

Henry: And so we're gonna continue to expand and evolve our system over time as we bring new processes online, as we establish communications with more and more intelligent devices. And I think for us, sustainability is kind of the key thing. So we're also focusing on knowledge transfer and documentation. So this can be passed on to the next generation of staff as they come on board with the utility.

Annie: So excellent points there. Thank you. So thank you to everyone. I just wanna take a couple of points, make a couple of points before we go to Q&A. First, we'd like to invite you to try Ignition for yourself. It's easy, just download the full version of Ignition anytime at inductiveautomation.com. Again it's free, unlimited time trials and installs in less than three minutes, so why not give it a try? Another great way to learn more about Ignition is to visit inductiveuniversity.com. We've got hundreds of quick Ignition training videos, you can watch anytime. Additionally, our online Ignition user manual provides a wealth of information on Ignition features and functions, and that can be found at docs.inductiveautomation.com. If you are interested in getting a personalized demo of Ignition, please call us at 1-800-266-7798 and speak with one of our helpful account executives. Now it's time for Q&A. Henry, I'd like to start with you. How are water utilities being affected by the silver tsunami?

Henry: I think if you were to look at the years of experience, maybe 10 years ago, of our plan operators combined, we had about 135 years of experience between them. And then a few years back it dropped to 60 as operators retired. And just last week we had an operator with 27 years of experience retire. And so we're now combined operator staff only has 24 years of experience total. So there's a big loss of knowledge due to the operators retiring. The SCADA system now is used in greater measure than ever before to ensure reliable operations and to protect public health. And so the tools of yesterday certainly couldn't meet the needs of the utility now, as we're getting a younger and younger workforce coming in to play.

Annie: Thank you, Henry. It is a significant challenge. And we're grateful to hear how Ignition is helping to meet those challenges. Jason, what advice would you give for the water, wastewater utilities looking at doing a SCADA project?

Jason: Download and play with the demo, look at what's out there, learn your systems, know what you have, right? Henry actually hit on a really good point. We got a newer and younger workforce coming in. So finding tools that are modern that have the flexibility that give you... More and more of the technicians coming out of school now are coming from a computer programming background. It's harder to find automation professionals that don't have a programming background. So things that leverage modern programming features tend to resonate very well with them. Your Millennials and younger can naturally start to work with these tools.

Annie: Definitely some key points. Thank you, Jason. I have another question for you as well, Jason. What was the context that you were describing when you were talking about the templates that you were using?

Jason: Anything that I use more than once. So motors, tanks, sensors, analyzers. If there's something that I have to display on screen that we have more than one of, I will use a template and a UDT for that. It makes my development faster. I can replicate them, deploy them and manage them from that one template rather than having to make those duplicated changes.

Annie: All right, Kent. Can you give us an example of the number of tags used? Oh, excuse me. Let's start with Henry. Henry, can you give us an example of the number of tags you used in your system?

Henry: Yeah, so we have a 9,600 tags, controlling the processes at the treatment plant and in the distribution system.

Annie: Jason, could you do the same as well?

Jason: Only about 2,500 tags. [chuckle] Give or take.

Annie: Alright. And Kent, this is for you. How do I keep up with the webinars you schedule?

Kent: Yeah, that's a great question. So, we do have our newsfeed. So, you can go to our resources tab on our webpage and sign up for our weekly newsletter. That'll keep you appraised of this kind of stuff. Also, it's announced on our social media channels. So, feel free to subscribe to us there.

Annie: Kent, can you do data validation when people are filling in forms on client screens alerting the user to questionable values?

Kent: Yeah. So, that question is from Parker. Parker thanks for that question. So, for data validation, certainly, on any client screen you develop an ignition, you can do Python scripting. And Python scripting allows you to do any kind of validation to make sure that it's an appropriate date, or that the values are within given parameters, or even you may only want them to enter in certain values, so it could be a drop down rather than manual entry. And that way, you're ensuring that you're getting quality data.

Annie: Thank you. And Kent, this question is also for you. Please address the impending Java issue, and what we will be facing.

Kent: Yeah. And so this one's a little off topic, but Michael, thanks for asking it. So, a lot of people are worried about Java because Oracle has changed their support structure and all that. We have released a white paper on that. We encourage you to read that. That's probably the best source to get an answer, but I just know that we will be pre-packaging Java with ignition going forward with, I think in 79-11 will be the first release that has that. And so, you will no longer need to install Ignition manually. It'll come pre-packaged with the server. And same thing with the clients. You will just be able to run an executable and that will include your Java runtime as part of that executable. And that'll run side-by-side with any existing version of Java, but also with that, this new functionality is a product of Java 9 that was released. They added a way that you can build your own Java that only includes the libraries and classes that we actually take advantage of. And so we're excited about that. Oracle changing their methodology for support has pushed us to do new exciting things with Ignition. And so the path forward looks better than ever.

Annie: Thank you, Kent. Alright. So, we're just gonna take a couple of minutes here to wrap things up. I'm gonna ask Jason, Henry, and Kent to take about 30 seconds each for any final comments or remarks that you have today. So, Jason. Please start us off.

Jason: Final comments. Ignition is my engine of innovation. That's what I turn to. It drives all of my projects. I kinda use that as the foundation. Any time I need to do something new in automation, I haven't found anything it can't do yet. I only find the limits of what I don't know how to do, which is when I can leverage my integrators.

Annie: Alright. I think we might find a new marketing branding campaign there. [chuckle] Thank you Jason. Henry, your thoughts?

Henry: So, Ignition really is world class software. It's very powerful. It's open. It's cost effective. There is nothing that the software can't do. And it kind of breaks the cycle of complex and costly upgrades that have to be done in a continuous kind of cycle. And so, Ignition has met these issues head-on. And it's reduced the effort to keep a SCADA solution running.

Annie: Thank you Henry. Kent?

Kent: Yeah. So, just to kind of wrap up here. I wanted to emphasize, once again, obviously, we do not get to everybody's questions today, but we will certainly make sure that you all get your questions answered. We'll have someone reach out to you individually, but also once again, you can always give us a call. And we'd be happy to talk and answer any questions, but we really are excited about the water/wastewater industry. And we think that it's an area that is ripe for improvement. And so, we're excited to continue to help you innovate in that space. And these steps really are gonna enable you to do that. And so, even if you cannot implement all 10 of these steps, even our presenters here today, Jason and Henry, they haven't implemented all 10 of these steps. So, don't be intimidated by the workload in front of you. Just get started. Do something. And you will find that it does benefit your system. And that you will be moving towards a new sustainable SCADA architecture. And thank you for coming today.

Annie: Wonderful. Thank you Kent. Alright. I wanna thank everyone again, Kent, Henry, and Jason, and those of you listening in our audience. Thank you for watching today. If you'd like to stay informed about our upcoming webinars and events, please follow us on Twitter, LinkedIn, or Facebook, and subscribe to our weekly newsfeed on our website. Thanks again. And have a great day.