Rising to the Challenge - Adventures in System Conversion

Speakers

TJ Holt

Ignition Team Lead

Flexware Innovation

Jen Conner

Senior Systems Engineer

Flexware Innovation

The folks at Flexware are no strangers to a challenge. When the opportunity to convert a large system over to Ignition arose, they took it head on. Join them in this session where they'll talk about the project and share their lessons learned, talk about custom tools, and describe their thought process.

Transcript:

00:02
Chaz Cooper: Hello, I'm Chaz Cooper from Inductive Automation Support, and I'm a Software Support Engineer. Welcome to today's session, "Rising to the Challenge, Adventures in System Conversions." I'll be your moderator for today. To start things off, I'd like to welcome TJ and Jen. TJ is a Team Lead at Flexware Innovation. He leads a team of seven engineers who focus on Ignition's Perspective SCADA application. He developed a Flexware SparkSCADA, agnostic framework of process objects, face plates, UDTs, navigations, and ad hoc trends that won Flexware a Firebrand Award in 2022. With over nine years of industrial automation experience within steel, water, wastewater, and more, he's always looking for a new challenge. In fact, TJ made a playable instance of Doom within Ignition's Perspective. And then next we have Jen. Jen is a Senior System Engineer at Flexware Innovation, where she has focused water and wastewater SCADA conversions to Ignition. She has 10-year history in SCADA systems and automation controls with a vast background in water and wastewater, steel, chemical, automotive, animal health, and food and beverage industries. She enjoys working with Ignition so much she developed her own Perspective mobile app in [Ignition] Maker to help take care of her indoor plant collections. Please welcome TJ and Jen.

01:28
TJ Holt: Thank you. And welcome. So this is "Rising to the Challenge." My name is TJ Holt. I lead the development of SparkSCADA. Fun fact, my first name is TJ, and as Chaz said, I have background about 10 years of controls automation. Yeah.

01:49
Jen Conner: And I'm Jen Conner. And I've been with Flexware for about four and a half years. I actually more recently moved to our Ignition team. I've been with our Ignition team for just over a year. But same as TJ. I have around 10 years of experience in all different kinds of industries. And a fun fact about me, my house is essentially a zoo. I have four cats, a dog, four snails, and a slug. And that house plant collection that Chaz mentioned with my Maker Edition app, I just hit over 100 house plants. So thank you so much for being here and spending your afternoon with us. I know we're right before the Build-a-Thon. That's the main event. So thank you so much in joining here with us today.

02:27
TJ Holt: Thanks, Jen. So before I go over the agenda, I wanna set the precedence here of what this is. So this isn't a very technical speech. This isn't a... We made this magic tool to convert all your systems into Perspective. This is a project workflow best practices, lessons learned through our experience as well as some utilities and some tools that actually built for our platform. And then I'll release to Exchange we'll kinda get to those a little bit later. But anything that you see or that we talk about and you're like, I wonder how they're doing that, or I wanna get more information. Come find us, we'll talk, we'll talk technical. But this presentation is not deep dive in technical. This is just basic workflow and case studies. So, getting to the agenda, we're gonna go over the projects.

03:14
TJ Holt: Jen's gonna talk about our eight-step process flow and lifecycle. In between, there will be some references to the tools and utilities that I've created. And then we'll have a summary, and then we'll open up the floor to Q&A. Jen.

03:29
Jen Conner: Two.

03:31
TJ Holt: That's me. Yeah... So who we are, right before we get into that. So Flexware, we've been in the Top 10 Integrators the past three years, 2022 Firebrand Award winner for our SCADA platform. We have 35 active projects, over 30 customers. Our team of engineers, there's 35 of us, all Gold Certified. The industries that we're in, we have life sciences, food and bev, water/wastewater is a giant customer of ours. And we have our typical architecture. So the SCADA platforms, you'll have your central hub, and then you have your devices. So like our SCADA platform and way that we do things, we try to make sure that it's tablet, mobile, desktop, all friendly. Utilizing MQTT and partners with Stratus. So there's just a little background of like who we are as a company and give us some I don't know, credibility if you will. Now I'll give it to Jen.

04:32
Jen Conner: Yes, thank you so much. Alright, so I'm gonna take us through what a general process flow for us looks like when we approach a SCADA conversion. We've kind of summarized it into eight steps. And what I'm gonna do here in the next couple about 20 minutes, is I'm gonna take you through what this process looks like for us. Before I do that, along with going through the process, one of the things I wanna share with you is some life lessons that we've learned along the way. We haven't done everything perfectly. It's really easy for me to stand up here and make us look good, but the reality is, and I'm sure, as you all know, humans make mistakes. So I'm gonna be sharing with you some anecdotes and some things that we've learned along the way that you can take those lessons and apply them to your own conversions.

05:12
Jen Conner: Flexware Innovation is a system integrator. So I'm gonna be talking from the integrator standpoint. I realize that some of you may be end users, and I'm hoping that by going through this process with you, you may learn a few things to go through your own SCADA conversion or at least some topics that you can approach an integrator with. So you can talk through maybe what their process looks like. So I'm thinking everybody can get something out of this here today. So let's go ahead and dig in. So when we approach a SCADA conversion, the very first thing we wanna do is define the problem. Now, you might be thinking to yourself, well, it's a SCADA conversion. Isn't that the problem? But the reality is, is that there's always a deeper problem that is underlying into why somebody is wanting to convert their SCADA platform.

05:53
Jen Conner: Is it old technology? Is it a poor user interface or a user experience? There's always something a little bit deeper and it's important to understand exactly what that is so that you can make sure that when you're developing your new solution, you are fixing that problem. So if it is a bad user experience, we wanna make it a better user experience. A lot of times too is, it could be just an obsolete application. There's still a lot of 32-bit SCADA applications floating around on Windows XP, and as we all know, it's gotta go. So sometimes, it's just a little bit more than just doing a plain SCADA conversion, taking this rectangle on this screen and making the same rectangle on the new screen. And it's just important to kind of understand that. Once we define what that problem is, the next step is to understand what the existing system is and what it does.

06:44
Jen Conner: It's important to understand this functionality because, of course, we wanna make sure that our new solution performs the same function. Documenting what exists doesn't necessarily mean you're gonna walk out of this step with a formalized document that you'll probably never look at again. Really the intent is to go through the existing system and understand exactly what it does. I'm talking about what tags are being used, what happens when I click this button, what scripts are going on, what hidden objects you have. As you're going through this process, and if you've ever seen a SCADA system that's been around for a hot minute, you know that at least 15 different people have approached that SCADA system with 15 different ideas of how to program something. And there's some weird stuff in there. We've learned the hard way. Don't ignore the weird stuff.

07:32
Jen Conner: I myself, for example at a project that I did last year, found a hidden button on a screen and it had about 50 lines of code in it. I'm like, oh, it's weird. It doesn't really look like it does anything, whatever. It turns out during commissioning, out of those 50 lines of code, there was 49 that did nothing, but one of them did. And it turned out it reset something in the PLC for the operator. They were looking for the button. My ego was a little hurt, but just remember, don't ignore the weird stuff. That kind of stuff tends to come back and bite you. So, like I said, this phase is not about coming out with a formalized document. It's really about understanding what the existing SCADA system does. Once we have a good handle on that, we can go ahead and design our dream SCADA.

08:15
Jen Conner: We wanna make sure that we approach design with best practices in mind. And what those best practices are can mean a lot of different things to a lot of different people. Some of the top hitters though, are how are we gonna navigate through our solution. Navigation's key. SCADA is generally a hierarchical design. How are we gonna get from the very top of our application to the very bottom in an efficient manner. And one of the tools that TJ's gonna be talking about a little bit later is the method that we use for navigation that we think works really well for us. So I'm very excited to share that with you guys. Also, remember that just because it existed in the original system doesn't mean you need it. We have converted system where there's a tank level and there's a cute little goldfish symbol in the tank.

09:00
Jen Conner: I know this might be shocking, but that goldfish does not add any value, even though it is adorable. So clean up the garbage, just because it exists doesn't mean you need it. Use your best judgment and you as a subject matter expert can define what that looks like what you need, what you don't need. Make sure that you're using consistent feel and language. How I start a motor over here is exactly how I should be starting a motor over here. I don't care what the context is. Make sure that an operator can look and understand from anywhere in your application how to start things, stop things, etc. If you are finding yourself needing more than one of a symbol, standardize it. Templatize it. I tend to draw the line at three. If I have more than three motors, then that tells me I need a template for motors.

09:44
Jen Conner: It's much easier to update one template instead of trying to find 50 of the same thing through an application. So just try to do yourself a favor, templatize and standardize whenever you can. If you're not familiar with what a best practice standard might look like, we would highly recommend you taking a look at ISA 101. So if you're not familiar with the ISA 101 standard, it is an International Society of Automation. That's what ISA stands for. And this is the high-performance SCADA standard that talks about best colors to use, best alarming methods, what your pop-ups should look like, what your navigational structure should look like, all that good stuff. So if you're not sure how to approach your conversion, I would highly recommend taking a look at that standard because it has lots of good stuff in it, although it is a long read.

10:36
Jen Conner: And also make mockups. I don't think it's a secret that a lot of SCADA screens can be pretty complicated. There's a lot of stuff going on, especially depending on the process. So if you have something that you need to take and just kind of slam out on a picture, I don't care if it's a bar napkin, Visio, Microsoft Paint, it doesn't matter. Make a mockup. It's much easier to talk through with a colleague or if you're an integrator, it's easier to talk through something with a client if you have a picture that you can reference. And also at this step, it's really important to understand what matters to operations. At the end of the day, I'm gonna guess that you aren't gonna be sitting there from 8:00 to 5:00 trying to operate the system. So what's important to operations?

11:18
Jen Conner: Last year, I commissioned a system and I thought it was the best system I've ever done. And I get to commissioning and the operator comes up to me. This was like the last couple days of commissioning, and the operator comes up to me and says, "Hey, I gotta fill out this report once an hour and I need like 20 data points," let's say. And none of these data points were together. They were all over the place because I didn't realize that they needed all these data points together. So I made it right, fixed the screen, whatever. But had I known that ahead of time, I would've approached design much differently and made sure that all that information was in one place for the operators. So make sure you talk to operations. Don't develop and design with your blinders on. Understand what's important to that end user.

12:04
Jen Conner: Once we've designed our dream SCADA, now we gotta pitch it. If you're an integrator, this is about pitching it to the client. If you're the end user, maybe you need to justify why you need CapEx money and you need to pitch it internally to management. So this is where you take that design that you just did and spent all that time on and sell it essentially. Use those mockups that we just talked about to help lead conversations. Again, picture says a thousand words. It's much easier to talk about things conceptually when you can reference a picture. Also make sure at this point you identify any misalignment. So if you feel like you're trying to talk through your design with somebody and they're not getting it, maybe you feel like you're talking right past each other, you probably are.

12:47
Jen Conner: So make sure you have a fully transparent conversation. You wanna make sure that everybody's on the same page before you start developing. Also, remember, and this is mostly for integrators here, the customer is not always right. This isn't Macy's, right? It is important that you as the subject matter expert, use your best judgment. Sometimes if a client approaches you and said, "I want a turquoise background with high hot pink text," I think we all know that's not a great idea. So make sure that you are having a transparent conversation using your own knowledge. You are the subject matter expert, and come out of this conversation with confidence in your design.

13:27
Jen Conner: Once we have pitched it and everyone's on board and super excited, we wanna go ahead and start developing. And I'll be honest, the development stage is actually only my second favorite step. We're gonna get to my favorite step. But the developing is really where the rubber meets the road. We've spent all this time doing the design. Now we wanna do development. We found as a best practice, develop against PLCs. If you have one in the office set up as a test bench, use that, download to it, develop against it. We wanna make sure that we are not developing in a bubble and we think we got all of our tag paths right, and it turns out we didn't get a single one right. Naming conventions are key. I don't care if it's a script, a tag name, a view name. This is not 1992 anymore.

14:12
Jen Conner: We are not limited to eight characters. Use fully described names. I wanna make sure that if TJ and I are on the same project and we're codeveloping that the script I just made, he's gonna know what that script does just by looking at the name of it. So use fully described names. It's also really important as an integrator if you're gonna hand this over to the client, that they're gonna be able to take that system and run with it. We don't want secrets. We wanna make sure that everything is clear as day so that they can take it, they can maintain with it, they can develop it. Remember too accessibility matters. So that ISA 101 standard that I mentioned, one piece of that standard talks about accessibility. You don't wanna create a system that is completely based on red and green and then hand it over to an operator that's red-and-green colorblind.

15:02
Jen Conner: So you're gonna have operators that might have colorblind issues, hearing issues, sight issues, things like that. You wanna make sure that anybody and everybody who is gonna be using your SCADA system can successfully use it. So always keep in mind accessibility matters. And one thing that we have had to learn the hard way is keep your eye on changes on site. So the SCADA system that you are converting is currently in production. So make sure that if new projects are happening, maybe breakdowns are happening, things are changing on site, that you keep an open conversation with the clients. Or if you're the end user, you keep an eye on projects so that the solution that you're developing has all of those changes already integrated to them when you get to commissioning. You don't wanna walk on site and realize that you are missing five screens 'cause that is not a good feeling. Been there.

15:52
Jen Conner: And also one thing that we found that really helps us is to have remote access whenever it's possible. So I understand that not all infrastructures or IT groups may allow that, but if you can VPN into the client's site or if you're on site 'cause you're the end user, it helps to push updates as you're going through development. It helps, one, so that when you're going on site for commissioning, you're not just doing a huge dump with a whole project. But it can also be very helpful from a troubleshooting standpoint. If I'm pushing screens on site ahead of time, I can go ahead and set up my device connections and make sure that all my connections work. You don't wanna be finding out when you go out there for commissioning "Oh, whoa, we don't have any IP addresses. I don't know what I'm looking at."

16:38
Jen Conner: Just get it done ahead of time, and it helps to have that remote access whenever you can. Once you are done developing, it's time to test. So I mentioned earlier that it's like a good best practice to develop against simulated PLCs. So you might be thinking to yourself, well, yeah testing of course. I actually just did a commissioning a month ago where we were converting a Ignition Perspective project that another integrator did to our framework. And we found that you had to start the motors by clicking the stop buttons and you had to open the valves by clicking the close buttons. So as much as it might be obvious to test your system, whoever made that system apparently did not to come to work that day. Don't let your ego get in the way.

17:27
Jen Conner: We all might be the best developers that we've ever met, but I promise you, you're not perfect. So just make sure you're fully testing your system. Test, test, retest. I cannot stress it enough. If you did not develop against a running PLC, now would be a really good time to test against PLC code. We just wanna work out any bugs and be really confident in our solution before we even get it on site for commissioning. So this is my favorite step. I love commissionings. I love them because the goal is for them to be as boring as possible. Everyone can remember a time when they had a really bad commissioning because those are the only ones you remember. A good commissioning, you're never gonna think about it again. So I love this. I love organized checklists. Like this is my favorite step of the process.

18:15
Jen Conner: It's a really good idea to get a checklist ahead of time. We have a cool tool that we created that TJ's gonna talk about that allows us to export any OPC tag, an entire SCADA system that we've created, that we use as a checklist. What passed the test? What didn't pass the test? If it didn't pass, what was wrong with it? If it did pass, what was the date of passing? And are there any notes? It's always good to approach on site with a very organized idea of exactly how you're gonna get things tested. Along with that, it's important to create a form of schedule. Again, this might be a duh moment but if you're an integrator, make sure you talk with your client about what you can and can't test. We've gone on site and thought we were gonna be able to test everything in December, and it turned out part of the process had been winterized. So make sure that you're having a good conversation with your client ahead of time to put a formal schedule together. And also don't forget training.

19:11
Jen Conner: You can make the best SCADA system on the planet, it doesn't mean I can walk up to it and use it. And same thing for operators, make sure that your SCADA system, when you hand it over, everybody knows how to use it and you're gonna have to do some really good training to do that. We've actually had issues where maybe we didn't involve operators in the right steps, we didn't do as good a training as we could have and operators were a little bit weary about using our system. We got past it. But it's much easier for operators to be properly trained, have ownership of the system, so that when you get on site and you're ready to be done, you are done. You walk away, everybody's happy. And then of course, our very last step is post-launch support. If there were any requests when you were on site that maybe you didn't get to if there was any kind of documentation that's required for you as a deliverable. Now is when you're gonna take care of all of those items, and that is gonna help you finish up that project on a good note. And that's it. That is our eight-step process. So I mentioned a couple of tools earlier that we were gonna talk about. I'm gonna hand it over to TJ, who's going to take it from here and talk about those tools.

20:21
TJ Holt: Thanks Jen. Recently I had a large project. It was done in ICONICS' 32-bit. There are over 1,200 screens, so that all needed to be converted into Perspective. Now it's commissioned and there are... I want to say it's like 800 screens still, 112,000 tags, and 114 devices connected to it. So it's a very large project and a lot of the sites were duplicates. So we templatized views and we were able to utilize history migration, and we were able to utilize the commissioning documents so that we can make sure that we're checking off every single one of them. Because the scale of this project was just so large. The first thing that we'll talk about... Is our location model. So this is our navigational structure. So this is a semantic hierarchy where you have like your enterprise, your state, your site, and then your process, and eventually down to your components. All of this is done in the backend on SQL. We also bring this back into tags so that if you lose your connection to SQL, you don't just lose all of your navigation structure. We do this in a way so that we can identify what location we're at and then dynamically update tags and bindings because we know where we live in the semantic hierarchy.

21:46
TJ Holt: So if I'm in this distribution system, I can follow that hierarchy through my tag paths and I can follow it through my view paths and I can dynamically create multiple views. Typically we create one view for some P&ID. Let's say that similar PID lives everywhere else. I make it once I just copy-paste it throughout my view structure and then when I launch it into the client or push it to the client, all of my tags, all of my view bindings, everything just starts working. So this view structures is really helpful for rapid deployment, for rapid development, as well as a clear understanding of where things live and how to navigate to them. It allows you to jump from all the way from the enterprise down to your process layer and just move around the whole location model just because of the hierarchy that there is. Next one, Jen's favorite, is documentation. So this is a commissioning export. What this does is it takes a tag tree and then it recursively goes through the entire tag list, and it pulls out the OPC item path.

22:49
TJ Holt: If you have a huge system, like we did with 114 devices and 112,000 tags, it's hard to say that you've touched every single thing. Oh, we turned that motor on. We hit the start button and it turned on. But you forgot that there's a motor overload alarm or you also have some in-service, out-of-service button or what have you. Well, this is going to eliminate some of that lost time or missing these pieces of data where export this all to Excel. You get your columns to know when this was checked off. You can go through systematically for each and every point inside of your Ignition project and validate it and check it off with the customer and make sure that someone's looking at the PLC code so everything is working together. The last is the tag historian migration. So if anyone's following the Exchange resources, Travis Cox developed a Vision app for tag historian migration, which is why this looks really familiar. This is done in Perspective. So I built this in Perspective 'cause not all of my clients have a Vision license.

23:58
TJ Holt: And yes, you can run it in trial mode, but I just really like Perspective more. So what this does is it takes an Excel backup. You have some Excel document that says, this is my tag, this is the timestamp, this is the value. You can shove it into here and then it'll create the partitions. It will backfill all the data for the tag, for the new tag and Ignition. And you can do multiple tags. I think I was able to do 1.2 million rows of data and it took like 30 seconds. So you can do years of data to backfill. So then when you're done migrating your system, you can kind of just go turn off that legacy thing and then your customer still has all of the historical data and they didn't lose anything. So, in summary, the workflow process, we have the document, the existing do more than recreate. We don't just want to take that rectangle and make a rectangle in Perspective. We want to innovate, wanna bring more to the client and to the customer. Ask questions. Don't develop in a bubble. You wanna bring more to the table and how... Get operations involved and get their insight so you can make their lives easier. When I do a system, I want to hand it off to them and I don't want them calling me. This is your system now. I want you to own it and have it.

25:23
TJ Holt: I'm here for support, but let's make sure that when we do these conversions that we're meeting all the expectations that may have been dropped from previous integrators or instantiations of your SCADA platform. And then consistency. I think this is like the biggest part. Everyone tries to be consistent with their naming convention. You're like, "Oh, I got motor one, motor two," or even your scripts, try to be consistent with the naming and what those things are. In your scripting you wanna do, if I'm gonna pull data, you're gonna get something. If I'm gonna push data, I'm gonna move or push. So that when you go into a system, there can be 20 engineers working on a project, but to the customer, it looked like one. If you can get that kinda consistency, it's gonna make a very streamlined process, allow the customer to be able to follow all of the different views, the scripts, everything that's inside of your project, and it's just gonna make everyone's life a lot easier. And then last is testing and testing and testing and testing and testing and testing and testing. So you're gonna want to just put it through its paces.

26:26
TJ Holt: You don't want to go to site and be like, "Oh yeah, that one thing, we forgot to do that one thing." And now your whole schedule is pushed back. There's no reason why you can't create testing in Perspective to go against your main SCADA platform or whatever project you're doing. Or if you have PLC code, you can write PLC testing to simulate logic and processes. This is a huge takeaway for anyone when they come to site, like, "Yeah, we made this pretty screen, I got cute buttons and I got motors. None of it works because we didn't test it against anything." So, in summary, those are the data flow or the process flow that we take some little antidotes that we had. And I appreciate everyone's time. We open the floor now to questions.

27:14
Jen Conner: Thank you.

27:20
Chaz Cooper: Thank you, TJ. Thank you, Jen. Can you guys make sure you guys ask the questions into the mic and then just raise your hand if you need any, if you have a question. It's kind of hard to see, though.

27:31
Audience Member 1: I have a question. So you talked about documenting existing applications. Do you have a tool that you like? Because I found I've got logic in this script. It talks to something over here and becomes such a mess that instead of documentation, I just have another mess I have to dig through.

27:49
Jen Conner: Yeah, that can be tough. And it also kinda adds another element of complication depending on what platform you're coming from, whether that's a FactoryTalk View, an iFIX, etc. We kinda kick it a little old-school. We do like an Excel spreadsheet. We do have kind of a general layout to our spreadsheets where we break down element by element exactly what's happening. And if there's a script, we summarize exactly what's happening in that script or mark something to follow up for later, and then everybody on the team who's on that project utilizes those. It also helps too with tag paths so that you can kind of formulate the proper tag path later in Excel, so you can just copy and paste it into Ignition. It's kind of a standard, that's kind of a loose term I'm using, that we've developed over time that has worked for us. I would not say that we have the best method, but really, once you get into digging into the screens and start getting things together, you might find a standard that works for you. I think that's fair to say. That our standard, like I said, we kick it old school in Excel.

28:56
TJ Holt: If you can get a place where you can organize all your data, I think it's gonna to be the best way to do it. So whatever that is, OneNote, Excel. I can't see any...

29:03
Jen Conner: Yeah, I can't either.

29:04
Audience Member 2: Hi.

29:05
TJ Holt: Hi.

29:08
Audience Member 2: My name is Lamar. So you're working with a customer... Sorry. You have a lot of stakeholders, maintenance, say, operations, even IT but one of these big stakeholders don't buy in the project. It's frustrating. How do you deal with such situations?

29:29.8 TJ Holt: We fire that guy.

29:34
TJ Holt: It's difficult, right? So typically as a systems integrator, they're gonna come to us because they want to do something. So they've either already got an internal buy-in, but there are some times where you get a little pushback from this one group who uses the SCADA platform for their thing. And I don't know, we just do it gently, I guess. And it really helps if you kinda build a relationship and you don't just force it down their throat like, "You're gonna use this thing." You go, "Hey, I know you like your old platform. What part of it do you use? Well, what part of it don't you like that you do use? Why are you so apprehensive of going to Perspective? Is it because it's scary, Python in Java? Well, let's fix that. Let's do a training session. Let me get you comfortable with how Ignition works." It's not VBA, it's not iFIX or very simplistic. If it's that, then train them. If it's because they've done a lot of work or put a lot of effort into this thing, well, you got to validate that and go, "Yeah, I appreciate that, but it's gonna be easier to maintain if we only have one system and be consistent throughout your entire company as opposed to having two separate things." Typically people come around, especially once you start doing little favors, like, "Hey, it'd be really nice if I can have this table of data on this one page, instead of having to go around and find all of it."

30:53
TJ Holt: You go, "Yeah, I can do that for you." And you kind of get a little more buy-in from the customer and then kinda eases that tension of coming in and overtaking their old system, like, "No, you're gonna get Ignition now, 'cause it's cool."

31:05
Jen Conner: And I'll add to that as well. We found that by doing regular cadence meetings during the development cycle, it may help where your point of contact brings in other colleagues into those meetings. Whether it's operators or production, management, whoever. Bring those people into those meetings and get their feedback, everybody likes their ideas to be realized. Like, everybody wants their feedback listened to. So by going and listening to their feedback and implementing that feedback, people are way less hesitant about things because they kinda already bought into the solution. So that might be a good strategy for you as well, is just be open to having people review during the development cycle, take their feedback, implement the feedback if it's good, and by going through things that way, we found that that kinda helps.

32:00
Audience Member 3: Do you guys do upgrade PLC code at the same time that you're doing a SCADA rewrite typically? 'Cause in my experience, often the stuff you inherit in the PLC often forces you into a corner with the SCADA.

32:14
TJ Holt: Yeah, it does. You are right. It really all depends on capital. So if they realize that they have this degradated SCADA platform and the code works. PLC code works and it's fine. So they're like, "We don't really wanna upgrade that yet, but we really want the better visuals. We want be able to have it on our phone or be able to get a text message or what have you." It does kind of paint you into a corner. Now you gotta deal with this stuff, this garbage that someone else did or whatever. It's hard to navigate the way that we do things with templatized views and UDTs is we make it all agnostic. So what that means is you're not tied to an Allen-Bradley processor, you're not tied to whatever it is, and you're not also tied to an AOI, Add-On Instruction, inside the PLC layer. I can pinpoint what tags I want to use within my UDT and I can disable the ones that I don't. So that one motor over there, some guy programmed, and it has these 10 things and this other integrator came in now that one has 20 of them. Well, I still have one motor UDT, and I just fill in the pieces that I do have, and I disable the tags that I don't, and then the UI will interact with that. So now I no longer have that in-service, out-of-service button.

33:33
TJ Holt: I no longer have that one alarm, doesn't exist in the PLC, but now I'm standardizing, I'm templatizing, and it's one source of truth so that you have some consistency throughout your entire project. It's a little more involved and time-consuming. Individual tags as opposed to being parameterized to an AOI or what have you, but at least you can have some consistent on what you can touch and then just leave that off to the side. No one wants to talk about it until they come back for more money, right?

34:03
Audience Member 4: Can you talk a little bit more about how you actually simulate PLC? Do you create custom code that sort of does that for you, or is there some other way that you do that?

34:14
Jen Conner: Yeah. Yeah, so I love simulation. So we either utilize Ignition or we also have utilized Wonderware to essentially act as I/O that interacts with our PLC code that's maybe running in the corner of the office, so that once you hit the start button on the motor, that simulation tells the PLC, "Hey, I'm running." So we can see that then affect the Ignition screen. You can also utilize, like, in the gateway, the Ignition gateway, the simulation. I know on the Exchange, like, if you're dealing with Allen-Bradley, there's, like, the L5K parser that we've utilized to kinda create the simulation in the gateway if it's Allen-Bradley PLC. So that's kinda the way that we've always done it. But really it's just about whether it's Wonderware, Ignition, what have you, whatever you're most comfortable with, kind of essentially turning I/O on and off and making that process run. And we've done simple testing. We've done whole-process testing as well for an entire production line. So depending on the project, we'll determine what level of testing you want. But that's generally how we do it. We wanna make sure that we toggle all the I/O, we start/stop exactly how we expect by talking to the PLC directly.

35:34
Audience Member 5: Hey. So I actually have two questions. The first one, so you've mentioned that sometimes you'll go from one Ignition integrator, and then you'll kinda take over. So if you're dealing with an Ignition project that is using, like, let's say the gateway has a third-party module installed in it, and somewhere inside their system, they're using API calls to this module, I guess. Have you ever ran into a situation where you're trying to diagnose what the current system does in order to get all the functionality, and then there's just calls to this random module. You don't know exactly what it does if you've ran into that, how do you account for trying to port that functionality to your system?

36:21
TJ Holt: It's tricky. So it really depends on if it's, one, well documented, and we can just take it over. There's at some point in time where you get too frustrated, and I will just stop everything, and I will sit down with the customer and go, "What do you want it to do?"

36:36
Jen Conner: Yeah.

36:38
TJ Holt: Ignition is very powerful. So what do you think this thing does? What do you want it to do? And then maybe we can find a better solution for what that is instead of wasting time trying to understand this thing that someone made that no one knows. That's just, "Oh, don't touch that." Don't touch it. Because things go bad. That's all we know. We're like, "Okay, but what breaks if I do?" And so then you can go that route. I think that's really the best method, 'cause the operators, the end customer, they're gonna be the ones who are gonna know what they want. So I would take that approach.

37:16
Audience Member 5: Thank you. My second question is, TJ, is there anywhere that I can find this Doom in Perspective?

37:26
TJ Holt: Not yet.

37:27
Audience Member 5: Okay, well, if that ever becomes public, I'll be interested.

37:39
Audience Member 6: What do you find is the best way to train the people in the factory? Do you prefer paper documents? In-person training? What have you found works the best?

37:51
Jen Conner: Yeah. So operator training is something I'm super passionate about. I love doing training. I love to talk. I'm a Chatty Cathy, and I've kinda developed a way that I do it, and I've shared it with my colleagues. Kind of just, start from the top. Assume nobody knows anything. You never know, depending on the site, if you're dealing with older people who may not know how to use computers very well, down to people who probably might be better at it than you are. So I like to approach it like they don't know anything. I go over the most basic functionality of our system all the way down to exactly where they can find pertinent information to the process. So I generally take that approach. Assume they know nothing. Try to pretend like you're trying to explain it to your kid, essentially. The other thing that we found works really well because one of the larger pushbacks we've had from operators is when you change colors. You blow minds when you change colors. People love their red and green. And when you change that to gray, like, if you reference that ISA 101 standard that I was talking about, they recommend grayscale. So going from essentially something that looks like a clown threw up on it to grayscale is very shocking for a lot of people. So we will embed like a legend into our SCADA system.

39:10
Jen Conner: So that, I'm gonna tell you, "White means it's running or the valve means it's open. But just in case you forget, here's where you can find the legend." And we have found that that helps a lot with operator acceptance as well. And maybe that legend will have what the different alarm priority colors mean or what the button looks like when it's clicked versus when it's not clicked. All kinds of just basics. And it helps when you go through that step two that I was talking about, the documentation step, where you get a good idea of what their SCADA looks like, will help you define what you want to put on that legend. But we have found a lot of positive feedback from the legend. It helps a lot with that operator just kind of getting their feet wet and using our SCADA system, that's been a really powerful tool for us.

39:55
TJ Holt: And I will say as much as we would hope that people would read documents that you put together, unless it's explicitly said, "No, we want a white paper of how this system functions," it's way easier just to put a PowerPoint together or sit down with the operations and walk through the steps. I think that's really the way we go towards it until someone says, "No, we want a binder that no one's gonna read and we're gonna put on a shelf."

40:22
Jen Conner: Yeah. Thank you guys so much. Thank you. Thank you.