The Making Of World-Class Control Systems

Speakers

Travis Cox

Chief Technology Evangelist

Inductive Automation

Travis Broussard

Lead SCADA Engineer

Edge Controls

Cédric Groc

Director of Activities

2Gi Technologie

Dominique Wille

CEO

Plantformance AG

Kevin Komara

Director, Data Acquisition Services

Customized Energy Solutions

What takes a controls project from run-of-the-mill to showcase-worthy? Join us on October 31 for an in-depth look at how professionals are using Ignition to solve industrial controls challenges in forward-thinking, out-of-the-box ways. We’ll explore some of the year’s most innovative projects from a variety of verticals, each of which provides a stellar example of the impressive things you can accomplish with Ignition.

This webinar will be a great opportunity to delve into some amazing projects and get inspiration for your own work. By learning from these examples, you can hone your innovation skills to attain greater success in your controls solutions, improve your business results, and truly make your mark on the industrial automation industry.

• Get a deep dive into showcase-worthy controls projects
• Ask engineers about their creation process
• See prime examples of innovative SCADA
• Gain inspiration for your industrial automation solutions

If you can’t make it to this free webinar, be sure to register anyway so that you automatically get the recording link via email!

Transcript:

00:00
Travis Cox: Hello everybody and welcome to our webinar, "The Making Of World-Class Control Systems." We're so glad you could join us, especially today. Happy Halloween to all of you out there. I know I got in the spirit here today. I hope you all are as well. And I'm looking forward to the day and to this webinar. We're gonna explore some of the year's most impressive, forward-thinking solutions in industrial automation here today. So my name is Travis Cox, I'm the Chief Technology Evangelist here at Inductive Automation. And I, over the years, have been working with a lot of customers, been fortunate to see the amazing projects that you are all building and pushing the boundaries of what is possible, and helping to really bring a lot of innovation to the OT world. So I'm excited to to be here with our presenters today, our professionals who work on some really innovative Ignition projects. Welcome everybody. So first we've got Travis Broussard, the lead SCADA engineer at Edge Controls. Hey Travis, can you tell us a little more about yourself and Edge Controls, please?

01:01
Travis Broussard: Sure. Good morning, everyone. I'm other Travis, Travis Broussard. I've worked as a SCADA assistant architect for about 15 years now, primarily in the oil and gas sector. I hold a master's degree in math from the University of Louisiana at Lafayette. Professionally, I also have project management professional PMP certifications alongside Ignition Gold and Cirrus Link IIoT certifications. I'm the lead SCADA engineer at Edge Controls, an engineering and technical service company based in Midland, Texas, focused on providing turnkey solutions for automation and SCADA. I'm really excited to be here. Thanks for having me.

01:40
Travis Cox: Thank you so much, Travis, and thanks for being here today. So also joining us today is Cédric Groc, the director of activities at 2Gi Technologie. Cédric, can you tell us a little more about yourself?

01:52
Cédric Groc: Hello, Travis. Good morning to all, or maybe I should say good evening, because 2Gi is a premier integrator that operates in Europe and mainly in France and in Spain. Thanks to our level of expertise, we work on highly complex projects like the one presented today. We started out 20 years ago, specializing in SCADA and MES, and have delivered projects in all kinds of sectors. So, yes, I'm Cédric Groc, and as Director of Activities, I have three main roles. So I steer the company's growth, ensuring that the level of quality remains at a high level. I also manage the customer relationships, and I act as project director on large scale projects. I'm happy to be here and I'm very stressed about being here in English language.

02:46
Travis Cox: Well, thank you so much, Cédric, for being here, especially it is the evening over there. We really appreciate you sharing your projects here with us today. And also over there, we've got Dominique Wille with us, the CEO of Plantformance AG, which is one of Inductive Automation's Alliance Partners. Hey, Dominique, please tell us about yourself and 2Gi or Plantformance, excuse me.

03:09
Dominique Wille: Hello, Travis. I'm the CEO of Plantformance, a long history in the heavy industry. And Plantformance is very simply a Swiss company that excels in managing and maintaining Ignition for large-scale enterprise across the globe. So we are currently maintaining around 100 factories on four continents in more than 30 countries. So that's cool. And the unique aspect of Plantformance lies in its comprehensive solution, delivering onsite hardware, software, and service seamlessly. And to make it simple, this integrated approach simplifies Ignition implementation across multiple locations worldwide, enabling efficient and effective operations. That's all. Good morning, everybody.

03:54
Travis Cox: Thank you, Dominique, and thanks for being here again and your evening. We really appreciate you being here. And last but not least, we've got Kevin Komara joining us today. He is the director of data acquisition services at Customize Energy Solutions. Kevin, can you share a little bit about yourself, please?

04:10
Kevin Komara: Yes, and I'm really not a Lego robot. Anyway, so I've been with Customize Energy Solutions for about 25 years. I'm sorry, for 15 years, the company has been around for 25 and Customize facilitates generating facilities and other market participants participating in energy markets around the world. When I started the company 15 years ago, we put in our first SCADA data acquisition system and onboarded our first two 100 megawatt customers. And I'll talk about those kind of customers during my presentation. But now we're at 30 gigawatts. So we have lots of generators worldwide. And I'm glad to be on this webinar.

04:53
Travis Cox: Well, thank you, Kevin, and thanks all of you here again for making the time today to share your amazing project. So now that we've got introductions there, let's take a look at the agenda here for today. So we're gonna start off by taking a look at a project Edge Controls did that enables Stonehill Environmental Partners to rapidly deploy future facilities. Then we're gonna explore the solution that 2Gi Technologie and Plantformance created for Saint-Gobain Glass. This solution lets Saint-Gobain leverage a hybrid architecture to deploy identical applications at multiple sites.

05:27
Travis Cox: And last but certainly not least, we've got a fantastic project from Customize Energy Solutions, where they built an asset information management system and use Ignition in a new industry segment, the grid-scale energy management industry. We'll finish up, as always, with the Q&A to answer your questions. If you think of a question during the webinar, please go ahead and type it into the questions area of the GoTo Webinar control panel, and we'll answer as many of them as we can. If we can't get to your question today, though, please reach out to one of our knowledgeable account representatives, and we'll get it answered for you. We'll have their contact information available for you towards the end of the webinar. Also, this webinar and its slides will be made available within the next few days. If you want to go over any of these tips again or share them with somebody who couldn't make it here today, of course, including the slides and the recording of the presentation.

06:21
Travis Cox: So to introduce Inductive Automation briefly for those of you who may be new, here are some fun facts about us: we make software for problem solvers, and we're focused on our software platform, Ignition. We've been in business for 21 years, and we have a large customer footprint, with 61% of Fortune 100 and 44% of Fortune 500 companies use Ignition, and you can see a lot of the case studies and references on our website, inductiveautomation.com. So we've got a very diversified customer base across all verticals, and there's installations in over 100 countries, and we have a network of over 4000 integrators worldwide that support and deploy Ignition solutions. We are an independent company with no outside investors, and our focus is on providing the best platform for our customers. Here is a little information about Ignition, Ignition is a universal industrial application platform for HMI, SCADA, MES, and IIoT. It acts as a central hub for everything on the plant floor and beyond; you can use it to create any kind of industrial application you can dream of, is web-based, web-managed, web-deployed, it has an unlimited licensing model.

07:33
Travis Cox: And it's fully cross-platform. It offers industrial strength, security, and stability, and of course is trusted by thousands of companies worldwide every single day. And around the world, there are millions of control systems that companies use to keep their automated processes running smoothly. Today, we want to look at some of the control systems that are so exceptional that we would consider them world-class. What makes a control system stand out as world-class? Well, first of all, it has a significant positive impact on a company or an industry. For example, it could increase efficiency, change the way a plant operates, or make life better for workers, customers, or communities.

08:14
Travis Cox: Uniqueness is also important. It can be very customized, or it can introduce a control system or a new technology to a situation where it hasn't been applied before. It should have a well-designed UI/UX, and that means that it was made with the user in mind, and it applies the best design principles and successfully merges form with function. We also look for sophisticated architectures and features. The architecture should be strong, efficient, and complex in a good way, and its features should be customized to the exact needs of the system and those who use it. And finally, the best control systems are the product of a high-level effort in planning, in execution, and in the number of people who worked on it. In today's webinar, we're gonna showcase three excellent projects; each one of them is an example of a world-class control system, and each one leverages the power of the Ignition platform. So let's take a look at our first project here, which is created by Edge Controls, and for that, Travis, I'm gonna hand that over to you. Take it away.

09:18
Travis Broussard: Thanks Travis. So, we completed this project in collaboration with Stonehill Environmental Partners, which is an energy infrastructure company focused on the entire life cycle of fixed water pipeline, recycling, and disposal assets. As a quick overview of our project, we standardized field equipment programming with the SCADA configuration, creating a unified data model, enabling the rapid onboarding of new facilities with a fully mobile-responsive, modern card-based UI, all while overall lowering costs. So let's take a deeper dive into the problem that we needed to solve. The existing assets were a mixed bag of brownfield solutions. There were multiple hardware brands, all initially programmed by different groups. These assets varied wildly in their configuration standards, making routine operations much more difficult than they really needed to be. Stonehill is also growing rapidly and has evolving business objectives. Field-wide changes were difficult to achieve, and so it became pretty obvious that more agility in field automation and SCADA was necessary to achieve their organizational vision. So how did we go about tackling this particular issues?

10:39
Travis Broussard: By first understanding Stonehill's organizational vision and the pain points they had with their existing solution, we were able to develop a new unified data model. We had two main goals during development of this particular model. The new model is flexible, and it's able to grow and evolve alongside Stonehill's changing business needs. It's also scalable, maintaining organization and being easily understood regardless of how large the system may become. This work led to a comprehensive framework that enabled us to simultaneously align the brownfield solutions I mentioned previously with the new data model and standardize our field equipment configuration, automation programming, and Ignition UDTs. All of this sounds great in a presentation, but how did we actually achieve it in the real world? We first needed to find the right hardware that wouldn't require us to compromise on our own vision for the system. We found that hardware solution with Opto 22. Their groov EPIC and RIO devices are extremely durable, surviving brutal Texas summer heat. They feature a powerful IDE in the CODESYS platform with the generous licensing and training resources. They also come with a great price, so it's pretty much a no-brainer. We've been extremely pleased with Opto 22's offering and support.

12:07
Travis Broussard: And before I get this question, no, I am not affiliated with them in any way; I'm just a fan of their products. We use a cloud-based VM infrastructure for scalability and cost savings. Our server architecture follows a pretty standard hub and spoke approach that you may be from the US from previous Ignition webinars. We've fully embraced the MQTT communications paradigm and rely on Cirrus Links Chariot MQTT broker for our communications heavy lifting. So let's take a look at what our architecture looks like. So starting from the left, we have our field devices communicating via MQTT Sparkplug B to the Chariot broker. We also have several instances of Node-red running in Docker containers used for data simulation purposes. This allowed us early on to scale our design to over 10,000 distinct physical sites with full data load during our performance testing.

13:17
Travis Broussard: We also have a redundant pair of back-end gateways that are solely responsible for tag IO operations. And from there, our history is stored in a highly available redundant PostgreSQL database cluster with primary standby and dedicated development nodes. We utilize a central gateway for other tasks that may require heavy processing loads, such as API functionality, data ETL scripts, and so on. This central server also houses alarm notifications via SMS, email, or good old-fashioned phone calls. We have a redundant pair of front-end gateways to handle Perspective client sessions and reporting duties.

14:03
Travis Broussard: From there, data is served out to clients. We utilize Okta as an external identity provider for some extra security measures, which I'll get to here in a second. So earlier, I mentioned a unified data model, and so let's dive a little deeper into what that means. While some think of a data model as nothing more than a tag naming scheme, for us, it's comprised of much more. In our system, each piece of equipment being used in the field has an associated data package.

14:37
Travis Broussard: This package, for the most part, consists of a few common elements. Its UDT, which obviously contains its tag configuration, including those naming conventions. It includes a particular view, which acts as a quick look or summarization of its data, which we call its card; another view that shows all tags associated with that piece of equipment called its detailed view; and its associated field configuration standards and/or automation code templates, depending on what we're talking about. These packages were designed from the ground up to be highly parameterized and reusable in a wide variety of situations. Adding copies of existing packages is seamless and quick, allowing us to rapidly onboard entire new facilities or add equipment to existing sites.

15:32
Travis Broussard: We can even build new types of facilities by simply mixing and matching existing data packages. After all, a tank is a tank regardless of what's inside. Perspective allowed us to deliver all of this functionality in a truly modern way that our users already instinctively understand how to use. Early in our design process, we used style guidelines published by Apple for iOS and Google for Android to design what I believe is a functional user face that looks great. I'm really proud of what we've done here.

16:10
Travis Broussard: Our system is fully mobile responsive and has full feature parity between our mobile and desktop versions. You can do anything in our mobile version that you can do in our desktop version. As you saw in the architecture diagram, we use an external identity provider, Okta, for extra security measures. One particular cool feature of Okta that I really like is their adaptive multi-factor authentication. They could intelligently allow or block certain connections based on suspicious activity.

16:43
Travis Broussard: So as a quick example, if one of your users suddenly logged in from Russia an hour after logging out from Texas, the system will simply deny that access, block the Russian IP, and notify our admin staff without any sort of intervention on our part. So let's take a look at what this guy looks like. Here is an example of a site detail page. You'll notice a site list on the left, which features filtering options. On the right, you can see that we have two charge pumps, two injection skids, and a few tanks along the top, each having their own card to display its data.

17:25
Travis Broussard: We use color pretty extensively to convey information, borrowing only a few ideas from the high-performance HMI crowd. Clicking the header of any of these cards or the right chevron will open that equipment's details view, which you can see here on the left. The options with the pencil on the right-hand side are editable by the user. Certain types of equipment may have control components here as well, such as start or stop buttons, LEDs to show statuses, set point input fields, things like that. Any tag in our system is colored with the typical blue text of a link in your browser to let you know that it's clickable.

18:14
Travis Broussard: Once you do, you'll see the details of that particular tag as shown on the right. We simply call this tag details. As you can see, there are several options for interacting with the tag, including a quick trend, current alarm information and history, and extended tag configuration metadata. You can also easily configure new Ignition alarms right here from inside of Perspective. These are screenshots of the same site details page on a mobile device, in particular, my iPhone 14.

18:49
Travis Broussard: As you can see, things are a bit less graphical. This helps with loading performance when data coverage may be spotty and also makes things easier to see at a glance. Our mobile version has all the features I just mentioned in the desktop version, such as the ability to show detail docs, tagging information, edit tags, perform control operations, and anything our security system may allow you to do. It can be difficult to quantify all of the results of such an ambitious project such as this, but here are a couple of highlights. We improved communication throughout Stonehill by arming every level of the organization with the real-time data they needed to make data-driven decisions instead of using pure intuition.

19:36
Travis Broussard: The modular design increased operational agility, giving us the ability to execute new initiatives or fix things in a fraction of the time. We're now able to build entirely new facilities by mixing and matching constituent data packages. As an example, I actually timed myself building a brand new saltwater disposal facility. From the time that the tags were published to our broker, the site was being displayed in Perspective, with users able to make control changes in less than 15 minutes. And that is no exaggeration; I literally timed it. Numerous forms of cost savings here from decreased windshield time, less time needed on-site for repairs or work, and less manpower needed to build and commission new sites into SCADA.

20:14
Travis Broussard: Our new associative data model made creating analytics and reports much easier than before. We actually used some of this cost savings to deploy Opto 22 energy monitoring units on our drives and motors. We're actually preemptively collecting the metrics needed for pump health and maintenance predictive analytics. Last, but certainly not least, we've proven that our system can scale alongside Stonehill's needs, further cementing a solid return on investment for their organization. Again, thanks for having me here. Back to you, Travis.

21:15
Travis Cox: Thank you so much, Travis, for sharing that incredible project there that you did for Stonehill Environmental Partners. As another reminder, if you have any questions, feel free for any of these presenters here to put into the GoTo Webinar area, and we'll get to those at the end. We've already started getting a few questions in, so that's great. All right, so let's take a look at our next project from 2Gi Technology and Plantformance. So, for that, I'm gonna hand it over to you, Cédric, and Dominique.

21:46
Cédric Groc: Well, thank you, Travis. So, Saint-Gobain Glass is part of the huge Saint-Gobain group that all of you may probably know and which is present in more than 70 countries. Saint-Gobain Glass specializes in manufacturing and distributing innovative glass products for various sectors such as construction, automotive, and industry. For this project, 2Gi Technologie worked with Plantformance in order to create and deploy Ignition applications in several countries. This project used hybrid architecture consisting of local real-time data as well as shared data in the cloud.

22:25
Cédric Groc: Ultimately, the goal was to accelerate Saint-Gobain Glass digitalization and consolidate its data at a global level. As you can see maybe from the dates on the screen, there was also a time challenge as we had only six months to set up the first scalable application and deploy it on the very first site. Saint-Gobain faced many problems, and I hand it over to Dominique to detail them.

23:01
Dominique Wille: Thank you, Cédric. So, actually, due to the global nature of Saint-Gobain Glass operations, they faced a number of problems. The first because of a variety of technological capabilities and no standardized data processes, there was an inconsistent digitalization across their plants. There were also fragmented data, and data silos were inhibiting efficiency when it came to global data analysis and decision-making. Also, their decision-making processes were slow due to their reliance on manual data, lack of access to real-time data, and the mix of global and local data.

23:32
Dominique Wille: Finally, since the data management and the reporting structure were decentralized, Saint-Gobain Glass wasn't able to get the big picture of their global operations. And I think Cédric will continue showing you this great project that happened.

24:00
Cédric Groc: Yes. Thank you. As you can see at the bottom of the picture on your right-hand side, the data sources are either local from PLCs and DCS, or at the top of the picture, you can see the global data coming from the cloud, such as the one in MicroStrategy, where sits some complex rules and calculations. Also, in orange, you can see data stored and managed in Microsoft Planner, where all tasks are managed across the organization. So, really, the key element to be able to deploy to multiple sites and multiple countries is the global data referential. This is called a GlassRef, glass for Saint-Gobain Glass, glass referential. You can see it as the UNS, Unified Namespace, focused on the description of the world organization.

25:03
Cédric Groc: There, inside it, you store all the plant contexts, such as the internal organization of the plant, the lines, the equipment, etcetera. And when an application starts up inside a factory, it first begins by accessing to the global data referential, and then it loads its own context in which the application will execute. So, the application customizes itself thanks to the configuration stored in the global data referential. So, really a key element there. Another one, another layer, very key, is the middle layer, middleware, sorry, layer in this architecture.

25:45
Cédric Groc: This is a layer you have to go through to get data or send data from and to the global tools. This layer is provided by Plantformance and is key, as it guarantees, in particular, performance and cybersecurity. For the software architecture, you also have to follow some rules, and all the development teams have to follow them in order to fit the deployment process. So, project inheritance first; this is a concept that you find in Ignition. It ensures that common functions are reused by all applications. Another one is the versioning strategy that you have to set up for all components. And it's key because you have to manage different versions in different sites.

26:31
Cédric Groc: For instance, you can have one plant with a new version ahead of the others to test it before this version is rolled out. So, you can have different versions in different sites. So, the versioning strategy includes also the master project and the shared scripts. Finally, in terms of software architecture, you have also to do with various practices applied by all the teams. For instance, to manage shared translation dictionaries across different applications. And last but not least, we have used the Enterprise Administration Module. Again, this is a module provided inside Ignition, and that enables deploying the same project everywhere in the group, or in a particular group of plants.

27:11
Cédric Groc: As you can see on the screen, all the plants have the same architecture, so you deploy your site at global level. I'm sorry, I have a small problem on my computer. I cannot change the slide. I don't know if Travis or someone else can change it for me, please. That I switch to the next slide. Thank you very much. One goal of Saint-Gobain was to automatically generate dashboard. Why? It was because they can use them for what we call in the industry the visual management practices. Or sometimes we call that part of the lean management practices. You will see that on the slides.

28:11
Cédric Groc: As the dashboards automatically connect to local and cloud data, the users now have everything on one single screen. This allows team collaboration and supports to foster a culture of continuous improvement. The connection with Microsoft Planner enables fast decision making and task management again on the same single screen. Lastly, the goal was to be able to create other applications easy to deploy. And that's what we have done with the next application, Energy management system, to manage electricity meters.

28:53
Cédric Groc: And in this application, you have another feature, which is to stream data to central data lake for global KPIs and comparisons. I have promised photos. On this photo, you can see that they have been taken in the real-life context of a factory. As you can see, users now can meet every day and apply the best practices of visual management because the KPIs are always available, not depending on any manual data collection. On this screen, you can have a view of the laminated line. This screen is divided into dashboards on the left-hand side.

29:44
Cédric Groc: These dashboards have been custom built using widgets. And they are defined at different levels. So you can define them at the user level, so individually, or at the team level, or even at the plant level. They can be reused depending if they are shared at the user, team, or plant level. And when an action is decided by the team during the meeting, it is immediately added on the right-hand side panel, which is the one connected to Microsoft Planner, and then the team can follow all the tasks interfaced directly with Microsoft Planner in the cloud. On this slide, what you can see is simply, if I can say simply, the screenshot of the pop-up, enabling you to see the evolution over time.

30:39
Cédric Groc: What is really interesting there is that this pop-up is automatically populated whether the indicators in the widgets are local in real-time, calculated by third-party tools in the cloud, or simply entered manually by the plant teams. So, one single pop-up to all widgets and automatically populated. So, Dominique, I'll give you the hand to summarize the results.

31:10
Dominique Wille: Yes, sorry. Thank you, Cédric, for the great job at 2Gi. Saint-Gobain Glass can now deploy a project in multiple factories anywhere in the world in just one hour. That's the key point. Cybersecurity and IT operations are now globally managed under Plantformance, which has boosted security. And also, since the data is now collected and organized in the same way, they are able to do local and global data analysis. This new system also helps speed up decision-making, which is an important result. As you've shown, it is used for real-time management meetings. And they have also a consolidated HMI that uses different sources and services. Now, what I'm very proud for both Plantformance and 2Gi with this superb Ignition product is that we were able to complete this project in a record time, deploying the Ignition application to over 10 plants in France and abroad in less than one year.

32:14
Cédric Groc: And in terms of scope, the project included over 10,000 tags per factory and more than 10 clients per factory. It has dynamic dashboards, as you can have seen, that are each user-generated and between 500 to 1000 alarms per factory. And because it is done with Perspective, you can access that to different devices, PCs, or tablets that are used in the factory. The architecture was an enterprise architecture with, of course, the Enterprise Administration Module. And what's key there also was the Plantformance factory brains, which are the servers that host Ignition on-prem inside the factories and make that easy to deploy, and the connections to the cloud services through Apache and HiFi middleware. In terms of databases, we both used MySQL, and Prostres and SQL. So I think it's all for the project. If you have any questions, no problem. And I will give you the hand, Travis.

33:30
Travis Cox: Thank you so much, Cédric and Dominique. That's a great project. It's amazing to see the ability to deploy a project to 10 factories in just less than a year. Pretty incredible to see. All right. So moving on to our third project, which comes from Customized Energy Solutions. Kevin, I'm gonna turn it over to you to explain how your team's general approach in your specific project for an energy company. So Kevin, over to you.

34:01
Kevin Komara: Good afternoon, everybody. My name is Kevin Komara. I'm Director of Data Acquisition Services for Customized Energy Solutions, and we're based out of Philadelphia. So we're a different kind of customer. We are a different kind of customer. We facilitate generating facilities in the markets in the United States, Canada, and other parts of the world. We facilitate those entities participating in energy markets. What I mean by facilitate is we have to provide the data. The data from these facilities in this case, it's a generating plant that I'm showing on the screen.

34:41
Kevin Komara: We get telemetry from the site. We bring it into our current systems. And if everything is normal and in the limits, we send that data out to the grid operators. So in the United States, a large part of our country is deregulated, and we have energy markets. So that's what our job is. Our job is to be the middleman, to facilitate the information flow from the site through our systems to the end users. So we don't own the generators. We don't own the information. We just facilitate it. And the big thing and why we have done a lot of work with IA is the data has to be fixed if it's bad.

35:24
Kevin Komara: That's what sets us apart from most of the other kinds of customers. Bad telemetry is not allowed. We have to fix it on its way through our systems. And I'll talk more about that in a few minutes. So now the systems that are typical for our industry segment are very large, very expensive, and they're rigid. You can't change the operating system. You have to forklift your old one and put in the new one. It's sort of the cell phone model where you buy a new cell phone, you transfer your data over, and you throw the old one away. Well, in this case, you throw away a several million-dollar system.

36:00
Kevin Komara: You buy an even more expensive system. You use it for a few years, and then you throw it away and get a new one. It's very, very, very expensive and hard to do, and disruptive. I use this analogy here as basically the shrink-wrap Lego solutions. In this case, all these examples are spaceships, but they do different things. They've got some common building blocks, but in reality, if you buy the Millennium Falcon, you can't build the Death Star. And if you buy the Death Star, you can't build the TIE fighter or whatever. That's what we're faced with in our industry, and basically all of the vendors are jack of all trades, master of none.

36:30
Kevin Komara: They all do a great job overall, but they all have good parts and bad parts. One system's database is better than another one's HMI or whatever. We didn't wanna have to compromise anymore, so we wanted to basically get a system that's master of all trades. And do that with a reliable and maybe cost-effective system, or we're asking for the world. Well, so we broke the problem up into the smallest component, a little Lego block, and we said, well, there's interoperability.

37:15
Kevin Komara: Let's see if we can break the energy management system up into a bunch of functional components and buy the best on the market of every component. And, of course, Ignition came very quickly to the top as the core, the building block of everything else that we're gonna build. So basically, if you're playing with Legos, this is that big piece you put on the bottom that everything else is dependent on and interoperability and standards. So we do pick different pieces of software. Some you may recognize, some you may not. The one most people probably on this call will know is Kepware, and I'll talk about that a little bit further down the road here. So now we're gonna build whatever we want. It's, from a it's, we're gonna build ships and cars and I don't know what that is, some kind of tank. But we can build whatever we want. We have unlimited growth.

38:06
Kevin Komara: So when the market changes, we can adapt very quickly. So building block again, we need to start off with standards, and the building block is, the Lego block is in Ignition. And this picture was how we started, but we quickly evolved, and this slide isn't as clear as I like it to show. Basically, we're using the brokering of data with Ignition as the foundation or the glue for everything, and we're using that to transfer data to and from various systems in in real time. So, that's where we started. We can build anything. So, what did we build, all right? So, we built an information management system. We did not build a SCADA system; it has SCADA in it, and we did not build an energy management system, which is what the vendors were offering. We built a system that was specifically for our industry segment that we operate in at Customized Energy Solutions, and we operate assets. We facilitate them in the markets.

39:09
Kevin Komara: We need to provide information as well as a lot of other services, but the system we needed, we actually kind of carved out our own name for it, called an asset information management system. So, this is a new energy segment IA, and we've created a completely distributed system. We broke the communications between different nodes up into its smallest parts, and the thing that I have to thank Travis for, a call we had probably back in 2020 or whatever, the use of some UDTs actually is the reason... Without these UDTs, we wouldn't be able to use Ignition and be able to intervene with the data as it flows through the system because the base tag structures, even with extensions internal to the tag structures in IA, really still didn't give us enough flexibility to be able to intervene. We built basically some Swiss Army knives with UDTs, and I'll explain a little bit more of that a little bit further on, but the last bullet, we basically now have a system that we can infinitely upgrade and expand, and we never need to replace it. We're never gonna need to do a forklift approach.

40:22
Kevin Komara: We're never gonna just buy a new cell phone and throw the old one away. We are going to be able to open up the cell phone and put a new, more memory, better CPU, bigger battery, replace the screen, stereo speakers, and just keep using that same basic platform forever. No single vendor could do that, so we embarked on a proof of concept. Again, this project has been going on for quite a long time 'cause it's very big. We have 30 gigawatts of customers, about 275 customers globally, so that would make CS somewhere around the seventh largest utility if we were a utility. We are not a utility or an independent system operator. We are a consulting firm. And so I do with five people what utilities do with 50. The goal, again, major goal of the project, integrate things together, various vendor components, best-of-breed, and make the system function as one homogenous system.

41:23
Kevin Komara: So we put together the functions, we scan RTUs, bring data into the core of the system to display it on different graphical formats, and then we disseminate that data as it leaves the system to the different grid operators. So in California, where IA is, our customer there is the California ISO. On the East Coast here in Philadelphia, the grid operator is the PJM Interconnection, where I used to work for 11 years. So these are the grid operators. The assets are all over the place. I actually operate a solar farm on a Pacific island called Guam from our control room in Philadelphia. So, to go a little bit further here, we built a system with the proof of concept system, with basically six servers. We wanted to get our feet wet. These are using some old names.

42:11
Kevin Komara: And we wanted to make sure that we could do all that man-in-the-middle and make sure we could create a system that would meet the energy management and the operator intervention requirements that we needed. So that system started off as a point-to-point, using IA's networking capabilities, but we had a power failure in our building a couple years back, where it became very obvious we needed to integrate in a more distributed model, even more than what we had originally. So everything now goes to and from brokers, and it allows us to have a system in multiple states. So I'll talk about that in the next slide or two, but the bottom line is, we basically scan RTUs with a pair of servers, bring that data in, publish it to a broker, then the next part of the system, the core, subscribes to that data, presents it on displays, and then the data flows back out to the broker, and then the output servers then send it on to the ISOs. And if you notice, Kepware is in here. And in Inductive Automation, we have special UDTs that allow us to have the stateless software Kepware and a high-availability and failover capability, which is not in their software.

43:27
Kevin Komara: We added that with UDTs in Kepware, and I can explain that if anybody wants to talk about that after the call, but we can actually do a failover and have a Kepware on one server, stop scanning RTUs, and a Kepware with the exact same database on the other server, pick up the scanning. And we do that in many, many other places or other software. So I'm running; I think I'm running a little bit behind here. So the solution worked well. We put together the POC, and we decided to go with it in production, and the system grew. We added development servers. We have basically 34 servers on our system. We have servers in Philadelphia and Indianapolis, Indiana. And then we also have production systems in Indianapolis and Philadelphia. We have a development system that's in the Amazon Cloud. So we are completely distributed across multiple states, and all the values on the screens when the dispatchers look at them, no matter where they are, update in real time together. So we designed that production and the development systems and test systems that are now are fully deployed.

44:44
Kevin Komara: And this is the system right here. We have a set of development servers in the cloud, and basically we can push databases to any of the other test or production servers. And then we also have a system called our migration system. And this was a key component for us to not disrupt 30 gigawatts of generators that are already in production right now. We can't just stop doing what we're doing and just wait for months or years to put in a new system. We have to migrate and not forklift. So our system looks like this. It's completely distributed. The broker is distributed. Our influx database will be distributed. And we have these servers. We have this in our test system, our production system, our development systems. They all are mirror images. So I can push tag databases from our development system to any of the other systems. And I just wanted to touch real quick on the migration. Right now, our current Siemens system is scanning. It's still in use. We extract all the data out of that system, put it into our new system. So all the primitive data in the Siemens system gets pulled out and distributed to our new system. And then we actually added a migration component so that basically our new system can scan all the equipment, connect to all the ISOs, but then our Siemens system can still be backfed the data so we can do a coordinated migration versus a forklift. This has minimized the risk of deploying this system by orders of magnitude.

46:22
Kevin Komara: So basically, we weren't even done with our new system, and we sold one to one of our customers, which was a really nice way to put some development money back into the company that we used to work on this. The initial cost of the system is about 25% less than our current vendor, quarter of one of the other ones. And then the total operating costs, because it'll never need to be replaced, is probably somewhere around 20% of all of the competitors that we've looked at or different other vendors. Quickly, a couple of screens. We're using Grafana as our dashboard mechanism and HTML-5 solution. And we were able to generate some really, really beautiful screens.

47:07
Kevin Komara: And I have developers that have actually created the ability when you hover over a point, you can interact with it. Again, that man in the middle. If a value was wrong on the screen, we can't let it stay wrong. We have to remedy it. So that's what happens. They can right mouse on the Grafana screen, bring up a dialog box, then block the value from updating in the field and then change it, and it leaves our system fixed. So I'll just go quickly through a couple of more screens. They're just absolutely beautiful technology. We will be looking at also at using Perspective in some of our solutions. And that's it for me.

47:47
Travis Cox: Well, thank you very much, Kevin. And thanks everybody for showcasing your projects. Real quick, if you haven't tried Ignition yet, you can download a free trial today. It's quick to download, takes about three minutes, and you can use it in trial mode for as long as you'd like at no cost. So you can dive right in and explore the platform that our guests today have used for their incredible projects. You can also check out our Inductive University, which is a free training website with hundreds of educational videos. It's a convenient self-guide way to learn Ignition at your own pace, but you don't have to take my word for it. Since we launched IU 10 years ago, there've have been over 10 million video views and almost eight million challenges taken.

48:27
Travis Cox: Try it out for yourself at inductiveuniversity.com. For those of you outside North America, we want you to know that we have a network of international Ignition distributors who provide business development opportunities, sales, and technical support in your language and time zone. If you wanna learn more about the distributor in your region, please visit the website listed here on the screen, or you can contact our international distribution manager, Igor. If you'd like to speak to one of our account representatives at our headquarters in California, please call the number you see on the screen here. And to reach our office in Australia, the number is at the bottom. So let's get to our Q&A now. And as a reminder, you can type questions into the questions area of the GoTo Webinar control panel, and we'll get to as many of those as we can.

49:09
Travis Cox: There were quite a few questions that have come in. So let's dive right in. I'll start with the first question here for you, Travis. And there was a question here from Haytham that says, "What's the reason for the many Ignition servers and separate tasks?" So when you show your architecture, you have the back-end, the front-end, and that separate server for, I think, alarm notifciation. Maybe you can explain the justification behind that.

49:34
Travis Broussard: Sure, if I'm understanding the question correctly. So I'll tackle that main server first. So we have scripts that sometimes take a while to run that we simply don't want bogging down our back-end servers. The servers that we have today, the server pairs that were on the architecture diagram today were simple disaster recovery redundant pairs, right? We bring one down, patch it, make sure everything's good, bring it back up, so on and so forth. The design does incorporate load balancing as well. So we are anticipating in the future, as Stonehill grows, if we need more back-end servers or Perspective client servers, we can just add those behind a load balancer as well. So right now it's just for disaster recovery, but we are anticipating needing load balancing in the future.

50:25
Travis Broussard: As for why we have the main gateway, and that's really just to offload the processing from the back-end tag IO servers.

50:36
Travis Cox: Perfect. One more question for you. And this is a question around auditing. "Did you or do you experience issues by putting too much customization or parameterization of UI devices? And how do you keep track of changes made, example like alarm changes that you can add an alarm on that tag dock window there?"

50:57
Travis Broussard: That is a great question, whoever asked that. And the answer is absolutely. We ran into issues early on in the process, but surprisingly, those were human issues, not really technical issues. And that centered around documentation and making sure that our team is on the same page with what we were doing around standards. So right now the level of customization and parameterization is really very similar to what a field tech would have to configure anyway. And so they're configuring it in the end devices. That information translates over to our tags, and we have logic behind the scenes to basically move that data where it needs to go. So we've actually found that it hasn't really increased the options that field techs need to configure. So the audit trail portion, we very early on saw the need for a sanitized, more end-user-centric audit trail audit table. And so we actually built a custom version of the audit table that we have. We use the built-in one for our SCADA admin staff to really get more verbose information as to what's going on. But we do have a sanitized, simplified audit trail that users of certain security levels can access as well.

52:22
Travis Cox: Okay. Thank you. So next question here is gonna be for Cédric and Dominique. So he said, "Did I understand correct, is the action management widget that you showed connected to Microsoft Planner, is it connected with Microsoft Planner? If so, how is that achieved?"

52:40
Cédric Groc: Okay. I think this one is for me. So, yes, the answer, the short answer, yes. So we are connected directly to Microsoft Planner, and the question is, how is that achieved? So you have endpoints on Microsoft Planner that you can access from outside. So they expose data through web services, and we have connected to this web services directly from the local Ignition. And we go through, if you remember, there's this middleware layer proposed by Plantformance that encapsulates the course for many reasons. The audit trail, the bandwidth, the security, et cetera. So, this is how it is implemented. So web services encapsulated in a layer that do all the logging and security stuff.

53:42
Travis Cox: Perfect. There's one more question for you, and that is, "I hear a lot about global access in your presentation. Any considerations for experience in regional/global compliance and how to implement it?"

53:56
Dominique Wille: Yeah, probably, I'm going to answer this one. I think it's the question is more about security, if I get it. And I think what we see from this global projects in general, you always find, the same, similar concept. The first one is onion security. So you are putting multiple layers to protect yourself. You have also different kind of layers. You have the networking layer, that is, where we're using firewalls and this kind of thing. And you have also the user or location definition. And on this one, I would say a good recommendation that we would suggest to everybody working in large groups is to differentiate users like operators. These guys are on the industrial network usually, and they need really to access anytime, the factory staff, because factory staff needs to have access to things locally and probably not to open to the entire group by default. So, considering the people that are in the group, but not from the factory, let's say a group of user that may access, but only on authorization-based.

55:12
Travis Cox: Okay. Perfect. Thank you guys for that one. This question's gonna be for you, Kevin, and that were... "How were the UDTs that you used to annotate or change data for that project?"

55:25
Kevin Komara: I'm sorry, what was the question?

55:27
Travis Cox: It was, "How were the UDTs used to annotate or change the data that you were mentioned how clean the data and all of that. How is that actually accomplished?"

55:36
Kevin Komara: So our UDT, and we call them Swiss Army knives, they kind of sit in the middle. They can get data. They connect the data coming from a broker. All these UDTs exist on our, what our, we call our core, the center of our system, the hub, instance of Ignition. And then the UDT usually has multiple tags. So we have parts of that UDT are reference tags that can grab data from the broker that the Kepware instance of the system is pushing up to the broker. So Kepware and Ignition push the data up, that UDT grabs it and brings it into the UDT's memory space. And if the, there's another tag in that structure called the block tag. And then there's the what you, the view the user, measurement on the screen tag. So if the block tag is zero in that sequence, then the value that comes into that reference tag is written by a tag event script into the "What's the value that will be displayed on the screen?"

56:35
Kevin Komara: But if the block tag is a one, then the value doesn't update; it doesn't get pushed to that other value. So now you're free to put whatever you want into that particular tag. And that tag then is part of the UDT, is published to the broker, and that's what the, is used to leave the system. So, basically, the sequence and the coordination of whether the block tag is on or off will allow the value to update or not on the screen and flow through. Did I answer that one well enough?

57:06
Travis Cox: Yeah, yeah. No, that's perfect. I think so, on that, so, one last...

57:10
Kevin Komara: I can explain if someone wants to see it.

57:12
Travis Cox: Yeah, for sure. Yeah. And please, if they, you could, reach out, right? To get more information about these. One last question here I'm gonna throw out to the entire group here, which was, for these projects where... The question is, did anybody use elements from the Ignition Exchange or resources from the Ignition Exchange to help with your projects?

57:37
Travis Broussard: I'll chime in first if no one else minds. Absolutely. I think one of the best Exchange resources I've ever seen is an alarm analysis dashboard. And I apologize if the author is on the webinar. I don't remember who made it, but that is such a great resource that I think I put it in every single project from now on.

57:57
Travis Cox: Beautiful. Any other from anybody here? Okay. No worries. Perfect. Well, again, I wanna thank everybody for participating here today. We'll be back on November 21st with our new webinar, but until then, stay connected with us on social media and subscribe to our weekly newsfeed email. You can also stay up to date through our blogs, our articles, case studies, and more. There's tons of helpful content for you to explore on the website, so be sure to check it out. So thanks everyone for joining us here today. Have a wonderful Halloween, and we'll see you next time. Thank you, and bye-bye.