Design Like a Pro: How to Pick the Right System Architecture

Speakers

Don Pearson

Chief Strategy Officer

Inductive Automation

Kevin McClusky

Co-Director of Sales Engineering

Inductive Automation

Whether your automation project has only a few tags or hundreds of thousands of tags, you need to make sure that it will work properly now and that it has enough room to grow in the future. Having the right architecture and server sizes are absolutely essential in reaching this goal.

If the process of determining the right architecture seems overwhelming, this is the webinar for you. During this one-hour presentation, we’ll provide many valuable tips for architecture and server sizing, and we’ll show you how the Ignition platform offers the right tools to build solutions of any size.

• Get database sizing tips
• Explore standard and scale-out architectures
• Find out how to handle more tags through optimization
• Learn about deployment with physical and virtual machines
• Hear best practices for cloud architectures and security
• And much more

Webinar Transcript

00:00
Don: Welcome everyone to today's webinar, Design Like a Pro: How to Pick the Right System Architecture. We really appreciate that you could join us here today. My name is Don Pearson with Inductive Automation. I am joined by Kevin McClusky, he's the Co-Director of Sales Engineering here at Inductive. Kevin, totally appreciate your time today. Do you mind maybe starting up by just telling us a little bit about yourself and what you do here at Inductive?

00:26
Kevin: Yeah, I don't mind at all. Happy to do so. My name is Kevin [last name here?], I'm Co-Director of Sales Engineering here at Inductive Automation. Along with the other Co-Director, Travis, I lead the Sales Engineering team. And I've been with the company for about a decade, a little bit over actually. Been through a bit of an evolution with the company as we were a smaller company when we started and now we're one of the major players around the world inside this space. It's been fun to be part of the journey. I've worked with hundreds of companies at this point on their architectures overall with Ignition on growth over time, and I'm happy to share some of that expertise with you here today.

01:06
Don: Thanks, Kevin. Yes, it has been a journey. That is for sure over the last 17, 18 years here, you've been here for a good part of it. I appreciate your time today. So here's what we have on the agenda for today's webinar: First, I will tell you just a little bit about our software platform, then we'll introduce today's subject by talking about the relationship between the architecture of your system and its performance. We'll go over some sizing tips for databases and a note about the amount of memory to allocate to Ignition. And we'll talk about Ignition standard and scale-out and architectures. We'll talk about some important security practices and discuss value changes and optimizations and running Ignition on virtual machines. As always, we'll get to our Q&A. So today we'll be talking a lot about the Ignition platform.

01:58
Don: Ignition's a universal industrial application platform for your HMI, your SCADA, your MES and IoT Solutions. Chosen by about 54% of Fortune 100 and I think about 35% of Fortune 500 companies. The latest version is Ignition 8.1. On the screen, you see some bullet points about some of the stand-out features of Ignition: Unlimited licensing model, cross-platform compatibility, IT-standard technologies, stable server-client architecture, it's web-based, it's web-managed, it's a web-deployed designer to make clients, modular configurability, and of course rapid development and deployment tools.

02:40
Don: Ignition is actually a development toolkit, because with these unlimited licensing and different module structures, it gives you the tools to build solutions. And the Ignition projects can be as small as a data collector for a few tags or as large as an enterprise solution with hundreds of devices, hundreds of thousands of tags, and hundreds of visualization clients. It's extremely important to find the right architecture and sizes of servers for your Ignition project so the project will work as you intended, and now, of course, some room for growth in the future.

03:18
Don: In this webinar, we intend to provide some tips to help you determine the correct architecture depending on your requirements. It's important to note that any architecture that you come up with needs to be fully tested and verified. Throughout that process, you can observe the performance characteristics of a server in order to make any necessary adjustments to the architecture. There's no guarantee on performance, since it is based on your design choices. Ignition's, when you look at its performance in sizing, is based on several different factors. Not all, but a number on the screen. Number of device connections, types of PLCs, number of tags, frequency of tag polling, number of tag value changes per second, number of concurrent visualization clients, SQL database throughput, deployment, networking latency, and of course, more than that.

04:14
Don: The features you use in Ignition also play a large role in performance. It's just like these require processing time and memory. And so device connections, OPC UA, tags, historian, alarming, transaction groups, scheduled PDF reports, sequential function charts, scripting, like tag change scripts which is running on server, visualization clients, number of tag subscriptions on a screen, polling queries, number of Gateway requests, Gateway Network, MES functionality, MQTT or cloud injection, and more. All those obviously have a lot to do with performance.

04:55
Don: Ignition is heavily multithreaded and can handle configurations with all of those features just mentioned at reasonable limits. Some features such as device communication and tags have predictable performance. Other features, such as visualization clients, have less predictable performance since they are based on how the user interacts with the system and how the project is configured. In most cases, a single server is sufficient to handle everything. However, when projects get large or when we push the limits of Ignition, we need to utilize multiple servers. Luckily, Ignition is modular and has the ability to scale out by separating different modules and features onto dedicated servers. At the end of the day, those separate servers work as one Ignition unit and are completely transparent to the user, thanks to Ignition's Gateway Network. And of course, the standards like OPC UA, SQL, MQTT, and more.

05:57
Don: So, when you think about it, what we're gonna do in this webinar is walk you through projects of different sizes, understand their hardware requirements and architecture. Since Ignition has a lot of different features, this guide is going to focus on what we're gonna do today anyways, focus on the number of devices, tags, and clients. Also, this webinar is based on a paper we recently published called, Ignition Server Sizing and Architecture Guide. So that's a good resource if you'd like more details about the subject that we'll be covering today. So with that as an introduction for today, let's dive into the subject. In fact, Kevin, over to you.

06:38
Kevin: Thanks, Don. We really wanted to answer the question that you're likely all here for first, before digging into some of the more deep details here. And that question is: How do I pick the right architecture? Now, we put together some questions. This isn't necessarily a 100% guide, you'll see that these are somewhat simplistic questions, however, they should give you a good direction as to where to start.

07:06
Kevin: And so, if you think about the architecture, completely... If this is a completely new project that you're thinking about rolling out and you're thinking about architecture, you wanna ask yourself some things as you go along. So, first think about those physical connections. Are the connections unreliable? Is there a geographic separation? Do you need resilience if part of that system goes down? If you said yes to any of these, choose Hub-and-Spoke. This is the Hub-and-Spoke architecture, every one of those different lines here has the potential of having problems going down, coming back up. And the whole idea behind Hub-and-Spoke is that it provides resiliency for those connections if there is a communication loss. It will do store-and-forward, hold on to information, send that information through when that connection comes back up, and you won't experience data loss just because of an unreliable connection or a low bandwidth connection.

08:02
Kevin: Next question, Do I need to run on the cloud? Well, choose a cloud architecture. This is a quick example of what a cloud architecture can look like, where you have the Ignition server in the cloud along with the database, and then different clients, designers are connected up to that. And there's some connection down to sensors in some way or different locations. The connections there will loop back and talk a little bit more about here in a few more slides.

08:32
Kevin: How do I pick the right architecture? The next question here. Do you have a small system and a reliable network? And can you run everything from a central office or a server room? This is our standard architecture as we call it, sometimes I also call it our basic architecture. And this would be the simplest type of architecture that you can have. Basically, if you have reliable connections, if you don't need to worry about data loss if the connection goes down, if you're not worried about someone taking a forklift through a network cable. Or if that happens, if you have bigger problems and it's not going to be a big deal if some of the connections go down or some of the screens go down at that time, then having a single server. And if you have a small enough system, having a single server, it can make a lot of sense. And we'll talk about some of the sizes, when I say “smaller system,” that I'm talking about.

09:28
Kevin: Smaller systems can still be fairly large with tens or hundreds of thousands of tags, going up to potentially 100 or 200 clients at the same time, but we'll get into some of those numbers here shortly.

09:43
Kevin: And what if you have a system that's bigger than a small system? What if you have a system that you need to do the same thing, but it's large? That's where the scale-out architecture comes into play. The scale-out architecture has to do with tags and IO Gateways connecting to devices, by having those individual pathways there that will split the load for device communication. Sometimes that's connected directly to devices, sometimes that's connected through MQTT, sometimes that's connected through other protocols or OPC. Whatever that connection happens to look like, these tags and IO Gateway split out the load for the tag processing, for the historian, for the alarm processing, separating that from the frontend Gateways. And if you have more than a few hundred clients, you can have multiple frontend Gateways as well. So there's the ability to run behind a load balancer and those frontend Gateways can split the load across all of the different clients.

10:44
Kevin: Maybe you need to do a variety of these functions. Can you mix and match architectures? Well, yes absolutely. So this is an example of the scale-out architecture in the upper right, inside a site that you already have. And maybe you wanna add on cloud to this and so that there's cloud visualization, there are cloud reports. Maybe it's a small number of folks who access the system through the cloud, or maybe it's a really large number. You can absolutely mix and match and extend your architectures here. Or if you need a scale-out architecture and you also need that data resiliency, and you need to be able to connect two different devices and more of a Hub-and-Spoke type of manner, you can absolutely do that.

11:29
Kevin: So these architectures are intended to be examples, not an exhaustive list of exactly what you need to do. And in fact, I work with companies every day, where we talk about what it looks like for enterprise roll-outs, for scaling up smaller architectures, where they may be starting from a standard architecture that we just showed with a single Ignition Gateway, and then they're moving to something bigger. And so it's very common that folks have many pieces of all of these different architectures. But these are a nice way to think of your final architecture, the different features that you need and why you might need those features.

12:05
Kevin: And the last question here, What about IIoT, Industry 4.0, and data collection architectures enabling digital transformation? Well, these layer into all of the architectures that we've mentioned here. So MQTT is often used for these types of applications, and in fact, more applications in that as well. So MQTT is used for resilient data collection and single source of truth, company adoption of open standards, low bandwidth connections, modern industrial and IT protocol support. And if there are initiatives for Industry 4.0, IIoT, digital transformation, it fits right into that as well.

12:48
Kevin: I should mention this webinar is focused on the Ignition Gateways themselves inside those architectures, and every one of the architecture diagrams that you see along the way can utilize MQTT for data collection and data transfer. So you'll see a lot of lines inside these diagrams that just show connections to devices, connections to PLCs, connections to other systems. Those connections, a good portion of those connections can be MQTT connections or can be some other protocols there as well. And if you're not familiar with MQTT or IIoT in general, I would encourage you to go to our website, check out our solutions page on IIoT. We do have webinars on digital transformation, on data apps using MQTT and a variety of other things. So the rest of this webinar won't focus on this, but this is certainly an important part of many architectures and architecture decisions going forward, and if you're thinking about scaling out or you're thinking about going to more modern architectures, I'd certainly encourage you to have MQTT as part of that discussion.

14:00
Don: That's great, Kevin. So now that we've gotten sort of a basic overview of the different Ignition architectures, we need to think about what size our Gateway and our server need to be, how many tags, clients and devices we can handle. So now we'll take a closer look at each of those architectures beginning with the standard architecture.

14:20
Kevin: So the most common starting architecture for Ignition as mentioned before is the standard architecture, this architecture that I showed a moment ago. So this consists of a single on-premise Ignition server connected to SQL database, PLCs, and clients. All the functionality is configured on the same Ignition server. If you're starting small, this is a really easy way to get started. Ignition has unlimited licensing, but it is limited by the size of the hardware. Larger hardware equals more device connections, tags and clients. You're able to do more if you have more hardware on a server like this.

15:00
Kevin: And so we've actually gone through and put together some recommendations for server sizing as you're looking at more tags, more clients, more device connections. We've included some of those here. As mentioned before, there's also a companion document so you don't need to be furiously scribbling down notes here or rewatch this webinar in order to get this reference information that I'm about to show here.

15:24
Kevin: If we're talking about a small Ignition project, and all we want to do is a handful of tags, handful of devices, a handful of clients, you can see right here in terms of hardware recommendations something as small as an arm device, that would be a Raspberry Pi or something else that's a tiny, powerfully industrialized, but maybe not even industrialized piece of hardware, Ignition can absolutely run on that. We recommend running very small systems on that, so if you're looking at anything more than 500 tags or a couple of concurrent clients or one to two devices, you probably want something bigger than that, but if you're just looking at a really small system, those systems work great.

16:12
Kevin: As we go up in terms of numbers here, two-core system, 2 gigs of memory, we always recommend an SSD. You'll notice that that's listed on each one of these, but if we're talking a two-core system, somewhere between 1 and 10 devices, 2500 tags, five concurrent clients could run on something like that, and then two-core, 3-gigahertz system, 4 gigs of memory, SSD, you might be looking at 1-10 devices still, but maybe double the number of tags and double the number of clients there. These would all be considered small systems.

16:48
Kevin: If we go up a little bit higher to a medium-sized Ignition project, you can see that these numbers start increasing pretty significantly, so four-core systems for all of these different amounts of memory and different amounts of speed, different speeds in processor, going anywhere from 1-25 devices for the smaller one that's listed here up to the largest one listed here is 1-100 devices, 10,000 tags to 50,000 tags, 25 concurrent clients and 20 all the way up to 50.

17:23
Kevin: And then if we start going to really large systems, we're talking about full Ignition SCADA systems or MES systems, if you're using the Sepasoft MES modules, for example, 8 cores, 16 cores, and then numbers scale up from there to 75,000 tags, 100,000 tags, 150,000 tags and then up to about 100 concurrent clients. But it should be noted that Inductive Automation in generating these, these numbers are somewhat conservative, but we wanted to make sure that we presented a view that was relatively conservative here if people are using this as guidance. We do have some systems that are over a million tags on the individual system. They generally are not going to be running a large number of clients at the same time, or possibly a lot of device communication might be to a few specific ones or connected up to MQTT and having that information piped through with some of that processing happening elsewhere.

18:28
Kevin: So these are just overall general guidelines that if you're looking at setting up a system of certain size that are a good starting point, and if you are looking at scaling any one of these numbers up more significantly than this, we'd always encourage you to reach out to Inductive Automation, we're happy to have architecture discussions. Our team, my team, the Sales Engineering team has a whole variety of experts who all we do is talk to customers about architecture and design, and we would be happy to have that conversation with you.

19:00
Don: Kevin.

19:03
Kevin: Yes?

19:03
Don: Kevin, I gotta throw in a quick... Just a quick question here.

19:06
Kevin: Absolutely.

19:07
Don: Matthew, he said, "Most Vision clients you've seen working on a single server, are there actually limits? We're at 541 right now." Matthew said...

19:18
Kevin: Great question. Yes, 541 is just about the largest I've seen. I've seen one system that's somewhere around 600. So where this says 100 concurrent clients, just to continue adding to what I mentioned a moment ago, this is based on an average type of design. So folks can design clients where the clients have 100 tags that are subscribed from the client to the server, or 1000 tags subscribed from the client to the server, or zero tags, and how heavy that client design is, is going to directly influence how many concurrent clients you might be able to have inside a system.

20:07
Kevin: Systems that don't have any real active communication back and forth from the server to the client, that only have updates once every 60 seconds or when users interact with them or press buttons in certain ways, those can normally scale really, really high, and then other clients that you have, if you have really heavy processing that's happening, if you have machine learning algorithms that are happening on the server as part of the support of that client, if you have updates that are happening every second with advanced algorithms, you're probably going to get a small number of clients because you're doing a lot and you'll be taxing the system quite a bit and so, it can scale anywhere between there. As I said, these numbers that we're showing here are normally relatively conservative. There's a good chance inside your project design that you can get more than this, but these are the ones that we have as just general guidance for folks who are spinning up on Ignition Gateway.

21:01
Don: Kevin, so one last question. I won't bug you a lot, you've a lot to cover, but this is relevant here too. What's the most significant: Devices, tags, or clients? For example, one to five devices, 100,000 plus tags, one to five clients, is there anything more significant than anything else in terms of that?

21:20
Kevin: Yeah, yeah, good question. So we do have a little bit more information on that inside the guide itself, if you download that and take a look at it after this webinar, but I'll answer that very quickly. The devices, it depends on the type of device. So some devices are really lightweight like Modbus devices; other devices, if you have drivers that are communicating Allen Bradley protocol, that's more heavyweight, for example. Sometimes OPC UA devices, if they're updating things all the time, are even more heavyweight because it's a very heavy protocol. If there are tags that you're subscribed over OPC UA to Siemens S7-1500 that are not changing, then it becomes really lightweight because OPC UA is a subscription-based protocol. So, number of devices can make a big difference. It's normally number of tags that's a bigger difference than the number of devices, and the number of tags, somewhere around every 50,000-100,000 tags is another gig of memory that Ignition generally needs, but that'll depend on if you're running expression tags, if you're doing scripting on every single tag or on a good proportion of tags.

22:30
Kevin: Or if you're not, if you're just doing simple tags that have simple value updates, then they're not a big load on the system either. And in that case, if you just have really simple things that are coming through, and you're not doing advanced scripting and you're not doing expressions and evaluations and calculations on a good portion of things, then normally it's the clients that are the heavier piece of the load. And clients generally take CPU as opposed to memory. They'll take a certain amount of memory, but it's CPUs, that heavier resource that's available for that. I've seen some folks go bigger than 16 cores on their systems if they have a large number of clients. So it's not unheard of to do 32 cores all dedicated to Ignition in order to have everything running on a single system or possibly split out with a Hub-and-Spoke type architecture there as well.

23:25
Kevin: Alright, I'll keep going here. So yeah, in terms of the single standard architecture, and notes about this on all of these architectures really is that results can vary based on design choices, and I've just been talking about that, but it's worth noting that explicitly here, so faster poll rates, increased value changes, utilization of other Ignition features like scripts, Transaction Groups, SFCs and more do have a significant impact on the system. Everything that you're adding to the system takes a certain amount of CPU cycles, and of course, the amount of CPU that you have available is going to affect how much you can scale up for each one of those items.

24:07
Don: So the SQL database is the most widely used relational database in the world, and it is a very important part of many Ignition architectures. Chances are, you'll be using a SQL database in your system, so let's talk about how to choose one that is right, the right size for your requirements.

24:26
Kevin: In all of the following scenarios, the SQL database should be on its own dedicated server. Sometimes you'll even have more than one SQL database, which is fine, but the SQL database normally lives separate from the Ignition Gateway. That's to keep the resources separate, that's to keep the processing and the memory separate, and that's to make sure that Ignition has all of the resources that are allocated to it or available on the server available to Ignition itself without the database impacting Ignition's performance.

24:57
Kevin: Here's some basic database sizing tips when it comes to the amount of history that you might have coming through. This would be for the database server itself. I've got two cores, 0-100 value changes per second. It has some numbers in terms of the amount of space that it requires per year and the amount that you can get through per second for all of this. This is completely uncompressed. There's some options that you can compress data in different databases in different ways. We do have a SQL database Ignition historian guide as well, that's also another companion guide that has this information inside there for optimization techniques, and we'll talk about a couple of those a little bit later, but this is fully uncompressed, just a standard install without any performance tuning, these are the numbers that you get: 100-500 value changes per second, it's disk, though requires about 60 gigs with 2% change, smaller with slower rates.

26:05
Kevin: So it gives you an idea that if you have 500 tags inside a system, but not very many are changing, you have a lot smaller storage requirements than if things are changing pretty significantly, and almost nobody has 100% of changes at a one-second rate. That's not a very realistic picture of how people use the historian, but as mentioned before, and as I'll mention again, this is more or less worst case scenario that we're trying to share with these numbers to make sure that if you do have a very high throughput, that you get an idea of what this would actually take. And once again, if you need more information on this, if you need to understand how you're going to size this, that historian guide is a great guide to give you the calculations for exactly how much space it would take, based on your actual expected rates. There's a companion historian calculator that will let you type in numbers and values to calculate out those value changes per second.

27:12
Kevin: And these value changes per second, if you have 5000 value changes per second, that's typically a system that has somewhere around 100,000 to 500,000 tags may be inside that system, because most folks aren't seeing value changes every second for every tag. So it gives you a quick idea of the sizing of this overall. Large historian, you're looking at a really big number, with things going through, this is still a single database in this case. 30 terabytes per year is a very large amount of space, that as I mentioned, if you're going this large with your historian, you're probably going to enable some compression options or pick a database that has significant compression there, so you don't need to store quite that much.

27:57
Kevin: And a couple of quick notes about this. The number of value changes can be pretty significant with some databases, a Microsoft SQL server can handle a lot more than MySQL, for example, PostgreSQL is pretty good in terms of throughputs, time scale is great, and all of that information is inside the guide as well. And a quick note about memory, you can adjust the amount of memory directly inside the ignition.conf configuration file. Typically, there's 1-2 gigs that you need to leave for the OS and then just use the rest for Ignition on any of those Ignition servers.

28:35
Don: Thanks, Kevin. As we said, there is a lot of power within a single Ignition server, but in some cases, you're gonna need more than one Ignition server to get the best performance. Our scale-out architectures offer a lot of power and a lot of options that you should consider. To help you with that, let's just take a look now at some ways that Ignition helps you scale out your system.

28:57
Kevin: So in larger systems, it's often easier to have multiple Ignition installations, and we showed that a moment ago. This is perfect for large systems that aren't split up to different sites, so this would be on-site, more or less taking a single Ignition Gateway and splitting it into multiple Gateways. Hub-and-Spoke is normally better fit for systems that are split into different sites, but for a single site, the scale-out architecture is great. Just for reference, this is the architecture that we're talking about, with I/O and frontend split out there, or multiple I/O's, or multiple I/O's with OPC UA or MQTT connected there.

29:38
Kevin: The scale-out architecture has at least one Ignition Gateway that handles backend communications, the backend talks to PLC's, does device communication, does MQTT communication. It also does history, and it does alarming. Frontend Gateway handles the client, so it serves up the data that is pulled from that backend Gateway, and is possible through the Gateway Network. That interconnection there is a protocol that we call the Gateway Network Protocol, that is a specific protocol that was created by our developers, and so that Gateway network communication there is what allows for the tag sharing, remote tag, provider sharing, history sharing and other communications.

30:25
Kevin: One of the nice things about the Gateway Network is that it is a dedicated HTTP channel, so it sets up as a node or it can actually proxy to other Ignition Gateways as well. In the cloud architecture that I showed earlier, you had multiple Ignition Gateways that were talking to each other, and they can proxy to each other through the Gateway Network communication. There are security settings, and so you can restrict connections, you can whitelist connections, you can set security on individual services, so you can set it up so that there's read-only access from the outside, or read-only from certain zones, or disabling access entirely, or read-write from certain zones. For the frontend, you'll wanna have read-write. From the cloud, you might just want that to your read-only connection and have zero chance or possibility or zero permissions for anyone to change anything if they're coming from the cloud, that's easy to set up. An SSL is available, so you can have full encryption and in fact, that's turned on by default.

31:24
Kevin: More Gateway Network features here, the Enterprise Administration Module uses the Gateway Network, and so it's used for monitoring things, centralize backups, license management and resource distribution, any changes that you might wanna make remotely, and then there's also remote real-time providers, historical providers, alarming, alarm journal, audit logs, and all of those are remote services that allow one Gateway to access information on another Gateway. So you can have a central designer pulled up, for example, where you can browse all the tags and all the rest of the Gateways, and in fact, you can configure additional tags on other Gateways over that Gateway Network connection, as if you were on that remote system. So you can browse its devices, you can browse its OPC side and you can pull in new tags like that. And we already mentioned security zones and service security there.

32:23
Kevin: So this is a quick example of that scale-out architecture, you might start from a single Ignition Gateway or the standard architecture we talked about earlier, and then split it up into a scale-out architecture like this, where you have the frontend Gateway and an I/O Gateway that are basically handling the load for those different pieces. So I/O is talking to devices or talking over MQTT, or talking over OPC UA and then the frontend is talking to the back-end of that I/O Gateway and it's also talking to the SQL database there, and clients can be anywhere. Often, in this case, with the scale-out architecture, these would be on-site but they could be connected from remote over a VPN or really anywhere that has IP-based access to that frontend Gateway.

33:11
Don: Quick question, Kevin.

33:12
Kevin: Yeah, go ahead, please.

33:14
Don: In the back and front end Gateways architecture, which one would handle database connection for history insertion?

33:24
Kevin: Great question. So the I/O Gateway does that. The I/O Gateway is going to be putting that data, the history insertion, into the SQL database. Basically, the Tag Historian Module runs there and the SQL Bridge Module. Since it's the one that is seeing those tag changes live, it's the one that is most appropriate to be doing that data insert into the SQL database. If you have 15 I/O Gateways, and there's certainly folks who have really large scale-out architectures, you can pair it up with one additional Ignition Gateway that's just dedicated to the historian, and I have seen some folks do that. But in general, that I/O Gateway is going to be doing that job, it's going to be putting that data into the SQL database, both for tag history and for any transaction groups, any event-based history that might be going in there.

34:17
Kevin: And then the frontend Gateway can be set up so it is pulling data either through the I/O Gateway or it can pull it directly from that SQL database in a way that's transparent overall. It's a really nice setting inside that Gateway Network, the remote history provider that might be set up there in the tag provider. You can say, "I want to pull history directly from this database," and that frontend will issue priorities directly against the SQL database instead of making an additional hop through the I/O Gateway, but you can set it up in either way.

34:52
Don: Kevin, one last thing. Since we're talking architectures today, I think I should interject this question right on topic now and it says, When using a scale-out architecture with a third party OPC UA server handling device communications, is it still recommended to have a separate Gateway purely for handling frontend clients?

35:11
Kevin: It depends on your load. So in some cases, this can make sense, in other cases, you might be able to do a single server architecture. So most folks actually will take a look at what they think that the overall sizing might be, how much history they're going to be storing to the database and how many frontend clients that they're going to be looking at and then decide based on that. The I/O Gateway, that one is going to be largely responsible for history insertion and if you have a really large amount of history, that load is going to be heavier than any sort of device, native device, communication mode on that server, and so it can make sense to have it split out. If you have 100 tag changes per second that are coming through or 500 tag changes per second coming through, so some number that's not too large, it's probably fine. And you've got 50,000 texts, it's probably fine to have all of that in a single Ignition Gateway here but if you're looking at really large numbers, it might make sense to split it out.

36:17
Don: Thanks, Kevin.

36:18
Kevin: Sure. Alright, and so one other architecture here, and this is actually exactly what was just asked. That OPC UA server there, whether that's Kepware, or whether that's Ignition, or whether that's Autosol, or whether that's a third party's OPC server, like B&R has their own, regardless of where it's coming from, this architecture can make sense. And so yeah, this was the question I was just asked, Does it make sense to have an I/O Gateway if you have an OPC UA Gateway? And the answer is, yeah, it certainly can if you have large enough numbers of clients and you have a large enough number of tags that have tag history that are associated with them or large number of transaction groups.

37:03
Kevin: In this case, this is showing... If you're using Ignition's OPC UA server, this diagram is showing that it can make sense to have that split off as well because that device communication, the driver communication, if you have a few hundred devices, it can be one of the heavy pieces overall in terms of the amount of CPU that's needed for that utilization and having that inside its own server right alongside the I/O server can be a nice way to have that isolated and have that dedicated to PLC communication. Whereas, the I/O server is dedicated to processing those tags and the history, and alarming, and possibly reporting, sometimes that's done there as well, and then sending any value changes to the frontend for the clients.

37:53
Kevin: One other architecture for scale-out is this architecture where you have two different I/O Gateways for three or four or five and you have a frontend Gateway that separated out from that. This is also useful if you have really large numbers of PLCs that you wanna be coming through. They might even have a geographic separation or they might not, they might be on different networks, and so folks often use this architecture overall. This is one of the more common scale-out architectures.

38:21
Kevin: A couple of best practices around this. For the tag providers that you have inside these scale-out architectures, you wanna make sure that you provide proper names to those tag providers. You don't just wanna call them default. That's where they start out but when you set up those I/O servers, make sure that you name them something that makes sense, and you can have multiple tag providers per I/O Gateway or you could have a single one. The important thing is that the frontend Gateway is going to see those tag provider names and having those consistent is going to help a lot in terms of the way that clients are designed. And if you did need to move some devices around or you needed to move an area to a new I/O server, if you have good names for those tag providers, you can move an entire tag provider to another I/O server and have zero work that you need to do on the frontend other than setting up that Gateway network connection, which is 30 seconds, and then the tag provider will be coming from that new I/O Gateway. So if you had properly designed it there, then that can make it very convenient for continuing to scale out on the I/O side.

39:25
Kevin: You do want to make sure that you use fully qualified tag providers as part of the path. This is done by default in Ignition 8.0 and 8.1. If you drag a tag out onto the screen, it includes that, but if you're typing this in by hand, having that fully qualified tag path makes it so that it is automatically portable. So if you export that project and you import it in somewhere else, if you move that tag provider as mentioned, if you have a project that is pointed at a certain location and then it gets pointed to another default tag provider, if you have that fully qualified tag provider names, it's not going to change or break any of your tag paths as part of that process. So, that can be a really important piece of a design choice as you go through.

40:20
Kevin: And then if we're taking a look at scale-out architecture with a load balancer, and you'll notice, I will mention this in a second, we added a little bit in the upper left there too. This scale-out architecture with the frontend Gateways behind a load balancer allows you to scale out multiple frontend Gateways. And so you can have three or four or five or six. You can have as many frontend Gateways as you want there, and those frontend Gateways are running exact copies of the exact same projects there. That load balancer takes any new client that comes in and connects it up to a different frontend Gateway. So this would normally be someone who's running over 100 or 200 clients that are with relatively typical client design. So clients that take a certain amount of CPU, maybe a 200, they're maxing out a frontend Gateway, and it makes sense to scale out. So adding additional frontend Gateways behind that load balancer will let you scale out in a clustered way and every time that you might be running low on CPU you can add another frontend Gateway and that'll just split the load there. Because all of the tags exist on those I/O Gateways, it doesn't add any tag load to the system, it only is adding more client load on the front there. And then of course, those have communication to the I/O Gateways, but that communication is normally really efficient and relatively lightweight because that's subscription-based as well.

41:49
Kevin: Now, what this is showing in the upper left there is the possibility of the I/O Gateways connecting down to Ignition Edge, so that would normally be an MQTT connection that's going down to Edge for data transfer there, although that can be the Gateway network as well. And then that Ignition Edge might be connected up to a few different devices that will have store-and-forward built into it and push information back up, if that connection goes down. So that piece is a part of the Hub-and-Spoke architecture here, and it's pretty typical as architectures grow.

42:20
Don: More and more organizations are leveraging the cloud in their systems. You may have a good basic understanding of the benefits and risks of using the cloud, and you also should understand it from an architecture standpoint. Kevin, can you talk about this a little bit more?

42:36
Kevin: Yeah, so this is a quick example of cloud architecture. So if you take a look at our standard architecture and you drop it into the cloud, this is basically what that looks like. So you have the Ignition server and then you have the database in the cloud. Often that's done with the database architecture from the cloud provider, so Amazon Web Services has their RDS service. Azure has their Azure SQL. They have a variety of databases actually, that you can spin up inside Azure now. It used to just be Microsoft SQL Server, but you can now scale up just about any major database there. The connections from the cloud server to what you see in the upper-left sensors, that's drawn with a dashed line because it is very undefined inside this diagram. There are a lot of different ways to connect down to sensors. Most often we see folks interested in connecting that over MQTT because it's such an easy way to go about that and provides open standards communication, but there's about 100 ways to get that connection going. And even if you're using MQTT, the sensor to the MQTT collection device, if the sensor doesn't have MQTT built-in directly can still be a consideration there.

44:00
Kevin: So on that side, this webinar isn't specifically focused on sensors, we can have a full webinar probably on the architecture that includes the sensors and the communication and all of that, but that's showing that there's some communication up to that Ignition server in the cloud. Some systems don't have sensors at all, some systems just have a database inside the cloud. Sometimes that cloud database is being fed from other systems and Ignition is just providing a visualization platform. That's entirely fine. We have a lot of folks who are doing that today and so that sensor piece might not even be relevant, but that sensor piece is relevant. Sometimes that is cellular communication that's going over that and data being pushed over cellular to the central Ignition Gateway, so outgoing communication, so it's firewall-friendly, and there could be some other good options there as well.

44:53
Kevin: I'd say on that piece, if you need more information on that or if you have considerations about what that connection looks like, I'd also say give us a call, reach out, we're happy to have our engineers talk to you to provide some details on all of that or to talk through the options with you and figure out what piece of that might make the most sense for you.

45:17
Don: Thanks, Kevin. I know we got a lot to cover and you're still trying to get to optimizations and running on VMs with today's webinar, but you can't miss the point of security. Like sensors, it could be a whole webinar on its own, but it's an issue that you can't get away from, especially as this cloud becomes more important. Every connection in our architecture offers new possibilities for gathering and sharing and getting more value from our data, but we also have to think about how to keep those connections as secure as possible. On the deep subject, obviously, can we at least go over some general best practices that everyone should follow?

45:52
Kevin: Absolutely, so if we take a look at the scale-out architecture with the cloud connected here, where we have a joint architecture and you have things on-site and you have things inside the cloud there, my general advice on security, and I've actually written a couple of articles about this that have been published in different places, I've had a talk or two about security that we published out. But my general advice is to take a look at each one of these lines and decide how secure it needs to be, and then make that security happen. So encryption is a very common way to do security. Ignition has a whole variety of switches inside that you can turn on encryption for all sorts of different connections that you might see. And so Ignition to Ignition communication is encrypted, Ignition to the Ignition clients can be encrypted, and we'd normally... In fact, our starting advice generally is to do that immediately on any new frontend Gateway that you set up. Set up the encryption there so it's going over HTTPS instead of HTTP using the same encryption that your banking website uses.

47:02
Kevin: And then on the cloud side, all of those pieces are in place and available as well, so those connections to the clients can be encrypted, the connection over VPN can be encrypted there as well, down to the site. And then in addition to just encryption for the communication, there are a whole variety of things that you can do for securing user accounts, securing sections of communication, securing the services that are running over the Gateway network. And then on the device side, you can encrypt those connections as well for certain devices. Almost all devices don't support that. A very small number do. Like there's OPC UA with encryption built into the Bedrock processors, for example. Beckhoff processors have support for that if you purchase the right package with them. The Siemens S7-1500 with the built-in OPC UA has that, but I think that that's the full list that I know of. Oh, and then Opto 22 PLCs or PACs are also in that list, but it's a very small number. The rest of them don't have encryption support. So what you can do to encrypt that side is put a small installation of Ignition Edge right next to those PLCs and then do encryption over that communication, and then have some isolation there.

48:28
Kevin: There are a number of other techniques available as well. We have the security hardening guide, which I would highly recommend anyone who cares about security to go through and read, and it talks about securing each one of these individual Gateways there, and what the knobs and tune, levers, and all the things that you can move and switch and configure inside there are to increase your security profile to make things as secure as possible.

48:53
Don: Thanks, Kevin. So far in the webinar, we've been looking at the big picture of different architectures and related security issues just now. So now let's focus for a few minutes here on some little things you can do to optimize performance of your system. There's a lot of small adjustments you can make that collectively can make a big difference. So give us a few minutes on this, Kevin.

49:14
Kevin: Sure thing. So if we're taking a look at optimizations, these can help you scale up your Ignition Gateways. If you're starting to run into anything that has to do with capacity, if your values are updating slower than you'd expect them to on an Ignition Gateway, if you're hitting near 100% CPU at any point, or really over 50% or 60% CPU, then you're starting to run near capacity there because Ignition does a lot of burst processing, so you wanna be able to have enough headroom there. If you're running up against your memory limits, if you're seeing clock drifts, for example, you'll certainly wanna be taking a look at your performance, and either switching to another architecture, adding more resources or resource allocation to the server, or take a look at doing some optimizations. And so these are some ideas for these optimizations that you can do. It's worth noting that when it comes to tags, the value changes are the most important metric to be looking at. So a lot of folks who maybe are doing their first Ignition project, they're not very experienced with Ignition, just drag all of their tags in and leave everything at defaults there.

50:27
Kevin: That doesn't necessarily give you the best optimized system because you can tune all of those tags to be coming in at different rates, you can also tune in the deadbands, so it only processes those tags when they see that a certain amount have changed. You don't really want to have deadbands in there that are going to log noise, and that's probably the most important thing. If your sensor has a value that it only has an accuracy of 0.1 and you have a deadband of 0.0001, you're going to be logging a lot of complete junk into your system. It's absolutely useless information because the sensor isn't that accurate, so it's just noise on the line. If you tune that down to actually match the sensor's accuracy, or maybe even a little bit more than the sensor's accuracy, if you think about sampling logic from the engineering classes, but you're going to end up with much better optimized throughput there.

51:27
Kevin: If you're taking a look at value changes for alarms, historian and all of that, the value changes per second, the amount of the rates that you're pulling things in, and all of that, and the values that are going into historian all make a difference for all of that. And by the way, I'm going through these slides relatively quickly. All of this information is inside that guide, so you can certainly go through and read those optimizations in more detail inside the guide as well. You've got different tags and different value changes, different percentages changing per second, that's still 500,000 tags, if you had a five-second rate with 10% changing, that's 10,000 values per second, same with some of these. All of these are 10,000 values per second. Just different examples of how to get there, so all of those make a big difference.

52:19
Kevin: And then when you take a look at individual optimizations, polling rates, deadbands, leased and driven tag groups, event-driven items, scripts, polling queries and caching, tag historian, data collection and data detection, vision, client tag poll rates, statement network optimizations, remote tag providers for the alarm subscriptions and the history mode all play into this as well. And so the guide has optimization recommendations for each one of these, including the polling rates and how to dictate what those look like, what the deadbands might look like, what the least and driven scan classes and tag groups might look like, they're called scan classes in 7.9 in tag groups and 8.1 and 8.0. Event-driven optimizations for different things that are coming from events, largely from tag changes, but they can come from elsewhere as well.

53:20
Kevin: Script recommendations, scripts are one of the things that have a ton of power, but if you write them in certain ways, it can certainly take a fair amount of CPU as well, if you're trying to do heavy processing, so there's some recommendations about that. Polling queries, we normally recommend turning off polling queries unless we need them on screen, so having it so that it just loads in data when you first load up a screen or navigate to somewhere, and keeps that static more or less until something triggers. You can have it polling but polling at a one second rate certainly puts items, puts additional load on the server. If you multiply that by 200 clients, you have 200 queries per second, and then that's going to put additional load on the database as well, and so having those polling rates be more reasonable normally makes a lot of sense.

54:11
Kevin: Tag historian stale data detection, I'll show this in about two seconds, but I wanted to pull this up as well, just because this is something that a lot of folks don't realize, and it's a very important setting, if you can do it. So if I have the Ignition Gateway here, you should be able to see my screen, and if we go to the configuration section, we go down to the section on tags history and configure one of these connections, and we go to the Advanced Options, there's this Enable Stale Data Detection, if you turn this off, this significantly improves the performance in the tag historian in certain types of systems, especially in systems that are struggling with performance already, because of the way that this works. I recommend reading the guide and maybe reading the manual to understand a little bit more about what this does. But if you don't have a reason to leave this on, I would recommend for every data connection that you'd set up, turning this setting off right there. So that's that optimization right there.

55:16
Kevin: The vision client tag poll rate is something you can turn down, Gateway Network can be optimized with a number of Gateway network settings. A remote tag provider alarm subscriptions can be used in a subscribed mode as opposed to a read mode, which will allow it to asynchronously send information over. Remote tag providers have a history mode and going straight to the SQL database can make a big difference there, and that takes us through the items that we have there. As I mentioned, there's more details inside the guide.

55:52
Don: That's great. Thanks, Kevin. I know we've got one last topic here, running virtual machines or VMs. Also, I just wanna emphasize that you will all get this presentation in addition to the guides available that Kevin referred to, so you can have a more detailed look at all of this. Get to VMs though, they've become a pretty main deployment environment for today's use of Ignition, especially for enterprise users, although VMs by design are slower than running on bare metal. Ignition runs well in VM environments, as long as the correct configuration is properly applied. So, here are some great tips for running Ignition on VMs.

56:30
Kevin: On the software side, so Ignition is going to be running on any hypervisor, and it is important that resources are appropriately allocated for the resource allocation, allocating CPUs in line with the guidance earlier here makes a lot of sense. As I mentioned earlier, very large systems could be up to 32 vCPUs with more than 64 gigs of RAM, and how much you allocate really depends on how big of a system you have inside Ignition and how you're designing your project, what you're asking Ignition to do.

57:03
Kevin: The most important thing on virtual machine settings is setting up dedicated resources. Ignition does a lot of burst processing so Ignition needs CPU available immediately any time that something comes in over the network. The speccing this out or even after you have something set up already going back and dedicating resources is something that we highly, highly recommend for anybody who's running on a VM.

57:34
Kevin: So inside VMware specifically, VMware is the most common VM platform. There's a setting called latency sensitivity, that latency sensitivity should be set to high. If you set that to high, that does a 100% dedication of those resources to the Ignition VM. And so, if you're running Ignition right now on VMware and you don't have that set, I would recommend it right after this webinar, immediately going to the IT group and asking them to do that. There might be some pushback because it takes away resources from other VMs that might be running on the same host, but that's the point, you don't want to be sharing those resources with these other systems, you wanna make sure that Ignition has all of those resources available, and especially if your system is struggling, this can make a significant difference there.

58:21
Kevin: For other hypervisors, there are other settings that do the same thing, but as I mentioned, VMware is the most common one, so we're aware of where that setting is and we can share that with you. And there's a little reference information for where that comes from as well. Alright Don, back over to you.

58:40
Don: Well, you covered a lot of material. As we knew when we started this, it was gonna be a big topic 'cause we came to the end of our time. Just a couple of things I wanna say. If you wanna try Ignition for yourself, please do. And the other resource is Inductive University, please take advantage of that, 600-plus videos, you can learn anything you want to about Ignition there. So I just wanna say that we appreciate, Kevin, you digging in deep. We totally appreciate the interest. Thanks for everyone's attention today, and thank you for your presentation. With that, everyone, have a good day. Thank you.