Design Like a Pro: Alarm Management

Many industrial organizations depend on SCADA systems to control and gather data from their industrial installations. A well-maintained alarming system is needed to ensure optimal performance and long-term stability of your SCADA system. If your alarms are not managed carefully, they can quickly lead to distraction, downtime, and disaster.

In this webinar, alarming experts from Inductive Automation will dive into the best practices of alarm management to help you improve the safety of your industrial system and reduce catastrophic downtime incidents. No matter which industry you work in, these time-tested principles will help you get the most out of your SCADA alarming system.

Learn more about:

• What alarms should and shouldn’t be used for

• Prioritizing your alarms

• Getting nuisance alarms under control

• How to reduce alarm floods

• How to easily set up complex alarming systems and remote alarms

• And more

Transcript:

(The moderator, Don Pearson, briefly introduces the topic, Inductive Automation and Ignition software, and then introduces the presenter, Travis Cox.)

Travis: Alright, thank you, Don. I'm really happy to be here today to present on this particular topic for best practices for alarm management. It's something that I'm pretty passionate about. When you do have your system configured correctly, I've seen a lot of ROI from just being able to really clearly understand what's going on and I've seen some really messy spaghetti mess situations, and I've seen some really clean situations when it comes to alarms. So, today, we hope to show you some of the real best practices to help shape how you would, at least, think about architecting your alarming system. But first, I'd like to start off with an all too common alarming scenario. One day, at 4:00 PM, in a wastewater treatment facility, the water pressure becomes too high in one of the tanks, which causes an alarm to go off. The alarm is set at priority level four, it's a critical alarm, but it doesn't stand out because most of the alarms are set at that level too.

Travis: Pretty much everything was set at the critical level. The operator can't acknowledge that alarm right away because he's dealing with several other alarms that went off a few minutes earlier which were caused by a pump that is turned off at this stage of the process. Then a few minutes later, another set of alarms go off. At the end of a long day, the operator is getting tired and he knows that most of the alarms in the facility are just nuisance alarms, so he just silenced all of the alarms at once. So, despite all of this alarm activity, the operator remains unaware of the high water pressure in the tank and this problem will get worse as time goes on. This was a fictional scenario, but situations like this frequently happen in the real world. Too often, alarming systems grow to the point that they make it harder for operators to solve problems rather than helping to solve them. Alarm systems have to be managed in a careful and consistent way over time.

Travis: Today, we're gonna help you do that by sharing some best practices from the alarm management handbook. These best practices have been proven over many years and are applicable across industries. We consider this book to be the standard for a SCADA alarming system and we built the Ignition Alarming System to meet these guidelines. We'll also show you some of the unique features that Ignition has when it comes to alarming. So, let's start by defining what alarms are and what they should be used for. Alarms are conditions that are evaluated with respect to a specific numeric data point. You can attach any number of alarms to any data point. Alarms are typically configured on a tag, but in many products, it can be configured on other things as well that don't have to come from devices. They can come from different sources of information. Let's take this moment to clarify a few basic alarming terms. Active or clear: When the alarm condition becomes true, it's active. When the alarm condition becomes false after it has been true, it's been cleared.

Travis: Alarm notification, a message sent to a person or group when an alarm becomes active or clear. In Ignition, this functionality is enabled by the alarm notification module. Alarm history or journal, this stores historical alarm information in a database. Info such as source, timestamp, associated data, values of the alarm properties of the time the event occurred, and more. Now, these are very basic terms, I know a lot of you probably already know them, but they're very important as we look at how we set up our system. So, why are alarms important? Alarms let you know when something is wrong in your system, which requires some type of corrective action, and that's really critical here. We wanna configure alarms that have a corrective action that we want to do something with. 'Because if they're not managed carefully, alarms can make problems worse. As the opening scenario showed, when there are too many alarms or too many that are at the critical level, they cover up the real problem and they're very difficult to manage.

Travis: While having an alarming system in general is important, the way you implement it is vital to your SCADA system. Make sure to create alarms only for real problems that require action. Also, they should easily identify the issue. Alarms should always signify an abnormal event or condition. They should not be used to confirm things that are running normally or just to do event tracking. We can do event tracking with alarming systems, but we should not notify people or put them into... On the screen as something that we need to have a corrective action for. Excessive amounts of alarms can be distracting and the users can eventually lose sight of their purpose. When there are too many alarms to handle, operators may choose to ignore them and that can lead to a potential problem. Here are some big problems that commonly arise in alarming systems, especially as they get bigger: Alarm floods, an alarm flood occurs when there are more than 10 alarms in a 10-minute period.

Travis: State alarms, or stay alarm. A stay alarm is one that stays in the alarm state continuously for over 24 hours. Chattering alarms, when an alarm goes from active to clear three or more times in one minute. Now, of course, these time frames can vary from industry to industry, but generally, these alarm floods, stay alarms, and chattering alarms are something that we have to deal with. In regards to that, here are some benchmarks that we should use for these alarms. For the total number of alarms per day, 150 alarms or less is considered manageable. Any more than that, it becomes a problem. So, if you experience 300 or more alarms per day, you should significantly reduce the number of alarms in your system. For alarm floods, there should be no more than 20 per week. In the long term, as you get your alarm system under control, you should aim to have zero. For stay alarms, you should have no more than 20 per week as well. And in a long term, you should, again, have zero. For chattering alarms, you should have no more than 10 per week and zero long term.

Travis: Later in the presentation, I'll show some techniques for dealing with these different alarm problems. Alright, so that brings us to probably one of the most important concepts when configuring alarming systems, and that is how we organize our alarms. There can be thousands or hundreds of thousands of alarms in a facility, and it's important to organize those alarms in respect to what an operator needs to see on screen and how we're gonna be potentially notifying or storing this information. So, you really need to think about a hierarchy for your alarms, especially when it comes to filtering. And that does mean that we may have to look at a hierarchical structure or adding associated data that is going to allow us to do that. So, there are really two ways that we can organize alarms for filtering in particular. One is that we can name our tags and folders with something meaningful to the process and use that structure to our advantage.

Travis: So, how we set up the structure of our tags can dictate how we're gonna look at the alarms. But not every time is that gonna work. Sometimes, how we wanna show alarms or filter alarms can be different than the structure of the tags that we have set up. So the other way to do that is through associated data. Associated data are basically custom properties that can be added to any alarm. And alarming systems should give you the ability to add any arbitrary amounts of associated data to these alarms, because the values of the associated data will be attached to the alarm event as it moves through the alarming system. If you make the value dynamic by linking this associated data to another tag or a calculation, you can go back and see what that value was when the alarm went off. So for example, when the alarm goes off, you can go back and see what the room temperature was or what happened upstream or downstream?

Travis: Now, if you make that value static, the associated data won't change, and so it's a perfect way of using that for more contextual information on the alarm, especially for filtering. This gives you a very powerful way to provide filtering on just about anything that you want and to have many levels of filtering without requiring a specific naming convention for the tags. So, organizing is, like I said, very important, and there's different ways that we can go about it. Now, what I wanna do is show you a small example of how we can do filtering inside of Ignition. Keep in mind, I wanna focus on these two techniques: The tagged structure, as well as associated data that can be applied to any product that's out there. So I'm gonna open up the Ignition designer here, and if you look over on the left-hand side, I've got some tags configured here in Ignition. Now, for all of you who don't have any familiarity with Ignition, it's okay, you guys could certainly come back later and get demonstrations on how all this works. But in particular, let's look at the first way we can filter alarms, is through the hierarchy, the tag path.

Travis: So if you look here, I've got a couple of folders: Area A and area B. So I've got all my area A tags inside of this folder, all my area B tags inside this folder. So if I want to simply look at my alarms by area, I would be looking at all the alarms that are inside of this folder. So I've got here two machines, and I've got a couple of tags in each of those machines. Same thing with area B; I've got two machines and a couple of tags with alarms on them. I also have down here at the bottom, tanks. So I have two tanks with their level alarms, and that's another area that we can work with. So these folders, I can use on the screen for filtering, I can use it for how we notify people, I can notify different people based on these different folders. So I've got a simple situation here, I've got a drop-down on the screen, where I can look at area A alarms, so I'm looking at machine one and machine two. Look at area B. I'm seeing now, machine three and machine four. So of course, if something was to go active in there, so if I make all of these things happen at the same time, maybe make this one go to a high alarm, I can see that all of those alarms from that area are now in this list.

Travis: If I look at the tanks, I can see all the tanks that exist here. And so, we're just simply organizing our alarms in this folder structure, that we're taking advantage of that. So on screen, we can only show alarms for those areas. Now, of course, in real situations, we might have an alarm banner at the very bottom of our SCADA project, where we're... It's automatically filtering based on the area that we're in, as far as our navigation scheme. So you wouldn't be selecting from a drop-down list, you'd have this behind the scenes. But the idea is that we can use that hierarchy, that structure to our advantage. Now sometimes, though, that structure doesn't always work, like I mentioned earlier, sometimes we wanna be able to organize and show alarms by some other arbitrary grouping. And so, this is where associated data comes into play. If I look at a particular alarm that I have here in Ignition, we can add associated data to that alarm. So it's this little green plus icon, when we configure an alarm. Down the bottom, I get this category called associated data.

Travis: I've got two of them here, I've got Group and I've got Category. So I just kinda put two arbitrary things in there. Category, I have, this one's called faults. My temperature has a temperature category. My tank level is a level of category. Then I have a grouping; group A, group B, group C. So again, arbitrary. You can add whatever you want in this. And so we can use this data that is associated with that alarm for filtering as well. And if I look in particular at this tank two high alarms, I look at the details, I'll see that this information is category, and this group is stored along with that alarm. So as it moves to the system from the status to history, of course, notification, we know what those categories are, what that information is. So we can take that into consideration when we want to filter on that. So here's a screen where I have two different drop-down lists for those categories and for the groups. And so, I can... I'm showing everything right now, here's all the alarms in the system. If I look at just the faults, I can see just the machine faults. And if I want, I can go further by Group A, Group B, Group C. And so here, I think everything's just in Group A.

Travis: So it'll go back to all. If I go to temperatures, I can see the temperatures. Again, we can filter that and the group. If I go to tank levels, I can look at tank levels. And this one, I can see group A, it'll have just one tank, group B would have... Oops. Group B would have the other tank there. So again, we can do any combinations of these as we want, and we can have more than one of these selected at any given time. So here's a screen, a different way of being able to filter. I have the categories over here and I can select the ones I wanna see. If I see faults, I'm gonna see all the faults. If I show temperatures, I'll see that and the temperatures. If I see that and the tank levels, again, now I am able to show the alarms that I want based on this arbitrary associated data. So, utilizing the structure, utilizing the associated data, we can really achieve any combination with those and it allows us to organize and to present things to an operator that's gonna make sense for them. We don't just want a list of 500 alarms that a person has to sift through to determine what's important and what's not.

Travis: Okay, so that brings us to the next topic which is priority. And so on that last screen, let me go back to the screen, Ignition. You could see this was probably an example of a bad use of the priority. Everything here is a high or a critical alarm, and that might not be the situation in the real world. So we want to use the priority to determine how we're going to present this information on screen. So, in the handbook, in a lot of applications, there's five priority levels. There's diagnostic, there's low, medium, high, and critical alarms. We have a lot of customers who wanna be able to add or adjust this list and add their own custom things, and they're missing the point of why we have the priority. It's not to be able to just show any tag, any value you want on a screen, it's to be able to categorize these so we understand what we're dealing with. For example, diagnostic alarms, we would typically use diagnostic for things that are troubleshooting or event based tracking that aren't gonna be things we're gonna notify or have any corrective action on.

Travis: But then we have low to critical, and this priority really affects how the alarms system is going to work with it, and how potentially that alarm is gonna be escalated. Most of your alarms should be at least... Should be set at a lower priority. The alarm management handbook says that only 20% of the alarms in the system should be at a high or emergency level. Think about reducing the alarms, if the alarm doesn't require response or action, then it should probably be diagnostic. So, let's take a look at Ignition here one more time, and as the example, for the priority. So I have it on screen here where I just wanted to show how we look at on the screen, how we can potentially filter on that. We can use this priority to our advantage, as well as, of course, we do notification. So I've got this alarm. Right now, if I look at the configuration of the alarm, the priority of it is diagnostics. You clear the screen, it's a different color, it's gray, and I know that this is that particular kind of alarm. Now if I was to make this the low alarm.

Travis: And let me make it go active again. We're gonna see that now it's yellow. And so I've actually configured this so that every state has a different color on the screen, so we can easily see and categorize what are these alarms in those different states. Furthermore, If I had multiple of them, you can prioritize or order what's shown on the list by that priority. And so, we can see the critical ones at the top followed by the high, followed by medium or low. And we may want to get rid of our diagnostic alarm. And there's the ability here to actually specify a minimum priority or you can look at a particular priority when you're filtering that. Again, if I go now to the alarm and I make it a medium alarm, in my particular case, it's gonna be an orange. And if I go to a high alarm, then it's going to be red. And if I go to critical, which... It's very common that you really want to get the person's attention when it gets to that level. And that in this case, I actually have it blinking and on the screen, so I know that's critical, and of course, the priority shows in the list.

Travis: Now, like I said, we use it for filtering, we can use it for how we display things, and it's really up to us on how we want to configure what those states look like. This is just a view of all those styles that we have configured here. So, priority is very important. Again, if I go back to the diagnostic for all the things you don't want corrective actions on, there'll be diagnostic. And by default, you can have it show on the list, but a lot of times, you want your minimum priority to be at least low. So you do at least low, those won't show on the list. It's not going to confuse an operator as to that they need to potentially do something with that. Furthermore, one thing that you should definitely take advantage of is the fact that we can add notes to the alarm. So if it's something that... If it is a critical alarm, you should have, "Here are the steps to the corrective action that we want," as the notes. And so, when that alarm becomes active, we're gonna see it in the list, and they should be able to see clearly what those steps are, how we could potentially handle that, or even open up a machine manual or a actual PDF that describes what we should be doing to figure out how to solve that problem or at least what to do with it.

Travis: Okay, so let's look at the next concept here, which is all really around, now, we've got a bunch of these alarms configured, we've got priority set in these alarms, we've got them organized well, but we still may have a whole lot of alarms happening at the same time or upstream or downstream events can affect other machines that cause these alarms. So, how do we deal with these things outside of the fact that we've organized them and to reduce some of those issues, alarm floods and chattering and the issues that we talked about earlier. So, one of those things is shelving. And shelving is an important way to temporarily silence an alarm. So an example, you can get a alarm notification while you're dealing with a bigger and more urgent problem, you may want to temporarily silence a set of alarms so that it's like you're pressing a snooze button in your SCADA system, so that we're not gonna be notified of those 'cause we know we're doing maintenance on the machine or something's happening that's gonna cause these alarms to occur. We don't want people to be notified unnecessarily, and we also don't want things on the screen to show there's an issue, if there's really not a problem.

Travis: So, shelving is a great way to do that, and it should very much be temporary, it's not used as an indefinite or long-term solution. There needs to be expiration on it. Now, in contrast, we should... For certain other alarms, we need to be able to disable those alarms. And this is really to combat nuisance alarms. So, disabling is another method which is stopping an alarm from being evaluated so it won't generate that alarm event and won't go into the system. This can be used for those nuisance alarms, which are alarms that are unnecessary and enunciate excessively. So for example, a machine that is intentionally switched off because it's still generating alarms, then those are just nuisance alarms. So a common type of nuisance alarms are also upstream and downstream effects. Now, this is when an alarm on a machine sets off an alarm for another machine, which is up or downstream from it. So we wanna be able to deal with those correctly.

Travis: An alarm should be disabled if a machine is intentionally turned off. For example, it's night time and the machine only runs during the day, we should just disable the alarms. In state-based alarming, also known as alarm flood suppression, alarm settings are dynamically adjusted to match their proper settings for each state. So there are a few caveats though, before we implement this kinda system. You should first assess whether your process is a good candidate for it because we are disabling the alarms automatically based on these conditions. It's very effective for batch or a semi-batch processes and for processes that normally contain variables in different sets. Also, it's not intended as a long term or an indefinite solution as well for these nuisance alarms.

Travis: Now, let me show you an example of both of these in Ignition, and I'll be a little more clear about what each of them are doing. So, let's go back into the designer here. And we're gonna go to a screen that is all about shelving and getting rid of these nuisance alarms. So here, I've got an alarm that's a ramp right now. It's just a simulated value that's going from zero to 100. And after it goes above 60, the alarm comes in. So you'll see every so often this alarm is gonna continue occurring. It's active and it's cleared. It's active and it's cleared. It's active and it's cleared. And one thing is, is that we might be doing some maintenance and I just don't want this to happen. And so, we can actually go in here, so when it goes back to being active, I can click on that alarm and I can actually shelve it four to five minutes.

Travis: So that means that now, for five minutes, or for some time period that we wanna select, that alarm is not gonna actually generate an event. As you can see, it's not coming back in the list, it's now shelved. If I look at this button here, I can see that it is shelved for that five minutes. So it's a way to... For a certain subset of alarms, when you want to, to temporarily disable those. And you can see that when we do have it, the time period wasn't like... You can't set a year because that just defeats the purpose of the shelving. It should be a short amount of time, that makes sense. Maybe 5 minutes, 15 minutes, an hour. So it's a way to get it out of your mind just for that little bit of time. If I un-shelve it, of course, if it's active it'll come back in the list, and I will see it right back in here. Okay, so that is one of the things that we can do. Now, there's also the ability, like we have chattering. So temperatures that are going above the threshold barely but only for a brief amount of time and then going right back. So, when you're right in the cusp of that threshold of the alarm and you have your state going in and out of it, it could potentially cause a lot of alarms. And we don't want it to be an alarm unless it's actually in that state for a certain amount of time.

Travis: So, there's a setting that a lot of alarm systems have, and in particular on this alarm, this chatter alarm, we have an active delay and a clear delay. So it's a number of seconds that we are going to wait to make sure that that alarm is in that state. In this case, I've got it to 10 seconds. And same thing with the clear, a number of seconds we're gonna wait to make sure that this alarm is actually clear. So this is a really important thing because, again, if we just go above that threshold and come right back. It's not really an alarm unless we stay there for a little bit of time. And so, to illustrate this I've got a tag, just a memory tag. The threshold's 90. So when it gets above 90. Right now, it's 89, no big deal. So if I go to 91, then you'll that it's not actually gonna come up in this banner until 10 seconds later. So, 10 seconds later, it's gonna be then considered an actual alarm event and we're gonna see chatter alarm show up on the list.

Travis: Now, let me go back to being cleared here, I'm gonna go back to 89. And again, you're gonna see it's gonna wait 10 seconds for that to consider that it's actually cleared. So just a moment here, we'll see that, now it's gone. Now, again, if I go to 95 and I went across the threshold, and I go back to 89 within that 10 seconds, it's not gonna be an alarm event that's gonna go off. You're not gonna see it show up in the list here because we're going to, again, filter that nuisance out. So those kinds of things are very common and we need to be able to take advantage of. And we gotta, a lot of times, do that on a per tag basis. Now the last one is the ability to enable or disable alarms based on other conditions or other events that are happening out there. So, here, for example, I've got an alarm on this downstream machine. So it's called downstream alarm. And basically, if I look at the alarming configuration, the enable property, whether or not this alarm is enabled is based on a calculation or an expression.

Travis: In this particular case, it is looking at whether or not the upstream event is... If it's equal to zero on the upstream, then this alarm will be active. If the upstream event is a one, then this alarm will be disabled because that was actually happening before this and we don't want things to show up unnecessarily because we know there's a triggering effect that happens that kinda filters down everything. And so, if I look at this, again, when it's equal to zero, this thing's active. So if I make this go true right now, I'll see it come right into the list. Downstream alarm is there because, again, this upstream is not the one. So this is now an alarm of just that downstream one. But if I make this go... The upstream event go true, and I try to make this downstream alarm go true, you can see that that's not coming into the list now because that other event happened and it's disabled that downstream alarm from even occurring. That event will not show up in the system until that condition was to go away.

Travis: So if I make that condition go away, well of course, now it's gonna come into the list because it is active. Now this is, again, memory tags here but it's very important to be able to do this, and on certain alarms to be able to conditionally set what we want that thing to be enabled or disabled because we know there are things that can affect it. Okay. So we've looked at how we can organize alarms, we've looked at the priority, we've looked at how we can deal with nuisance alarms or temporarily silencing alarms, and then it comes to notification. And we still might have lots of alarms that are gonna be sent out, that we can notify people on. And I've been in situations where an operator gets 1000 or 100 text messages every 10 minutes with these alarms and he just pretty much ignores them, 'cause he can't deal with all the messages that are happening.

Travis: So, this next topic is about consolidation. It's about being able to put multiple alarm notifications into a single message and this can help reduce alarm floods. A lot of alarms happen at one point, we wanna be able to prioritize and send out these messages that are important. So for example, if 10 alarms go off at the same time, we wanna get one message sent to the user with all 10 of those notifications instead of 10 separate notifications. This makes it more manageable for the operator who's getting the alarm notifications and it's incredibly important if there are a 1000 alarms that can go off at the same time. So, I wanna explain how typically systems work with this kind of consolidation and behind the scenes, how this is actually... How does the system know to consolidate, how does it know how to actually deliver that to a person? So, there's two settings typically in alarming systems: There's the delay and a frequency, and these are used together to ultimately determine when we're gonna send a message out.

Travis: The delay is the amount of time that we're gonna wait. When the alarm comes in, we're gonna wait that amount of time for another alarm to come in behind it. So if we look at this top one, my delay is 15 seconds in this case. Message one came in, and if nothing comes in after message one for 15 seconds, which would be this gray bar, then we would send a message out. But in this case, you can see message two came in just a few seconds after message one, it reset the clock, and now it's 15 seconds after that. So at this point, nothing came in for 15 seconds, so the system said, "Okay, I'm gonna deliver a notification containing both message one and message two." So, that's one example. Now, there are times where messages can keep coming in consistently and we would have to indefinitely reset that clock or continue waiting for more to come in and we never get a message sent out. That would be bad. So, frequency is the longest amount of time that we're gonna wait before a message is gonna get sent out.

Travis: And in my case, I have it set to 60 seconds. As you can see, message one comes in, message two comes in. We reset the clock, message three comes in. Again, reset the clock. If things keep going in, 60 seconds later it says, "Okay, no more. I can't wait. Let's send a message out with all five of those at the same time." So, it can be both for a burst amount of alarms coming at the same time and to handle a lot of messages that are just happening in sequence that are spaced apart at some delay, we can handle both of those scenarios for this consolidation. And I thought I would actually demonstrate this in a real scenario for you. I actually have the ability here to... If I look at my system, I've got this whole folder of consolidation alarms and I can actually press this toggle which would make both alarms happen at the same time. And when that goes on, it's gonna go into, what at Ignition we call, a pipeline. And pipeline is how we're gonna handle who is going to receive that alarm.

Travis: And so, this is a simple pipeline. This is gonna alarm us when it comes in, this notification block. The notification block though has consolidation on it. So, it's turned on with a delay of 15 seconds and a frequency of 60 seconds. And so, I can illustrate this by showing... First, I'm gonna make alarms happen and we're gonna see that it's throttled waiting for the 15 seconds. Then it's gonna notify... It's gonna email a user. Then, the second time, I'll make it happen and then just a few seconds later, I'll make another alarm occur which will make it wait from that point. So, 15 seconds after it. So, let's go back here. The way that you can show this is I've got Ignition setting to two email addresses, or in this case, ioperator1@mailinator. So, I've got this inbox ready to see the alarms. Let's go ahead and make this... Toggle this to make multiple alarms happen at the same time. Now multiple alarms have happened. If I look at the pipeline, they're all throttled right now and you can see the time in the block is this time here, 10 seconds now.

Travis: And when it goes to 15 seconds, at that point nothing's come in, so it's gonna notify those people. And again, it's gonna be one consolidated message. So when I look at this, I get the alarm notification here and I get a consolidated message that three alarm events have occurred and here they are. Now, of course, we can configure out how we want this message to actually look, but the idea is I've got a consolidated message. Alright. So, that was the first scenario. I just waited 15 seconds and nothing came in, so we handled it, we've sent it out. Now, let me make it go back to clear. Let's try it one more time. So, we're gonna see that now they're in there throttled and I'll wait a couple of seconds, to 15 seconds, but I'm gonna go now and make another alarm occur. And if I go back here, you're gonna see now another alarm has occurred and these ones have gone past 15 seconds that they've been in there, so not gonna notify yet because we're waiting for this one now, see if anything comes in at this point for that time period.

Travis: Now, 15 seconds are gonna come up and all of those are gonna be notified here. And we're gonna see a message and another message with four alarm events have occurred, including that fourth one. And this can go on, again, until that max frequency of 60 seconds. I don't wanna necessarily do that. It's a long time to wait, but you get the idea that that's an important way to be able to just send out these messages. And of course, we can make it so that they can acknowledge and we can acknowledge all of those alarms at once, and we do consolidated messages in Ignition's case. Okay. So, the last scenario that I wanna talk about for notification is escalation. And the example I showed earlier is just consolidated messages, but we just sent the message out once. And if we want a corrective action on that alarm, we want somebody to do something with it, it's in our best interest to make sure that somebody's gonna handle it, that we in the notification system notify different sets of people and continue to escalate to different people until something actually happens. We want that action to be there. And so, escalation becomes very important for this.

Travis: We can escalate it by any type of notification, an email, SMS or voice. We can also send it to different list of people. And basically, instead of sending the same type of notification to the same person over and over, we can start doing different actions until that alarm is either... Well, typically, alarm's acknowledged. Even if it becomes clear, we still may want that person to acknowledge the fact this thing happened. Ultimately, we want to have that corrective action take place. So in Ignition, we handle escalation through these pipelines. I already showed it with the consolidation, but that pipeline I showed was very, very simple. The pipelines are basically a graphical design feature for quickly building the logic for handling these alarms. We can drag and drop different notification scenarios from simple to sophisticated. We, at Ignition, realize that there are many different possibilities for how you wanna notify people, and instead of just giving you a few settings, we wanna give you the ultimate flexibility with this where you could hook up alarms to different people, with different schedules, different rosters, notification profiles, types, all that into one. And so, these pipelines become very important.

Travis: So, let me show you an example of the escalation here. And I've got a pipeline set up that is a little bit more detailed, as you can see. So, these blocks allow us to do different things. And so, initially, I'm gonna come in and set a variable to num sent to one. I'm gonna notify the operators through email that these things have happened, and then after that, I'm gonna wait five seconds. I'm gonna increment my num sent by one and I'm gonna check to see if that is less than or equal to three. And if it is, I'm gonna go and notify them again. So, I'm actually gonna notify them a few times with a five-second delay in between or whatever delay you want, especially if you're calling people. And then, if after three times, they don't handle it, they don't acknowledge it, we're going to email supervisors in this particular case. So, here's a way to show that, is I wanna show the actual look at the pipelines. And again, I have the operator and supervisor. That's why I have the supervisor email over here.

Travis: So, let me go and make this alarm go to true. And now it's in the pipeline. And so, we're gonna notify the operator, as you see, the operator got notified, there's one, two, and then one more is gonna come in here. And after that, five seconds later it's gonna then go to the supervisor. So it's three and that one. Now, the supervisor is gonna get notification that happened. And again, we could... At this point it drops out, but we can continue doing that over and over again, until actually somebody handles that alarm. So it's a very good way for you to more or less guarantee that these things are going to stay into a system and every alarm that happens is gonna be independently running in their own pipeline until something happens. I also wanna mention that there's one other feature in Ignition, we have a lot of... I know this is best practices of alarming in general, I don't wanna... Ignition has a lot of these things built in, but we're now seeing customers who are going to enterprise solutions, who are looking at alarming across multiple locations, have a lot of remote locations, and it comes down to, do I have the ability to notify somebody at a particular location through email, SMS, or voice? And sometimes we don't.

Travis: And so, there is the ability for us to send out alarms remotely from Ignition to Ignition or we can send it to a system, a central system that does have the email, voice or SMS equipment that we can do that. So these pipelines, we can connect to multiple Ignition services together with these pipelines, and it's perfect for these kinds of enterprise scenarios. I just wanted to mention that here, and I'm not gonna show a demonstration of that, I just wanna give you a sense that that is a possibility within Ignition. Alright, so before we wrap up and go to the Q&A, I'm gonna leave you with two things the alarm management handbook recommends you do to maintain the improvements that you've made. And that is an alarm philosophy document and a master alarm database. So after you've thoroughly reviewed everything related to your alarming in your organization, you create an alarm philosophy document. It will serve as a comprehensive guideline of how an alarm should be developed, implemented, prioritized, monitored, handled and modified.

Travis: It should be the go-to document for every alarm topic and situation for anyone in the organization, not just those who are familiar with the alarming. It's all too common a scenario where the initial developer doing the project has really put thought into how they're gonna organize and how they're gonna work with these alarms and they get a system in place. And as we continue adding more things to it, more alarms are getting added from other people and they're not following the same philosophy. And now our alarms are gonna go out of control. We're gonna get into the same situation that we potentially were at the very beginning. So, it's important that everybody who's working on the system is under the same guise of that philosophy and understanding what they're doing. Now, that's one part of that, that's just how we set things up. But how do we even know what these things... Or what priorities should be or who should we notify and what should it all really work with? And that's gonna be working with operators working with people in your organization, ultimately creating a master alarm database. I also commonly refer to alarm rationalization, where we are rationalizing why we do certain things with that alarm, what the corrective action should be? Should the priority be critical? Should it be low?

Travis: Should we notify them? Should there be escalation on that? All these kinds of things, we gotta rationalize that. And that's gonna be done on a per-alarm basis. So creating this document that looks at all of that is very important as well. So, whenever we make any changes to the system, that should be documented, it should be... We should rationalize things, we should have communicated to everybody and approve just as we would for any changed or actual operational process when it comes into putting new equipment or new things in place. I can't stress how important this is. If you do it, and you do it well, you'll have a much better maintained, and a system that people can rely on rather than just being a nuisance and they just ignore it all. And so, last but not least, you should be continuously auditing your system overall, and making sure that things are set up the way that they should be, because guess what? There's things that happen out of our control, equipment starts to die, things start happening that we should be looking at why are these things happening a lot? And we'll come into looking at KPIs alarm system and seeing how things have been logged. Alarm history is very important to coming back to see what's been going on and what are our big actors of our alarms.

Travis: So in a recap here, for things that we've discussed today, think about alarms and manage them carefully. Alarms can get out of control and it can cause serious problems. It's always best to set up your alarms the right way from the start. Alarms should be easy to identify and should signify a real problem that requires action. Avoid unnecessary proliferation of alarms in your system. And it's very easy to take existing systems with tags, I realize it'll be a lot of work to go through if they've already been configured to go and then apply these philosophies to it, but it is easy to export typically to an export file. We can look at that and outside of a product by going for a tag, and then we can determine how that should be, change this file, import those alarms back in, so we can get to that state very quickly. We wanna think about organizing our alarms; Focus on the alarms that you need to see in your potential local area or how you want to filter or what hierarchy you need to use for those. Prioritize alarms, make sure that no more than 20% of your alarms are set at a higher emergency level, critical.

Travis: If you have alarms that don't require a corrective action, consider it as diagnostic. Know how and when to shelve alarms. You can deal with important or somewhat important alarms later by temporally silencing it. Disable nuisance alarms such as chattering or upstream/downstream effects alarms when necessary. Consolidate the alarms to cut down on alarm floods. Put multiple notices within one message. Utilize schedules to ensure the notifications are only sent to team members who are currently on shift. Use escalation to ensure that alarms are properly responded to without sending them to everybody at once. And lastly, maintain your system for the future with the alarm philosophy document and the alarm rationalization that you can do. So, that we're gonna... I'm gonna send you back to Don when we will be going into the Q&A.

Don: Travis, thank you. That was great, I really appreciate it. I just wanna... As a takeaway... I know you had an opportunity here to see a lot of different aspects of alarm philosophy to the documentation and a few demos along the way. So, just a takeaway I'd like to emphasize is, really, that Ignition... If you take a look at it, it provides a fully robust and capable alarming system, has everything you need to implement all of the best practices we've discussed. It includes drag-and-drop alarm notification pipeline builder, which you saw, it makes it easy to develop advanced notification systems. When they're coming, you can receive and acknowledge alarm notifications being the email, voice, text. It offers dynamic alarm customization SQL database, support and comprehensive analytics and reports, and it ain't that expensive. So, I'm saying that to you because you saw it on the demos today but if you're gonna check out Ignition, you can. You can download it and put it in trial mode for two hours, and look at anything you want. Please, take a look at that. If you want a further demo, more detail of how alarming might work for your system, then please don't hesitate to give us that chance to do that demo for you.

Don: One more thing I wanna say is this was a Design Like a Pro Series event focused on alarming. If you'd like more Design Like a Pro tips on other subjects, we have a whole series of these webinars and white papers on our website. Design Like a Pro is all about just really... What we're trying to do is provide you fundamental ideas and tips and best practices to help you build the best HMI, SCADA, MES and IoT projects for your organization. So, we have content on a lot of topics; alarming today, also launching projects, HMI optimization, templates, reporting, dashboards and really, the list goes on. So, with that, we're gonna move into the Q&A. I will, basically, just go as many questions as we can, Travis. We've got a pretty long queue here, so I'll ask 'em and you can answer 'em, and then we'll just take it from there. First question, would it be possible to link an alarm to a page so that if the operator clicked on the alarm, it'd go automatically to the page where the element is in place, where the element is placed?

Travis: That is a great question and the answer to that is, in line with one of the topics we talked about today, when it comes to that associated data that I mentioned earlier, we're looking alarms on a particular screen, we're filtering on that associated data, that associated data is not be used just for filtering, it's any contextual information you want to add to an alarm. I've seen it in many situations where we can specify a file name that we wanna open up, specify the window we want to be able to go to within the system, and all that information is given to us. When I click an alarm, I can have a right-click event or I can double-click on the alarm and it can do something in the product, again, looking at that associated data. So, that's commonly how we do it, is adding information to an alarm that we can then use on the screen when it's active to know where to potentially go or what to do at that point. So, they can be used for a lot of different things, not only for just statically doing things I mentioned, but also for dynamic data that will give us information as to what happened... What's associated with that event at that time that it occurred. So, associated data is the most powerful thing and tool you can have with any alarming system.

Don: That's great, Travis. Here's another question: How do you set it up so that users can pick and choose what alarms they receive or at least give them power to opt in and out for alarms?

Travis: So, that's a good question. With Ignition, we are notifying people based on a roster, which is a group of people that are gonna potentially receive that alarm. Typically, rosters are set up ahead of time. And so the user is not opting in or out of the particular alarm, they're just set to a set of... "I'm an operator, so I'm gonna get these sets of alarms," which isn't always the thing you want. There is the ability... I'm not gonna go into too much detail here, but there is the ability for us within the notification system to determine the actual people that we wanna send it to from a database, where we can define... A user can opt in or out of individual alarms and store that into a SQL database, and with the new notification block, there's a new type in 794 for us to bring back users in scripting from a database, so we can identify much more freely how he wants to notify people and utilize something else than the roster system that's built into Ignition. So, it's a good technique that you could potentially use.

Don: Cool, thanks. The next question is, just says: One, alarm latching and two, alarms acknowledging.

Travis: So, alarm latching and... For us to consider a value to be an alarm, this is what I talked about at the beginning. When the alarm is clear and then it moves into the active states, that rising edge is the latch of the alarm. It's when the alarm is gonna be active. When it goes out of that state, it goes back to clear. Of course, now, it's cleared and we have to go clear back to active before we consider it to be active again. And so, that's when we... So, when we see the alarm actually occur, then when the alarm is on our screen, we can certainly acknowledge it from here. So, like this one here, I can acknowledge. We can also force them to put in notes when they acknowledge it. So, we have some... But why did they do that and ask for logs in the system?

Travis: And lastly, we can do alarm acknowledgement through the notification system. So, email, SMS and voice are all two-way in Ignition. We can acknowledge them from those sources. You can send an email back, you can reply to the message, you can send a text message back and you can also put a user PIN in there. So, they're required to type in a PIN, so they can't just arbitrarily acknowledge something. And also, we can do it through voice. We can call them on their phones and say, "Hey, these alarms are occurring. Do you wanna press one to acknowledge?" All those things can happen. So we can do it in a number of places, but ultimately, that one alarm backend system will become acknowledged and will get tracked. There is an audit trail or alarm history with all of this. I'll just mention it since it's applicable here, and I'll just bring it on the screen. This is all the alarm events that we've had happen on this server so far and it's going to the database. We could filter on that and look at what's going on, look at KPIs of that system very easily. And in particular, with the KPIs, we have a built-in template that people can use in the system called Cloud Templates. It's called Alarm Analysis.

Travis: You can bring in this template, and it'll show us these KPIs. So here we can see alarms by frequency and by duration, by out of the day and the various KPIs are average time to clear, to acknowledge. All the information stored is in databases. The screen, just looking at it, and it automatically will work. So, a few things, little teasers and things you can do in Ignition to help really continue to make this better.

Don: That's great Travis, thanks. The next question is, does notification require a specific Ignition module to be installed?

Travis: Yes, the alarming is part of the ignition core platform, so configure alarms and doing history of that is actually just a part of platform, and you don't need a module for that functionality. The notification itself does require a module, and that's the alarm notification module, which gives you the pipelines and emails given for free in that module. There's also two additional modules: SMS and voice, where we can then notify to those systems. So, if you want best of all of it, you should get those three modules and you can also... You can use it standalone or it could be a course with everything else in Ignition.

Don: Good, thanks. And you've got one here about do you recommend hardwired alarm annunciator windows for most critical alarms?

Travis: So, I mean that hardwired is... POTS lines, of course, there's always the doing our voice calls. People would prefer to have those because it's local and they know it's gonna work. Never relying on Internet connections or anything like that, but nowadays, VoIP is pretty common and people are getting a lot of stability with that. So, we certainly do recommend having something on site. There are companies that are choosing to use Skype for business or Twilio for SMS notification, and that does require Internet connections and potential for, if your connection goes down, then your notifications aren't going out. If you do have that, maybe have a backup or something, so that you have the best of both worlds. So there's some redundancy in place, but you know that those alarms are gonna get out to the right people.

Don: Thanks Travis. We've got time for just maybe a couple of more questions. One is, are they compatible with other gated systems and standards currently in use?

Travis: So, I mean, that's a good question. We currently are not using OPC HDA... Sorry, OPC alarms and events, that particular standard. We are doing everything at Ignition ourselves here and we are exposing... We can expose our alarm states and we can expose the history of alarms of course goes to the database, so any other third party application can get our system, but we do plan on eventually here supporting OPC A&E. So that there's a standard way of being able to talk to us and get the alarms that we have. We've always kinda standardized in the fact that we use databases, now you can get it from there. But we will be using more standards. Currently, we are following all the alarm management handbook standards, but not with delivering it to the third party.

Don: Okay. Okay, so maybe a final question, can you export the database group of alarms for a second machine and then import the new alarms? What tools are available for rapid alarm creation?

Travis: So, there is an important export of tags inside of Ignition. If I take this tag here for example, the state tag, which has... I'll do the temperature tag here. This actually has two different alarm conditions on it. I can export one tag or all of my tags, doesn't really matter, but if I export this to my desktop and I do it as an XML file. There, let me open it up here, and I'll bring it over. It's on a different monitor, real quick, here we go. Okay, hold on, here it is. This is what the XML looks like, and so, we can see the alarms are configured as a block of alarms. Each alarm you can specify, you've got multiples per tag and all the conditions, all the settings, all the properties are in here, including the associate data. So all we gotta do is basically create this kinda file.

Travis: You can generate this file through a simple little script that we can write, we can generate it a lot of different ways. We've also written conversions from other CSV files to our format. All we gotta do is just get into this format, it's pretty easy. Once we have it, if I wanna make a change, say for example, that I want this to be a setpoint of 60 instead of 90, I can make that change. I want this to be Group D instead. I can save all that. I can come back in here, I can import this in. This is just that one tag, but if I look at temperature here, it's gonna overwrite that. You'll see that high is now 60 and the group D is down there, so easy to import and export in Ignition.

Don: That's great, Travis. Now we're at the end of our time here. We'll go back to the screen for contact information so that as I just kinda wrap up, I just wanna thank you for getting to so many questions. We didn't get to all the questions, which is why I want the screen up here. So, you have Travis Cox as the contact there. Melanie as Director of Sales and all the account executives, if you

For our white paper on SCADA alarming, click here.