Inductive Automation Blog

Which new innovations will prove vital for future success and which flash-in-the-pan trends are destined to be forgotten by ICC 2024? During this panel discussion, some of the Ignition community's most successful integration professionals share how they are responding to emerging technologies and techniques that are driving the evolution of the automation landscape. 

Transcript:

00:06
Chris Fischer: Nathan de Hoog is a Senior Automation Engineer at Cromarty with 10 years experience in industrial automation and application development within a wide range of industries, including water/wastewater facilities, food and beverage, gas utilities, manufacturing, and renewables. Over the last six years, Nathan has led Cromarty's renewables team to deliver many solutions powered by Ignition, including enterprise and local SCADA systems, containerized and cloud-hosted deployments with Git and version control, API development and integration, historians, database and data warehouse systems, MQTT, and edge connectivity to over 35%, 9 gigawatts, of the renewable assets in Australia. Welcome Nathan.

00:57
Chris Fischer: Sydney Bosworth is an MES Programmer at NorthWind Technical Services. Sydney has been with NorthWind Technical Services for seven years. She started on the service team, troubleshooting customer systems and programming small projects, then moved into the engineering department, programming and commissioning control systems. While in these positions, she developed her knowledge and experience with a variety of processes as well as a variety of software packages, including Ignition Vision and Perspective. From there, she went on to become the Primary Standards and Templates Programmer, where she was responsible for maintaining and developing new templates for NorthWind and OEM partners. She's now bringing her experience to their MES team to develop their MES offering using Ignition Perspective. Welcome, Sydney.

01:49
Chris Fischer: Alright, so let's get into some questions. Let's go, start going right down the line here with you, Jason. What is currently your biggest challenge, period?

02:02
Jason Hamlin: That you wouldn't let me have whiskey on stage. No. Recruiting people. Recruiting talent right now, that's our biggest challenge. We're water industry, and our pay scales don't always match or align with others, so it's been... We've lost some good people to other industries, and yeah, I think the biggest challenge right now for us is recruitment.

02:29
Elizabeth Hill Reed: I would say, in my opinion, our biggest challenge right now is maintaining high quality standards as our Ignition team quickly grows. I think we had 30 to 50 people on board this summer, and so trying to keep the high quality of finished product while also growing our Ignition expertise across the company.

03:00
Barbara White: My biggest challenge is finding time to get to all the work that my customers want done right now. We're overwhelmed with work, it seems, and you don't wanna hire too many people because you don't know how long this is going to continue. So it's kind of a balancing act at this point.

03:20
Sydney Bosworth: I would say, or is this kind of finding the balance of we would like... There's a lot of new ventures we would like to get into and stuff we'd like to develop. We have a fairly large, concentrated R&D team, but I think there's a lot of things we want to get into. And so the MES stuff, we're finding the best way to break into some newer things and deciding which technology to do, like cloud stuff, and putting the adequate resources towards developing that part, but then not getting distracted from the customer jobs and finding that balance between R&D and customer work, and that's a fun game.

03:54
Nathan de Hoog: We're similar as well, like Jason said, resourcing. There's a lot of work with limited resources that we have, so that is our biggest challenge. But like you said, the R&D and finding time to develop and utilize these new technologies and features that are Ignition-ready.

04:22
Sean McFarlane: I got the same sentiment, recruiting the right talent. Sure, a lot of people in this room, as long as everybody on the panel we gotta wear a lot of hats as integrators, recruiting the right skill set, the right attitude for this kind of work is definitely a challenge. We're also, we consider ourselves platform agnostic, with a big caveat that everywhere I go, I'm trying to get them to switch over to Ignition, and it's a battle sometimes trying to force out some of those really entrenched software platforms that everybody here knows, but we're getting there.

05:00
Chris Fischer: I'm hearing a little bit of a theme with talent and recruiting. Let's linger there for a second. What's your strategy to attract new talent to your company and the industry as a whole, anybody.

05:12
Nathan de Hoog: Yeah, we try and have a tie with some universities or colleges; I think they're referred to here. In Australia, we have several offices across multiple states, so bringing on students that are doing fine on your projects and giving them an opportunity to have a bit of real-world experience, that helps us and helps them as well.

05:42
Elizabeth Hill Reed: Yeah. Similar, DMC hires a lot out of colleges as well, and I think the biggest thing is finding people that like to learn. As system integrators, you're not only learning new platforms, but you're learning new processes. Be flexible with what industries you can support, and so people that can learn and learn quickly, I think, is really important with recruiting.

06:09
Barbara White: We've been hiring people. Generally, what we'll try to do this hire them as interns while they're still in school. So we can give them the tools to start. We throw Inductive University at 'em and say, "Hey, take a look at this for a little bit," and see how they do, and that gives you a chance to check number one, if they like working in control systems, and number two, if they're going to do well with that, and that's what we've done the last few years.

06:39
Jason Hamlin: Yeah. I was gonna say, I learned something from my former company too. We look at alternate resources, so we went and did a presentation at Job Corps, where they have kids learning IT track services, and now we're working out to intern some of them. And starting to look outside of the traditional methods of where we're finding people is bringing us a pretty wide variety, which is awesome.

07:03
Sydney Bosworth: Yeah, we've had to kind of pivot to more of a long game. That's what I can call in it. We've gotten more of a concerted effort to go into the local. So I guess NorthWind is in a, I should kinda start this with NorthWind is in a very... Headquarters is in a very rural area in Kansas, so it's a town of about 2,500 people. So you take a kid from Kansas City, a larger city, and say, "Yeah, move to this town of 2,500 people," and it's a hard sell, unless like I'm from a very small town. Those are the easiest ones to get fully integrated into our company atmosphere and enjoy working there, and really the long term, like I'm gonna work there for two years and go find a job in the city. So we have a partnership with a college that comes... They do a automation camp in the summer at NorthWind, and we bring in local high school kids because they don't know what automation is. Most of the time, we don't have a ton of tech programs in the school, so we have to show them what it is. We bring them in, and we kinda teach them what we do when they're in high school, kinda trying to put the feelers out there very early, so you're talking more long game, so we're kind of trying that a little bit more here recently. And then, of course, the college fairs like everyone and trying to get a good company atmosphere going.

08:16
Sean McFarlane: We put in a golf simulator. So, I make sure everybody, when they come in to interview, we walk through that work from first, like... But we've had to change how we recruit. We're also partnering with local colleges. We started up a partnership program with them where we take in some of their students for three months, bring 'em into the panel shop so they get experience doing that. And if they have interest in software and IT, we can kinda go from there. But as far as recruiting, it's a different skill set that we need to recruit for now. It's not just PLC HMI systems; it's web technologies. A lot of kids in college these days learn Python, which is great, so we can immediately advertise that and say, "Hey, you can work in Python and you can build these great systems to control things," and that seems to work.

09:15
Chris Fischer: And since the time that you all started in the integration business, have customers pain points changed, or are they pretty much the same? Let's start with you, Sean.

09:25
Sean McFarlane: I think their pain points haven't changed much. A lot of what we hear is like yearly maintenance costs for the systems that they have. And some of the systems we come across, their costs are just astronomical, and it blows us away, like, I can't believe you guys are still doing this and you're tolerating this. We have such a better way of doing things now, and that's a big selling point for us, the licensing structure that you guys offer, it really helps us present to our customers a new way of doing things that's way more cost-effective, but that's been the same story ever since we started working with Ignition. It hasn't changed.

10:09
Nathan de Hoog: They have changed. That's the nature of the beast, and what we see a lot of is the number of companies are having less skilled staff and engineering staff. So they don't necessarily understand the systems that they need, if they're scoping them or wanting to change things. They have limited knowledge. For us as well, cyber security is a massive pain point that's changed over the last couple of years, and it's continuing to change. So, cyber security is a big one for us and for the clients that we serve.

10:48
Sydney Bosworth: It's a little hard for me to speak on it, 'cause relative to many people in here, I've been around for... I'm an infant, so about seven years in it. From my perspective, it's a little different too, 'cause I started out doing more boots on the ground start-up working directly with the operators, and now a little bit removed that from that, so I'm seeing different pain points. But I think one of the things I've noticed too is there's... Six years ago, when I would be on-site a lot, there'd be kind of... They use trends and all that, but I think now there's a lot of places that have a lot of data, and now they're like, "Okay, what do we do with all the data that we've spent the last five years? Like knowing we need it, but now we need to figure out what we're gonna do with it." So that's kind of the biggest observation I've seen, but haven't been around a long time.

11:36
Barbara White: I've been around a long time. A lot of my smaller customers, at least their problem is they don't have the talent in-house to do what they need to do, and some of them are trying to get it, but it's like us; it's difficult for them to hire somebody who understands controls and IT and can do all of it. Now, the bigger customers have different pain points, but the smaller ones, I think that's their biggest challenge: trying to know what to do. And when they bring in an integrator like me, then I try to point them in the right direction to what are some of the things they need to do to upgrade their systems and get more secure systems. But I think not having that in-house makes it a difficult sell for their management teams because they don't have somebody they trust to, that I am just not trying to sell them something. So it's sometimes a difficult sell to get them to upgrade the systems they need to upgrade, but...

12:40
Elizabeth Hill Reed: Similar to echo Sydney's point of... My first project was food and beverage. I think they had very basic reporting, no historical trending, and nowadays I did another food and beverage project about a year ago, and they have all this great data, very detailed data, and I swear I'm on calls with them, and I'm talking about this report, and they have no idea that this report exists. And I was like, "You guys asked for it. The data is there. Please use it." So having a lot of difficulty getting people to know the data exists and use it in meaningful ways.

13:23
Jason Hamlin: Of course, I'm in water, so we lag behind, which is beneficial 'cause the pain points we hit somebody else has already solved five years ago. So we just adopt that, but one of the biggest ones we see is a change in the class of operator or the class of the end user. I'm not judging anybody's age here at all, but when I started, the iPhone didn't exist, so the end user is using SCADA at the time. Compared to the end users we're seeing now that when you're recruiting people into the water sector and they're coming in, they're coming in knowing how to use technology in a way that never existed in an industry that's still really lagging. So pain points of getting automation still exists. Now they're struggling with recruitment 'cause you're bringing in people who are looking at, "What is this archaic stuff? Why can't I run this from my iPhone?" I literally had a customer ask me, "Well, could you gamify our system to keep our younger people engaged?" I'm like, "You want me to gamify water treatment? What happens if they don't high-score? Do people get sick?" So yeah, that's some of ours.

14:31
Chris Fischer: Just a little trophy.

14:32
Jason Hamlin: We'll give them little trophies. We could do that. Gamifying water treatment is the scariest request I ever heard.

14:41
Chris Fischer: Alright, so what industry trends are most relevant to your business today? Anybody.

14:48
Sean McFarlane: I think integrating the business systems with the plant floor is becoming extremely popular. Basically, every new customer that I talk to, that's one of their top requests. They don't wanna have separation between the automation systems and their ERP or their MES up above. So the tools that you guys give us in the guys from Sepasoft, those are opening up doors that just weren't there a couple of years ago.

15:20
Elizabeth Hill Reed: Yeah, and I think that people are becoming more comfortable with the cloud, so I think that's finally getting a little bit more traction and adoption that we're seeing.

15:32
Barbara White: My customers seem to want, they want their data and they want it wherever they are, so if I'm in the middle of the night, I get a text that says I have an alarm, I just wanna look at my phone. I don't have to go get my computer out or anything of that sort, so that's what they are asking for.

15:46
Chris Fischer: On demand.

15:47
Barbara White: Yep. On demand, where I'm at.

15:49
Jason Hamlin: Analytics for us, AI, and analytics in the water sector starts to really enhance the power of treatment and what the operators can do and then... So that's probably the biggest one. And then for us, personally as an integrator, more automated tools for converting and doing projects faster, those type of accelerators. Because we can't recruit people, utilizing somebody who can automate conversion or those type of high-level functioning people that can speed up some of the process of integrating is probably the biggest thing we would love.

16:21
Sydney Bosworth: Yeah, that's with us too is the, kind of seeing the centralizing everything, being able to connect plants that are physically the same or in the same location, and kinda being able to look at everything in one place and where you want it, when you want it, and then kinda in-house automation too. We're getting a little bit more automated, like our panel shop has automated systems in it, so where we need less panel builders and doing more with less people. It's pretty big everywhere, it seems like.

16:51
Nathan de Hoog: It's similar in renewables as well. It's been a lot of data collection at a wind farm or a solar farm, but that's never been... Data haven't really been used very well, and everyone now wants to use that data and see that data wherever they are, and provide meaningful KPIs with that data.

17:16
Chris Fischer: So thinking about the new and emerging technologies that we see today, whether it's containers, digital twins, AI, blockchain, whatever it may be, in five years, which ones do you think will still be relevant and which ones will be old news?

17:39
Sean McFarlane: I'm curious in what context, Jason, you've been able to get results with AI in actual deployments, or is it something that customers are still just feeling out, they wanna know what is available.

17:49
Jason Hamlin: We had some successful pilot studies. I'm trying to think of what I'm gonna get myself in trouble saying.

17:56
Sean McFarlane: I didn't wanna put you on a spot or anything.

17:57
Jason Hamlin: No, no, no, it's not. We have some successful pilot studies because we have massive amounts of data to draw from, so we're so focused in water/wastewater, so for instance, managing and running a wastewater plant. Some of those tests take five days to run, or you're gonna run a BOD test, it's gonna take five days in the lab to get that. We now have a company we're working with that is using machine learning to predict those loads in real time by harvesting other sensor data and doing it fairly accurately. So we're starting to see some of that. Chemical usage would be like the biggest one for us. And then some, like flow predictions on East Coast CSO plants. So now I'm just speaking in garbled terms nobody knows, combined sewer overflow. But anyway, we're seeing some minor successes on that.

18:47
Sean McFarlane: We've had some customers reach out to us, and they ask about AI and machine learning for doing predictive maintenance. And most of the time, my response is something along the lines of, do your maintenance techs actually want this? Is it something they're gonna utilize if we put it in for you? And it'll be a lot of trouble to implement something like that. But it's really the application that matters. If you sense that a motor is going bad, if they don't replace it until it fails, then it doesn't matter anyways. But if it breaks, that's when they replace it, so I'm curious to see how that technology develops. One that we're actively using and very excited about is containerization. We work with a lot of different customers. We have many different projects going on at once, being able to spin up a new container for Ignition. Shout out Kevin. I don't know if he's here. But the work that Kevin Collins did is so helpful for us as integrators. I mean, we use it all the time.

19:51
Barbara White: It makes spinning up development servers so much faster. Yeah.

19:57
Nathan de Hoog: Containerization helps us in our production systems to manage our deployments. It's a lot nicer to upgrade and to monitor those solutions. I can see AI... It's a hot topic now, but it's hard to see where it can go. I would love to be able to see it being used in the Ignition. A user, for example, wanting a dashboard on a wind turbine or an inverter or a motor, and they can type in a prompt, and it could build a little dashboard or widget. So I can imagine that being really powerful. I don't know whether that will ever happen, but...

20:45
Jason Hamlin: Talk to me after the session, 'cause I can't sales pitch here. Of course, wind turbines aren't water, so I don't really care about that problem.

20:55
Elizabeth Hill Reed: I don't know if this is in the next five years; might be a little bit longer, but I'm curious to see how like AR plays out, especially with operator training, work instructions, things like that. I think that has the potential to be pretty powerful. I think we're a ways away from that, but I'm curious.

21:17
Chris Fischer: All right. Well, thinking about the theme of this year's ICC, elevate, what's one way that Ignition helps you elevate the customer experience?

21:28
Jason Hamlin: I mean, I've been promoted to Cloud Services Manager. How much higher you want me to get?

21:42
Barbara White: Well, my customers, they can do more with what they have. I mean, in the past, if they wanted to add tags or servers or whatever, they want more data, they had to get more licenses or whatever. And right now, with Ignition the way things are, they can elevate and get as much as they need and as much as they want and get it in their hand right now. And that's what they're asking for.

22:10
Elizabeth Hill Reed: I think it really to build upon that idea; it expands the user base of it. It's not just a subset of people. You have five front-end client licenses, you can really get more visibility into the data. You can have the plant managers, like more of the business-side people, accounting, quality. They can all look at the SCADA system. People can have it up on their computer when they're in the office and just monitor it and so yeah, expanding that, I think, is huge.

22:42
Sydney Bosworth: Yeah, I think on the main MES drive we've been working on, it's been great 'cause there's a few there kind of in-house database people that we've given the LinkedIn Inductive University, and they've gone and watched the videos, and I think they've been playing around kind of with the reporting, and it gets them to buy into it a little bit more when it's something they can look at it, log in. And they have enough background; they can kinda look at the designer and start to figure it out and see the possibilities of it themselves. And not just rely on us saying, "Oh, did you know you could do this? Oh, you could do this." And it's just like, take your word for it. You can see what you can do and play around with it. And it's gotten them to be a lot more on board with it and easy to pick up and all that.

23:22
Jason Hamlin: My serious answer is gonna be helping end users that see how much more Ignition is. It's not just SCADA, and that's especially in the water industry. Again, there's a Firebrand Award here for the Room in the Inn where that whole project is... That's not SCADA at all. That's not control. That is amazing. And that's built on Ignition. And like, we have a water user who look at that, like, "Wait, that's the same SCADA software I have. Well, could we use that to manage employee timekeeping, and could we build something to like, check people coming in and out the door?" And I'm like, "Yeah, actually, Inductive does that at their office. Like, let me call someone and see if they'll send me that project." And they were like, "No." I'm like, "Okay, that's cool," but...

24:00
Jason Hamlin: That's fair. But when they understood, yeah. That elevates their experience 'cause they, suddenly, there's a whole platform of new software you don't have to buy and manage. We can build that or contract to somebody that knows that side of it better than we do. Partnerships, that's the other thing... Yeah, I should mention that too. The other thing that really helps us elevate is partnerships with other integrators. And we've worked with other integrators because they have skill sets and expertise that we don't, and like this community is where that exists. I've never seen that anywhere else. I won't name any competing product names, but I would never be able to like, share with one of their integrators to do a job. They think I'm crazy, but on the Ignition side, yeah. All day long, which is really cool.

24:46
Sean McFarlane: Yeah, I think Ignition's embracing of open technologies helps 'cause they don't... Customers don't have to stay within the Ignition ecosystem, even if I really badly want them to. They can use other tools to access the same data, and the whole philosophy that other companies have of having like a walled garden just doesn't exist anymore when we give a customer access to Ignition's powers. And then I mentioned it earlier too, the maintenance costs, they get really prohibitive with other software packages, and our customers are very, very happy when they see the numbers for a competing Ignition system.

25:28
Chris Fischer: Yeah. Before we get into a little bit of Q&A, I want to hear what's a lesson or a success story from the last year or two that you'd like to share. Anybody?

25:42
Nathan de Hoog: For us is moving production systems to cloud-hosted and containerized solutions. For us it was a big step, and I'm sure Jason's done all of that but it's very nice.

26:00
Elizabeth Hill Reed: I think from this past year we've really made a push to do more of an agile method with projects. And so getting that high-impact feature to the customer early on so they can really see the value of Ignition. They get excited about it, they wanna do more, and you just continually build on all of that. So moving away from more of the waterfall where it's the high upfront spec phase and it's a long time to where you see results and moving more towards the agile and a lot more of our projects, I think, has been pretty impactful.

26:34
Barbara White: Well, with my company, we're still very heavy in some of the other SCADA systems, unfortunately. But one time I was headed up to a customer, and my boss told me, "Well, he really wants this one," and I said, "Well, can I just mention it? Can I just mention Inductive?" We got up there, and I just mentioned it and said, "Oh, by the way, just go to this Inductive University... Go and take a look at it. All you gotta do is sign up; it's free, go look at it." Two hours later, the customer came to me and said, "This is what I want. This is what I want." So, part of it is just breaking through these last few years. I'm really starting to break through some of those barriers of people who are hooked on to something that's been around for a long time and not changed, and to go to something new, it's not so scary, I think, as it was years ago. So definitely a success there.

27:34
Sydney Bosworth: I think ours would be. We have a customer we've done a lot of work with and kind of done all the automation in their plant so far. And this past year we've been, mainly my job has been kind of working through their MES system with them, building out some custom stuff for them. And we've been able to get rid of a lot of their pen and paper equipment checks and all this stuff and automate a lot of stuff for them and centralize a lot of the things that they're going to their ERP system and then our batching system and three different places to get. And it's helped them a lot being able to do that. And then it's helped us a lot being able to learn more of the ropes of that and a lot of... Doing a lot of Perspective development, for me personally has been great.

28:12
Sean McFarlane: Yeah. Earlier this year, we got introduced to a new customer. I went in to meet with them. Originally, I was just supposed to be looking at upgrading. It rhymes with schmackberry schmuck.

28:27
Sean McFarlane: And first, it was just that. And then, so yeah, basic HMI application; no problem, I could do that. Oh, there's also this data collection system over here that we need to upgrade because it's being maintained by some guy that doesn't work for the company anymore. And it's all visual basic, it's completely obsolete. And, oh, by the way, there's another HMI over here that we really need you. And so it got to a point where I was like, "Guys, the solution is to move to Ignition." And so we've been working on that for the last six months, and we didn't have access to any of the source code for that visual basic application, but it really didn't matter. We could replicate all of the functionality and stuff that they were doing before. And they were very happy with how quick we were able to get this new system up and running on top of replacing their other HMIs that desperately needed to be upgraded.

29:14
Jason Hamlin: So my biggest win is being invited up here by you. I can't believe you're that brave. You just don't even know what I'm gonna say. No, actually, we're in the middle of merging four companies. So, life is chaos. I couldn't even say what our biggest win is, but I'm gonna say my biggest win was actually here at ICC. We contributed something to the [Sparkplug] Data Dash. Little camera sending images over MQTT, just something we put together real quick. People have seen it. Well, I reach out to my friends at Opto 22, and I'm like, "Hey, would you throw this at your booth?" And he says, "Yeah, sure." So Benson [Hougland] does that. Well, then Alex [Marcy] from Corso [Systems], like, posts some stuff that he's on an airplane. He's like, "Yo, that looks really cool, but it would be way cooler if it's stored the images and we could do something with it."

29:57
Jason Hamlin: And like, he messes with it on the plane, lands here. All of a sudden, they have taken this [Ignition] Exchange resource, enhanced it already. And he's like, "Hey, here's the code we use to store all these images." And like, while that's really cool, when everyone's loving it and you go up there and take pictures with yourself, what he doesn't know and anybody else knows is that we actually have a contract with a customer to build this thing and store the images. So I just got all that free labor.

30:22
Jason Hamlin: Biggest win.

30:26
Chris Fischer: Smart guy.

30:26
Audience Member 1: 45 minutes.

30:29
Audience Member 2: Is that a recruiting strategy?

30:33
Jason Hamlin: Ask me after the session.

30:34
Chris Fischer: Well, on that note, let's get into some Q&A. We can hand mics out to you guys here. I see the first one right here in the white shirt. Oh, go ahead, Don.

30:52
Don Pearson: I can start first. Okay, good. It may be you that I want to get an answer from, Elizabeth, but I wanna go back to the earlier question about recruiting and personnel. And it's a little bit self-serving, so I wanna get some insights. I wanna get some free labor, like you just got, Jason. But one of the things that we have seen when people become integrators with Ignition, they have this struggle. Maybe that you just said, Barbara, of like you do everything you're... We say we're vendor agnostic, but not vendor indifferent. But somehow or another, you gotta be able to say, as Steve [Hechtman] used to say, value engineering; we'll just give you an option and you make the call. It's difficult to do that, to get through an organization and grow up, but our integrators tend to grow up, and they have teams.

31:35
Don Pearson: You started, as you said, doing programming, doing the work, whatever. But all of a sudden, if you recruit it and just recruiting, it's onboarding, it's development, and your job description can change. All of a sudden, I mean, Colby doesn't get to do any of the things he used to do that he loved. That was the biggest barrier to him wanting to do what he does now is, well, wait a minute, I love doing this. So how do you address scaling? We have integrators with 200 certified people; didn't start with 200 certified people. How do you scale an integration firm to gain Ignition expertise, spread it, bring on, onboard? 'Cause people have to change their job description and evolve in their careers, and they have to want to. You can't force that. How do you develop the desire to grow Ignition teams of multiple levels of expertise and change your own job description along the way? Does that question make sense?

32:34
Elizabeth Hill Reed: Yeah, and I think that's a really good question that we are still figuring the answer out. So DMC has a really big emphasis on technical capabilities. All of our interviews are very technically focused, but like you said, like the job description changes. I do mainly project management and sales now instead of engineering, like I don't really program anymore. And when you have an organization that you're trying to scale, like we're in a problem right now where we have a lot of, like, more senior people and they all wanna do technical things, so how do you develop the interest in doing sales or more project management and both in an organic way and maybe a little bit more of an encouraged way? I think that we've gotten better at hiring external senior people. That's a shift that we've done in the past couple years where we try and make more of an effort to hire more senior people that already have some of that skillset. We still don't have separate... We don't have sales engineers. All of our engineers or salespeople are very technical still. Yeah, I think that's still a challenge. I'm curious if anyone else on the panel has solved it, but...

33:46
Barbara White: Well, mine still says controls engineer, because I don't wanna even want that senior word in front of them. I said, "No, no, that just makes me sound old." So I don't want that at all. So I'm still a programmer, and that's what I wanna be. And I mean, we have other people who have moved into program management and that, but it's only if you want to. It's not something that we're pushed to have to do.

34:10
Don Pearson: Well, I think you make a good point, and I wasn't even insinuating that we want someone to do something they don't want to do. You love what you do, and you're really good at it. The challenge with Ignition is it's accelerating fast in organizations, and it forces the need to have those other skill sets if that integration firm is gonna be able to build out their expertise.

34:29
Barbara White: I think part of it is you gotta have to hire the people who have some of those skills. And I think, I don't remember who said something about you have to look in different places instead of just looking for a controls engineer, let's look for IT people and bring them on.

34:43
Don Pearson: Good point.

34:45
Sydney Bosworth: We've played around; we did the, I don't know if anyone's familiar with the PI, predictive, not predictive. I don't remember the name of it, but it's a PI thing. It's kind of a personality.

34:53
Audience Member 3: Predictive Index.

34:54
Sydney Bosworth: Yeah, it is that, okay. We did that within our company, and it's a call it personality, professional personality thing. So take that for what it's worth. But everyone did that, and it kind of gives you this scale. It gives you like an identifier word, like operator, maverick specialist. There's certain... It'll tell you what your maybe weak points are, kind of tell you what maybe your job should be, so we kind of went through that as an experiment this past year. Everyone in the company did it, and then it kind of revealed some people that like, okay, well, their job description lays out as this PI profile, but when they filled out the question, they lay out as this profile picture.

35:34
Sydney Bosworth: So I think the idea in theory is to maybe try to use that to be, okay, well, this person is showing some values that would be very good in a leader. This is showing some values that'd be good for that. So then you can kind of let those people have a little more... I learned some stuff about myself from it, so get some insight into where you might wanna go in the company based off of that, and then also as a hiring tool to get an idea of where someone might fit in the company. So we're hoping maybe that kind of helps clear a little bit of that up, but...

36:01
Don Pearson: Cool. Thanks.

36:03
Jason Hamlin: If I could answer that too and actually tag right off of that. Yeah, the direct answer I'm gonna give for your question, it's gonna be the most serious thing I say this entire week. How do we build and accelerate these teams? It's servant leadership and managers with the right soft skills. Because as soon as we have managers that realize people are more than just assets; they're more than just resources, and they manage holistically, the people naturally can grow. Case in point: somebody calling me, "Hey, I know you're at ICC. I've got these really bad technical questions I need to answer. Yeah, no problem. I've also got some stuff in my personal life, but you probably don't have the time to hear about that." And I said, "Whoa, back up. Let's talk about that first." Like, let's do that. And when you... When they realize we want to help you with every aspect of your life as an employee, that makes the difference.

36:49
Jason Hamlin: And then the growth scales organically because your team wants to perform. And thanks, Don, for making, like, Inductive University and easy tools for us to teach people. Like, that's amazing. But yeah, that would be the biggest thing I would say. Managers with the right soft skills.

37:04
Don Pearson: Thanks.

37:04
Elizabeth Hill Reed: Yeah. Well, like giving people room.

37:06
Sean McFarlane: I wanna piggyback... Oh, sorry, go ahead.

37:08
Elizabeth Hill Reed: I was gonna add, just add like giving people room to grow if they want. Not micromanaging, like giving them opportunities to expand their role if that's something they're looking for.

37:20
Sean McFarlane: Yeah. I wanted to piggyback off of what Jason said. When I joined the company seven years ago, we were about 20 people. Now we're about 70. So we've had that growth that we've had to manage. And it's a balancing act between, like, do you feel like you're too top-heavy with senior guys? You gotta hire more juniors. Do you feel like you need to hire more seniors to mentor the more juniors? So we've had that battle constantly over the last few years. But Jason said it starts with good management at the top. Our retention rate, I think, is well above industry average, and our average years of service is pretty high, and it comes down to management and then growing the team.

38:00
Sean McFarlane: What I did was I identified all of the things that I had to learn over the past few years in order to be successful at developing Ignition projects like the database skills, the Python skills, the user interface skills. And I started using tools like Codecademy and Udemy. I'm sure people in here are familiar with both of those. And I just, I came up with a training course of like do Inductive University, and then once you're done with that, do all these, and you'll be ready to go. As opposed to being like me spending four years just grinding it out, trying to figure out what I actually needed to learn to be good at this stuff.

38:37
Don Pearson: Sure. Thanks.

38:38
Chris Fischer: Yeah. All right, next question. I saw a hand go up over there. You could shout it out if you want to. It wouldn't be a problem.

38:51
Audience Member 4: Piggybacking on the same thing. Through COVID and maybe working in different offices, remote teams, as your businesses have been growing, what strategies have you guys tried to use to build and maintain a good culture? I know in controls, you get a melting pot of personality types. Some people are people people and some people want to be put in the dark room and write stored procedures all day, so with all of those challenges, what sort of... What's in your toolbox for building a good company culture?

39:26
Nathan de Hoog: Brunch Club.

39:30
Nathan de Hoog: It's hard during COVID. In Melbourne, I think we had nine lockdowns. I think and it was hard for everyone, and that culture has to be built up now that we can work in the offices again. But during COVID, as... Well, I'm not a manager. I'm an engineer as well, but there are people under me that I would regularly call to see how they're going. Always ask them about how they're feeling, and not necessarily personal personal stuff, but having that connection before then starting to discuss the technical or the work stuff. So COVID was hard, and continuing a good working culture was even harder. But I think we've come outta that now.

40:30
Sean McFarlane: Let's hope so.

40:33
Elizabeth Hill Reed: I think we're just starting to bring some more stuff back from COVID. There's kind of a pause. Where it's like, alright, we're gonna stop doing in-person events, and then we're slowly starting to bring that back, we try... So we have, I think, 14 different offices across the United States now, so we're very spread out, but we try and do like office gatherings to where like, you can meet people from different offices, like we just had one in Texas this past weekend where we had a bunch of people fly from different offices. They all get to meet and hang out in a bit more relaxed, non-work-focused way to build those relationships. 'Cause I think that is an important thing to have is those relationships that aren't just, hey, what's this technical question? What we've all been saying is have that personal relationship to where you almost, you feel connected with them and you want them to succeed and do well both professionally and personally.

41:34
Jason Hamlin: You can't force culture. You can grow it and foster it and nurture it and trust me, merging four companies together, like, it's not just a, okay, how do we grow this really fast? You asked for a tool; specifically, we use the StrengthsFinder assessment tool to identify like people's top five strengths. You can Google it, but we use that, and our managers use that to try to isolate and understand, if I'm giving somebody something that they're not naturally strong at, they're already setting them up for failure. So, we tend to try to align and merge our teams to work using and complimenting strengths, and that's ours.

42:15
Sean McFarlane: Yeah, I think COVID definitely changed things for us. It was tough at times to maintain kind of the comradery of your organization, but I think it also revealed to us that it's okay to be a little more hands-off and let people work and let people do their job, and you just trust them that they're gonna get the job done. I was working full-time from home for about a year when the lockdowns started in the US, and one day I just messaged one of our admin people and I said, "Hey, do you know when they're planning on, like, bringing people back to the office full time?" And she was like, "That was three months ago." And I was like, "Oh, no one told me. I was just at home, getting my job done." But the company culture thing, it's really important. It is something that comes from the top down, and I think we've done a good job just maintaining a healthy work relationship with everybody that... Everybody feels like their time is valued. They feel trusted to get the job done. They don't feel micromanaged. That's kind of the key, I think.

43:27
Chris Fischer: Yeah. Well, folks, I hate to say it, but we are at time. I want to thank you all for joining the Integrator Panel. Thank you, panelists, for joining today. Find these guys, ask them more questions. Enjoy the rest of ICC. Enjoy the Build-A-Thon, everybody.

With the release of Cloud Edition, it's never been easier to get Ignition running in the cloud. But are you ready for it? From security concerns to misconfigurations, there are plenty of pitfalls to stumble upon when managing applications in the cloud. But fear not, as help is on the way. Join the experts from 4IR in this session where they'll provide helpful tips and tricks for deploying and managing Ignition in the cloud.

Transcript:

00:04
Susan Shamgar: Hi. So my name is Susan Shamgar. I'm a Technical Writer at Inductive Automation, and I'll be your moderator for today's session, "Don't Get Lost in the Cloud: Tips & Tricks for Successful Ignition Deployment and Management." To start things off, I'd like to introduce our speakers for today. First up, a longtime member of the Ignition community, Joseph Dolivo. Currently serves as the CTO of 4IR Solutions, an Inductive Automation Solution Partner focused on cloud, Digital Transformation, and life sciences. For more than a decade, Joseph has focused on modernizing manufacturing by intelligently adopting state-of-the-art technologies and principles from the software industry. James Burnand is a 20+ year veteran of the industrial automation ecosphere, who has now turned his focus toward providing the infrastructure for manufacturers to reap the benefits of the cloud for their plant floor applications. He weaves cybersecurity, operational requirements, and management into 4IR Solutions' offerings and provides education and consulting for companies looking to begin their journey into a cloud-enabled and a highly automated OT infrastructure. Please help me welcome James and Joseph.

01:20
James Burnand: Thank you, Susan. Your payment will be after the session. We really appreciate that. Hi, everybody. Welcome to the session. Hello people live streaming. So Joe and I are here to talk to you about the cloud today. So we've talked all week about what we do in the cloud, but what we really want to do today is help you understand what are some of the considerations, what are some of the tools, and what are some of the methodologies that you should consider if you're going to be doing deployments in the cloud. So to start off, I'm going to review a little bit about that and go into a little as to why the cloud is in use today, what are some of the benefits, where are we seeing adoption taking off. And then from there, Joe is going to go into the real deep technical details about what things you can do, what tools you can use, and how to actually go about doing that.

02:08
Joseph Dolivo: Yep. We're excited. We'll get as deep as we can with the time that we have, but definitely save your tomatoes and everything else for the Q&A session afterwards. As long as my voice holds out, I will answer as many as we can, and we'll have contact info provided for future questions.

02:22
James Burnand: Alright. So let's get started. So why do people care about the cloud? I know we've been talking about it. It's become this huge discussion point. There's a lot of attention around different opportunities that are opened up, be they AI, be they flexibility, but ultimately one of the most basic things that's important about using the cloud is you only pay for what you use. So you're not buying a set of servers and computing resources that will have the capacity you need for the lifecycle of those assets, you're not buying five years worth of storage that you're eventually hopefully going to use five years from now plus your safety factor. You're literally paying for just what you're using and as you consume it, that price and that cost goes up. So controlling cost is really... If you think about why people are using the cloud in the first place, that's the biggest reason.

03:15
James Burnand: But the other benefits you get is that you are able to scale things. So not only do you get to only pay for what you use, but you have the ability now to theoretically endlessly scale those resources based on what the growth of a system is or the growth of the amount of data that you collect or the collection of different applications that you deploy. It also opens up opportunities with capability. So there are things that are just hard to do that you can go and install a service from a cloud provider that they do it for you. There's managed services, there's application functions, there's third-party plugins. There's all sorts of things that become remarkably easier to do when you take advantage of those precompiled and prebuilt resources that you can buy from a public cloud provider.

04:03
James Burnand: So what do we see people using it for and what are good use cases? So a lot of organizations that use the cloud, our folks, what we've seen in this conference quite a bit is people who have very distributed systems. So telemetry-type systems, places where it doesn't matter where my server is, everything that I'm collecting from is remote, that's a really great use case for the cloud. Or where there's a lot of focus on data and processing, and I need to be able to use more advanced functions and features to be able to provide the insights that I need. The other thing is that when you look at some of those services I described in the last slide, things like time series databases, AI applications, data warehouses, Snowflake, these are all things that become very easy to integrate with and use and take advantage of when you have the cloud.

04:48
James Burnand: So those data-centric applications just make a lot of sense to be able to use those resources for them. And then one of the things we... One of the most basic things we love using the cloud for is backing things up 'cause it's really hard to back things up in a way that it's easily recoverable, testable, and you can be sure that when it's time to go and restore those backups that they're available. The cloud is a fantastic and very cheap way to store long-term backups of systems that you're running on the factory floor. So what I will say though is just like playing soccer in scuba gear, it's not a... Just because you can, doesn't mean you should. You don't use the cloud for everything. And so what we found is that one of the really great opportunities, one of the really great options that people are starting to explore a lot more now is hybrid cloud.

05:38
James Burnand: So I grabbed a definition off of... I forget I Googled it, but a hybrid cloud is a computing environment that combines on-premises data centers, also called a private cloud with a public cloud, allowing data and applications to be shared between them. Really what it means is you install a piece of cloud in your building. So you put hardware in that provides a conduit, access, and ability to deploy those really cool applications that are precompiled, those services that the cloud providers give you into a piece of hardware that happens to live inside of a building. So a factory or a transfer station or wherever the local needs might be. So you get that low-latency, high-capability system that's running locally on site. You have the ability to cut the cord to the Internet and it still runs, but you get the benefit of running those cloud services down inside of the building.

06:35
James Burnand: I see it as being fairly revolutionary. I think it's still really new for a lot of folks. It's a concept and a way of thinking about deployment that not a lot of people are really that deep into yet, but I personally see that it's... I think it's going to be the future for a lot of the bigger systems. So who's using it today and what are they using it for? SCADA systems for distributed telemetry systems. We're seeing a lot of MES systems being cloud-deployed, especially things like OEE. We're working with our friends at Sepasoft on a number of different opportunities right now where there's, I want to be able to deploy across this fleet of facilities, I want to be able to create a consistent fabric of OEE application access and Ignition in databases.

07:22
James Burnand: And to be able to do that in some plants, it's super easy 'cause hey, they got great resources, engineers that understand what's going on, but it's really difficult to do in facilities where there's maybe not any sort of local support or they don't have people that are really understanding exactly how to build and maintain those systems. Using cloud or hybrid cloud for those sorts of solutions really makes it an equal playing field for all the users and all the locations that are going to have access to that application. The other piece that we're seeing is a lot of ingestion. So we saw some Snowflake stuff this week, which was really, really cool. We're seeing that there's this pull of all this information up to these data warehouses. Analytics tying together sales data and financial data in with production information in new and innovative ways that lets you make better business decisions and it's only being unlocked by the type of solutions that people in this room are putting together to ingest that information in. The other kind of piece to this is tying together with existing cloud services, things like ERP systems, cloud-based databases. There's just a ton of opportunity in pulling those things together. So that's what we're seeing today.

08:35
James Burnand: So challenges and risks, I would say the one thing to remember is the cloud is public. So when you go and you do a deployment, yes, you get access to all this really great technology, all of these applications, all of these things that you're able to do. But ultimately, if you're not careful, you are deploying those things in a publicly accessible location. There's lots of ways to remediate that, lots of ways to manage that. Really, what we find is the most critical part of that is making sure that you have a plan for how you're going to manage those assets. There's ways to be able to deploy in public clouds and have no external access to them, only internal to your facilities, but you have to plan all that stuff up front. So Joe's going to walk through all kinds of technology pieces around that.

09:21
James Burnand: I'm throwing the warning flags up and saying, just remember that it's public and that it's something that, yes, there's a policy in place for most major organizations to be cloud-first because of that first slide around cost savings, but it's not as simple as deploy and forget because if you do that, you're potentially opening yourselves up to all kinds of new risks and challenges that will unfortunately be potentially costly. I would also say that it's difficult to dabble in this space. So there's a big difference from what we've seen in being able to get something working versus having something sustainable and maintainable over time. So tools like cloud formation templates, which I know Joe is going to talk about, these are things that make it real easy for us to be able to build up an infrastructure in the cloud very quickly.

10:12
James Burnand: Even Ignition Cloud Edition lets you just start a virtual machine and run Cloud Edition and it's there and it's going, but you really do need to make sure that you're following best practices, hardening guide, best practices from the cloud vendors to ensure that you are putting in security as a consideration even for systems that you're testing, even for systems that you're just trying to figure out. Because what tends to happen, as I think many people in this room have seen, is I'm just going to start off small. I'll install Ignition here, and that's all it'll ever be used for. Six months later, it's like, "Well, I can use it for that. Well, I can use it for this. Well, I can use it for that." So you end up creating this burgeoning and growing set of applications. And when it's on-prem, the risk is a little bit... Well, it's a lot less because you don't have this public access. When you're doing that in the cloud, unfortunately, you have to be more careful. I believe Joe is going to take over talking now.

11:05
Joseph Dolivo: Well said. I think we're trying to differentiate between the ease of getting started, which is great for demos and learning and testing, and then production-grade systems. So we know a thing or two about production-grade systems. If you guys have seen the Data Dash that's going on right now, all that Ignition infrastructure is part of one of our managed service platforms called FactoryStack. What we're going to try to do is to take you through some of the lessons learned that we've had in working in this space for a long time before Cloud Edition was a thing, but then to give you some very practical takeaways that you can implement in your own systems, and also give you a little bit of insight behind what we've done and productized. And I will just say, coming out of the Technical Keynote, there are a ton of things that are coming in Ignition 8.3 that we are super excited for because it's going to make a lot of the stuff that we have to do now manually a lot easier for all of us.

11:52
Joseph Dolivo: So very, very exciting. Tried to categorize this into five different categories. Again, we could spend days talking about all of this, but they're largely broken down into networking, security, access management, data management, and cost management. And of course, especially with regards to network and security and access management, there's some overlap. So we've come up with a couple of different examples from each of these that we'll talk through. And again, as you have deep questions, please let us know and we'll go down into the weeds during the Q&A if we can. So I'll start with networking. So encrypt all the things. You hear a ton about encryption really in two different categories. There's encrypting things at rest. That's obviously important for data storage, making sure things aren't getting changed after the fact.

12:37
Joseph Dolivo: But also when it comes to networking, we're talking about in transit. So Ignition as a tool has great support for SSL certificates so that any traffic that's going into or out of your Ignition system will be encrypted, but it's not just Ignition. When you're deploying these production systems, you don't just have one Ignition gateway. Typically, you're going to have multiple Ignition gateways in a gateway network. The Ignition Gateway Network uses something called gateway network certificates that you can use to basically encrypt communication between Ignition gateways using the same principles that you use to encrypt your web traffic and all of that. So that's really key. And again, Ignition isn't just talking to other Ignition systems. It's also talking to databases, for example. So when you're configuring your databases, very important to enforce SSL encryption. There's a setting in the Ignition gateway configuration to do that.

13:27
Joseph Dolivo: And even more so, you can go down to the level of basically restricting access to certain ciphers. So I'm going to use certain cryptographic ciphers, I'm going to require TLS 1.3, for example. So focusing on encryption is a key part of everything that you're doing, is really, really critical. The other thing that you'll tend to hear about which is still very important and a good step one is to use a VPN. VPNs have been popular for a long time for good reason. They're a really nice, easy way to extend, let's say, an on-premise network into the cloud. Cloud providers have really good tools to make that easy, but if you just rely on a VPN, then you're doing what you call perimeter security, and we'll touch on security more in a minute, where you're securing the outside, and then as soon as somebody gets in the door, you now have... It's kind of free reign.

14:16
Joseph Dolivo: So a VPN is a tool, but it's a tool in defense and depth. So don't rely on a VPN by itself. Encrypting traffic, whether or not it goes through a VPN is important. So that's encryption. Limiting external connectivity. So we've got Ignition running in the cloud. Again, you probably have a database, for example. Best practices would suggest that you don't provide external access to the database unless you need to and typically you won't. So your Ignition system can be publicly accessible via web browser, mobile device, designer access, things like that. The database, you would probably want to be locked down inside of a virtual private network or a VPC depending on your cloud provider. I'll use both terms interchangeably.

15:00
Joseph Dolivo: And then there's a bunch of these cloud-native services that James had alluded to that are things like data lakes, digital twin services. And again, depending on if you're going to funnel all that data through Ignition, you don't want to have outside access to those systems. And the cloud providers provide really good tools, private endpoints, private link. Those are things you can use to basically expose even some of those managed services into your private network without having to go out through the public Internet which is the default. So highly, highly recommend that for anything that you're going to be doing which requires access from the outside. And the last one here is about minimizing hops. So especially for production-critical systems, getting data in a timely manner is very important.

15:44
Joseph Dolivo: And now we're not just talking about, oh, I'm sitting across from my server in my plant. I'm talking about having to go up to a cloud system and back in order to communicate. And the cloud is global so you can pick regions and then you can deploy things. I could be sitting here in California connected to a cloud server in Arkansas, which is actually what we're doing for the Data Dash here. And so by default, when you're starting to add these different layers of networking complexity into your systems, you risk introducing a whole bunch more latency to applications like Ignition. So one of the recommendations that we have if you're going to be deploying this inside of, let's say, an orchestrator like Kubernetes, which has been talked about a couple times, would be to look at the network interface that you're using to expose those workloads.

16:29
Joseph Dolivo: So for example, if you're using Kubernetes, by default, it deploys an overlay network called Kubenet, and it's got this virtual address space that's disconnected from everything else. It's introducing another network hop. The cloud providers provide integrations with something called the Container Network Interface that lets you expose the same IP addresses, same address space you're going to use for your virtual machines or for other kind of workloads, also for the containers that are going to be running Ignition. That reduces the network hop, makes your application more performant. Same thing when it comes to these complex architectures where you have load balancers in place. Every hop, every proxy you put in place is going to slow that down. So be very careful and selective about where you're introducing those kind of latencies. So we could have a whole session on networking.

17:14
Joseph Dolivo: That's a couple of highlights. Security, natural progression from talking about networking. Keep your systems up-to-date, and you're saying, "Well, of course, that's obvious." But when you actually look at the scope of systems we're talking about, let's take Ignition as an example. You've got your application, so you're going to be making changes to your application to fix bugs, to implement features and all of that. That application resides on Ignition, so keeping Ignition up-to-date, for sure. Doing that in a production system where... I love IT people, but you can't just push down security patches at any point in time. You've got a production system. You can't do that. So Ignition is a component of that and most applications are also built on a database. You're using the Sepasoft MES modules. It's built on a database.

17:57
Joseph Dolivo: Now, you've got to do those updates in tandem. So I need my database and my Ignition system to be in lockstep and if one of those is not in step, you think you're taking backups. We'll get to backups in a bit. Are they in sync? Are they cohesive? And now you're going down to a level below Ignition that's running in an operating system. Whether it's containerized or not, I got to patch that operating system. Maybe I've got an orchestrator like Kubernetes, maybe I've got add-on modules for providing other functionality. So looking at these systems as something that is living and breathing and you don't just set it and forget it is incredibly important. And to James's point, it's so easy to set something up once and then you forget about it and say it's good enough.

18:38
Joseph Dolivo: These air-gap networks don't really exist anymore. Maybe they never did, but nowadays it's not something to look at, especially when you're talking about the cloud. So reducing attack surfaces, the more stuff that's available on the public cloud, the more targets there are for attack. You go to shodan.io, you can see all the industrial OT network traffic that's available. It's terrifying, but you should check that out if you haven't heard of it before. So we want to do everything that we can to minimize the exposure to applications, to data from the outside looking at limiting external connectivity like we talked about as part of that. One thing I want to highlight within the Ignition ecosystem, Ignition has first-class support for containers. Containers are great because when you distribute a container, there's a couple of sessions on that at the conference. You're basically just distributing the minimum set of files that you need to run an application, and that's it. And you're decoupling it from everything else that's required like a kernel and everything else to run an operating system, Windows updates, all that kind of stuff.

19:39
Joseph Dolivo: So if your kind of target that you're deploying is basically these containers that have minimal packages installed, you're not having everything out of the box, you might get with a Windows updates WordPad, calc. So that really, really helps you to minimize that attack surface and it's, again, one less set of targets that attackers are gonna be able to go after. And then, of course, there's monitoring for breaches, and I can't tell you how many times two years down the road, somebody will find out that, oh yeah, somebody has been in our systems and they may have modified our data. We don't know what happened. We're gonna have to do a product recall or put out an announcement. So doing active monitoring is really, really important. It's something that there's a number of tools available to do that.

20:20
Joseph Dolivo: There's some that are kind of OT-specific, and you'll see 'em inside of OT networks from companies like Claroty and Nozomi and things like that. But there's also a lot of IT-centric tools that really work well in the cloud environment. A lot of them are based on machine learning to do like anomaly detection. So I'm gonna kind of pick... These are the sort of typical traffic patterns that I might be seeing in a cloud environment. If all of a sudden I see a huge spike in network traffic, or if I see access logs from users or accounts that I don't tend to see, maybe I raise a flag, I send a notification, I require manual intervention. And then tuning that in a way that you're not getting so many false positives, that is the same problem we talk about with alarms all the time.

20:58
Joseph Dolivo: It's, "Oh I've got so many alarms, I'm just gonna ignore 'em all." So there's a balance there, but the fact that you don't just kind of set this up and ignore it, you have to be actively monitoring for breaches. So super, super important. Again, we could have a whole session on security alarm. Let's talk about access management. So there was a question that came up in the Technical Keynote talking about using YubiKeys for authentication with Ignition and things like that. Access management is hugely important. And another universal principle that you'll hear, and it ties in really, really nicely I think with Ignition is to practice the principle of least privilege. So in terms of user accounts, that means if I'm gonna be authenticated and authorized to use a service, I wanna be provided with the least amount of access that I need to be able to do my job.

21:44
Joseph Dolivo: And that's for two reasons. One, in the case of kind of a malicious actor, that reduces the damage that can be caused if that account is compromised. And it also just helps people from kind of shooting themselves in the foot or doing something by mistake that they wouldn't ordinarily try to do. So for example, in the kind of Ignition roles, you may say, well, I'm only gonna give an operator certain roles so they can't accidentally change the configuration of the system. If I'm an administrator, I may have elevated roles, but we also tend to just say, you know what, I'm just going to use an administrative account that has access to do everything because it's too much work to go through a process and then you end up getting in trouble when that happens.

22:24
Joseph Dolivo: So enforcing roles in a way that is consistent and clear is really important and there are tools that you can use to do that especially if you are taking the management of that outside of, let's say, just Ignition. You can use something like... Entrada [Entra] ID is what it's called now, but I never get it right. It used to be Azure AD, so basically the cloud extension of Active Directory, and you can have all of your groups and roles centrally managed across your organization. And then you can have the concept of, let's say, a supervisor and a supervisor can have certain access granted in Ignition, certain access granted in other applications, your ERP systems, your CRM systems and things like that, and you have that all managed in a single place.

23:03
Joseph Dolivo: The last part on principle of least privilege is that it doesn't just apply to named user accounts. It also applies to, let's say, service accounts. And so this is an example. We'll talk about databases more in a minute, but when you're configuring access to a database, that database may not need, or that database user account may not need the ability to delete records. Maybe I can only do inserts, especially for audit trails. I'm gonna be able to insert into the audit log. I don't wanna have somebody that can update or delete from those. So think about the principle of least privilege in terms of the system accounts as well in addition to named users.

23:37
Joseph Dolivo: Password management. I'm super excited. Again, Technical Keynote talking about using a system like HashiCorp Vault, where you can have the dynamic password authentication. Right now, there are certain accounts like the database connection in Ignition, which is more or less kind of hardcoded. It's sort of encrypted in the configuration, but some of those things are kind of hardcoded. But for other things like logging into Ignition, the safest way to manage passwords is to not manage them, and again if you're using a system like Entrada [Entra] ID, or AWS, IAM, or OKTA, or Duo, or some other system, you've got an enterprise security company whose stock price and revenue is based on them doing a good job with all of that. So we recommend not having to manage it yourself. It's one less thing you have to deal with. So for our platform, we don't see any passwords at all from users. We say, nope, we don't wanna deal with it.

24:30
Joseph Dolivo: And then of course, monitoring and auditing access. So Ignition by itself, you configure an audit log. It logs a whole bunch of different events that are occurring by default, which is great. You also have a script function that you can use to add additional logs manually based on things happening in your application. And depending on again, the system you're using for identity and access management, you could also have sort of a central audit log in the cloud that you can use to monitor. So every time somebody logs in, every time somebody asks for elevated privileges, so there's tools like PIM, Privilege Identity Management, where maybe I'm gonna be given read-only access to a service, and I have to go through an approval process to give me temporarily elevated access rights to some other system. Well, that's gonna be audited and logged and it's maintained for a certain duration of time and then that'll be it. So again, active monitoring, similar to threat management when it comes to security. Really important for access management.

25:23
Joseph Dolivo: A couple more here, data management. So take backups and again, that sounds great in theory. Backups include a lot of different systems. And Ignition's actually really, really great in the fact that you can go in the gateway configuration page, you can schedule backups to be taken on a schedule, and if the volume to which you are storing those backups is, let's say, cloud-replicated, that's great. You can get cloud-based encrypted backups, multiple availability zones and multiple regions out of the box really, really easily. Again, most systems aren't just Ignition. There's gonna be a database component, there's gonna be other systems that you have to take and some systems are not as nice to... They're not as kind of allowing for doing live backups like Ignition has and the official kind of application process for doing backups is I'm gonna spin down a workload, and then I'm going to copy a volume somewhere else and I'm gonna spin it back up.

26:16
Joseph Dolivo: So we have to do some of that with manual pipelines and things like that. But if you have the ability to kind of coordinate the backups of all your systems together, really, really important. And then the backup's no good if you take it and then two years later you need to get it and you realize that the backup failed, or the backup was incomplete. So it's really, really important, especially for production systems, that you are doing regular verification of those backups. A really easy way to do that, especially if you're using Ignition in containers, take a backup of a database, take a backup of the Ignition gateway other stuff, and then spin up a brand new environment. I'm gonna say, okay, this is now my dev environment. I'm gonna restore a gateway backup, I'm gonna restore databases, and I'm gonna do some spot checks or automated testing to confirm that those are all still working.

26:53
Joseph Dolivo: So we do that regularly for all customer instances. It's something you should do as well. Really, really important. Data residency requirements, so especially when you're talking about production systems, again, in the cloud, you've got all these different regions you can deploy into. Certain cloud services you'll find are only available in certain regions as well and certain regions have availability zones or don't have availability zones. It's really important to know where your data is going and where your data is being stored at all times. And there are a lot of industries, a lot of companies that have very specific regulations to say, my data cannot leave the United States. For example, my data cannot leave Canada, my data cannot leave this particular geographic region. So keeping that in mind is really important 'cause you may say, well, yeah, my workloads are running inside of US-East-2, but to get there, it has to go up through this other system running somewhere else.

27:45
Joseph Dolivo: And now if the data's being... Even if it's encrypted, my data's going somewhere where it's not supposed to be. That's a big no, no. Same thing with storage. You could say, well, if the cloud providers have the concept of paired regions where you could say, you know what, I'm gonna store most of my data in US-East-2, but it's paired to something in Canada-West-1. So for disaster recovery purposes, that may or may not be okay depending on what your team's kind of requirements are due to regulations or company policy or anything else like that.

28:17
James Burnand: And maybe I can just quickly add to that if my mic comes on. When you're also architecting your solution, availability zones and regions become a huge important consideration. So for example, you can buy storage that's mirrored across three of those. So availability zone for everyone's benefit is a completely separate data center that has a separate power feed, it has separate network connections, but it's inside of a region. So US East, for example, for Azure has three availability zones that you can buy services from as US East. So depending on the reliability requirements of the application that you're deploying, you need to choose the services that have the right level of reliability. So by default for us, for example, when we do storage, we'll actually have storage that's mirrored across three availability zones in a single region so that way we can tolerate two buildings burning down before your system will stop. So just to kind of put a little perspective around that is that there is also a cost consideration as a part of that. So if you're going to buy something that is available across regions, for example, it's going to be more expensive than if you're getting something that's dedicated to a single availability zone in a single region. So your application architecture matters from a cost perspective.

29:29
Joseph Dolivo: We are definitely getting the cost as the next big pillar here as well. So well said, James. And the last point on here is just data integrity and retention. So I need to maintain data for seven years, 10 years due to regulatory purposes. The storage providers inside of the cloud, or the storage accounts inside of the cloud providers allow you to do, for example, immutable data. So I'm gonna push data into an archive storage tier. AWS Glacier is an example, Azure Storage account has an equivalent, where nobody's gonnae able to touch it, and it's gonna reside for some extended period of time. So that's really, really, important for compliance purposes and it doesn't even necessarily have to be data in your live system. You may say, you know what, having a 10 terabyte drive on this managed database service is really expensive.

30:17
Joseph Dolivo: But I need to maintain the data, but I'm not actually gonna query it unless an auditor comes and starts knocking on my door and says, "Show me the data." So you could store all of that older data in kind of much cheaper archive storage and then if you need to restore it to say, "Hey, look, I've got it," then you can go through a process to do that when you need it. A really good way to save cost, which is our final category for today. So cloud makes it so easy to get up and running, and the cloud providers wanna incentivize you to just pump all the data up. We're not even gonna charge you. If you're not going over an encrypted connection, we'll ingest all your data for free. That's become pretty much a standard. But once it's up there, they're gonna charge you for using it.

30:52
Joseph Dolivo: And there's a lot of stuff in the news recently. Hey.com recently talked about how much money they're saving by going out of the cloud and there's a lot of... So we talked about some of the reasons you may or may not want to use the cloud, but once you... You're really paying for sort of the flexibility and scalability that you get. So for the Data Dash, we said we're gonna spin up five servers. Give me five servers Azure and boom, we have five servers up and running. But you're paying for that dynamicism and flexibility. So if you know, for example, I'm gonna run Ignition Cloud Edition for a year at least, you go to the AWS marketplace, you go to provision Ignition Cloud Edition, it'll tell you if I know I'm gonna run this workload for a certain amount of time, I can basically commit to paying for a year and I'm gonna get a pretty sizable discount on the infrastructure cost.

31:36
Joseph Dolivo: 30%, 35%, something like that, that's huge, especially when you're talking at scale. And it's not just Ignition systems that can do that. You can do that with databases typically, you can do that with storage. So trying to estimate the workload that you have and then being able to kind of predict what you're gonna need is really, really useful as you've been running. Again, not so much for experimenting. When you're in a production system, that's important to consider, and it's something we do as well. So we actually will forecast out based on our customers. We're gonna commit to using this amount of resources and we get a cost savings from that. So that's reserving capacity up front. Another thing is called... And different cloud providers have different terms for it or basically spot instances. So this is where maybe I don't need a workload running all the time.

32:18
Joseph Dolivo: Maybe I need to do like a... I was gonna say batch job, but batch means something else in our automation industry, but I'm gonna run a report at 2:00 AM every week, for example. And it's something that's gonna run for a while and then it's gonna shut down. I don't need it running all the time. Or maybe I'm gonna just spin up a temporary dev system. I don't need it for a long period of time. If it goes down, it's not a big deal. You can leverage these cheaper spot instances where you basically will say, well, I only want to pay for a compute between this price and this price and if it becomes available, great. If not, shut it down. Or if somebody else is willing to pay a higher price for it, they're gonna steal my VM out from under me.

32:55
Joseph Dolivo: You can have incredible cost savings when you do that. It's also good for like a lot of GPU-based workloads like ML and AI training. So that's, again, not so much for Ignition production systems, but certainly for either dev and test systems or if you need some kind of temporary scalability like, hey, I need to add another frontend node to my Ignition server 'cause I'm anticipating more load during shift one, or something like that. So that's something else to consider. Huge, huge implications on cost if you do it right. And then I can't tell you how many times I've heard from customers saying, "Well, I got the bill at the end of the month and it was 10 times higher than I expected." So making sure that you're putting monitoring in place and alerting in place so that if you're starting to exceed your typical usage trends, you're able to identify that quickly and early.

33:39
Joseph Dolivo: So this has saved us a number of times. I talked to a couple of folks in the room about this where we had logs that we were aggregating that basically hit a trip wire and our system alerted us. We were able to make a change so that we didn't get $3,000 cost after that. And the cloud providers themselves and a lot of the cloud-native tools have ways of doing that. We'll talk about our tools in a minute. We use Grafana Cloud as an example for aggregating all of our metrics and logs across all of our systems. So you can set up alerts and notifications. You can do it in Azure, AWS, and GCP so that way you won't be surprised when the bill at the end of the month comes. So super important.

34:19
Joseph Dolivo: Just to kind of give you some insight, if you're kind of looking like, "Well, where do I kind of get started with this?" These are tools that we use. There's a whole bunch of them. It's really hard to pick, but I'll just kind of go through some of the icons so you're aware of them. Obviously, you know Ignition right in the center. Everything that we do and most of everything that you do is built all around Ignition. If I start at the top left, there we go. There's a laser pointer. So that is Kubernetes. We don't recommend that for most folks. It's one of those things if you have to ask, you probably don't need it. Something that we use internally, and there's a really great session that Kevin Collins did earlier today talking about kind of the nuts and bolts of that.

34:56
Joseph Dolivo: We use that because we're orchestrating Ignition across tons and tons and tons of customers. So if you're a bigger customer, you have a lot of Ignition instances to deploy, a lot of other workloads alongside a single gateway you need to deploy, a really good tool to consider. If you need to run one Ignition server, it probably doesn't make sense. Going clockwise I guess, Grafana is the next one. So this is what we use. I kind of hinted at it for metrics and log aggregation. It gives us really good deep insight into our containerized workloads as well as all of the kind of cloud provider-native services. So we can see how we're doing on cost, we can look at our CPU and RAM performance, all that kind of stuff. It's really nice to have a single pane of glass. And there's other systems out there that can do that.

35:37
Joseph Dolivo: We like Grafana. Great visualizations as well. Git, so when you're making changes, especially in kind of an enterprise space, it's not a cloud-native technology. I call it a cloud-adjacent technology. It's kind of in the same realm doing version control. Again, super excited for the changes coming at Ignition 8.3 that will make this more comprehensive beyond projects. We did a whole session on it last year. We're doing a workshop on it in a couple of weeks. But we basically run Git inside of the cloud to maintain backups of our project configuration, both for Ignition as well as other services. And then currently we support AWS and Azure. I love GCP as well. That's a great one. And then finally, the one that you may not recognize this logo here, this is called Pulumi.

36:17
Joseph Dolivo: So there's this whole suite of tools called Infrastructure as Code is the buzzword. Terraform is kind of the market-leading most popular one. They've been in the news recently due to some licensing changes that they've made around their open source offering, but we've been using Pulumi, which just lets us use our programming expertise that you'll have from Ignition Python, for example. You can use that to provision all of your infrastructure. So we never manually go and download a VM and download Ignition and go do the installer, even though it's only three minutes. We never do it. We use everything as containers and it's all provisioned using this tool called Pulumi. So there's a ton of good tools out there. We highly recommend being in automation as we are that you leverage some of these where it makes sense for you. I think...

36:58
Joseph Dolivo: So we've got additional resources. We made reference to some of these. So there's best practices, obviously the Ignition Security Hardening Guide, concepts for Kubernetes. AWS and Azure have their own. GCP also has some. These are links inside of the PowerPoint, which will be sent out. Definitely take a look at all of these. And then the two sessions, there was a good higher-level one on Ignition in the cloud. If you didn't get a chance to see it, watch it on the Livestream or the recording afterwards. And then the "Deployment Patterns for Ignition on Kubernetes" that Kevin Collins did. So really, really good sessions with really, really good, good info. And question mark means questions. We're ready for the tomatoes.

37:41
Audience Member 1: I didn't bring my tomatoes today, but one question I have, you guys alluded to it earlier that this is a space that's difficult to dabble in. So many of us being integrators or service providers here, what offerings do you guys have for providing a sandbox environment for people to get familiarized with your platform and potentially show it off marketing material style for potential clients?

38:08
James Burnand: I'll take that one. So we're in the process of hopefully soon announcing some really cool local versions of what we offer that you'll be able to actually run locally on your machine as a test environment. As it stands right now, we set up demos for integrators all the time with their own separate subdomain on our development system. So then you get gateway and database access. You can throw your projects up there, you can test playing with them, and you can make sure that they work. But one of the cool things about how the products that we built work is all of this complexity is kind of encapsulated in those. So you get designer access and you can get database access, and it looks just like a normal Ignition project. So from our perspective, we're trying to help make this technology easier to be able to adopt and that's kind of been our business model from the beginning.

39:02
Audience Member 2: Getting into cloud and cloud infrastructure and tools is... Can be a scary thing. And I think I've seen that with a lot of customers and even with myself thinking about how do I even get started? Can you guys talk to what you would say to somebody who wants to get over that fear and even just get their feet wet with cloud infrastructure and how they can start seeing those benefits and how do you overcome that first step?

39:31
Joseph Dolivo: Something that I would say is cloud is a spectrum. You don't either adopt it or not adopt it. There's kind of a spectrum of adoption. And so the easiest way that we've seen to kind of justify the use of cloud is just use it for offsite backups. Are you really, I like to say, take the tape drive down to the bank vault everyday. Is anybody doing that? Some people are doing that. Use it for encrypted multi-site offsite backups. That's kind of the Trojan horse, if you will, to kind of cloud adoption. And then use it for the things that it's really, really well suited for and tailored for like scalability. You know what, I'm gonna spin up a dev system, for example. I'm gonna play around with it. That's a really nice way to get companies more comfortable with it.

40:09
Joseph Dolivo: We spend a lot of our time with heads of IT and security folks kind of talking about why this is okay, how this can fit within their kind of existing IT landscape. It's actually kind of interesting because I'll say prior to maybe three or four years ago, the cloud was a scary thing for almost everybody. And we've really had this excitement that we've seen from a lot of customers I think driven by let's say Ignition's use of a lot of IT technologies, for example, where all of a sudden you talk to a chief security officer and they're like, "Oh, you're using containers, you're using this. I get it. You're speaking my language now." So that's actually helped I think to make it a little bit more palatable. But yeah, start from offsite backups. Super, super simple would be my point around that.

40:49
James Burnand: Yeah, I would only add to that that I actually think probably the best place to kind of focus learning attention if you're a traditional automation person and you're looking to figure out kind of how does all this work is I would focus on containers, learning the different container architectures, how networking works, how you actually set up those systems. And Kevin Collins' GitHub page is fantastic for anybody that hasn't been to it. Absolutely you need to go to it. I don't have the URL handy, but certainly it has so many resources that will help you learn about how to work with these architectures. And then really what you're doing is you're taking that Docker-centric architecture and you're using these prebuilt functions and tools to make it easier to actually do a more coordinated deployment.

41:34
James Burnand: One of the things Joe didn't mention is our Grafana system that's providing us all that alerting and monitoring, what it often is telling us is that it fixed something. So Kubernetes had a problem and it took care of it, and we get a teams message that says, "Yeah, the problem happened and the problem is taken care of." So like that part of kind of the progression and the ability to automate and take advantage of these tools at scale is the ultimate goal, but none of that happens if you don't first start focusing on things like containers.

42:04
Joseph Dolivo: Yep. The last part I'll add is looking at containers, it's another one of those kind of cloud-adjacent technologies. You can run containers on-prem and you can run 'em in the cloud. So start doing the things that will work well in the cloud, but just do 'em on-premise. So we've seen a lot of that's kind of hybrid cloud is kind of a similar idea with that. Thanks.

42:24
Audience Member 3: Do you have customers that are ingesting or exgesting? What's the opposite of ingest? I don't know. Doing that thing...

42:37
Joseph Dolivo: Expulsion?

42:39
Audience Member 3: In other cloud technologies. Like IoT Core, for instance. Are people using IoT Core to get data into your systems or then beyond just a normal database thing? Are there other places where data's going out of your environment?

42:56
Joseph Dolivo: For sure. So there's... And it is funny 'cause IoT Core is something, a service that AWS and other services have had. GCP made a lot of news recently where they actually sunsetted one of their IoT products. And so Cirrus Link is here. They have a great broker. HiveMQ is here. There's a number of kind of broker technologies, I'll say, for getting data up into the system and then also kind of pushing it back out. So Ignition is a good fit for integrating with all of those, a lot of those kind of event-based systems. Again 8.3 is coming and it's gonna make this easier. But you can ingest into Azure Event Hubs, you can ingest into AWS IoT hub, IoT Core. So those all work. The one thing to keep in mind too is that not all of those services, they may support MQTT, but they may not be fully compliant with things.

43:44
Joseph Dolivo: So for example, we went down a whole road with like store and forward and avoiding data loss. Going up into MQTT, there's some nuances to the TCP Keepalive Timer and all these kind of things that could result in data loss. A lot of systems that are sort of compliant, somewhat compliant outside the ecosystem don't support all of those. So that's something to keep in mind for sure. Once you get data up into Ignition in the cloud, then you can kind of push it out, but we found... We've seen a lot of benefit. If you're gonna push data into Ignition running in the cloud, whether it's [Ignition] Cloud Edition or whatever, keep it in there to do all of your visualizations and stuff like that if you're gonna use an Ignition and then push it out after that. So I hope that helps.

44:25
Susan Shamgar: Alright. Thank you, everyone. I believe that is all the time that we have for today. So can we get one more round of applause for James and Joe?

44:40
James Burnand: Thank you. Thank you, everybody.

0:44:40.6 Joseph Dolivo: Thanks everybody.