First Steps to DevOps

Speakers

Kent Melville

Director of Sales Engineering

Inductive Automation

Mark French

Director of Operations

Sepasoft

Ryan Crownover

Integration Manager

Vertech

The DevOps methodology integrates development and operations so that system changes can get rolled out quickly without causing unplanned downtime. Industrial organizations that successfully implement DevOps will have a strong advantage, but knowing how to get started can be a real challenge.

In this webinar, experts from Inductive Automation, Vertech, and Sepasoft will help you take those important first steps. They will show you how DevOps practices can prevent costly mistakes throughout the software lifecycle, and share valuable knowledge about implementing and maintaining DevOps as an ongoing practice at your organization.

• See how DevOps vastly improves planning, visibility & productivity
• Learn how to leverage DevOps in different architectures
• Find out why Ignition is an ideal DevOps automation platform
• Get expert answers to your technical questions

Co-Presenting Companies

Transcription: 

00:00
Kent: Hello, welcome to today's webinar, “First Steps to DevOps, Continuous Development and Continuous Delivery for Industrial Organizations.” Thanks everybody for being here. My name is Kent Melville, I'm Sales Engineering Manager at Inductive Automation, and I will be moderating and speaking for the webinar today. And I'm fortunate enough to be joined by two other speakers. They are Mark French, who is the Director of Operations at Sepasoft, which is Inductive Automation's strategic third-party module partner for MES, and Ryan Crownover, who is the Integration Manager at Vertech, which is an Ignition Premier Integrator. And so, Mark and Ryan, thanks for being here today. Could you just take a minute and just tell us a little bit about yourself and what you do for your organization? Starting with you, Mark.

00:47
Mark: Thanks, Kent. It's good to be here today. Yeah. As Director of Operations, I get to oversee our customer success activity. Very much like yourself, I get to get involved in project design and assisting customers be successful with their Ignition projects using the Sepasoft MES modules.

01:07
Kent: Perfect, and over to you, Ryan.

01:09
Ryan: I'm Ryan Crownover, I'm the Integration Manager of the Tennessee office of Vertech. My background is in building and architecting Ignition MES and SCADA systems, and now I lead a team of people in trying to help level up their game as well.

01:28
Kent: Perfect. Yeah, and I'll tell you a little bit about myself, everyone. So I'm Kent Melville, like I said, and I've been with Inductive Automation for about five years now. I get to meet with customers and help them figure out best practices and architectures for their applications. And when I get lucky, I get to help them build some stuff and get my hands dirty, so I have a lot of fun with the Ignition platform. But for those of you who may not be familiar with Ignition, I wanted to give you a little background to it. So Ignition is the first industrial... Universal industrial application platform for HMI, SCADA, MES and IIoT, as an unlimited licensing model, cross-platform compatibility, runs based on IT-standard technologies, and is a scalable server client architecture. It's web-based, web-managed with a web-deployed designer and clients.

02:20
Kent: It's modular, so you just kinda pick and choose what you want as far as licensing is concerned, and then has rapid development and deployment tools. But if you wanna learn more about that, there's lots of information on our website. That's not primarily what we're covering today. So what are we gonna talk about today? Well, I just gave you the quick introduction to Ignition, now we're gonna be jumping into why you should use DevOps practices. And so Mark and I will be talking to you about that. And then we're gonna go into a story about a common example of how a release can go wrong and how DevOps prevents those kind of problems from happening again. Then Ryan is gonna show you a live demo of how to execute code changes and deployments with DevOps architectures. Then we'll wrap it up by telling you how you can learn even more and by taking some of your questions. But with that being said, let's get started. And so over to you, Mark.

03:16
Mark: Thanks, Kent. So when we're talking about DevOps, we need to lay a little bit of groundwork, and why use DevOps practices, we've gotta get on the same page. So whenever you go about work, you have an approach to that work, and in software, historically, there's two major ways to approach those projects. Waterfall, which we're not really gonna get into today, and Agile, more recent development. And a lot of us have fielded projects using these methods, these approaches, but compared to DevOps, they typically can... While they can produce good outcomes, they typically take longer or miss out on some of the features that we want to add. So we're going to go from that and start looking at: What is DevOps specifically? 

04:08
Mark: So DevOps is short for development and operations working together. And that takes two forms here, there's two main aspects. One is a personnel aspect, and that's involving operation staff, typically with testing. So the end user that gets their hands on the end product of the software project, your Ignition application, getting them in the loop testing is typically something that's not done, right? Instead we throw it over the fence. Well, we wanna change that. In order to do that effectively, we need an environment that replicates production. There's no sense in a test that doesn't really apply. So that's the personnel side. On the technical side, yes, there's technical involved in that replicated environment, but on the technical side we need some mechanisms for moving changes from development all the way through to production, so that we can properly evaluate that. We'll talk more about that. But we need those mechanisms for managing those changes. So DevOps is a bit of a hot topic in the software world. Historically it's been something that fell to other folks, or a variety of folks, maybe a developer, QA, IT. Now it's becoming its own space, with its own practices, and kind of maturing. And part of the evidence of that is this webinar, I guess. But really we wanna think about some key things here, this process approach to delivering end user needs, new features, things like that.

05:58
Mark: And I think that this is a logical next step from an Agile framework for fielding software. And this last point here on this slide, there's a dichotomy between stable software and feature delivery. Right? We could have a very stable project maybe that's initially put in place, and then we never change it, right? Well, there's an opportunity that's lost there, because the needs of projects often change over time, or the needs of users change over time. But on the flip side of that, we could have a very open and dynamic project, and DevOps really strikes a nice balance here. So we want highly stable software, but we also want rapid feature delivery. Well, the Ignition platform specifically is very well-suited to the task, for several reasons. Obviously, the licensing model is very advantageous in this sense. Also the flexibility about the host systems, as we'll see here in a little bit. But the fact that you can pretty much run Ignition anywhere is always a big plus. Also with 8.0 and 8.1, integration with Git and other such tools, really empowers this DevOps framework. Okay, so why use DevOps practices? We've talked around DevOps a little bit, some of the background, but now let's talk about some of the benefits of the practices. Obviously, change management is part of your risk management, and DevOps provides this framework for deploying code.

07:54
Mark: First item here is really isolating development and test from production. Boy, if you saw all of the examples of failure in this point that I see from tech support, right? The old joke that “I only test my code in production.” Well, that's not acceptable for your organizations or your customer's organization. So by utilizing a DevOps approach, utilizing this framework, we can document code movement, and project resources and changes, between environments, and those gates or milestones where we move resources between environments give us review opportunity as well. So hopefully this is taking shape in your mind here, we'll see some practical examples here in a minute, how we can properly break up this problem, and using this approach, achieve both higher stability and better feature delivery.

08:58.5
Mark: Now, one of the key failure points that I see in a lot of customer projects is with deployment. So the old joke, "Hey it worked in test," or, "It worked on my system," it's a tired joke in software circles. We've probably all said it. But that's not acceptable for your production use, right, especially when other people, your business, is relying on this. So one of the key wins that most people walk away with from adopting a DevOps framework and approach is the ability to rigorously test those deployment procedures and to test deployment to ensure success there, so we don't have that angry Monday morning kind of situation where production is down. Now, at this point, it's worth pointing out that there is a learning curve with the DevOps approach and framework. And, myself and Kent, I think both would recommend that if you're not familiar with this, this is a great time to involve a certified systems integrator.

10:15
Mark: So there are specific tools that will be on display today. There are of course other tools, as well, in the space. But a systems integrator that has experience leveraging several projects like this can really accelerate your organization's project environment capability using this framework. So typically, these are shorter-term projects, really great opportunity to leverage someone else's experience, get your team up to speed here. So with that, I'd like to hand it back over to Kent. Kent?

10:54
Kent: We're gonna identify a story here of a fictitious example of the types of things that Mark had said he sees in support, people who have not implemented these following best practices and that they encounter issues. And so our example goes through the life of Developer Dave. Developer Dave is tasked with improving their existing Ignition solution, and he's asked to create a new screen to allow the user to create manual alerts that send SMS messages to a specified roster of users. And so Developer Dave goes in and he builds the screen and creates a new alarm notification pipeline inside Ignition to process those alerts, and then he tests the functionality in a development environment against a roster, with just his phone number as the recipient so that he can test it without actually sending out notifications to the everyday users. And when he's happy with that, he exports the new manual alert view, and the updated navigation sidebar view with a link to the new page and the new alarm notification pipeline, all into the production environment. And then he sends out a test message to a small roster of users. It works great. So he heads home for the day thinking he's done a job well done. Now he wakes up the next morning to chaos, the production line is down, the SCADA screen needed to reset, the fault has disappeared.

12:31
Kent: How did this happen? Well, he realizes that the navigation sidebar view that he exported was missing the link to the page controlling the new piece of equipment that had those faults that needed to be reset. And the reason it was missing is 'cause an outside vendor had developed the view for the equipment, and that had been added directly to the production environment, but it had never been back-ported to the development environment. And so when he went and made a change in development and pushed that to production, it overwrote those changes that had been submitted by the outside vendor. And so now that new equipment, there's no way to control it. And so when they have this issue, they're without a paddle, so to say.

13:16
Kent: Now, unfortunately, this kind of story is very common in our industry, and the solution is not always super simple, of, “Oh, I just need to go and fix the navigation.” The bug could have easily caused any of the following worst outcomes. It could have produced a non-conforming product that goes out unnoticed for hours or even days, forcing recalls or whatever else. It could have prevented critical sensor measurements from being recorded that are necessary for product certification, invalidating some products. Could have sent erroneous data to an ERP that's still valid, causing wildly incorrect financials. So there can be a lot of major impacts that come from just having a little issue with syncing between dev and production environments.

14:11
Kent: And so what is the solution to this issue? Well, DevOps. DevOps changes everything. And so any one of these following DevOps practices could have prevented the situation. One, they could have had a better process that was controlled for how they migrate between test and production. So they only release modified code through... Or to production through a control process, which would have prevented this outside vendor from having their view screen and their modified navigation deployed directly to production without going back to the test and dev environments. They could have had a library of regression tests to be executed in a test or staged environment. Or they could have properly implemented version control. And so these could be solutions that would prevent these kind of costly issues from happening for companies, and it's like, "Alright, well, that's great, but how do I do that?" And so to show you that, we're gonna pass it over to Ryan for a demo here, where he's gonna show you some DevOps practices actually implemented using the Ignition platform. And so, Ryan, I'm gonna go ahead and make you presenter now.

15:32
Ryan: Alright. So thanks for handing over, Kent and Mark, and cueing that up for us. First, I wanna talk about Azure DevOps itself. So Azure DevOps, or GitLab, or Atlassian and JIRA, all of those products, these are all CI/CD or Continuous Integration and Delivery platforms that support a DevOps approach to building software. In this demo, we'll be using Azure DevOps. And there's a lot of functionality in DevOps that we're not gonna touch today, you can manage tasks and boards, bugs, track issues, manage test plans, deal with requests from different people, do automated code building. But today we're gonna focus on two pieces that will give us the tools that we need to start with DevOps with Ignition.

16:19
Ryan: So the first of those two tools is Git repos and release pipelines. So, Git repos, or repositories, are source control. They allow us to store our code, all of its changes and all the variations of that code in a central location. And then release pipelines, these are perhaps the single most powerful component of DevOps as it applies to Ignition. Release pipelines allow us to connect to remote machines and build automation workflows to execute tasks from each of those machines. So we can transfer files between those machines, we can move code from DevOps down to those machines, all sorts of different things. And this keeps us from needing to SSH or remote desktop into our Ignition servers. In most scenarios, it'll keep us from even having to go into the Ignition Designer in various environments and to do all sorts of tasks manually. And that means we can make all the steps in the deployment process happen as fast as the computer can handle them. And it'll also do them the same way every single time without any variations. And then all these pipeline stuff, all that we just talked about, gets packaged into a user interface that self-documents the procedure you build, and it pulls all the logs into Azure together, so it's...

17:46
Ryan: We'll look at that a little bit later. But it's gonna be super powerful. And then I kinda wanna give an overview of the environments that we're going to have. So in today's demo, we're gonna be using a four-tier, or four-environment deployment structure, with dev or development, test stage or staging, and prod or production. So starting on the end, we've got the production environment. Pretty self-explanatory, that's what's actually running our facility or our operation. Then we've got staging, and this is an environment that is as close to the clone as production as we can possibly make it.

18:28
Ryan: It is a shameless replica of production as much as it can be. So this is the location where you're gonna spend time building a sophisticated simulator, or maybe even have a small prover production line with an actual PLC where you can test connectivity in different values. Staging is also where you're gonna wanna do your regression testing, which is where we're gonna test that our changes didn't make any unintentional changes. And if you've ever had a deployment fail and then there's an action item that gets created, right? "Oh, okay, we're gonna make sure that gets tested every time we deploy from now on." The staging environment is where you can actually do that testing and where you can actually keep that library of regression tests to run against, so that you really can test that every time from now on. And the staging environment is the key piece that would have saved Developer Dave in the story from earlier.

19:31
Ryan: Then we're on to the test environment. So now we start deviating from production. Test is where we're gonna evaluate the actual changes that we're making for this push of the code. Here is where we'll do very rigorous testing of what we're changing and adding to our project, but we're gonna ignore the rest of the application. And this allows us to have a laser focus on the changes we're making right now. And if there are any issues, we can go back and forth with dev really quickly and easily.

20:03
Ryan: And then to dev or development. And this is where we are actually in the Ignition designer, turning code wrenches and making things work. We make these changes here, send them up to DevOps, then use DevOps to push all those changes to the other environments later. And the cool thing is, because of Git source control and those Git repos, we can have as many development environments as we want to. Developers could even create their own development environments locally on their own laptops. So now we'll get into the real demo. So here, I've got sample servers of these four environments. First we've got our production environment, you can see the system name tag is listed up here in the top bar. We've got the staging environment, we've got the test environment, and we have dev.

21:00
Ryan: So this is the Ignition Quick Start project that comes with the application, with Ignition on install. And we wanna add another menu item here to the top for another page that we can navigate to. So the first thing we'll do is jump over to Azure DevOps, and we'll use a release pipeline that we've created ahead of time to create a new development branch in source control and push that down to the development machine. So while Azure is deciding that it is going to work, to give you a preview of what we're going to do here, when we first run this release pipeline, this is going to create this new Git branch down on our development machine. Then we'll make the changes in the Ignition designer on dev, and that designer save is going to push those changes back up to Azure DevOps and allow us to push those changes into the rest of the environment.

22:09
Kent: And while that's pulling up for you, Ryan, we had a question that came in, saying, “Is Git the only thing you could use here for managing your branches and things like that, or could you use other tools like Team Foundation Server?” 

22:26
Ryan: Yeah, that's a great question. So with the introduction of Ignition 8.0, now we're on 8.1, and there's advances that are I know coming in the future as well, Ignition has moved all of its project code into the file system. And Ignition will actually track the changes to the file system, so you can use best-in-class or any text-based source control system of your choice to manage code in your environments.

23:01
Kent: Perfect. Thanks, Ryan. And some other questions that were in here I'll talk about a little bit, and then, Ryan and Mark, you can kinda fill in the gaps where I leave it wanting. But people are asking a little bit about how you control different types of resources inside Ignition. So one is, “How do you handle Vision windows in version control?” And then another question was, “How do you recommend using Git to track gateway configurations and tags?” And so we do have a resource on our website that's called Ignition 8.0 Deployment Guide, I believe, that talks about these different resources, and the people asking these questions have probably viewed that and they know Vision windows, they are stored in the file system, but they are stored in a binary format. And so they certainly can be managed by Git, but you're not gonna be able to actually go in and see the text changes. And so if you're in Perspective, it's stored in a JSON format, so it's easy to go in, see what's been changed. But admittedly with Vision, you'll see that it did change and it can be tracked and rolled back and things like that, but you won't actually see a text, human-readable format of that.

24:14
Kent: So still can certainly be managed in this workflow, but admittedly it won't be human-readable from Git. And then when you have the gateway configurations and tags, in future releases we hope to add those to the file system as well, so that they can be all managed there. Right now, a lot of people will have it be a more manual step, that they'll take gateway backups and they will take that file then and manage it with Git, or use Git to manage the automatic backup folder for Ignition, or manually export tags or have a script that exports tags to the file system and then manage it there. But yeah, this is a continually developing process here, and we at Inductive Automation are continuing to analyze how we can change the platform to be more compatible with these tools for things like partial gateway configuration backup, so you could just back up your database connections, or just your users or rosters or something like that, rather than the full gateway backup. But, Mark or Ryan, any comments there about the different resource types within Ignition and managing those inside Git or other version control tools? 

25:31
Ryan: Yeah, and I'll just hit on that real quickly, Kent, before I get back into it. As far as gateway configurations, we actually want those gateway configurations to be different for each environment. And so while there certainly, if it was in the file system and we could do that directly that would be a great thing, it's actually not a huge disadvantage, because all those gateway settings are really well separated out, and so that's typically not an issue. And also, inside these pipelines, something I'll point out is that we can run and execute SQL and database changes as well, and manage those. So really when we're talking about limitations with Ignition right now, using source control and other CI/CD tools, really tags are the one piece that still leaves us doing some manual changes and processes. But if you design and architect your solution to know that that's the case, to know that your tag sets are gonna be different in each environment, just like you expect your database schema content, the actual data in your database to be different in each environment, it's actually a solvable problem.

26:53
Kent: Perfect, thanks, Ryan. But sorry, I've been distracting you, you wanna jump back into...

27:00 

Ryan: Yeah, of course. No, I'll get back in. Yeah. To reorient us here, we're gonna do this first step where we create a new Git branch on the development server. So we'll call this new branch “dev-newMenuItem”, and we can hit Create. And so what this release pipeline is going to do is it's actually gonna reach down into our development server and check out a new Git branch according to that name that we gave it down on that machine. And so this is the pipeline view, we'll see some more advanced ones here in a minute when we do some of these deployment pipelines. But what we're gonna see here is that we're gonna be able to jump into this guy and actually see all of the logs that are happening down on that machine where we're executing those Git check-outs.

27:56
Ryan: So once that code hits and shows up and refreshes, we're able to see everything that happened down on that machine, and this allows us to consolidate all of our logs into one place and see everything that happened. So as soon as this branch jumps in, we will hop into our Ignition Designer. So that succeeded. So now you can see I'm going into my dev Ignition Designer. I'm gonna do a project update to get the latest and greatest changes to make sure that we don't create any merge conflicts. And here I've got this header navigation, and I'm gonna come over to this menu, and here is where I'm actually gonna make my change. And so we will change this to look to the screen, and we'll call this “Canvas” and that's gonna be our target. So those are our changes, this is a screen that already exists. So now I can hit Save, and what that project save is gonna do, Kent mentioned it earlier, actually, is that Ignition Deployment Best Practices Guide.

29:13
Ryan: And we set up a script for that to actually push those changes back up to Azure DevOps. And so we're gonna... Any time we save, we're gonna do a Git commit and then we're gonna do a Git push, so that all of that code ends up back up in Azure. So now we will come back over to Azure and Git, and we can see, “Hey, you updated.” We'll go to our new branch that we created. And it'll say, “Hey, you updated this branch just now. Do you want to create a pull request?” And in fact, we do. So a pull request, or sometimes people will call them merge requests, and this is basically, if you're not familiar with version control, this is a way for a developer to say, “Hey, these changes that I made, I'm confident in them, let's try to push them another move down the line and see how they work.” And what we're gonna do is take this new menu item and these changes and we're gonna push them into the test branch. And what's cool about merge requests is we can protect branches and set up requirements for approvers and reviewers and people to actually make sure that this is legitimate changes, so that not anyone can just go in and modify the code. And we'll create that pull request.

30:51
Ryan: DevOps is gonna validate that there aren't any conflicts for us. This is the page where we would do approvals, people will get emails for that. And then once we're confident that it's what we want, we will complete that merge, and that will actually merge those changes into the test branch. And so once that is complete, we will go back over to our release pipelines, and now we are going to deploy these changes to the test environment. So, to make sure that you guys all know that I don't have anything up my sleeve, here's that change on the dev environment, where we have this Canvas button where we can click that guy on the menu and come over here and we can then go back to home, and now it'll be on this side. That's there. On our test environment, we don't have that button yet.

31:54
Ryan: So we can come over to the release pipelines or deployed a test, create a new release, and hit create. We'll see that “Release-8” has been created. And now if we jump over to this, we can see this workflow. So this one is a little bit more advanced. And each of these cards is called a stage, and these stages can be organized in any fashion, you can split and merge and do all sorts of different things. So as we push this code to test, what we're gonna do is look at this and see, we took an Ignition backup of our test system, we actually took SQL backups of the test system, and we had an opportunity to take tag backups, we didn't in this case. And then we're going to actually move our Ignition code into test, so that's the point where we move all of the project files over. Then this has stopped, so this is the time I'll show this, this is a manual intervention step where I can hit this resume button, I can actually write instructions to go here.

33:03
Ryan: So this is an opportunity for me to make sure that everything is square before I actually run my SQL updates. So I can hit resume on that, and now it'll run whatever SQL updates are a part of my changes against the databases that I've configured it to change, as well as, we mentioned talking about tags earlier, we'd have a manual intervention step to do those manual tag updates if we had them. So we don't have any here, we'll go ahead and click on that, and now we've deployed the test. So if I jump back over to our test environment, you can see we've got this Canvas button right here. So now we deployed these changes to test without opening the Ignition Designer. We did an RDP into the Ignition server. We took backups. We didn't open SQL Server Management Studio, we took SQL backups. We did all of that in a matter of minutes, that we definitely couldn't have done manually by ourselves in that amount of time, or with that level of consistency and reliability.

34:13
Ryan: So jumping back over to our sort of map of where we're going, we made our changes on dev, we pushed those back up to DevOps, then we deployed those down to test, and then we tested our changes on test, right? We made sure that, hey, when we went over to this page, we were able to do everything that we intended to do. If we had different functionality inside of these changes, this is where we would evaluate all of those, and test is the environment where you're permitted to make changes to data and the database, and sort of tweak things and push things to the limit. And you don't have to be worried about changing settings back for stuff later, because you're gonna end up pushing the same code that you deployed to test we're gonna deploy to stage, which is exactly what we're gonna do right now.

35:09
Ryan: So once those changes are considered good to go, we'll actually create another pull request, because we pushed those changes from this dev-newMenuItem into test, and now we're going to push them into staging. So same thing, we will create another pull request, hit complete on there, and merge those changes. And once that merge is complete, we'll go back to our deploy to staging pipeline, and run that as well. So we can go over to our pipeline view and actually see this progression happen, as it happens on all of these individual machines. And so when we set up these environments, it's important to note as well that in our example here, I've got Ignition and SQL, and in this demo they're on the same server. You could have eight Ignition servers and 10 databases and an ERP emulator VM, and whatever else, in each of these environments, and DevOps is gonna be able to handle deploying all of them.

36:34
Ryan: And any of the CI/CD tools are gonna be capable of doing those things as well. And so this is moving that Ignition project code. So while that's happening, we'll come over here, and now we can see in our staging environment, we've got this as well. So once we're to our staging environment, we're gonna go through moderate testing of our changes, but the big thing is this is where we're gonna run our library of regression tests, or a library of tests that we know, hey, we need to run these tests to make sure that whatever we did didn't mess anything up. So if we look back to our Developer Dave story, they lost a menu item that was a critical page of something that they really needed in order to run their operation in their facility, and not having that messed that up. So that gets on to a checklist or on to a test plan, and this is the location that you actually work and manage all of that.

37:40
Ryan: The other significant thing about making this environment, its only purpose is sort of validating that all of those regression tests still work and that it's as much of a copy as production as possible, even all the way down to including digital twins for critical equipment, or PLCs, or this is also where you could connect... If you've got ERP integrations, where you could connect the staging or QA environments of those ERP systems, this is the place where you can do that, is that this deployment, we're now not only testing the changes that we're making. The code we changed, it does work, but we're also testing the deployment process. So we're making sure that we're moving all the scripts into the place that they need to move, that we're doing it all in the correct order, that we're not forgetting any of those deployment steps. And if we test and we validate that here, then that means that when we release to production, which is the next thing we're gonna do, at this point the demo kind of feels, I'll admit it, a little bit repetitive and boring.

38:58
Ryan: You're telling me that the next step in this demo is we're gonna create another pull request, and this time instead of going to test or stage, we're gonna go to production. We're gonna create a pull request, and then we're gonna hit Complete, and that's gonna queue and then we're gonna run this release pipeline that's gonna be identical to the one that moved to staging, only now it's gonna move to production.

39:29
Ryan: And so we've got our deployed to production, we will hit create, it will start, and we'll jump over here to our production server, and while this is running, we'll kind of keep talking about this is that your deployment to production is going to be boring by the time you've done this. We're doing all of that testing and all of that to make sure it works in the staging environment where the risk is low and we're not threatening production downtime. And the great part is is that DevOps doesn't care if your code changes are 10 lines or 10,000 lines, this process is gonna take the same amount of time, it doesn't matter if it's one screen or 50 screens, this code is gonna all push down, the procedure is going to be the same every time, and you're gonna earn a consistency and a speed that would never be accomplishable without some sort of CI/CD tool.

40:31
Ryan: And so here, now you can see we've got our Canvas in production and everything works really well, so that's the demo, and really wanted to show that to you all this morning to highlight when somebody suggests... If you're a customer and you've got an integrator that suggested this or an engineer that suggested it, or if you're an integrator looking to do projects and... It's a lot of work to have three environments, much less four environments, utilizing DevOps and utilizing the CI/CD tools is actually what makes that possible to do it makes it a reasonable amount of effort that you have to put in, because we're letting the computer do all of the work for us, and that's just a huge advantage to reducing risk, reducing downtime, and getting to work on the problems that really need solving.

41:29
Kent: Perfect, thank you so much, Ryan, for that presentation. And there were a plethora of questions that came in during that process, and we'll just hit the last couple of slides and we'll jump straight into the Q&A, and so as a recap, what did we just see? More or kinda the highlights there? One, you should have multiple environments. He showed dev, test, staging, production. Dev is the only place that he launched the Designer, so he's not launching the Designer to then manually make changes in test or to even import changes from inside the Designer. And so that's a big deal here, is making the changes in dev and then letting DevOps do all the migrations for you. He also showed us the source control for the visibility of the changes, we could go even deeper in that, there's only so much we can show in one hour here, but showing, opening up Git and comparing text changes seems like that's all possible with this process. And then as part of the deployment process, it gave him a standard way to migrate these changes, he even said it's getting boring at the end as we migrate to production, and that's a great problem to have, to have that consistency so that you know what happens the same every time.

42:44
Kent: And because of that, these are self-documenting pipelines, they're simplified traceability, you can see what happened, it's really easy to see what's going on, and because you're eliminating all those manual steps, you're reducing user error, you're not taking all these individual backups going and restoring them, maybe you restored the wrong one, things like that, or you didn't export what you were supposed to export, and so overall, you're minimizing downtime and operational risk, and so as a conclusion here, DevOps provides that structure for all these different tiers for source control, isolated development, deployment process and CI/CD overall, which gives you those flexible priorities, gives you those rapid turnaround times, so much faster, and also coordination with multiple developers and minimizing that risk, and so I went through all that really quickly because we wanna get to your questions, but a lot of this is asking how to do it. And maybe the best way to figure out how to do this is to talk to the expert, Ryan and his team at Vertech. And so, Ryan, you wanna talk a little bit about the slide? 

43:58
Ryan: Yeah, absolutely, so we're offering free design consultations right now for setting up DevOps within your Ignition environment, and so we'd love for you to schedule an appointment and talk about that, this is a process that... There's a lot of nuance, there's a lot of little gotchas, nothing that's hard, but a lot of experience that we have doing this for customers and pushing forward in working with IA on making sure that this is something that can get out into the world, and so we'd love to have a conversation with you and help you have your development and deployment be that much more reliable and that much more risk-free.

44:40
Kent: Perfect, and if you guys wanna schedule time with Ryan and his team, there's the link that's right here on the slide, another thing you can certainly download Ignition and try it from our website, if you're new to Ignition, you wanna go see how this all works, definitely download it, play around with it. And if you wanna learn a lot more, attend our conference, we have ICC coming up here again, and ICC is our annual conference. What's exciting about ICC is it's a free virtual event, 'cause we're still in the middle of a pandemic here, so instead of hosting it in Folsom, California, it's gonna be online again this year, we'll have a Keynote address, a Developer Panel, panel discussions, community sessions, and really what everybody's been waiting for is the third installation of the Build-A-Thon where developers go head-to-head against each other to build something in just under an hour on the Ignition platform, and I've been part of that last two times, and it's always been Travis and Kevin going head-to-head from Inductive Automation on who can build the cooler stuff.

45:51
Kent: If you've been following our blog posts and our social media, you'll know that this year we're changing things up a little bit instead of having Travis and Kevin go head-to head, we're actually having two of our top integrators compete, one of those being Vertech and their champion who will be representing them is no other than Ryan Crownover who you just saw present, and so Ryan, any comments from you on how it feels to be one of the first competitors that's not named Travis or Kevin? 

46:20
Ryan: Thanks, Kent. It's definitely humbling and super daunting, but I'm up to the challenge, been working on my typing words per minute, more code you can write in less time, the better, and so we're gonna be ready.

46:37
Kent: Well, perfect, well, everybody, we're hoping that you can come tune in to see Flexware Innovations versus Vertech in that Build-A-Thon, so stay tuned, and you can register for free at icc.inductiveautomation.com, also, if you have questions about our product, certainly reach out to us, reach out to our Sales team, and also you can reach out to Ryan and Mark as well to talk about what we've discussed today and about our products. And so with that, we're gonna go ahead and go into the Q&A here. Mark, I wanted to start with you, for this demo, we've shown being able to do this in some of the Ignition standard functionality, being able to add navigation or something like that. Does this process change at all when you bring in the MES suite of modules? 

47:32
Mark: Yeah. Of course, there's some details there, it may or may not be different, it depends on the nature of the changes that we're migrating from dev through that process. We do have some particular features around specific import and export of MES objects and any data model changes that were made, so for most of the time, no, probably no change to that process that we saw today, but depending on the nature of the change, we might be using some database features, we might be using some MES-subject export type features. So devil's in the details there a little bit, that's the kind of thing that we'd be happy to do with design consultation call, that's our free design meeting, just like IA provides with Kent and others. We're happy to do that, so feel free to reach out if you've got specifics on that. Thanks.

48:33
Kent: Perfect, thank you, Mark. Ryan, can you talk a little bit about this process? Is it operating-system agnostic or does this only work on Windows or Linux? Any comments there? 

48:46
Ryan: Yeah, it works on... Definitely works on Linux. That is my preferred system of choice, and so all the tools, whether it's DevOps, GitLab or the Atlassian Geo products, it's all gonna work on Windows and Linux and just fine, obviously, your pipelines and your scripts are gonna be tweaked, but the process and the workflow is the same and totally doable in both platforms.

49:19
Kent: Perfect, and Ryan, can you also comment on integrating some automatic tests into this process? Is that possible? If so, are there specific tools that you would recommend as people try to do automated testing or should that be in Ignition or in the DevOps tool? Any comments there? 

49:40
Ryan: Yeah, so there's a few ways to approach automated testing, Ignition is... We are limited to some degree as far as what we can hook into, we can't build this whole test harness and connect with it, but we can build test inside of Ignition, we can use tag interfaces to test as well. So let's say you've got Ignition talking to a PLC in your test environment or your staging environment, you may not have a PLC, but you might have an OPC server that runs your staging environment and it can actually run through test programs that would trigger functionality inside of your... Inside of Ignition to replicate scenarios a machine might go through, and those could be historian replays of real data from production, so there's ways to test there, building REST API harnesses for automated testing is always something that we've done in the past. There's lots of different options, and it's not been explored a whole lot, but now that Perspective is in the browser, there's all sorts of doors opened to testing against the browser level and hooking in that way.

51:03
Kent: Perfect. And a lot of people were really interested in understanding a little bit more about the migration of SQL changes, could you go a little bit deeper into the magic behind the scenes of how that happens? 

51:19
Ryan: Yeah, so managing SQL changes between environments is a challenge in any application platform, Ignition or non-Ignition, and so a lot of the wins there are going to be based on how you design the process and what you actually include in your SQL changes, and I'll just... I'll give away the big one, and that is that you want those scripts that you run, and I think someone in the questions actually pointed out, there is a script that's packaged into source control that's run, if you set up those SQL scripts to only modify the structure of the database and have it be responsible for not harming the data, then you can manage those scripts from environment to environment a lot more easily, so there's a lot more that goes into that, but that's... The first step is changing how you think about how you apply database changes from environment to environment.

52:23
Kent: A couple of last comments. Do we have a step-by-step guide for this process? No, we do not, and that's not just because we're too lazy to build one, it's because for everybody's environment, this process is gonna be a little bit different, and so it's hard to create a one-size-fits-all solution, which is why we recommend working with Vertech or a partner like that, to have somebody come in who knows what they're doing to help you analyze your environment and figure out what's gonna be the best solution for you, and so certainly take Vertech on their offer for free design consultation on this, that's gonna give you a good jump start there, and we do anticipate continuing to release more resources around this topic, it's been a very popular topic lately, but no, as of this time, there's not a step-by-step guide or a white paper of how to recreate what you've seen today. As we wrap this up, I just wanna first go to Mark for any last comments, and then we'll go to Ryan for any final comments here, so Mark, any last comments before we wrap up.

53:32
Mark: Well, just thank you to everyone for tuning in, and we hope that you take this information and utilize it to again, get that balance right of stability and feature delivery, and that it results in better Ignition projects out there in the Ignition community. Thanks again for your time.

53:54
Kent: Thanks, Mark. Over to you, Ryan.

53:55
Ryan: Yeah, just to appreciate everybody joining in and watching the demo, listening to the presentation, and just to add... 'cause I know it's a question. The same thing we did in Azure, you can do in any platform that provides the CI/CD tools, people talked about open source, GitLab is one of those, the Atlassian Suite, there's a few others, the GitHub tools would also work as well, so all sorts of different options for which system actually does the pushing of the bits.

54:31
Kent: Well, perfect. Well, thank you so much for your time, Mark, and your time, Ryan. I think this has been very valuable to the community, certainly, we've had a lot of positive messages in the questions as well, so thank you everyone for attending, and with that, we'll go ahead and wrap up for today, but I hope everybody has a great rest of your day and we look forward to many more webinars with you. Thanks,bye.