Building Effective Plans for Risk Management

Today’s guest is Technology Director Mike Walden, who’s here to discuss what New Frontier Technologies is doing to help customers identify and solve security risks during process design, project implementation, and in existing systems. Mike covers the process for building risk mitigation plans, shares the most common mistakes and biggest trends he’s seen in preventing cyber security threats, and offers tips for those struggling with risk management. We also talk about where documentation comes into play. 

“We have developed some methods that are tailored to the automation control, SCADA, IT systems that contribute to our client's process safety management regime.” – Mike Walden

Bio: Mike Walden is the Technology Director at New Frontier Technologies (NFT). NFT designs, implements and maintains Operational (OT) and Information Technology (IT) systems for midstream operators. Mike has been working in the oil & gas industry since 1989, specializing in automation and information management systems. Mike holds a Bachelor of Science degree in Mechanical Engineering from Kansas State University.

Episode Transcript:

00:00
Justin: Welcome to Inductive Conversations. My name is Justin Reis. I'm the Integrator Program Manager at Inductive Automation. And today, I'm excited to be joined by Mike Walden from New Frontier Technologies. They actually have two offices, one in Kansas City, one in Houston. Mike, thank you so much for being here today.

00:17
Mike: Thanks. Thanks for including us. We're honored to be on a podcast.

00:22
Justin: Yeah. That's great having you guys on. Could you just give us a little introduction to New Frontier? Your services, your specialties, industries it serves?

00:31
Mike: Sure. We're New Frontier Technologies. We are a systems integrator. We think we're a little bit different from traditional integrators in a couple of respects. The tagline we've chosen to describe our company is "Where operations meets IT," which is kind of a cute little marketing phrase somebody came up with. But it does encapsulate what we think our specialization is, which is helping industrial clients bridge the gap that exists, in almost every industrial organization, between the people who operate and make things work and the IT department. There often, those two groups of people don't get along. Sometimes they do, but very often they don't. There's at least animosity or difficulty. And so we often use the analogy that we are technical therapists. We were there to hopefully be able to understand and interpret what the operations team says they need in language that the IT team can understand and vice versa. Taking IT standards or jargon and the expense that always goes with IT, investments, make that relevant to the operations team.

01:51
Mike: We work in a variety of industries, predominantly oil and gas, food and beverage, chemicals, agriculture, and something we call OEM. We have a number of clients who build their own machines, and we help them automate or standardize those machines. Generally, we offer solutions in a handful of categories; automation to control, which is we program PLCs, HMIs, distributed control systems, SCADA obviously, which is how we've had an excellent relationship with Inductive. And in fact, SCADA is its own category for us. What we call SCADA and industrial IT, we deliver virtualized systems, address cyber security concerns, enterprise SCADA. We also do training. We have something we call design services, we build control panels. We think that's one of the thing that makes us a little bit unique, is we cover everything from the IT side of operations, down to control panels, actually wiring up PLCs and relays and really glamorous, exciting things like that. Yeah, and then we have some packaged solutions, some things we've used Ignition to build applications that address a real specific customer problem. In the agriculture industry, we have a platform called NFT Grow that helps automate some agricultural products. We have something in oil and gas we call Smart LACT, which is a utility that helps a oil truck, people that drive trucks and unload oil, we automate those processes. That's probably way too many words, but that's what NFT does.

03:41
Justin: Yeah, that's quite a resume. I especially like the technical therapist one, that's a new one for me. If it's a need though, and I think that's a great term for it. Can you tell us what your role in the company is?

03:51
Mike: I am the Director of our SCADA and Industrial IT solution. So my responsibilities include finding business and making sure that once we have business, a contract with a client, that it gets executed well. That we have the right people in place, that they're well trained, that they have the tools they need to get the job done.

04:09
Justin: Great. And I noticed, I was going through your website before we jumped on the podcast, and I noticed that, in your core values section, it mentions New Frontier strives to bring new concepts to each discussion with clients, with the goal being an increased profitability by implementing better, more cost-effective solutions through the use of advanced technologies. With New Frontier Technologies being an Ignition Premier Integrator, I have to believe that Ignition has played a role in providing some of those new concepts and cost-effective solutions to clients. Can you speak to how Ignition is aligned with your company's core values?

04:41
Mike: Yeah, exactly. One of the best ways we convey that message practically to our clients is we take them into our lab. Unlike most systems integrators, we make a pretty significant investment in what we call R&D, which is not necessarily creating products. We don't create our own PLCs or our own SCADA software. But we've got a pretty extensive lab with samples of all sorts of things that our customers use, and Ignition is an integral part of that. For example, Inductive's decision, it's been quite some years ago now, to invest in MQTT and be one of the, really, the leaders in our industry and embracing that new, what was I know it's not really new, it's been around for 20 some years, but that's been an important component for us. We've been involved with several of the companies, particularly in the oil and gas industry, who've been the pioneers in cutting their teeth and learning the hard way how to implement MQTT. And Ignition's commitment to that platform, that protocol in particular, and as well as several other key things about Ignition, the way you package it, the fact that R&D is easy when you can do it in two hour chunks and a couple of clicks and you're restarted, and it allows our people to just be creative and understand the client's problem solving in our lab. And then take it out and show it to them.

06:11
Justin: Right. That's great. And then one little plug I'll put on the Integrator program here, being the Integrator Program Manager that I noticed you guys are really pouring the coals on the certification. I noticed you have five new engineers taking the Core Certification in Inductive University. So well done on that.

06:27
Mike: Yeah, thanks. We've actually, within the last year, we've incorporated the Ignition training as almost an onboarding process for new people in our SCADA and industrial IT group. Yeah, we expect essentially everyone here who is gonna be involved in SCADA and IT projects to have in their certification.

06:50
Justin: Well, that's awesome. That's my ideal scene, trying to get engineers trained in Ignition and with Inductive University, that was the goal setting up so that you guys could get free certification, free training with the University. So I appreciate you taking advantage of that. So switching topics here. Can you share with us some of the successes and challenges that you've had, or that New Frontier has had working in the various industries during the COVID pandemic?

07:13
Mike: Sure. We have been fortunate during this pandemic, nearly all of our clients have continued to make the investments that they were making in us and in their projects prior to. So we haven't had any project cancellations, which I think is rare and may just reflect the fact that we're lucky. I wish I could say there was something real clever we've done. But like everybody else, we've had to adapt quickly. It sounds like you've read our core values. One of the other key things that has been integral both to our success, and it's something that we've worked hard to groom into the fabric of the company is this concept of collaboration. So pre-COVID, each of our project teams, literally, were working very close together, sitting at tables and desks right next to each other physically.

08:03
Mike: And we've had to adapt that, obviously, like everyone else in the world, to doing it online. I would say that for us, the challenge that we continue to have, and we haven't really figured out is we're a company full of engineers and real technical computer scientists. And those are not the sort of people that are naturally expressive and naturally want to talk about themselves or naturally want to speak up on a conference call. They're the type of personalities that are happy to just sit still and let someone else do the talking and I'll get my work done. And so we have had to consciously work on essentially like small group dynamics online, trying to encourage our leaders to draw out input from the people that really have it, but who are naturally too reserved to insert themselves dramatically in a conversation. So we've done some specific things to try to facilitate that and make particularly our leaders a little better at online meeting guidance, but we're a long way from having figured anything out, to be honest with you.

09:14
Justin: Gotcha. And you guys are pretty diversified in terms of what you do, has that played any role with all the different things that you do in helping the company stay successful through this time?

09:24
Mike: Yeah, yeah, diversification has been a key. We have a couple of clients in industries that they're making machines, for example, I mentioned we do some work for OEMs, and COVID has brought them significantly more business. And they've had to ramp up and needed our help to support that. Whereas some clients, COVID coincided with a crude oil crisis that caused maybe some of our oil and gas clients to step back slightly from some of the future investments they're planning to make. So it definitely has been affected. And to your point the having a diversified client base has helped us, has contributed to the fact that we haven't had to lay anybody off, no drastic negative effects.

10:15
Justin: Right, that's great. Well, I'm glad you guys are doing good through this pandemic and keep that up. Let's focus a little on some of your areas of expertise, starting with the work that New Frontier does to help customers to identify security risks. What is your process for identifying a risk for organizations?

10:31
Mike: Yeah, that's a really broad subject. So as I said earlier, most of our clients are industrial, and they all adhere in one form or another to this concept. They have a process safety management program in place usually in some fashion that's designed to identify physical and operational risks to whatever process they run. And so we have developed some methods that are tailored to the automation control, SCADA, IT systems that contribute to our client's process safety management regime.

11:12
Justin: And what are the most common areas of risk that you help your customers deal with?

11:17
Mike: The approach that we take generally falls into three categories. So we identify risks in what we call a process design basis, where we will analyze a control system’s design. It's usually comes to us in the form of a control narrative or a cause and effect diagram. The client describes to us either... And sometimes just verbally, but hopefully in writing, or we help them put it in writing. And the goal there is to flag obvious shortcomings or issues that are not clearly defined with the process itself. How are we going to control whatever it is this particular client needs to control and identify the operational risks. And once we have those, then decide which of these things can automation solve for you. How can we protect you and your operators and your neighbors by automating something and what things can mainly be mitigated either by some manual process.

12:20
Mike: So the first category is looking at the process design. The second risk area that we help our clients with, we just call the project plan. And what we find that most of our clients overlook is the most expensive and the most dangerous time in a project is its initial implementation. Like the customer X, Y, Z is automating an old facility. The facility is already running, we're building a new control system, that new control system has to go in. If that control system takes two extra days to get installed, we may have just wasted more money than 10 control system costs. The project implementation plan is itself riddled with risk and that's an area we help customers try to identify. How are you going to implement this new system, how will it affect what your current operations are, and how can we mitigate all of those risks? What back-up plans do we need to have in place? We help them lay out an incremental sequence of how their new application will be deployed, so that at every step there's a fall-back position. If things don't go well we're still safe.

13:37
Mike: If things don't go well you're still making money, you're still operating. And then the last big category of risk, we address is cyber-related risks. So we've developed, really, a specialized consulting practice and a process that focuses on cyber threats. We work with our clients to help them scope or define the scope of their cyber issues. Are we looking at an enterprise-wide SCADA network, a couple of facilities, a single compressor station, a single-unit control panel. Once we define the scope, then we follow a pretty detailed process to document the systems from a cyber security point of view. And once they're documented then we can flesh out detailed risks and mitigation approaches for each risk.

14:27
Justin: Right, that makes sense. And I think you kinda touched on my next question a little bit, but I wonder if you have any more details. So can you tell us a little bit more about how you're working with the customers to build plans for overcoming these risks?

14:38
Mike: Yeah, I'll focus particularly on the cyber aspects just because I suspect that's the more relevant or the most relevant of those categories. Or that's the place where we're getting the most traction, I guess, from our clients. So the specific things that we do, I would bet more than half of the engagements we get involved with are related to identifying cyber risks for an existing facility. Something that's already there either as I said earlier, an old compressor station or a manufacturing plant that's been running on a good old system that hasn't been touched for 12 years or 15 years, whatever. And so, clients in those situations, they come to us either because they've had an incident or they're contemplating a major upgrade and they know that cyber is something they need to take seriously now that they didn't have to 10 years ago.

15:34
Mike: Or like I said earlier, we get called in because an OT person got a call from the IT Department and said, "Hey, what's with this network down here?" And they've got questions that can't be answered. Or the IT department is trying to figure out, "We have all this stuff and we don't really know what it is." So your question is, “What do we specifically do?” generally falls into two categories. We help customers assess what they have, because there's lots of existing equipment, and over the years, particularly, networks have exploded. If you own or operate an industrial facility over the course... Even probably if we just said, "Hey, over the course of the last few months, almost every facility somewhere a technician has come in and needed to connect something new, so he brought in a little switch and he plugged in a cable, and he plugged it into the router at the site, and he just figured out how to get it working."

16:34
Mike: And nobody has time to track or realize the fact that there's real threats there. So an assessment just helps a client identify what he's got and keep track of all those things that probably got cabbaged together by accident, all with people with totally good intent. Nobody's trying to do anything bad. So an assessment helps the client identify what he's got, the next step is often an audit, which sounds like something to be avoided. But in this case, a cyber audit is really just taking the results of that assessment and selectively identifying whether or not real problems exist. It can include network testing, simulated network attacks, and an extraction of configuration information in a control system. We have a partnership with a company in South Dakota called CSI, but it's not crime scene investigators, but they help support us in these audit processes that are tailored for industrial applications.

17:42
Justin: Yeah, that makes sense. I think probably the answer to my next question varies a bit, but when is this process typically happening with the customer?

17:49
Mike: Yeah. Of course, you're right. It depends. They're either wanting to do this because they've had some incident. And it's rare, we don't get a whole lot of contact from... The incidents are rarely, "Hey, we know that some bad international actor from another nation is trying to overtake my mixer." It's rarely something like that, but the incidents are usually the result of some clumsy network design or... So something bad has happened and a client is calling us, that's one. And then the other categories are kind of this idea of, "Hey, our corporation is trying to define what we have in these control systems and no one here really knows. Can you help us figure it out?" Or something worse. My corporation IT department is breathing down my neck, and I don't have time to gather everything that they want. Please help me.

18:48
Justin: And then you guys come in as the technical therapists, to put it all together.

18:51
Mike: Right, right.

18:52
Justin: Love it. Is it usually about problems with new SCADA systems or with older systems that have had problems for years? What do you usually see?

19:00
Mike: I think right now, the most common issues are related to existing systems. That's by far the most frequent. Nowadays our company, and I'm sure many others like us, have a reasonably well-defined approach for new systems that address cyber issues, but the older systems are far and away the most common.

19:19
Justin: Right. And where does the documentation come into play with risk management? I know that's another specialty of New Frontier Technologies, and also a critical piece of SCADA systems.

19:29
Mike: Yeah, exactly. It's very important for cyber-related issues and really for SCADA systems, the first thing we do is map existing and planned networks. I know I keep referring back to the same thing, but that's a key thing, to actually have a document. We use some tools, some utilities, that connects to our client's network at key points on their network. And these tools will come up with an enumerated detailed list of every host on a known network, and that allows us to document what's there. After that, we take a list of hosts and apply something called the Purdue Reference Model. It's something common, it's been around for a very long time.

20:12
Mike: But we use the definitions in that model to assign each host to a network zone based on that host's function and purpose, and then we compare what that host's designated zone is with what it actually is connected to in the network. The result is a document that spells out, "Here's plainly what we have. Here's at a minimum how it should be given best practices." So that documentation process, particularly in cyber-related and SCADA systems are elaborate networks, every single one. So the documentation piece is integral to helping improve and fortify all of those control systems.

20:56
Justin: Right. Thanks for that. And now New Frontier Technologies does a lot of consultation with oil and gas to prevent cyber security issues. What would you say are some of the biggest or most common mistakes companies are making when it comes to putting themselves at risk?

21:08
Mike: I'd say a couple of categories are the most common things that we discover. One is, I'll just call it random interconnected networks. As I hinted at earlier or referred to, it's really common in any operating application, things change every week, every day. We have to add new pressure transmitters, we have to add new valves. And sometimes in the course of hastily making improvements to the process, the control systems and the networks are just necessary portions of making those improvements and they have to get done quickly, and particularly in the area of cyber affected things; networks, servers, computers, things get added in a non-standard way or without being done in accordance.

21:58
Mike: So the most common things we see are things like I/O networks intertwined with PLC HMI networks, intertwined with DMZ networks, all of these things that should be separate and protected. We have a near tragic story from one of our clients who had deployed a control system with remote I/O, a clever use of putting one PLC and the main cabinet and remote I/O racks all around his facility. And those remote I/O racks happened to be connected to the same network as his HMI and an email server. And when that email server wound up getting flooded, it flooded the network, the I/O modules couldn't communicate with the PLC. The I/O modules, whoever had configured them had not correctly assigned the default position for some digital outputs.

22:56
Mike: And a very big compressor got left running in what's called a runaway mode, they couldn't stop it. And the core cause was a poorly designed network, where someone had just interconnected I/O networks with PLC networks, and eventually an email network that flooded the whole site and very nearly caused a very tragic incident. Luckily that was avoided, with some daring technician work. But that's by far the most common thing we see is poorly designed networks and interconnected networks. The other very common cyber risk that almost all of our clients, we discover in an assessment or audit, is outdated equipment. That, for the last 10 or 15 years, almost every piece of automation equipment is a piece of hardware, a PLC or a flow computer or whatever, but it also has software on it, and most of the time that software needs to be updated.

23:53
Mike: And usually the updates exist to address some security or network risk. And those are really difficult things to manage. So a lot of the solutions we wind up deploying our methods to help our clients keep those firmware and software revisions up to date at at least some regular intervals.

24:15
Justin: Right. And what are some of the biggest strategies you have seen as far as solutions to cyber attacks?

24:21
Mike: Yeah, addressing those two key concerns. It has become very common. There are a number of tools now that are available that can either run on a network as a dedicated appliance or run on one of a server or a workstation on a network, that help manage the two issues that I just mentioned, runaway networks or random interconnected networks, as well as firmware or software revisions of key automation components. So we work with a couple of those vendors. Some of them are the big vendors, like Cisco. Cisco has some utilities. Others are smaller operators that you may have never heard of, a company called SCADAfence and others like them that have solutions that are tailored for industrial networks that help address these risks. And so deploying those is becoming a more and more common piece of a solution. Certainly something we recommend. Not every client is willing yet to spend extra money for a cyber protection solution, but it is becoming more and more common and more and more affordable.

25:30
Justin: Awesome. And how has New Frontier been utilizing Ignition for cyber security and risk management and perhaps documentation as well?

25:38
Mike: You hit the nail on the head there. So some of those tools that I mentioned that perform some of the network monitoring and network awareness, they have standard interfaces. So for some, we have used SNMP and another Ignition partner has a nice add-on module that we have used. The folks at Kymera have an SNMP module that bolts right onto Ignition and allows us to map to data that's being collected by these third-party network monitoring utilities. And some of them also have an API that we've used ourselves to create an interface, so that we can use Ignition to be the visibility, so that our clients can use...

26:25
Mike: They're using Ignition to monitor and control their process, but we typically wind up with extra screens, extra Ignition graphics that are dedicated for maintenance, management and support. And those report on the health of the Ignition system itself, but as well, of the entire network. And we can enunciate alarms, route alarms using alarm pipelines to distribute network-related events across an enterprise. People who would otherwise never need to touch the Ignition SCADA system, are receiving alarms on their cell phone or in their email, for a server that's running out of memory or a firmware version on an OMNI Flow computer that is so old that it's risky. So, yeah, Ignition has become an integral piece of propagating and displaying the analytics that are created by these network monitoring utilities.

27:21
Justin: That's great, yeah. I love hearing the different ways Ignition can be used. It was really cool to hear all the different ways. I hadn't really explored or seen how Ignition has been used for cybersecurity specifically, so it's really cool to hear what you guys are doing with it.

27:33
Mike: Yeah.

27:34
Justin: Well, do you have any unique projects coming up for New Frontier Technologies that you'd like to share?

27:40
Mike: We've got several Ignition applications in development right now, and a couple of them are in the crude oil industry. We are taking the Perspective app and creating, for crude oil operators, essentially a hand-held device that gives field operations much quicker access, or knowledge rather, about equipment that's failing or something that needs to be addressed in the field, so that they short-circuit what used to be a milk run, where a guy in a truck drives around, in the course of a month, he visits 300 pump locations or what are called locked LACT units, and just checks on them.

28:19
Mike: Instead of doing that, we optimize his time by using Ignition alarm pipelines to send him messages about the three most urgent places he needs to go today to keep that company profitable. But that crude oil Perspective app is being expanded now to allow the management of these crude oil operators to use Ignition to help improve their process. And what I mean by that is folks that carry crude oil have to meet certain quality obligations. And if they can combine crude oil of one quality in just the right proportion with crude oil of another quality, they can deliver exactly what their customers want in a far more affordable way and make themselves, of course, more profitable. So using Ignition to automate some of the calculations and the work that those leaders and the operators have to do to really optimize their business, we're letting Ignition do some of the hard thinking and present the results. So that the folks operating these crude oil facilities can just simply have to decide, "Are we blending north to south or south to north today?" Things like that.

29:31
Justin: Yeah.

29:32
Mike: Ignition, for us, has become a very important part of a number of things that we are offering to our clients, and as I said earlier, it has become the integral piece of our lab, and how we build and test things for potential new solutions. And because it's so easy to get started with, sometimes we just use it as an entree for all of our new staff to learn about automation.

29:57
Justin: Yeah, absolutely. It's the unlimited platform, so its uses are really infinite. So that's great. That's awesome to hear what you guys are doing with it. Well listen, Mike, we're about to wrap up here, but before we do, is there anything else you wanted to say in our conversation today? Any final thoughts or tips to our listeners who may be struggling with risk management?

30:15
Mike: Well, first off, I wanna say thanks. Thanks very much for including NFT in today's podcast. And as I said earlier, we're really honored to be a part of it. And then, yeah, as far as risk management and mitigation and particularly cyber-related risks on one hand, it's not rocket science, it's just like all the decisions people make all day, every day about mitigating risks, simple risks, in our own lives. But in this case, a formulaic approach. Something that is structured that guides an industrial client through the process is really valuable. We find that, as we talk to our clients, aside from all the benefits of working with what we think we have a great team and all of our people are lovable, smart people, that do a good job, one of the most valuable things I think our clients recognize is just that it's such a vague and general area. When we show up with a plan, that's the most helpful thing, I think, is having a method, a place to start. So I guess whether industrial clients use us or not, I think finding a plan and a recipe to actually get started, it is the most compelling thing that needs to happen for most industrial applications.

31:33
Justin: That's a great insight. Mike, thank you, it's been awesome. I really appreciate all your insight into the industry, and what you guys do, and I'm learning a lot more about New Frontier.

31:41
Mike: Yeah, thanks again. Thanks for the privilege of being here.

31:44
Justin: No problem.