Water Infrastructure Company Replaces Point-To-Point VPN With MQTT

According to Goodnight Midstream’s Chief Information Officer, Kevin Cooper, “For anybody who has done large-scale networks, especially distributed out to edges that are way out in the middle of nowhere, VPN tunnels can be the bane of your existence.”

That’s especially true for Goodnight Midstream, a produced water infrastructure company for oil & gas producers, because unlike traditional methods for produced water transportation, Goodnight Midstream has an extensive network of water-gathering and transportation pipelines across the US. This network allows for saltwater to reach geologically sustainable disposal wells while eliminating greenhouse gas emissions produced by trucks and protecting freshwater resources.

Finding An Alternative To Point-To-Point VPNs

Goodnight Midstream’s SCADA architecture consisted of multiple point-to-point VPN tunnels for securely connecting their facilities and central hub. Managing each VPN connection’s firewall became complex as each required configuration, monitoring, and maintenance. This traditional point-to-point VPN setup also limited the growth of Goodnight Midstream’s infrastructure because of the additional resources necessary to individually connect and reconfigure every existing site to each new VPN. “It was not uncommon for us to come into work in the morning and have seven or eight tickets at the help desk because reports weren't working the way they were supposed to and inevitably we would trace that back to a VPN tunnel,” said Cooper.

Goodnight Midstream’s existing SCADA software provided some challenges for them as well. Simple changes like adding objects in the application screen required users to access the development environment. Moreover, maintaining version consistency — such as performing patch management across endpoints, historians, and data centers — proved cumbersome. If a data center was updated or a patch applied, the endpoints would become “orphaned,” meaning that they were invisible until they could be updated to the matching version. Manually updating each endpoint (over 100 in total) was not only time-consuming but required external contractors, which added to the operational complexity, use of resources, and overall cost.

A Deep Project Scope

These opportunities for improvement prompted Goodnight Midstream to seek help from CSE ICON, a professional services company specializing in customized solutions for industrial automation, SCADA, and Digital Transformation. From the beginning, Goodnight Midstream prepared a robust scope document outlining what they wanted to achieve, including:

• Remove the point-to-point VPNs to decrease complexity and administrative effort.
• Implement MQTT to transfer data faster and benefit from secure Internet connections.
• Replace the existing SCADA system with Ignition.

Goodnight Midstream chose Ignition — an industrial automation platform for SCADA, HMI, IIoT, and more — because it could fulfill several requirements:

• Data mining and business intelligence work on the system backend.
• Powerful Linux-based edge deployments.
• Easy upgrades that could be performed by administrative staff.
• Location data driven by an external database.
• Template support.

There was also the monetary benefit. “Ignition is very cost competitive. When you're looking at what your total cost of software is over time, having Ignition be very robust technologically, but also very affordable is incredibly important,” said Cooper.

Based on Goodnight Midstream's goals, CSE ICON defined the project scope along with the key objectives. The main focus was to design a gateway architecture and implement MQTT in the field to bring data up to the enterprise. To accomplish this, CSE ICON needed to perform:

• Installation and configuration of gateways.
• Installation and configuration of an MQTT broker.
• Creation of UDTs (User Defined Types) and instances for Ignition.
• Migration of data from their existing SCADA system to Ignition.
• Streamlined patch management.

“We felt that Goodnight was very thorough in their scope of work documentation and it made it a lot easier for us to implement the functionality that they required,” said Binh Vu, SCADA Solutions Manager at CSE ICON.

Building A Gateway Architecture With Ignition, MQTT & Moxa

Goodnight Midstream strategically positioned five Windows servers at the enterprise level and approximately 50 Moxa AIG Edge Intelligence Gateway devices — each installed with Ignition Edge — at their facilities. “We chose the Moxa platform because it came out of the box preloaded with Debian, which is Linux for the edge,” said Rick Hornung, Senior Manager of Information Technology at Goodnight Midstream. “Moxa provides a set of tools to manage those endpoints centrally. We were able to leverage that platform because of the portability of Ignition.” This combination of technologies enhanced data processing and communication capabilities at the edge of the network.

Next, the focus shifted to the communication protocol. To make an informed decision about which MQTT broker would best meet the needs of the project, CSE ICON and Goodnight Midstream’s technical staff conducted load testing to assess system performance under varying levels of demand. Goodnight Midstream ultimately implemented the Chariot MQTT broker as the results indicated that Cirrus Link’s cost-effective and easy-to-use broker exceeded the project’s requirements by a wide margin.

UDTs & Alarms Syncing

With the previous system, every tag needed to be manually created, so each facility had its own unique tag list. With the Ignition system, CSE ICON was able to leverage UDTs — an iterative tag data type in Ignition — to standardize tags from the ground up.

To establish a consistent real-time data exchange between client and server, CSE ICON created two types of UDTs, one for a pump and another for a tank, totaling over 100,000 tags, each of which included the OPC (Open Platform Communications) item path. However, there were also disabled tags within the UDTs, which Goodnight Midstream did not want transmitted. Fortunately, MQTT only transmits data when changes occur to minimize data transmission.

In addition to the key objectives, Goodnight Midstream also needed to receive remote alarm notifications from their pipelines at remote mini-hubs, which are manned locations in each field where a trained operator can monitor regional information. To accomplish this, the alarms (and acknowledgements) must stay in sync between the edge-based Ignition deployments at the mini-hubs and the central Ignition system. While this feature was not directly supported by Ignition out of the box, through some clever scripting, CSE ICON was able to successfully extend the system’s functionality.

A System For Data Handling, Alarming & User Interface

The Goodnight Midstream gateway architecture project was initially planned for 12 months. However, due to the database-driven metadata model and the close collaboration between CSE ICON and Goodnight Midstream, the project was completed in just eight months.

“I don't think we pulled a single item out of that scope document,” said Cooper. This was an impressive feat, considering it involved the replacement of Goodnight Midstream’s full architecture. Ignition Edge is now deployed at multiple facilities, pulling data directly from PLCs in addition to a polling gateway at the enterprise level for high-speed cellular-enabled skids (which measure the flow of the pipelines).

Goodnight Midstream no longer needs point-to-point VPNs as all communications are now transmitted through MQTT, complete with TLS setup. Not only is the architecture simplified, the resultant system is more secure, reducing the overall complexity for IT, and allowing for remote acknowledgements from one gateway to another, sent out via Twilio.

“Ignition allowed us to move to MQTT, which is optimized for low-bandwidth connections and allowed us to remove all those VPNs,” said Hornung. “We just pump it out over the networks, over its own secure protocol, and it just dramatically simplified the deployment.”

High Performance As A Standard

Using Ignition’s Perspective Module, CSE ICON designed screens that are responsive to any size, including mobile. “CSE ICON uses a high-performance HMI as a standard for our customers just to help them visualize and understand their data quickly without having a lot of clutter and a lot of other information that may not be necessary,” said Vu.

Additionally, to give an overview of Goodnight Midstream’s extensive pipeline network, the system features an interactive map and pipeline views with real-time mapping. This is particularly useful at the different levels of the architecture. “Using the templates that we have in Ignition, we can distill that information down to a KPI screen for our primary control room, but we can also broaden that view and give a little more information to the mini-hubs, since they're looking at fewer resources, and they can take a more detailed view,” said Cooper.

Enhancing Operational Effectiveness

The new Ignition system allowed Goodnight Midstream to easily overcome many common concerns in the oil & gas industry. “The reservoirs determine where you are, not where you want to be,” said Cooper, emphasizing the extreme temperatures at remote sites (from -40℉ to 120℉!). “The Ignition platform runs well on Linux devices, and when you pair those two things together, a really robust software platform with good hardware choice, [it] allows you to deploy low-power, temperature-stable devices into very harsh environments.”

The dynamic, scalable Ignition system has allowed Goodnight Midstream to use the same platform across their entire operation and primed them for expansion. With the old system, it was necessary to pull up the development environment to add objects to a screen. In contrast, the new Ignition system leverages an automatically populated tree view, which is driven by metadata, allowing Goodnight Midstream to enter new data by simply filling in a database table. The process, which used to take half a day to complete, now takes 15 minutes. This has empowered Goodnight Midstream to add new sites themselves, and the project is expected to grow to 30-50 sites in the near future.

Project Scope

• Start date: 1/2023
• Deploy date: 9/2023
• Tags: ~100,000
• Screens: 83 templated iterations of 12 unique screens
• Clients: Varies throughout the day; anywhere from 3-20
• Alarms: 59 Alarm Pipelines
• Devices:
  • 5 Windows Server machines
  • ~50 MOXA AIGs
• Architectures:
  • Enterprise architecture
    • Database exists only at enterprise
    • No databases on sites
  • 5 gateways at enterprise (Windows Server)
  • ~50 gateways at facilities (Moxa AIG)
• Databases:
  • 4 (MS SQL Server)
    • IgnitionDB
    • IgnitionHistDB
    • IgnitionHistDB_Temp
    • MetadataDB
• Historical data logged:
  • ~20 billion rows
  • ~70k unique tags

Transcript

00:09
Kevin Cooper: My name is Kevin Cooper. I'm the Chief Information Officer here at Goodnight Midstream. Goodnight Midstream provides integrated water management to oil & gas producers. The oil & gas companies will send us their wastewater and if they need water, we will provide it back to them in a reuse capacity. So Goodnight is using its Ignition SCADA system for managing its remote facilities, its remote pads, and all of our remote assets, and feeding that data into our field personnel, our SCADA personnel, our control room, and our engineers. The project that we used CSE ICON to assist us with was to replace Goodnight's existing SCADA system with a new updated and more functional SCADA system.

00:43
Binh Vu: My name is Binh Vu. I am the SCADA Solutions Manager at CSE ICON in charge of all of Ignition projects. CSE ICON is a professional services company, specializing in providing customized solutions for industrial automation, SCADA, and Digital Transformation.

00:56
Kevin Cooper: As part of preparing for this project, we spent a significant amount of time building a very good spec and scope sheet for what we were trying to accomplish. A few of the top features were mobile and desktop access were needed to be very easy to use and responsive. That needed to have a very, extremely robust historian 'cause we mine our data and do quite a bit of data mining and business intelligence work on the backend. The edge deployment needed to be incredibly robust and preferably Linux-based. The upgrade process needed to be very transparent and easy for my admins to be able to support. And the SCADA location data needed to be driven by an external database. And we also needed to be able to do simple templating. So for us to adopt the Ignition platform, these were all important factors.

01:41
Binh Vu: One of CSE ICON's goals was to develop a baseline architecture for using MQTT for our future customers. Main thing is reducing the complexity of the network architecture, so implementing MQTT out into the field to bring the data up into enterprise was a big goal for them. So minimizing the use of the point-to-point VPNs and just making the system more manageable from the IT standpoint.

02:02
Kevin Cooper: Ignition was able to meet these goals for us largely because it is so modular and customizable.

02:08
Binh Vu: Most important module, I'd probably say is the Cirrus Link modules that basically connect all of their remote field devices to the enterprise via MQTT and also the EAM Module as well to allow their SCADA admins to push changes down from the enterprise down to the field devices. They wanted a way to build a responsive HMI for all of their users, from the field level down at the control room, and Perspective to help facilitate building out these screens in a modular way. CSE ICON uses a high-performance HMI as a standard for our customers, just to help them visualize and understand their data quickly without having a lot of clutter and a lot of other information that may not be necessary for their requirements.

02:41
Rick Hornung: My name is Rick Hornung. I'm Senior Manager of Information Technology for Goodnight Midstream. Migrating to Ignition from our previous platform opened up a lot of different end compute options for different operating systems and hardware platforms. We chose the Moxa platform because it came outta the box preloaded with Debian, which is Linux for the edge, which is what we were looking for. And Moxa provides a set of tools to manage those endpoints centrally. We were able to leverage that platform 'cause of the portability of the Ignition platform.

03:09
Kevin Cooper: One of the things that is somewhat specific to oil & gas; you're almost always dealing with really out-of-the-way places. The reservoirs determine where you are, not where you want to be. So you're dealing with extreme hot, you're dealing with extreme cold, and I think that the Ignition platform running well on Linux devices and you pair those two things together — a really robust software platform with good hardware choice — allows you to deploy low-power, temperature-stable devices into very harsh environments.

03:40
Kevin Cooper: And we certainly experience that, we see everything from 120℉ to -40℉ and I'm able to use the same platform across my entire operation. There are a number of things that I think we're incredibly positive about this particular project and one of them is that Ignition is very cost competitive. When you're looking at what your total cost of software is over time, having Ignition be very robust technologically but also very affordable, it's incredibly important. Anybody who has done large-scale networks, especially distributed out to edges that are way out in the middle of nowhere, VPN tunnels can be the bane of your existence.

04:16
Kevin Cooper: Some SCADA systems will require to use VPN to propagate that data transmission from point to point. We had over a 100 of those VPN tunnels. Occasionally they would stick and if a tunnel gets stuck in the up or down situation, then you're not getting traffic and you're not getting volumes. It was not uncommon for us to come into work in the morning and have seven or eight tickets at the help desk because reports weren't working the way they were supposed to. And inevitably we would trace that back to a VPN tunnel. By moving away from VPN and using MQTT as our primary data transmission method, that has fixed all of those problems.

04:51
Rick Hornung: Ignition allowed us to move to MQTT, which is optimized for low-bandwidth connections and allowed us to remove all those VPNs. And we just pump it out over the networks, over its own secure protocol, and it just dramatically simplified the deployment.

05:05
Kevin Cooper: I think that the data resolution has been incredibly important to us because it really allows us to deliver to the business these higher-resolution data points. Previously, let's say a well engineer comes to me and says, "I wanna look at that well and I wanna look at it at like a five second data resolution." Well, I would have to take that entire location and bump it down to five seconds and now your historian is getting flooded with data that you don't really need. Now I can group just a couple of instruments and I can say, "I just want those to poll at five seconds." So I can just run that for a day and I can do things with my data engineers, working with my physical engineers, and give them answers that they couldn't do before.

05:41
Binh Vu: Previously every tag that they had in the system was manually created, so each facility had their own unique tag lists, and managing them was difficult. But by leveraging UDTs, it was standardized from the ground up.

05:54
Kevin Cooper: User response to Ignition: end users obviously are roaming the field, they need something that's very quick and easy, and we've actually developed some screens that are very specific to somebody who's rolling up on a facility needs to see their top five. We also run mini-hubs that are located in each field, where we have a trained person who's watching just their regional information, and then of course the control room's watching everything. Using the templates that we have in Ignition, we can distill that information down to a KPI screen for our primary control room, but we can also broaden that view and give a little more information to the mini-hubs since they're looking at fewer resources, and they can look, take a more detailed view. The way the Ignition is able to display that information makes that very easy and it makes it easy for me to make my customers happy.

06:36
Binh Vu: Now that they've implemented Ignition, their system's easier to manage. They can have a SCADA administrator come in and look at their system and manage it without having to bring in a team of developers to look at their remote facilities.

06:46
Kevin Cooper: What impressed me, and I think my team, about Ignition is we went pretty deep. What do we need? What do we want to do? What are we trying to accomplish? And the scope document that we built was very robust. And the fact that I don't think we pulled a single item out of that scope document and it fit inside of the Ignition platform and went very smoothly and we got what we wanted out of it, although it's what I wanted, I don't know if I expected that to happen, but at the end of the day, that's what we have on paper, that's what we have on our screens, and that is, I think, very impressive that a platform is flexible and robust enough that without breaking an arm, you're able to get what you need out of it.

07:24
Binh Vu: We felt that Goodnight was very thorough in their scope of work documentation and it made it a lot easier for us to implement the functionality that they required.

07:32
Kevin Cooper: CSE was very good at communicating with us and we understood when things were moving along, we understood when things were successful, and we understood more importantly when things were challenges and how to overcome those. And those were very positive experiences and we were very happy with the Ignition platform and CSE in that we got exactly how we speced out our project.