Cloud, Edge and More Enable a Fast Solution for Numerous Sites

Project Scope

• Tags: 10,000
• Screens: 12 overview screens plus one per site (Total of 115)
• Clients: 97 Ignition Edge Nodes plus 15-20 view clients
• Alarms: 3,500
• Devices used: approx. 145 (Rockwell, Koyo, and ROC)
• Architectures used: Hub and Spoke - Hub includes AWS, EAM and Cirrus Link MQTT
• Databases used: 3 plus redundancy
• Historical data logged: 3,000 historized tags

Project Overview

ARB Midstream acquired a new pipeline asset which includes 36 RTUs and central gathering locations. As part of the agreement, ARB was required to completely take over pipeline SCADA in four months. Challenged with this aggressive schedule, ARB contracted Industrial Networking Solutions (INS) to deliver a SCADA solution offering cloud-based reporting, management, visibility, control, and a communications network with failover capabilities and store-and-forward technologies.

ARB was evaluating a migration from legacy SCADA software packages to newer technologies on existing systems. The company’s traditional SCADA systems are slow with a high communications cost and a high annual maintenance cost with SCADA vendors. This new project offered an opportunity to implement improved SCADA and communication technologies.

ARB wanted SCADA reporting and monitoring to use web-based technologies for a broader audience within ARB, and cloud technologies to lower the investment into on-the-premises server hardware and infrastructure.

The ultimate objective of the new system was to create a standardized network infrastructure and a centrally manageable and scalable SCADA platform. This system should include modern networking standards and be reliable, expandable, and secure. This system should integrate as much reusable assets and intellectual property as possible. However, the system should enable ARB to quickly acquire assets in the future and be a unified SCADA system.

INS recommended Inductive Automation's Ignition Pro hosted on Amazon Web Services, Moxa UC8100 with Ignition Edge MQTT, and Cradlepoint Netcloud Perimeter SD-WAN.

Starting at the edge, the previous system had multiple PLC vendors with a variety of protocols. To address this problem, INS installed a dedicated networking enclosure at each RTU with:

• Cisco IE-2000 Managed Switch – Allows INS to create segmented VLANs to separate local PLC communications from the SCADA communications.
• Moxa UC8100 Computer – Allows INS to run Linux-based applications with additional memory for store-and-forward capabilities in an industrial environment.
• Ignition Edge with Cirrus Link MQTT – Allows INS to develop local monitoring and control application for the RTU with protocols for multiple vendors. With Ignition Edge, the software allows for store-and-forward capabilities upon a loss of communication. The Cirrus Link MQTT allows for reduced data rates and flexible automation for the addition of new devices in the field. Another benefit was the compressed data, allowing for more efficient cellular communications.
• Cradlepoint IBR600 – The Cradlepoint cellular router provides WAN failover for both Verizon and AT&T. Additionally, the routers have a feature called Netcloud Perimeter. Netcloud Perimeter is an SD-WAN feature allowing PLC-PLC and PLC-SCADA communications without the need for advanced routers or legacy VPN technology. Another key feature of the SD-WAN technology is the WAN NAT capability. Essentially, this allows deployment to multiple RTUs without the need to re-IP all the LANs and default gateways for each location. This technology was a key feature in deploying the solution in a short timeframe.

Cradlepoint’s NetCloud Perimeter allows ARB Midstream to build and deploy a virtual overlay network connecting all its IoT devices, remote LACT locations, and mobile workers to business-critical resources. Offering cloud-based security, the NetCloud Perimeter service eliminates the hardware, complexity and operational costs of traditional WANs or disparate IPSec tunnels and hardware required to maintain these tunnels. It enables easy deployment of changes at the network edge while maintaining security, visibility and control as ARB Midstream’s business and operational requirements evolve and grow.

NetCloud Perimeter works with ARB Midstream’s existing network and security infrastructures. It scales instantly and is subscription-based. The overlay network is tailored for purpose, to scale and perform independently based on business needs.

• Through Netcloud Perimeter, private IP addresses will be assigned to edge and remote devices within the overlay network
• All devices and clients will be connected to one secure overlay network
  • Encrypted data end-to-end is 256-bit AES
  • No data stored in Perimeter’s cloud
  • Cloaked IP addresses hidden from public
  • Certificate-based Auto-PKI (X.509 CA)
  • Runs on top-tier cloud providers around the world
  • Fully redundant architecture
  • Seamless failover (carrier neutral - AT&T or Verizon)
• This network and security help protect against many network-based attacks such as:
  • IP address-related attacks (port scans, spoofing, DNS poisoning, and DDoS)
  • Packet sniffing exploits (Firesheep and other sniffing programs)
  • Authentication hacks (unchanged passwords, brute force, single factor)
• Remote clients can be installed on support technician laptops, giving them controlled access to the Perimeter network
• The VPN client will be installed on the AWS Ignition server, providing both the host and application secure access to the edge MQTT device data, while allowing control at remote locations

ARB has selected Inductive Automation Ignition SCADA software. The system is architected to use multiple components of the software suite. The software is broken up into the following functions:

• Amazon Web Services (AWS) – Host four Windows 2016 Servers:
  • Ignition Enterprise Servers (two servers) – One server acts as the primary application server and the other hosts the redundant hot swappable gateway. Each server is hosted in different AWS regions for further redundancy.
  • MS SQL Server Standard Edition Server (two servers) – One server acts as the primary hosted database server for historical and configuration management and the second acts as mirrored redundant hot swappable failover. Each server will be hosted in two different AWS regions for further redundancy.
• Lucerne Plant – This is a standalone server hosting an Ignition Local Client Fallback to the primary enterprise Ignition gateway. If the primary hosted application in AWS cannot operate, the Lucerne fallback gateway takes over primary monitoring and control functions. An additional MQTT Broker is hosted on this client fallback gateway to manage communications to the LACT locations.
• LACTs - Each LACT unit hosts an Ignition Edge MQTT application running on Moxa UC8100s. Each Ignition Edge license will include the enterprise management module EAM for remote monitoring and management. Additionally, the MQTT Transmission module will include both the primary broker at the enterprise cloud location and the redundant failover location at Lucerne.

This application delivers applied edge CIP, MQTT Sparkplug B, store-and-forward, cellular SD-WAN and cloud-hosted SCADA. All these items combined make for a great project. This application uses the best-in-class vendors with Inductive Automation for the SCADA, Cradlepoint for cellular communications and software defined networking, Cisco for managed networks, Cirrus Link for MQTT, and AWS for cloud-hosted services with redundancy. INS believes this architecture helps set the baseline for future SCADA architectures in the IIoT market.