Cirrus Link Exhibitor Demo: Everything Cirrus Link MQTT and Cloud Connectivity

This session provides an overview of Cirrus Link to include MQTT Architectures, the MQTT Modules and their use cases. It will also touch on MQTT SparkplugB, the Unified Namespace as well as cloud connectivity through the cloud injector modules and IoT Bridge products.

Transcript:

00:00
Nathan Davenport: My name's Nathan Davenport. I'm the Director of Sales Engineering here at Cirrus Link. What do we do? What do we build? How do we integrate into the Ignition platform? And where did we come from? So, Cirrus Link provides MQTT centric software for industrial automation solutions. We've been doing this stuff a long time. We have 85 plus combined years of experience of MQTT on staff. We have the co-inventor. You guys know who Arlen Nipper is. The co-inventor of MQTT. He is our president and CTO. And he himself has tons of experience. We were founded in 2012. As you guys know, we are strategic partners with Inductive Automation. We build a lot of different modules. MQTT and Sparkplug centric for the platform. And we also created an open source, the Sparkplug specification. So, created it, and then we basically gave it away to the Eclipse Foundation, and we helped kind of shepherd that stuff through. What is MQTT? So, the spec itself describes how to implement a message-oriented middleware Pub/Sub infrastructural. What in the world does that mean? I'll talk about that a little bit more here in a second.

01:05
Nathan Davenport: Where is it used? It's used all over the place. Every major cloud provider has some type of MQTT endpoint, right? You have AWS IoT Core, you have Azure IoT Hub, and you have Google, well, you used to have the Google endpoint, but I think they retired that thing. But we do have the IBM Watson IoT, right? They all use MQTT. What is so special about MQTT? Well, it was originally designed for real-time, mission critical SCADA systems. It's simple, it's efficient, it's stateful, it's open. What does the basic MQTT architecture look like? So, in this particular example, I have two clients and a server in the middle. So, the first client connects. The other client's connected as well. Client number one subscribes on #. # is the wildcard, right, topic within the MQTT topic namespace. So, any message published on any topic should be received by this client, MQTT client one. So, client two publishes on a topic of hello/ICC with a payload of hello ICC 2024 bang. The server then gets that message, looks up in its table to figure out who has the appropriate subscription such that they should be getting that message, delivers said message to MQTT client one, and it gets the message on the topic hello ICC. Hello ICC 2024 bang as the payload.

02:29
Nathan Davenport: So, we have three core MQTT modules for the Ignition platform. I will give you a summary here. We'll look at a little bit of a topology diagram next, and then I'm gonna pivot over to the Ignition web portal so you guys can get a little bit of context, and I can show you some of the features within engine transmission and distributor. So, what does transmission do? Transmission is the MQTT client that runs on the Ignition platform. Its job is to basically consume tags. It consumes tags, converts that data, tag data, into Sparkplug messages, puts that stuff on the wire, sends it off to MQTT distributor. What is distributor? Distributor is your on-platform MQTT server, right on the ignition platform MQTT server. It can handle up to 250 clients simultaneously. Engine is primarily a consumer of data. It accepts, it subscribes on the appropriate topics, gets Sparkplug messages in, converts that data basically into tags, and renders those process variables as tags in the engine tag provider within the Ignition tag subsystem. So, what does an Ignition-based MQTT architecture actually look like? Well, you have to start with a server. So, we have the MQTT server here in the middle, and we have a host client as well.

03:47
Nathan Davenport: That's a Sparkplug host client. Those two, the server and the client, are both on the same gateway here in this particular architecture. And then for this topology diagram, I have two edge clients. One's running the Edge SKU, one's running full-blown, talking to your PLCs, RTUs, and so forth. What does that mean? Well, engine's the host application. Distributor is the MQTT server. Transmission is the Sparkplug edge client. So now, I'm gonna pivot over here so I can talk a little bit about the features within engine. So what I wanna show you guys first are the default namespaces with an engine. So, the namespaces really define which MQTT topics the engine client subscribes on. Some Sparkplug B, for example, means if you have this namespace enabled, the engine client is going to connect. It's going to subscribe on spBv1.0/#. It's gonna get all the Sparkplug messages in from all of your edge clients across all of your plant floors out in the field, and so forth. These others are a little bit less interesting, but really, all you need to know is that the namespace defines the subscriptions that the MQTT client makes.

05:04
Nathan Davenport: That's how we get data into the platform. I don't think we talk enough about custom namespaces, really, within engine. Custom namespaces, however, are specific to taking in generic MQTT data on any topic. So, this is non-Sparkplug data, right? So, let's say you have a bunch of MQTT devices in the field. None of them speak Sparkplug at all. They're all publishing maybe JSON payloads, right? So, custom namespaces allow you to define the topics that you want to subscribe on in order to get this data from your edge clients and route it to the appropriate location within the engine tag provider. So, this one happens to be subscribing on B/#. Not super interesting. And within custom namespaces, we have the ability, if it is a JSON payload, we can basically parse the object, take each individual object property, and make that a separate tag within the subsystem itself. If you don't have JSON parsing turned on, you're basically gonna get the entire string payload represented as a string within the tag, right? So, if you're publishing JSON, take advantage of the auto JSON parsing. Know that it's pretty strict. If you don't have JSON that is exactly formed properly, we're gonna choke, we're probably gonna throw some errors, and you're not gonna get your tags into your engine provider.

06:26
Nathan Davenport: All right. So, I'm gonna talk about sets and servers briefly, and then we'll move on to transmission. So, what is a set? We use server sets in order to define redundant sets of MQTT servers. In this particular case, engines connected to two MQTT servers simultaneously, they are in two different server sets, and therefore, we connect to both of them. But if I were to put these two servers into a single server set, so for example, if I swapped out the chariot server set and I put default in, we would connect to the first server available within that set. If that guy dies, we're gonna walk to the next server. We're not necessarily gonna walk back, we're gonna move to the server that's available, we're gonna stay there until something changes, and then we will, if that server dies, walk to the next server. This can be two, this can be three, this can be N. Doesn't matter. All right. Transmission. So, an engine, like I said, it's primarily a consumer of data, right? Transmission, on the other hand, it is publishing tag data. So, it has servers too, just like engine does. It has server sets as well, just like engine does.

07:41
Nathan Davenport: And it uses those exactly in the same way. The edge will, you define your redundant pairs on transmission, on engine, in exactly the same way. The engine server sets came along, I don't know, one or two versions back. And it wasn't really a use case we were thinking of. Logs were pretty spammy if you had engine pointing to multiple servers, and we weren't able to connect all of them simultaneously. So, we thought, well, we already have a mechanism for that. Let's port server sets to engine. So, we did. Like I said, the server set really is kind of the glue between the server definition and the thing you're trying to consume or publish. So, in this case, the server set binds a transmitter, we call it, to a server. So, in this particular case, I have two transmitters. I'm kind of cheating. I'm making it look like I have two edge gateways, two edge nodes on two different boxes, when in fact, I really have two edge nodes on the same gateway. As you can see, you're pointing to a tag provider, you point to a tag path. Tag pacing period is kind of a cool thing. That is the amount of time that we wait in order to aggregate all tag changes into a single message.

08:53
Nathan Davenport: So, imagine if you have 10,000 tag changes within a second. We're gonna package up all 10,000 tag changes into one message, put it on the wire, on the appropriate Sparkplug topics, publish it out to your consumer. We have some optimization settings in here in order to do things like maybe you wanna compress your data. Maybe you wanna use aliases instead. You have very long tag names. We can swap in an integer representation for a particular tag name and just send that integer ID, if you will, across the wire. And so, we'll publish the birth, full context along with your alias once it hits MQTT Engine. Engine caches all of that stuff in a lookup table, and it knows exactly which alias corresponds to exactly which tag. There's a few other things in here, too. We do natively support UDTs. So, if you wanna send UDTs as true UDTs, you have a UDT defined on the edge. You want that UDT at Engine, you would turn convert off and let those UDTs be published. Very valuable for cases where the models, the UDT definitions need to flow along with the data so that the consuming side can rebuild those models, right?

10:08
Nathan Davenport: Rebuild instances of those models. All right. And then a few other settings around history, right of course, we support history. If the edge client has a disconnect at the edge, so your network drops, we'll start storing every tag change either in memory or on disk so that when that connection is resumed, we can package that data up, put it on the wire, get it over to your consuming client. Also, even if your connection to the MQTT server is good, but your consuming client Engine is not there, we use what we call a state message between Engine and transmission so that transmission can basically realize when Engine has gone away. So the server's there, but maybe Engine has died. Gateway failover, somebody pulled the power cord, right? Whatever. Transmission will still basically take itself offline and store that data until the primary consumer comes back. Because if you're publishing live data, but there's nobody there to consume it, it's gone, right? It's gone forever. Hopefully you have history at the edge. Okay. All right, you guys, let's get back to the presentation at hand. So we went through this already. So which markets are we in? We're in about every market that you can think of.

11:28
Nathan Davenport: This is just the top markets for the last 12 months. We've seen a significant uptick in manufacturing, which is great news for us, right? Manufacturing is really starting to pick up this technology. Use it broadly. Deploy it broadly. That's good news. We have what we call Cloud Injector Modules as well. These do not rely on MQTT infrastructure in the slightest. You don't need a MQTT server. You don't need Engine. You don't need transmission. You simply need the ejectors to point to your tags and consume tag change events, convert that to basically Sparkplug JSON, and send it off to whichever cloud endpoint you wanna push it to. So we support Google, Azure, AWS. Then all the endpoints there, I'll tick off a few Kinesis on the AWS side. That's probably the most popular endpoint to publish data to. We support DynamoDB, but I think that's used far, far less frequently than Kinesis Streams. We support the Firehose configuration within Kinesis as well. On the Azure side, it is IoT Hub, Azure IoT Hub and Azure Event Hub probably as the two most popular endpoints to publish data to. We also support Azure Edge and IoT Central, but those are far less popular.

12:54
Nathan Davenport: So what's so great about these things? You can deploy 'em anywhere you want to. You wanna put them on the edge and push data to the cloud? Go ahead. You wanna put it on the central gateway where we aggregate all of your data and push it up from the central gateway? You can do that as well. There are some benefits to doing that. We can more tightly pack the data on the central gateway and ensure that those messages are as large as they can be so that when we push it, we're right up against the max message size without having to start splitting messages. That's saving you money, right? Every time you push a message, the little cash register turns over. So better to put it on the central gateway. We can better pack your data. We can better optimize those payloads, and we can ensure you're paying the smallest amount of money for the data that you're pushing. Same efficient tag reporting scheme as transmission. It's just that we basically put it into a JSON payload for you. The Chariot MQTT server.

13:50
Nathan Davenport: So in the case where maybe you need to split your MQTT server off from your Ignition gateway, maybe the load is too high. Remember, Distributor has a max client count of 250 clients. So if you're talking thousands of clients, chances are you are gonna need Chariot. So what's so great about Chariot? Well, we've had previous versions of Chariot. This is Chariot V2, we call it. Chariot V1 was kind of using some libraries that we didn't write from scratch. So what did we do? We clean room, wrote an MQTT server from the ground up. Purpose built for OT and industrial applications. Built-in MQTT and network debugging tools. This is probably the coolest thing in Chariot, in my opinion, is that it is both Sparkplug aware, right? And every time we find something crazy in the field, we're debugging production outages, we're debugging network connectivity issues, configuration issues, we go build in what we call alerts so that we have basically monitors that can run against your data flow and identify issues like colliding client IDs at the pure MQTT level, right? When an MQTT client connects, it has to have a unique ID.

15:00
Nathan Davenport: If it does not, you come in, the old client's getting bounced. New one wins, the currently connected client loses. So we can identify those things for you. In Sparkplug Land, we have what are called group Sparkplug ID collisions, or group plus edge node collisions. Your unique identifier within the Sparkplug namespace is your group, plus your edge node ID. And so we can run into cases where either your static configuration or your dynamic configuration basically stands up two Sparkplug edge clients with exactly those same IDs, group plus edge node. And when that happens, consuming applications like engine, host applications, get confused because they're getting two duplicated data streams from two different clients, and the sequence numbers that should be flowing in order which are across the two duplicated data streams eventually get out of order and engine doesn't know what to do. And so it says, hey, I got a sequence number out of order. Actually, engine has the ability to reorder messages too for broker implementations that won't deliver messages in order, but it's never gonna make it out of this hole. And so what you get stuck in is what we call a rebirth storm. Sequence numbers come in and out of order, engine squawks, requests a rebirth, those clients spin up again, publish messages again, we're back in the same state, it's a nightmare.

16:24
Nathan Davenport: So anytime we find this stuff, we try to go build it into the product. I'll show you that here in a second. Easy to install, easy to configure, built on top of Java just like Ignition. So it's cross-plat, any Windows distro, any Linux distro. Excuse me. We have either one-time perpetual licenses where you can buy the license up front and do the install yourself, or we offer Chariot as a marketplace offering in a cloud environment like Azure, a cloud environment like AWS. So in that case, you just go to the marketplace, you click, you deploy, you don't install anything, software's already there. License is basically a no-op because you're already licensed and you are getting billed via runtime by your cloud provider. Web-based administration, we got a full web front end. It's fully backed by REST. Anything you can do in the Chariot UI, you can do via REST. Great for maybe we have a UI feature you want that we don't have. Hit the REST endpoint. Maybe you wanna spin up MQTT credentials on the fly after you've deployed. Let's say you have a thousand of 'em. You don't wanna go type that stuff into the UI? Hit the REST endpoint.

17:34
Nathan Davenport: Very, very, very flexible and valuable at deployment time, in my opinion. We support LDAP and Microsoft Active Directory integration. Highly secure. Of course, we have the ability to use TLS, right? Both for HTTP and MQTT. We're always making sure that we have all of your data being encrypted, whether it's web traffic, whether it's MQTT traffic. Now, before we get to that slide, let me show you Chariot here. So this is the Chariot front end. I'm purposely starting on the Sparkplug page because I kinda think this is one of the more interesting features within Chariot itself. It's Sparkplug aware, guys, right? So when your Sparkplug assets come online and publish messages, we know about them. We can discover exactly how many edge nodes you have. You have three. Only two happen to be online. They belong to one group. You have three host clients for some reason. We'll talk about that here in a second. And you've got three devices. Two of which are online. So you can get some information here, but it's far more interesting if we pivot over to... Oh, hold on. I think I lost. My session has been timed out, so let me log back in. There we go. Back to Sparkplug.

18:57
Nathan Davenport: Same data, just in list form. And I'm going to have to speed up because I'd like to give you some time for questions. I've got six minutes left. So I'm gonna try to blast through this stuff if I can. Cool stuff within the edge node space. You can see the edge node, exactly what its Sparkplug IDs are, whether it's using a primary host ID. This guy's not. That's probably a bad thing. What it's client ID is, which IP it connected from. When was it last online and offline? How many metrics? What is a metric? A Sparkplug metric is an ignition tag. This particular edge node has four tags associated with it and a single device. We also cache your birth messages and have some other interesting stuff in here. So if you have somebody complaining that I'm not getting a tag value at engine, you must not be publishing it. Well, go check the birth message. Is it in the birth message? If it's in the birth message, I published it. Let's get to the engine side. Let's go figure out what's going on. Maybe the message didn't make it. Maybe engine didn't like it. But clearly you can see exactly what we published here. If you were publishing UDT definitions, they would be in this node birth as well. You can copy it out. You can hit that via rest as well. I'm running out of time, so I can't tell you about some of those other cool features there.

20:08
Nathan Davenport: Raw MQTT view, this is kind of your underlying clients, right? For every Sparkplug client, there really is an underlying MQTT client. This is the basic MQTT data per client. So MQTT engine's connected from this IP, subscribed on these topics. Here's the alerting that I was telling you guys about. It's about half MQTT specific. The other half is Sparkplug specific. Diagnostics is not all that interesting. Those are threads and so forth, so I'm not gonna show you that. Accounts. This is your admin accounts. Log in via rest. Log in via the UI. Manage your chariot server, right? MQTT credentials. These are your credentials that you use to be able to define exactly which topics your clients can publish and subscribe on. If it's # #, that guy has root privileges. They can basically do anything that it wants, get all messages published on any topic, right? Server config. Just pure server stuff. Are we doing MQTT over TLS? That's typically 8883. Do we wanna use WebSockets, secure WebSockets and so forth? Do you wanna allow anonymous? Please don't do that in production, folks. Let's not do that. The MQTT client. I'm gonna tease this thing I can't show you now. We've added a thing cliented to the chariot server.

21:26
Nathan Davenport: Arlen and I will talk about this tomorrow at 2:45. Come check it out. It's gonna be awesome. These are kinda like your MQTT spy equivalent within the chariot server. Licensing. Kinda boring. You can add your license. You can deactivate it. We can handle offline. We can do online activations. System config. You need to upload some certificates, right. Do backups. Do restore. All of that stuff is done right here. All right. Let's keep going 'cause I'm running out of time. So just like we have the cloud injectors for the ignition platform, we have what we call IoT bridges for different cloud solutions. We have three of them. One for Azure. Specifically, it hits the Azure digital twin endpoint. One for AWS SiteWise. And one for Snowflake. And These are the solutions that require the UDT definitions to be published from the transmission side because if we don't have those models, we can't create your digital twin in ADT. We can't create your instances of said models in SiteWise. And we surely can't build the right views dynamically in Snowflake without those models. These bridges allow you to basically consume Sparkplug messages natively and then forward them off to the cloud endpoint of your choice.

22:41
Nathan Davenport: These are all deployable through cloud marketplaces. We don't yet offer an installable package that you can drop anywhere you want to. What are the new cool features that we're going to announce that I keep teasing but not showing you? Guys, we keep getting a ton of requests around UNS. UNS is super popular, right? How in the world does it work with Sparkplug? What do I do with my Sparkplug IDs? Why is it that I can't get my metric path or my tag path into the topic? We've tried to solve all of these problems for you guys. Engine now has a UNS namespace configuration piece within the Sparkplug B namespace. It will allow you to lay your tags out exactly as you've always wanted to without the Sparkplug overlay. We're gonna demo that tomorrow. Come check it out. We have a UNS publisher for transmission. This thing will take each one of your tags and publish out one message per tag change where the MQTT topic is the full tag path. So now you can say, well what if I wanted to cherry pick out five tags, push 'em out to my server and have them retained, and have the entire tag path as the MQTT topic? Because maybe I want one guy in IT to be able to consume these 10 tags and another guy in IT to be able to consume these 10 tags.

24:07
Nathan Davenport: Now via MQTT credentials and access control list, you can lock that stuff down and you can publish exactly what you want, one tag per topic, one message. We have alarms over MQTT now. People have been asking for this for years. We finally got to the point with platform support. Thank you guys at IA. So we can get alarm, active alarms from transmission to engine. We can act them back and forth. We can clear them, and so on. Then as I teased, we have the best in class Chariot MQTT client available for free. So you guys can download the Chariot MQTT server, install that thing, run it without paying us any money, and that we typically have a two hour trial timer under which almost all of our features run, but the thin client, the MQTT client in Chariot does not adhere to timer. So you can run that thing 24 hours a day for the rest of your life, and you don't have to pay us any money. Thanks you guys. And I don't think we have time for questions, but I am at booth number two, right down the hallway. Come talk to me. I would be happy to talk to you guys about this more. Thank you.