Moving Data from the Edge to the Gateway

Brad Byrum and Jared Weakley from Texas Automation Systems (TAS) join us to share an MQTT implementation story for an oil and gas company. They discuss the challenges and successes of starting from scratch to design a new control system with site-to-site communication in a field covering many miles, and making it secure. We’ll hear about the benefits of MQTT, and TAS’ advice for developing an MQTT-based system. We’ll also discuss working with Cirrus Link Solutions, TAS’ growth with Ignition, Core training, and becoming Gold certified.

“MQTT was an eloquent way to solve a very complex problem, and the customer ended up very happy.” – Brad

“Being able to publish data from any site, also subscribe to any data that you want, that’s a big deal.” – Jared

Guest Bio

Brad Byrum joined Texas Automation Systems (TAS) in 2015 to serve as their President. TAS has built a strong reputation within the automation industry by providing best-in-class automation, SCADA and wireless system design, and development and commissioning services. In his current role, Brad divides most of his time between his self-described roles as the company’s “Chief Roadblock Clearer” as well as “Chief Strategist.” Brad began his career at Amdahl Corporation, where he worked as a Systems Test Engineer on mainframe computers. He later joined Yokogawa as an Application Engineer within the Controls group. Over the following 25 years, Brad held many positions within Yokogawa such as Sales Representative, Regional Sales Manager, Product Manager, Marketing Manager, Business Unit General Manager, and Vice President of Sales & Marketing. Brad received a Bachelor of Science in Electrical Engineering Technology from DeVry Institute of Technology.

Jared joined Texas Automation Systems (TAS) in 2012 to serve as their Director of Automation. Texas Automation Systems has built a strong reputation within the Automation Industry by providing best-in-class automation, SCADA and wireless system design, development and commissioning services. In his current role, Jared is responsible for leading a talented team of experienced PLC and SCADA programmers and staying on top of the latest IIoT technologies, as well as, PLC, SCADA and Telemetry hardware and software. Jared began his career at Anadarko Petroleum Corporation (APC) based in The Woodlands, TX where he worked as a Plant Operator at a large natural gas plant. He later joined the Corporate Automation Team leading and working on large corporate automation projects across all land- based assets.

Episode Transcript:

00:00
Justin: Hello and welcome to Inductive Conversations. My name is Justin Reis. I'm the Integrator Program Manager at Inductive Automation, and we are joined today by Jared Weakley and Brad Byrum from Texas Automation Systems. Brad, Jared, thank you very much for joining us today.

00:14
Jared: Thank you.

00:15
Brad: Yeah, thanks for having us Justin.

00:16
ustin: Absolutely. So let's start out with some introductions. Brad, you're the president of Texas Automation Systems. Can you tell us about your role and background in the company?

00:25
Brad: Sure, sure, yeah. I've been in the automation business about 30 years, spent 25 years with Yokogawa in the instrumentation side of it, and about five, five­ and­ a ­half years ago came over to join TAS, and it has been nothing but a good and crazy ride for the past five years, so it's been a lot of fun. But yeah, I describe my role mainly as the chief roadblock remover. And so, we got technical guys like Jared and lots of other guys who are truly more talented than I am technically, but I get to spend my time helping them make sure they've got the time and resources to be effective doing their jobs.

01:05
Justin: That's great. Alright, and Jared, you're the Director of Automation. Can you tell us your role and history with Texas Automation Systems?

01:11
Jared: Sure, yeah. So I am the Automation Director. I was employee number one, so I've had the privilege of watching this company grow over the last seven or eight years. And day to day I wear a bunch of different hats, like Brad. We assist the guys in the field and help them any way we can. But I spent the good majority of my career at Anadarko Petroleum, was part of the corporate automation group there, and so brought a little bit of that expertise here, and Brad and I together have grown that.

01:44
Justin: That's awesome. Alright, well let's go ahead and orient our listeners to your company. Can you give us a little bit of background on Texas Automation Systems?

01:51
Brad: Sure, I'd be happy to. So Texas Automation Systems, if you couldn't guess, we're in Texas, and we started about... Jared said about eight years ago, when he came aboard, he was employee number one, and we have steadily grown over the past seven years to, we're currently about 50 employees. And so I think that we started off part of another company, then we kinda spun off about five years ago, and that's when we really took off, 'cause we were completely independent. And we've grown, we started off doing a lot of oil and gas, a lot of RTU work in the field, and then we started building up our PLC integration team. And then several years ago, we really wanted to get into SCADA, and so we started building our SCADA team several years ago and we've now built that up pretty substantially. So that's, besides oil and gas, we do food and beverage, water and wastewater, general industries. But our background, we've got a lot of experience in oil and gas.

02:49
Justin: Great. And you had quite a bit of growth over the last couple of years with Ignition. I'd love to hear a little bit of your take on the recent success.

02:56
Brad: Yeah, yeah, so it's an interesting story. We have some big strategies that we put in place, and then the rest of it's mainly tactical, but probably four years ago, we made the decision, "Hey, we wanna be a significant PLC player in the market." So we started hiring PLC guys, and we now built up to where we've got 25 PLC programmers. And then about two and half years ago, we made the decision, we really wanted to get in the SCADA space. And so, I had some experience with SCADA, Jared had some experience with SCADA, and we set off on a couple-of-month review of all the SCADA packages on the market and did a deep dive. If we're gonna hook our wagon to a SCADA platform, which one do we wanna hook it to? How do we go about this to become relevant and significant in the market?

03:43
Brad: And after doing our research, we really felt like Ignition, Inductive Automation with Ignition was... You guys were doing a lot of really cool stuff and we saw the growth you guys were experiencing. We really wanted to be part of that. And so we started committing ourselves to getting guys trained up and Core Certified, and now we're Gold Certified, and we just kept adding to that and have built up now a significant... We've got, in fact, I did a count earlier, we got... Currently have one Gold, but we got three in Gold testing, five Core, and six additional in Core testing. So we've got some significant Inductive resources internally, and it's been a lot of fun. We've enjoyed working with you guys.

04:28
Justin: Yeah and same here, we love working with you guys. And I love your dedication to the training in getting your guys certified. That's awesome. And you mentioned you guys are Gold Certified. Can you tell me a little bit about what that has meant to your company?

04:41
Brad: Yeah, I think for us, it's really substantiating us in the marketplace. There's lots of SCADA people out there. I guess the entry barrier into the SCADA business is lower than it used to be. Used to be a much smaller market, much more dedicated learning. There wasn't online learning, so you had to make all this, you had to build relationships, and it was a larger commitment, I think, to get into the SCADA space. And now, it's a little easier to get into the SCADA space, partly because you guys offer this free Core training to a lot of people in college and other places, can start the training. But for us, getting that Gold Level was really substantiating our... We're not only committed, we're really committed, and we're dedicating resources to be the best and the brightest within this space that we can be in order to serve our clients the best we can.

05:28
Justin: Yeah, that's awesome. I really appreciate that, thanks. So, I know that we're here to get into the MQTT implementation story you guys are here to tell us. So why don't we begin by discussing the challenges and requirements that the customer brought to you?

05:42
Jared: Yeah, I'd be glad to take that one. So we started with an existing field, a fairly large field, this is oil and gas. So, as you can imagine, the geography is pretty widespread. And basically the initial challenge was how are we gonna implement the new control, how are we gonna monitor that control system and how are we gonna talk back and forth between the PLC and SCADA? Like I said, all these sites are very remote, they cover a lot of miles. And so the customer wanted to start from scratch. These are existing sites with existing control systems, a little bit antiquated, so we started off by designing the new control system. And once we got that designed and approved, we started implementing that piece of it. And part of that was, "Hey we need security." Security is a big thing now, so I don't wanna jump again here, but that's one of the reasons why we started considering MQTT. And so security was a big concern and a lot of what we were doing required site-to-site communications. So instead of the traditional SCADA server polling devices we also needed sites in the field to be able to talk to each other. And so that was one of the requirements that was a little bit different than what we had done in the past. And so those three things there, some of the original challenges.

07:02
Justin: Yeah, it sounds like quite a challenging project. How did you go about tackling the challenges and requirements?

07:08
Jared: So first, we evaluated the technology that was out there, and some of the technologies that were being used in the upstream oil and gas space, and a lot of those technologies are a little bit antiquated as well. So first, we kinda talked about the standard kind of master­/slave polling network where you've got controllers in the field and you wanna talk to every point in the controller and we would poll on a timed interval all of these points out of the controller every 5, 10, 15 minutes, whatever that may be, and that's a little antiquated. So we kinda quickly move past that, we considered using native drivers with some kind of a middleware that would go out and natively talk to different controllers and things like that. The OPC ­UA and that's a good solution as well. And then we really started to look what people were doing with MQTT and, more specifically, what Ignition was doing with MQTT and that's kind of the path that we went down.

08:09
Justin: I see, so what were the reasons you guys decided MQTT?

08:14
Jared: Well, we kinda talked about the traditional polling, just kind of polling all points out of a PLC at once. It's not really efficient. I may have some values that are not changing, I may have some discrete points that hardly ever changed state. So we liked the value of MQTT being able to push (i.e., publish) on exception. Another thing I might mention, too, is all of these locations connected to the internet via a cell modem. And so if you've ever been to West Texas you'll know that the cellular connectivity is not great, but that's really the only way that we could come up with this system quickly and have the bandwidth to be able to do what we wanted to do. And so really we liked the idea of MQTT being able to push the values that needed to be pushed on exception that we're updating. And we also worked with your team there on a custom license to be able to also subscribe to data at the edge. So that was another nice feature because we needed to grab information from other sites in order to make decisions on this network without having to write some kind of a hook through a traditional SCADA system. So that is really what sold us on this.

09:25
Justin: Nice, that's great.

09:27
Brad: One of the things that always means a lot to me on the MQTT is the store-and-forward capability, which we didn't mention. Any time you use another system, store-and-forward is not native and so you've got to code it in. And when you've got a system like we design for, you've got a hub-spoke and a spoke-to-spoke communications. There's a lot of data grown there, and to try to figure out who knew what last and when and backfill that data without a tool like MQTT would have been monumental. I imagine it's possible but I can't imagine the number of hours it would take 'em to do that.

10:01
Justin: Right. In the course of developing this system, what kind of obstacles did you come up against?

10:07
Jared: Well, we did find a few issues in software that Inductive and Cirrus Link were able to work out phenomenal support on that. We did have some stale values, that for whatever reason would not update but that got worked up, works very well. And we talked a little bit earlier about being able to not only publish data, but also subscribe to other data at the edge and I believe we're one of the first integrators to actually implement that and so that's a big deal. And like Brad said, being able to publish data from any site, also subscribe to that, any data that you want, that's a big deal, that allows you to really fine-tune the control in this case on the sites that we were working on.

10:54
Brad: And I think just to add to that, and to do that with the store-and-forward made it really simple. Again compare it, if you didn't have that capability inherent into the system, then trying to keep up with who knew what last and how we have to backfill that data, or loss of communications, which occurs all the time in West Texas, it would have been really, really difficult. And MQTT, just it was an eloquent way to solve a very complex problem, so I think and the customer ended up very happy. But you guys, as you've mentioned, you guys were phenomenal and there were many nights there in the early phases where it was almost a daily, ‘we got a problem, Houston,’ we need some help here, because we were doing things, like Jared said, that had never been done before. And you guys and Cirrus Link together, were able to jump in with us and we were able to quickly go through those issues and get them resolved and we were back to fighting the next day, so it was awesome.

11:52
Justin: That's fantastic, yeah, thanks for that feedback, Brad. The Inductive Automation guys are gonna love that. And I know Arlen's gonna love that as well.

11:58
Brad: Yeah, yeah, I hope so.

12:01
Justin: So you mentioned, I'm sure the customer is happy, but what have been the end results for the customer since implementation of the new SCADA architecture?

12:09
Jared: It's been great. I mean, like Brad said, we're doing some things possibly that nobody else is doing and it's still growing. So, the project's not quite over, but we're up to around a million tags, updating on exception and we're doing a lot of intuitive control based on some of the site-to-site communications. And so far it's phenomenal, it's something that we're certainly proud of. I know the customer is certainly proud of it. It's something unlike I've ever been a part of before in my career.

12:38
Justin: That's awesome. Well, I really appreciate you guys sharing all that information on the project. So, is there any advice you would give other integrators when developing an MQTT based system?

12:48
Jared: I've talked to Brad about this. We will not go into another project and not use MQTT. It will all be our go-­to architecture. And I think we did a great job in the beginning, but really laying out your namespace, thinking that out, really... upfront, do the upfront work to make sure that the architecture is sound, that you come up with a good namespace architecture, that's really it. I tell you, MQTT's that buzzword that we talked about for, what, Brad? Like two or three years? And it was one of those things that we talked about a lot and we never saw anybody implementing it. We never saw a real life demonstration of how that works, and so it was a little bit intimidating. But now having this project, at least the good portion of this project, under our belt, it is so simple. Ignition makes it simple to set up, and it just makes sense. The way that it moves data from the edge back to the main gateway is just... It makes sense to us and that will be the base of our architecture going forward.

13:48
Justin: Brad, do you have anything to add to that?

13:50
Brad: Yeah, I was just gonna say, I think what Jared says is true, and the adage I'd heard used once, which I like, is a carpenter will say you measure twice, cut once. And I think with the namespace with Ignition, it's that way. Spend double the time creating your namespace, being thoughtful about it, so that when you get into development you're ready to go. Because if you don't spend that time upfront doing that work you're gonna spend a whole lot more time on the back end fixing mistakes and having to redo it. So, measure twice, cut once.

14:18
Justin: There you go. Good advice. All right, and one thing I do want to mention to the listeners is that this project you've been discussing will eventually be a case study which will be available on our site.

14:28
Brad: Correct.

14:29
Justin: Yeah, so look forward to that. They will also be in the Discovery Gallery as an entry for ICC 2020. So definitely be on the lookout for that. All right, and as we wrap up this conversation, is there anything else either of you would like to add?

14:42
Brad: I would just say that we really, again, we really appreciate the collaboration and teamwork that both Inductive and Cirrus gave us on this project and we really, just really appreciate the partnership there. And I think that this time your ability to do the training for Inductive has really... It's just a great offering for everybody and we've tried to really take advantage of this COVID downtime to invest in that because it's something that's easy to do from home. And so I wanted to give you guys a plug for that because it makes learning your product very, very easy.

15:18
Justin: Well, I appreciate you putting that plug in for me so I didn't have to do it, so thanks for that. How about you, Jared? Anything else you wanted to go over?

15:27
Jared: No, I'll just reiterate what Brad said. It's been a pleasure working with you guys on this project and hey, I highly recommend anybody that's sitting home with nothing to do during this COVID situation, get your Core training out of the way.

15:42
Justin: Exactly. Get your Core training out of the way, and if you're Core, do your Gold training. Do that as well.

15:46
Jared: That's right.

15:47
Brad: That's true.

15:47
Justin: There you go. All right. Just one last personal question for you both. So, do either of you have plans once the stay-at-home orders end?

15:54
Brad: We're in Texas and it is, the orders are kind of coming to an end now. And just so happen I am leaving tomorrow morning to go to my daughter, my only daughter's wedding in Kansas City. It was supposed to be in Austin. And we're gonna have a big group of people. We have had to modify that so we got a COVID 15 for the wedding instead of 150, but that's the first thing we're doing after COVID. We've been at home every other weekend and now we're gonna go get our daughter married, so it's awesome.

16:28
Justin: Wow, yeah, yours is a little bit more adventurous than mine. I'm just going to a restaurant.

16:32
Brad: Yeah.

16:33
Justin: So I'm pretty pumped about that. How about you, Jared?

16:36
Jared: Yeah, the restaurant thing was a big step, man. Poor Brad. As soon as... And they did, but as soon as they open back up, I'm just gonna go chill for like six hours.

16:46
Justin: Not leaving.

16:47
Jared: Yeah, just because I can now. But my son just graduated yesterday, high school, and so we are going to head to the beach. That's what we got going on.

16:57
Justin: Congrats. Wow. So some pretty big milestones over at Texas Automation Systems on a personal level.

17:02
Brad: Yeah, absolutely.

17:02
Jared: That's right.

17:03
Justin: Weddings. Graduations. All right, guys. Well, listen, I appreciate your time. I know that you guys are busy and I really appreciate you joining us today and I look forward to working with you more in the future, guys.

17:13
Brad: Thank you, Justin. We really appreciate the opportunity to be here today.

17:15
Jared: Thanks, Justin.

17:17
Justin: Absolutely.