Tyson’s Smart Factory Journey

Speakers

Chris Windmeyer

Lead Controls Engineer

Tyson Foods

Geoff Nelson

VP of Technical Solutions

SafetyChain

This session provides an overview of how Tyson has standardized operations with Ignition as a SCADA platform, highlighting and detailing how consistent data and dashboards allow for faster implementations.  The talk will also include best practices that Tyson has developed, and will identify some of the key integrations that have helped simplify and streamline data collection processes.

Transcript:

00:05
David Grussenmeyer: Hello, everybody. I'm David Grussenmeyer, I'm the Industry and Education Engagement Manager at Inductive Automation. Welcome to today's session, "Tyson's Smart Factory Journey." I'll be your moderator today. To start things off, I'd like to introduce our speakers today. We have Chris Windmeyer, Lead Controls Engineer for Tyson Foods. Chris has over 20 years of experience in the CPG space working directly at production sites, with more than 10 years of that in controls in SCADA. He's designed, deployed, and supported multiple MES solutions over the last 15 years.

00:44
David Grussenmeyer: I also have Geoff Nelson, VP of Technical Solutions with SafetyChain. Geoff has always been interested in how things work, which led him to a degree in mechatronics from CSU Chico. After college, he worked as an intern at Csys Labs for a few months before landing his first job as an automation engineer at USS-POSCO Industries. Geoff worked there for five and a half years before moving on to SafetyChain Software, where he is currently the Senior Director of Customer Engineering. Geoff loves his work and enjoys being able to make people's jobs easier and safer. So please help me welcome Chris and Geoff.

01:32
Chris Windmeyer: Alright, thanks everybody for showing up today. I'm normally the man behind the curtain, so public speaking isn't my forte, so if you all hang in there, I'll get through this. Alright, today I'd like to walk you through an overview of Tyson's journey from disconnected systems to a standardized, scalable data collection and SCADA dashboard system. When we started our journey, every site had their own SCADA solution. We had installations at Rockwell, Wonderware, there were even some Ignition out there. These systems were designed and developed by integrators, OEMS, and even some in-house developers. The data that was collected was data that was important to the plant at that time, but there were no standards, so reporting on an enterprise level was complex at best, so the decision was made to create a team that would define standards and build that solid foundation that all other platforms could utilize. The platform we chose, Ignition.

02:31
Chris Windmeyer: Currently, as with many sites, there are a number of challenges, the first one being outdated infrastructure, including network equipment, servers, and even the controlled hardware. The next challenge we had to overcome was the multitude of different equipments, you might have two plants that are making the exact same product, but they use two completely different pieces of equipment. Next, the issue of inconsistent data that was collected in a format that was not standard or normalized. It was collected manually and either entered into a homegrown system into a database, an Excel sheet, or it was data that just wasn't collected at all. And finally, the issue was, there's no overall vision. There were multiple projects, multiple teams that would all be working to achieve the same goal, but they were all operating in silos, there were no communication between them. That's where the Smart Factory Foundation team was formed to create that programmatic collaborative approach across the entire enterprise with a unified vision to create a standard, scalable solution for enterprise reporting.

03:41
Chris Windmeyer: A solution is only sustainable if the site is ready for it, so we did surveys and assessments to validate that the site was ready for our solution. If a site was not ready, we defined a plan to get them ready, whether that involved hardware upgrades, infrastructure upgrade, network upgrades, or even training for the on-site technicians that would ultimately support that solution. One thing we learned early on was that if the site did not have ownership of the solution, it would not be used or maintained, so from the start, we made sure that every team member, every team group saw the benefit of this solution. To ensure the on-site support teams would have ownership, we made them part of the development team. Our team rolled out the configs and the data sets and set the base configs.

04:31
Chris Windmeyer: After that, we invited the on-site tech to configure the tags and actually develop the screens. That way, they had ownership in it, they built something that was visible to the entire corporation. So, as you can see, there's more to creating a solution than just creating a project and walking away. A solution is only as good as the people supporting it and if there's no ownership, it will not be supported or used. So enough with the philosophical backstory, let's get on to why you all got here. So the first thing we decided to do was we created a tag naming standard and a standard folder structure. We started out, as you can see, with the plant code, which is our ERP system's code. Underneath that, we defined an area; now this has changed a little bit, so we've decided to use our ERP resource as the area and then adding machine, the equipment underneath that. This way when we pull in data, which we now have data coming in from our ERP system, we can tie that resource back to process orders, back to material specifications, runtimes thresholds, all that information can be displayed on the SCADA screens.

05:48
Chris Windmeyer: Underneath each equipment layer, we define three separate folders. We've got a machine info folder, which basically is just some static data with a machine description and a serial number, things of that sort. The raw data folder is where we actually pull in the tag data from the PLCs themselves. Within the standard data folder, we take the data that we got from the raw folder and we normalize it. Let's say equipment vendor one is reporting temperature in Celsius, vendor two in Fahrenheit, so within the standard data folder will show a Fahrenheit data, same would be true for a line speed if it's feet per minute versus meters per hour, so we pick a standardized value and that is what we go forward with.

06:38
Chris Windmeyer: Along with this, we also partnered with... Tyson just joined the OMAC, which they help develop standards and best practices across the industry. Tyson does sit on the advisory board, and we're trying to promote that and move it forward as we go. Currently, the PackML model that we're using is only batch process, we've actually been able to get them to agree to start working on a technical spec for continuous processes as well. So this is a machine state model defined by the PackML. We use these... Not every single one of these, but we use these to calculate machine uptime and throughput, so most of the ones you use is execute like a running state; you've got a held or an idle state, a stop state, so you use those within the templates to define your machine parameters and running stats. This is one of our screens that we developed, and we chose a four-quadrant grid for all of our screens across all of the enterprise. The upper left-hand corner screen is always showing equipment metrics such as uptime, quality, performance for the current hour.

07:55
Chris Windmeyer: The second screen, we chose the critical statistics for that piece of equipment also over that same hour. In the lower left-hand corner, we take those same critical points, but we map them for the entire shift, so you can see how that shift has been running with those critical points. The lower right-hand quadrant, we chose to show alarms and downtime for that same shift. Another example of one of our screens showing the same four quadrants, a different piece of equipment, still the same form metric for watching. It looks a little different, but we were able to use our EAM server. We defined a centralized development server, so we define all of these components up on the development server and then we can push updates out to all the projects, if a change was needed because every component is inherited from a global project that gets shared down through the EAM.

08:57
Chris Windmeyer: We've defined this simplicity of scale; we were looking to build a solution that was scalable, so when we went from that first screen to the second screen, we changed two tags, we changed the tag path and the name of that piece of equipment, and referencing back to that standard data folder, as long as it is the same type of equipment that we've defined those equipments types as our base. So if you go from one oven to another oven, you pass in that tag path, that screen comes alive. That's all the changes that's needed. So, one of the main highlights I'd like to call out, our team was able to develop, we do a lot of, in the plants, there's a lot of human interaction with a piece of paper, and they're taking metrics out on the floor: temperatures, speeds of conveyors. It's all a piece of paper; somebody either goes and enters it into an Excel sheet or, even worse, it just gets thrown in a drawer and nobody even knows it exists.

10:02
Chris Windmeyer: So what we developed was an actual process controls and monitor form that's following the TPS 2.0 methodology. Every form, every cell on here is able to be defined by the plant at runtime, so they can totally define what checks they wanna see. Every check has upper and lower limits, it has alarm values, the tags that you see here, the cells that you see here gray are actually pulling in directly from the PLCs. So an operator doesn't even have to go look at an HMI that's... Oh there's my value, they fill in the information they want. And they hit the submit button. The other thing we've been able to do, you can see we've tied in the SKU data, which is pulled in from our ERP system, so these metrics actually get tied back to that order that was running at that time, so that when this gets ingested into our cloud infrastructure, we can report on it across the corporation to see what kind of issues we had. And like I said before, these are entirely customizable. We built them so that we can actually have reoccurring time, so let's say we needed one of these checks that happen every hour, so if that hour passed and that check was not done, there's actually an alarm that gets sent out and say, "Hey, this check is past due." The same goes for a setpoint that's out of spec, so when they submit that and it's out of spec, it can actually send an alarm on to the supervisor or FSQA.

11:39
Chris Windmeyer: So one of our next and one of our really good integrations we had, SafetyChain is our quality control piece that we use in Ignition or not in Ignition that we use currently at the plants, so we had a really highly complex system. We were pulling data from the local Ignition database up into the cloud, and we were going to Monarch, which was pulling metadata from SafetyChain, overlaying those, building that data, submitting that SQL query back to SafetyChain, and submitting that data. And what we've been able to do with the new Ignition module that's just come out, is we're able to go straight from Ignition straight to SafetyChain. This will pull the form in directly from SafetyChain; all the fields at the bottom are coming in from the form in SafetyChain. It can pull data in from any data source that is available to Ignition. You've got the full expression language scripting available to you as well, so this was a huge time saver and for scale and rollout. So if we have this template, I can take it, copy, paste, change my data source, done. Five minutes versus days in the old format. So with that being said, I'd like to turn it over to Geoff and...

13:05
Geoff Nelson: Thanks Chris. So I've been working with Tyson about five years. I've been with SafetyChain for 10, and we've been partnering with Tyson to deliver this quality module. It's usually where we come into a place and we're replacing paper clipboards. Chris did mention that a little bit. Tyson's case is a little bit more unique. They had a homegrown system called Plant View. Plant View right?

13:32
Chris Windmeyer: Yep.

13:32
Geoff Nelson: And it did digital collection already, so there was high expectations for what SafetyChain would come and do and replace Plant View. So as a quality system, we came in and became part of the Tyson tech stack to make sure that they're within regulatory compliance for quality. And we set up a lot of integrations because their current system had it already, so they were pumping data into SafetyChain to make sure their temperatures are within compliance or weights, or what have you, through production, through quality. But it was difficult; like Chris said, they had the system called Monarch, which is a homegrown system.

14:09
Chris Windmeyer: It is. That's correct.

14:09
Geoff Nelson: Home-built system. So they built the system in Monarch to do these queries, pull this data, create that JSON payload, and send it to SafetyChain, very complex. Took a lot of time, and there were some issues with some of the queries.

14:19
Chris Windmeyer: Yeah, very unstable.

14:21
Geoff Nelson: Very unstable. And so what we did is we said, "There's gotta be a better way for us to accomplish this." So we sat down, SafetyChain. So I went and met with Tyson, Tyson architects, engineering, IT, also with Travis Cox came and joined us, to discuss what can we do. We also had, actually, Cam Bergen, CEO for mode40, came to help. We just sat in a room to say, "What's the right approach here?" How can we make a scalable solution here? And came up with this Ignition module. So, we decided with the plant level, the enterprise level, the way that the Ignition deployment had been done and was going to be done at Tyson, we would leverage the gateway and the data that was available there and create a module that made it a lot easier to pass the data to Tyson easier, more scalable, reliable. It's configurable now instead of how long would it take to create something like this in Monarch?

15:19
Chris Windmeyer: Probably days.

15:21
Geoff Nelson: Days, yeah. Now it's minutes, right? Potentially.

15:22
Chris Windmeyer: Exactly.

15:26
Geoff Nelson: So much easier. And then it leverages Ignition's framework. So we have store and forward, there's queuing. So if anything fails, you can see why it failed, if the connection is lost for whatever reason, it will queue and send back up when the connection is restored. So a lot more reliability, a lot of scalability there moving forward with the module. So it just shows the partnership piece of the equation. SafetyChain's a digital plant management platform. We're there as that quality system with Tyson. But now we've been able to develop this module to allow any of the system data to come in. But the main point was the partnership coming together. Inductive Automation, along with Tyson, who's a great partner to create this module and extend scalability and reliability of the data. That's all I have.

16:23
David Grussenmeyer: Okay. So now we're gonna open it up for Q&A. We have a mic runner. Please, if you have a question, you can raise your hand. If you're in the front rows, you can come down to one of these mics down here or just one. And make sure to state your question into the mic so that we get it for the Livestream audience as well. So, any questions?

16:42
Audience Member 1: Hi, so I just had a question with regard to how SafetyChain is able to streamline the integration from when it was done from Monarch. Can you maybe elaborate a little bit more on the details of how it streamlined that?

16:58
Geoff Nelson: Yeah, so Tyson has Ignition. Tyson had decided on Ignition as a standard for their enterprise. And so the data could become available through Ignition. Monarch was a tool that was essentially, it was homegrown, it was essentially a way to pull queries on data. So very complex queries to pull data and create a JSON payload to call an API. Through the module, we've made it configurable to actually download form configuration from SafetyChain so that Ignition knows what's my data set from SafetyChain; what is my thing I'm looking at. What are its temperatures, where it's at, what it's about. And it can pull any of the data sources from Ignition. So it can look at tag, the historian, name query, SQL query, and perform expressions to manipulate the data potentially and pump it directly to SafetyChain. So it's a bidirectional integration, essentially through the module to know what SafetyChain needs, downloads it, and then sending it back up to SafetyChain. So you could trigger like time-based temperature checks, for example, in an oven, or you could do it. You have some really complicated triggers, right?

18:06
Chris Windmeyer: Yeah. So we've got queries we've defined that, say in an oven, for example, where a temperature has to be at 128 degrees for 60 minutes, and then what we send up is the previous 60 minutes, every minute temperature of four different probes in each oven. So we've really been able to define that trigger within the form itself. So it can use any, like I said, any data source within that's available to Ignition. And you can define those triggers in the tags. You can define it in the form itself as what triggers it. You can have it time-based. It can be a, you could say, let's do a temperature check every 60 minutes every time this tag goes true. I mean, it's very, very user-friendly and easily defined.

18:55
Audience Member 2: So, Chris, I like the story you told about the way that you would take the application to plants and then get individuals at those plants to learn how to build screens and things like that. Can you kind of go more into that process and kinda like what was involved to get their buy-in? Did you have to get them Ignition certified? Did you walk them through it? Just how did that process work out?

19:12
Chris Windmeyer: Sure. So, like I said, from the start, you have to get buy-in from ELT teams. So you've gotta make sure you've got buy-in from the business unit as well. Plant management all the way down to the people that are actually gonna be doing the support. So once we got that and we got approval to go to a site, we would do our validations. Okay. Is the network up to speed? Is the virtual infrastructure okay? How about your automation techs are gonna be supporting it? Do they know Ignition? They do. Okay. They're certified. Great. We go, if they're not, we've actually taken and cherry-picked out classes that from Inductive University and said, "Okay, you guys need to go through this. We built a tier-based solution, so four different tiers, and before we get to the site, you need to be through tier three."

20:01
Chris Windmeyer: And then once they're through that, we get everything set up, we get the server spun up, we get the base project created, and then we build, say the first line 100%, and then line two we work hand in hand with the onsite techs and say, "Okay, you're gonna build this solution." So they create the tags, they map the tags in from the raw data into the standard data. They change the tag and the views, and they're rocking and rolling. And that really gets buy-in from the site. They've now got ownership; they develop something on their own that's showing real-time data, live data that everybody can see. And that is a key point. You've gotta have buy-in from every level, or it won't be sustained.

20:46
Audience Member 3: Was there anything that was like a catalyst within Tyson to commit to this sort of big strategic project? Or did you have to champion that? And how did you get traction for a big automation strategy instead of each site being sort of content with the status quo and just getting by?

21:06
Chris Windmeyer: Well, I mean, that was the big piece, is the plants were fine with it. They had the data they wanted, but at a corporate level, there was no visibility into that data. Yeah, you could get the ERP data, but that's really, really high level. We wanted to be able to see, from a corporate level, that granular level. So we get the ERP data up, now we've got the Ignition data tagged to it, and you can really see it at that level. So it was pushed down from the higher management, said, "Hey, we need a solution for this." And the solution was create a team and go out and figure out how to do it.

21:44
Audience Member 4: The question from me is, you guys standardized the screens that you wanted to be able to, let's say, compare maybe a plant to a plant. Did you also enforce the standardization for maybe projects that the guys that you cherry-picked would say? You know what? It's probably good if I wanna visualize this other process that I have here that's not tied to those standard screens. Did you standardize that as well?

22:08
Chris Windmeyer: So what we've done is we've given them templates. So if they do create, come up with a new piece of equipment, they've got our standards, and we've told them, "Okay, go ahead, build it on your own. Once it's complete, come back to us. We'll make sure it fits our standards, make sure it rolls with everything else." We give them the feedback, we allow them to make the changes. So they develop that component, which would then get rolled out, should be to multiple sites. So then that really builds that buy-in. They're now not just building something for their plant; they've just built something that's just spread across the entire corporation.

22:48
Audience Member 5: How's it going? Specifically, if you don't mind, what's your experience been with SafetyChain specifically? My company is a food manufacturer as well. And there is a lot of paperwork with USDA and all the other... And I'm just curious, can you speak to like your experience with SafetyChain specifically? Are you extremely happy, moderately happy? Is there a range?

23:12
Chris Windmeyer: That's kind of a loaded question.

23:13
Chris Windmeyer: So I don't... I don't really use SafetyChain myself. I've learned enough about it, just so I can see that when I submit data, it showed up over here. Okay, great. I have worked with the team that does some of the report building. They seem to be really impressed with the reporting capabilities and how they can see the data they need to be able to see.

23:32
Audience Member 5: Alright. Thank you.

23:39
Audience Member 6: Hey, Chris.

23:40
Chris Windmeyer: Hello.

23:40
Audience Member 6: Your tech paths are pretty cryptic for the average person. And so I wanted to ask what the process was like for your design team to choose between easy to read, kind of anybody would understand versus the engineering or ERP tag structure that you came up with?

23:54
Chris Windmeyer: So we went through a couple renditions, but since we did ultimately want to get that ERP data down in an easy way, that's kind of why we chose to go with that more cryptic kind of definition. Yeah, it's a little more complex for the plant engineers that have to support it, but really it is defined. I mean, step by step, we've got documents to define. Okay, this piece of equipment goes in this spot, and it's very easy to follow, and once you've worked with it for a few days, you kind of start to understand it.

24:28
Audience Member 7: Yeah. I saw that you were bringing in raw data, right? And then you're normalizing in your tag structure. Are y'all historicizing and operating off of both of those data sets, or are you strictly operating and historizing the normalized data?

24:38
Chris Windmeyer: So it depends. So if the data coming from the PLC is already normalized, it stays in that raw data folder. If we need to make some changes to it and normalize it, then it'll go into the standard data folder. So we are using both, depending on the case.

24:54
Audience Member 7: Gotcha. And follow-up question: would there be valuable or value, excuse me, to you to consolidate like your tag counts by having some kind of tag type that would allow you to transform that data before it entered the system. Instead of having to maybe say, reference it and do calculations on it.

25:10
Chris Windmeyer: Yeah, I mean, a lot of times we don't have access to the PLCs themselves, so we have to basically take the data we get as it is. There's a lot of options we're still pursuing. I mean, this team is not even a year old yet, so we're still kind of getting our feet wet and still defining those standards and redefining them. So anything's possible and under consideration at this point.

25:43
Audience Member 8: Hey, Chris, you talked about a little bit about the start of, about your Digital Transformation journey. I'm curious, if you were to go back now and you had all the stakeholders in the room, if there was any one thing that you would do differently or if you'd want to emphasize for people that might be kicking off smart factory journeys themselves in the coming week.

26:07
Chris Windmeyer: I kind of came into this team about six months after it was spun up, so I don't really 100% know where it totally started from. But I think that was a big thing that we learned kind of after the fact. We did kind of in the beginning, try to show this to a plant and say, "Here's what you're getting." That wasn't quite received so well, so we stepped back a little bit, made sure we had buy-in, made sure that that support was gonna be there. So I think that really is the key to this whole thing, is getting that buy-in and making sure everybody's aware that of what's coming.

26:41
David Grussenmeyer: Alright. I think we got time for one more question.

26:43
Audience Member 2: Oh, okay. Well, I get a second one then. So now you have Ignition at all these plants. You have all these technicians trained in how to use Ignition. It's piping all this data up to the enterprise, but do they have the leeway to do other stuff with the system? And are they playing with their new toy and what are they doing?

27:01
Chris Windmeyer: Yeah, so we've given onsite techs a lot of leeway. They basically, they're free to do, it's theirs. I mean, we put our solution out there, our standard data is there. As long as they don't mess that project up, they're free to go do as they want. Maybe they wanna make a one-off for their wastewater solution. Well, we're not monitoring wastewater in our enterprise reporting, so go have fun, play.

27:28
David Grussenmeyer: Awesome. Well, let's give a round of applause for Chris and Geoff.