Project Summary:

Initially, Zendal approached Optomation Systems to tackle their needs for monitoring unmanned storage units holding pharmaceutical final products for their customers. When Optomation introduced Ignition, it rapidly evolved into a larger project with a broader scope to encompass the data acquisition of varied equipment, process control and supervision, alarm notification, data historization, and automatic report generation.

The main objective of the project was to implement a system that acts as a data repository for the information collected from the dedicated equipment in the different production areas and auxiliary services, both to satisfy regulatory requirements and prove compliance with CDMO obligations to customers.

In addition, the system integrated new production areas into the system and expanded functionality to include control and supervision of several processes. Field devices and sensors are connected to controllers that execute automated sequences, and from Ignition, operators have full access to the operation, just like a traditional SCADA system.

Problem:

One of the main problems for Zendal was that the freezers and cold rooms for the final products are spread throughout the site and are mostly unmanned, even during working hours. To ensure safety and compliance, set-apart personnel were required to regularly inspect the installations to record data and check for equipment failures.

New products and regulatory requirements also brought in new process units in the form of dedicated equipment (autoclaves, incubators, particle counters, etc.) or skid units with their own data retention mechanisms. This created a labyrinth of data formats, device connections, and communication protocols leading to information silos that were really hard to sort out.

Lastly, the increased production heightened the demand for operations such as connecting skid units or cleaning, which are normally performed manually, to be automated to increase speed, repeatability, and product safety.

Solution:

Selecting the proper architecture design along with Ignition’s unique features, provided the solution for network and area distribution concerns. The hub & spoke architecture with gateway nodes placed locally in each production area ensures that all data collection is independent of the network backbone. Using Ignition Edge in Opto 22’s groov EPIC controllers, combined with Ignition’s Store and Forward mechanism, safeguards historical data in the event of communication problems and guarantees that the central node receives all data for long-term storage.

The system leveraged I/O and programming capabilities of the groov EPIC controllers to achieve the full potential of this segmented design. In practice, each groov EPIC becomes a node in the gateway network that can perform data acquisition, process control, operator interface and data synchronization in a self-contained unit that is independent of the network and other areas.

The gateway network provides the appropriate protection and cybersecurity for data traveling the network backbone, while the physical separation of the IT and OT networks protects all the data sources such as standard laboratory or production equipment with legacy protocols or unsecure access.

For the non-process controlled, unmanned areas, the combination of Ignition’s Alarm Notification and Voice Notification modules provided the perfect solution for the maintenance teams. By monitoring both device alarm contacts and process conditions, the system can provide notifications to selected users using visual elements, e-mail notifications, and in some critical cases, by phone calls to on-duty staff and on-call crews.

By carefully using the unique features of Vision and Perspective, combined with the ability to provide multiple projects within the same license space, the system can easily tackle the different requirements of all its users. 

• Operators, production supervisors, and quality technicians gain access to production using screens in Vision clients created for each plant area. 
• Managers for production, product, or quality have access to all the plant’s data with Perspective where views were created to unleash the amazing customization of Power Charts, so users can create and save their own charts, check dashboards, or export data for advanced data analysis.
• The Quality Assurance department gets automatically generated and pre-formatted reports for key equipment.  

Results: 

Besides the evident benefits from going from a manual and non-automated data recollection and supervision to a new and modern automated system, the project also showed the amazing scalability of Ignition.

The extensive use of UDTs to template analyzers and other typical pharmaceutical equipment permitted new equipment to be rapidly incorporated anywhere in the plant, saving time, resources and, ultimately, money. Also, the breakthrough licensing model of Inductive Automation gives the customer the confidence and predictability to keep growing the system.

As a CDMO, Zendal is continuously adapting their installations to its customers’ needs, not to mention that for Zendal and its customers the speed to market is crucial for the business. Ignition’s flexibility and scalability has proved to be the right choice to adapt to Zendal’s ever-changing needs.

Start Date: 11/2020

Deploy Date: Ongoing

Project Scope:

Tags: 187,900 (Atomic Tags), 28,600 (OPC Tags)

Screens: 217

Clients: Unlimited. 5 fixed Vision clients, and other Vision and Perspective clients connect freely

Alarms: 9,500 (Atomic Tags with at least one alarm configured)

Devices used: - 18 Opto22 groovEPIC, 25 Siemens SIMATIC S7-1500, 2 Allen-Bradley CompactLogix, 10 Bürkert Valve Islands (160 Solenoid Valves), 8 Schneider Motor Variable Frequency Drives, and 60 Lighthouse Particle Counter Analyzers

Architectures used: Hub & Spoke

Databases used: Two. One per site, MSSQL Server 2019

Historical data logged: 36,700

Project Scope:

• Tags: 5,801
• Screens: 22 Pages, 50 Views
• Clients: 5 HMI Thin Clients (3 more future scope this year, plus many more for the building expansion project next year), Perspective session via browsers in office
• Alarms: 390
• Devices used: 17 total; Rockwell, Siemens, Delta V; All communications via Kepware
• Architectures used: Redundant, on-site, standard with ability to publish to cloud gateway.
• Databases used: Microsoft SQL Server Cluster and OSI PI in Redundant configuration
• Historical data logged: 4,640 Tags logged in OSI PI Historian

Project Summary:

National Resilience Inc. offers a broad mix of customized manufacturing and development capabilities at their Mississauga, Ontario facility to serve the needs of biopharmaceutical companies at all stages of the drug development process – from pre-clinical development to commercial supply. Resilience requested Grantek’s assistance to develop an Ignition SCADA to support new GMP manufacturing processes in the building area known as the “H-Area.” This solution was needed to rapidly scale a brownfield contract pharmaceutical manufacturing facility, allowing Resilience to maximize their productivity and meet their commercialization goals.

Problem:

National Resilience Inc. offers a broad mix of customized manufacturing and development capabilities at their Mississauga, ON, facility to serve the needs of biopharmaceutical companies at all stages of the drug development process – from pre-clinical development to commercial supply. Resilience requested Grantek’s assistance to develop an Ignition SCADA to support new GMP manufacturing processes in the building area known as the “H-Area.” This solution was needed to rapidly scale a brownfield contract pharmaceutical manufacturing facility, allowing Resilience to maximize their productivity and meet their commercialization goals. With Resilience focused on broadening access to complex medicines, this solution needed to come online as soon as possible. This urgency was heightened as the COVID-19 pandemic and global military conflicts increased the threat to supply chains. Resilience’s clients rely on them to contract and manufacture important pharmaceuticals that help improve and maintain the health of patients across the globe. Ignition SCADA is a vital need to ensure operations function efficiently and these pharmaceuticals are manufactured successfully while maintaining compliance.

Solution: 

Grantek reviewed the UX development including the overall look and feel of the Ignition Perspective SCADA user interface with all stakeholders prior to beginning SCADA development. This review was part of the ongoing preliminary engineering engagement. Grantek also suggested milestone design reviews at the 30%, 60%, and 90% stages of the H-Area SCADA development. All functionalities were based on the approved Grantek Functional Specification (FS). 

Grantek believes a proper Ignition-centric deployment model should include both an immutable component (verified via digital signature) of the software installation and base-level configuration, as well as a version-controlled mutable component. National Resilience Inc. agreed with this belief and realized the benefit this would provide to their overall operations. To do this Grantek developed the Ignition SCADA application and a new PLC program for direct control of the water valves for the 5 POU stations in the Upstream, Downstream, and Formulation Areas. The new SCADA/PLC coordinates directly with the existing Water Manager PLC (PU70) for Water for Injections (WFI) request/authorization functionality. Grantek also updated the PU70 PLC program to include the new POUs in its water manager prioritization system. 

Grantek developed the Ignition SCADA application for direct connection to ethernet enabled OEM equipment and will account for the new SCADA platform will use Ignition across Development, Test, and Production environments, with the Test and Production environments implemented on redundant Ignition gateways. This allows for the solution to evolve in the future, as it could see the three environments be connected to an enterprise-wide cloud-based instance of Ignition. The new SCADA platform supports WiFi distribution as well as data acquisition and monitoring of OEM production equipment and instrumentation. This solution has hardwired connections via the new remote I/O panels for visualization and logging of process values and states. To do this, National Resilience Inc. provided a finalized list of tags per equipment category for integration into the SCADA system. Grantek provisioned a Git-backed version control system to track mutable changes and worked with National Resilience Inc. to define procedures for tracking and approving resource modifications.

Results:

The SCADA solution Grantek provided in the “H-Area” was able to utilize Ignition to not only deliver results, but also integrate with other aspects of the facility. By using a unified software solution like Ignition, National Resilience Inc. also has access to data that can be leveraged in future projects at their other North American facilities. The solution Grantek delivered included over 5,800 tags and 390 alarms. The solution also logged over 4,600 tags in OSI PI Historian. 17 total devices were used in this solution, including Rockwell Automation, Siemens and Delta V – with all communications done via Kepware. Ignition provides a single point of reference for all of their OEM production equipment, no matter the supplier. A redundant, on-site, standard Ignition architecture was utilized, with the ability to publish to cloud gateway. A Microsoft SQL Server Cluster database was selected, including OSI PI in Redundant configuration. While PI is the historian, Ignition is the reporting tool chosen to present that historical data.

Grantek’s solution is helping National Resilience Inc. quickly scale up their unique brownfield contract pharmaceutical manufacturing facility. The result will see National Resilience Inc. produce pharmaceuticals and vaccines available to patients all over the world, especially important as the COVID-19 pandemic and supply chain issues continue to hamper the timely access to needed medications.

Transcription:

00:09
Doug: I'm Doug Yerger, a Principal Engineer with Grantek. Grantek is a systems integrator and a business solution partner with over 40 years of experience in the life sciences, CPG, and food and beverage industries. We are a Premier and Enterprise Ignition Integrator with over 10 years of Ignition experience.

00:30
Doug: We did this project for National Resilience, Inc. Resilience is a technology-focused biomanufacturing company dedicated to broadening access to complex medicines. Founded in 2020, the company is building a sustainable network of high-tech end-to-end manufacturing solutions to ensure that treatments can be made quickly, safely, and at scale. Resilience offers a broad mix of customized manufacturing and development capabilities at its facility in Mississauga, Ontario, Canada. Resilience serves the needs of the biopharmaceutical companies at all stages of the drug development process from pre-clinical development to commercial supply.

01:10
Doug: Grantek helped Resilience develop an Ignition SCADA system to support new GMP manufacturing processes in the building known as the “H-Area.” This solution was needed to rapidly scale a brownfield contract pharmaceutical manufacturing facility allowing Resilience to maximize its productivity and meet its commercialization goals. This SCADA filled the gaps in the new area's automation systems by providing the Water for Injection prioritizing, point-of-use valve control, OEM equipment status monitoring, various environmental and process input monitoring and alarming, alarm statistics, trending of OSI PI data, and reporting.

01:49
Doug: The design tool Figma was used to concept the screens and review with the customer. This is the overview for the Water for Injection system. The upper portion of the page shows the status of the two WFI loops, including their current prioritized queue requests. This page provides the operator and support personnel with key information around the systems so they can more quickly diagnose any issues. On the bottom of the page are point-of-use controls for the point-of-use valves themselves. And in the lower-right corner is a small area that actually is the control for each of the valves. This is a pop-up that is controlled by selecting which area you wish to see.

02:32
Doug: In addition to the overview, the detail of the status of each of the water systems is also available. This provides additional insight into the external water systems, which are not local to the new systems in the H-Area. This is an example of an OEM status screen. Resilience and Grantek work with the OEMs to determine available connectivity methods and available tags. Resilience then combs through those lists to determine which tags they wish to see displayed. This still left a very large number of tags for most pieces of equipment. These tags were then divided into logical groupings, which are displayed as tabs on this page. This page is fully dynamic with all the tabs and tag lists generating automatically from the tag browser.

03:16
Doug: If a tag is added to the directory for a piece of equipment, it will automatically show up on the screen. In addition, the user can pin commonly used tags, which will appear at the top of the page. The lower section is also collapsible to allow for a larger viewing area for the pinned tags at the top. The miscellaneous environmental and process inputs and outputs are displayed by sub-area in the process. This page uses a similar design to the OEM page to provide a better user experience. Once again, this is a fully dynamic page based on the user tag screen. Once again, this is a fully dynamic page based on the tag tree structure. One key addition on this page is the collapsed alarm banner at the bottom of the page. And here is an example showing the alarm banner expanded. P&ID pages are available as well for more detailed troubleshooting.

04:07
Doug: The system includes reporting shown as the alarm report. On the left is the filtering system for the reports allowing you to set start and end date and filter by equipment or alarm name as well as priority and state. The report will update when you click the "Generate Report" button in the lower left. In addition, we have an alarm statistics screen. On this screen, we show the number of alarms occurring over the course of the time period, in addition, the top 10 highest occurrences during that time period as well, showing both the alarm itself, as well as a Pareto showing the ranking of that alarm.

04:46
Doug: A final important feature of this project is the audit trail. We've enhanced the standard audit trail to include previous value, reason, and electronic validation signature when required for audit roll actions. Our workflow for this project made use of the Figma design tool. This facilitated conversations around themes, styles, and user experiences without diving into the development environment. This greatly shortened the design process. I personally think Figma is a great design tool. Resilience's commitment to Ignition is highlighted by this project being expanded for a new greenfield facility, including full control for the WFI and EMS systems. In addition, Resilience is using this project as a basis for a global cloud-based solution.

Project Scope:

• Tags: 1,322
• Screens: 12
• Clients: 2
• Alarms: 91
• Devices used: 10 Microsoft Tablets with Windows 10 Pro
• Architectures used: Redundant Cloud-Based
• Databases used: Microsoft SQL Express, Microsoft SQL Server Standard, OSI PI Historian
• Historical data logged: 44 tags OSI PI

Project Summary:

This project involved using Ignition in the personalized medicine industry to handle multiple communication protocols in one 21 CFR Part 11-compliant environment. The same regulatory data integrity requirements exist in personalized medicine as in large-scale productions and Ignition has proven to be a valuable tool because of its flexibility, mobility, and above all — device integration.

Problem:

The industry of personalized medicine faces challenges as lab equipment is brought into commercial manufacturing. The equipment consists largely of closed systems that process individual (patient) batches and generate a large amount of GMP data. The equipment is unlike traditional industrial automation equipment, posing a challenge to many different communication capabilities and protocols. The same regulatory requirements, such as 21 CFR Part 11, in large-scale productions persist in personalized medicine, in that data collection and integrity are required for commercialization of the product. Ignition has proven to be a valuable tool with its flexibility, mobility, and above all — device integration — that will serve this industry well.

Solution:

Barry-Wehmiller Design Group chose Ignition because it required a software that could handle multiple communication protocols such as MQTT, OPC UA, and Web Services, all in one environment. Ignition allows users to use portable devices like tablets and laptops, and embedded wall-mounted monitors with web browsers to visualize and interact with the application in clean rooms where traditional HMIs are not ideal because they are difficult to clean and comparatively large. Simple, intuitive screens provide a constant overview of processes, as any minor disruption in an operation can cause a batch to be discarded in cell and gene therapy manufacturing. 

Licensing costs were also a major consideration in choosing Ignition over dedicated software designed to perform a single activity, which often have high licensing prices. This level of flexibility allowed Barry-Wehmiller Design Group to develop custom dashboards for each room, along with floor-plan screens that mirror real-world layouts.

Also, Ignition allows for easy integration of a sitewide Active Directory, which directly helps to ensure secure user access to the Ignition application, by maintaining the policies and requirements (such as password requirements, account lockouts, etc.) defined within the Active Directory. Also, Ignition provides a robust audit trail feature which is very important for 21 CFR Part 11 compliance. In addition, the Ignition application was easily validated during IOQ. All of these items help to ensure Barry-Wehmiller Design Group meets and maintains compliance with 21 CFR Part 11.

Results:

Ignition’s ability to work with a variety of technologies was of utmost importance. This was a key factor, because there are so many different types of equipment. The industry is fairly new, and many companies are still developing their processes, and these processes are established in hospitals and universities across the globe. Depending on where a company begins this journey will determine what process equipment it ultimately ends up using in its commercial manufacturing facilities. Ignition’s “multilingual” capabilities made a significant difference in this regard.

The purpose of 21 CFR Part 11 regulations is to ensure that electronic records (and signatures) can be trusted as much as paper records and ink signatures. Cell and gene therapy processes generate many electronic records that are used for regulated purposes and are therefore subject to the regulations. The equipment used in the process, and in-scope of the Ignition PCS, do not utilize electronic signatures, so we are focused on the electronic records, i.e., information in a digital form that is created or used in some way by a computer system. Our Ignition PCS satisfies a few aspects of the regulations pertaining to rendering records, document storage and record retention, system access, authority checks, and audit trails.