Inductive Automation Blog

In an industry like pharmaceuticals, maintaining quality and compliance is a strict requirement, not an optional goal. Outdated or ineffective technologies and methods can not only hamper efficiency, but also stunt growth and potentially affect product quality.

That was the situation Sandalwood Systems Integration Group found itself in with one customer who was having difficulty bridging the gap between their scheduling application and their ERP system. Sandalwood leveraged Ignition — an industrial automation platform for SCADA, HMI, IIoT, and more — to deliver a solution that completely revolutionized the customer’s production processes.

Minding the Gap

The customer was struggling with a recipe control gap. While their ERP system provided meticulous high-level recipe control and monitoring, the plant floor lacked real-time data as well as access to scheduling and optimization tools, relying instead on an in-house Excel-based application to schedule multiple sterilization lines.

Sterilization scheduling can be a complex process: components must be sterilized in time to feed fill lines, which are predicated on the downstream demand of the final product. On top of that, the existing inventory of sterilized components needs to be considered when determining production quotas. Relying on Excel macros for batch preparation did not offer the flexibility or scalability required to effectively manage the sterilization lines, resulting in a system that could only schedule a few days at a time, hindering long-term planning efforts.

“We were initially given a copy of the Excel-based application and asked to duplicate it in Ignition, which acted as our initial requirements,” said Bob Sloma, Digital Transformation Lead & SME, Systems Integration Services at Sandalwood. “As you can imagine with an Excel-based application, figuring out what they were doing by analyzing the VBA code was a challenging task.”

Even consulting with the application’s developer, it was a challenge to dig through layers of code and determine which reports were actually being used, as it became evident that certain reports were more vital than others.

The Excel application determined the best overall schedule by calculating all possible routings and their overall durations. This brute-force approach could not produce a full schedule without overloading the machine running the application. Clearly, it had reached the limit of its efficacy. “I have significant experience with MRP and production scheduling with my 30-plus years of experience and knew there was a better way to do this,” said Sloma.

Batch Production Scheduler

Once Sandalwood and the customer clearly defined the objective of the project, Sandalwood was able to deliver the Batch Production Scheduler: an application that would allow the customer to create a schedule of the sterilization lines that covered the requirements for the full horizon of the fill lines’ schedule as well as enable them to add products and lines over time. The core functionality of the Batch Production Scheduler included efficient component sterilization scheduling, component quantity determination, as well as reports and metrics.

The Batch Production Scheduler transformed the customer’s production process with real-time scheduling and optimized resource utilization, leveraging Sandalwood’s algorithm to intelligently evaluate processing routes for each component, selecting the most efficient plan. The improved scheduling delivered accurate component quantities for caps, vials, and plungers based on the fill line product schedules, not only ensuring timely production, but also reducing waste and avoiding unnecessary downtime.

Beyond that, the new application delivered adaptable scheduling, allowing users to define multiple routings for component categories. This increased flexibility was especially useful as the new system introduced long-term scheduling, expanding the time window to weeks ahead. As production introduces new products and adjusts current offerings, the customer can now add new products easily, with bill of material-defined component quantities.

The new system also revealed some unexpected benefits, highlighting how difficult it was to implement changes in Excel and how little connection the data had to other applications. “Ignition was chosen as the platform to enable future interoperability starting with integration to directly query a database for inventory data,” said Sloma.

The system displays metrics for each sterilization machine’s schedule and its ability to meet fill line requirements. These detailed reports empower decision-making. “There are over a dozen middle managers and the reporting is being utilized by over three dozen people across a 24x7 operation,” said Sloma.

Sandalwood developed the solution’s UI in the Ignition Perspective Module. Perspective, which leverages mobile-responsiveness and web-based deployment, combined with the Reporting Module enables operators and managers to access reports via a web browser, eliminating the need for local files and allowing the customer to control and constrict access to specific screens and critical data.

Sandalwood’s “lab on a cart” speeds up the development process

Lab À La Carte

A solution like this doesn’t come out of nowhere. At Sandalwood’s corporate headquarters in Michigan, they have a “lab on a cart” where they develop the basis for their applications. The lab consists of a two-unit rack-mounted device running two Windows PCs — one is on the corporate network while the other is connected to a manufacturing network with a firewall between the two — in addition to other devices like industrial PCs, RFID readers and antennas. Sandalwood runs a VM to simulate multiple PLCs publishing data via OPC UA with other VMs running various development environments, one of which contains a shared Ignition Gateway, MSSQL Server, and PostgreSQL.

“Ignition’s use of Python as a scripting language was also a big advantage,” said Sloma, who developed a prototype coded in Python for the scheduling algorithm, including specialized rules for how often the sterilization machines need to go through a wash cycle between production cycles or component changes. “Most of that code was able to be used within the Ignition application by the Ignition developer without significant changes. That saved a lot of development time.”

Ignition’s designer does not limit the number of concurrent developers for a single project. While Sloma was the project lead for this application, Ignition’s shared environment allowed multiple people to perform code development, reviews, and functional testing. As a result, Sandalwood was able to complete most of the development within their lab before implementing the solution in the customer’s environment. “Ignition is a breeze to install,” said Sloma. “This was handled by the customer directly. It takes more time to get the infrastructure in place (VM, network access, etc.) than it takes to install Ignition.”

Ignition’s capacity for rapid development and implementation greatly expedited the project. “The ability to start development sooner within our lab while our customers get their environment in place is a game-changer for us and allows us to deliver solutions more quickly than if we had to wait for clients to set up the resources and grant us access,” said Sloma.

Planning Long-Term

Even with the successful implementation, Sandalwood is already looking to improve the customer’s downstream visibility and advanced scheduling while utilizing the same UI template. Even though this project is not currently connected to any devices, it includes the SQL Bridge Module. Sandalwood plans to connect to a SQL database at some time in the future, with the goal of eventually obtaining the live component inventory data directly by querying the database via Ignition. The Sandalwood team hopes to utilize Ignition to develop line side user dashboarding and annunciation (Andon board) and create a digital whiteboard for the operators. Another goal is to integrate the customer’s Microsoft Active Directory for user management in place of Ignition’s built-in capability as well as their badge-in systems.

The Batch Production Scheduler has streamlined the customer’s scheduling process, and by implementing the Ignition platform, created a foundation for innovation both short and long-term.

Project Summary:

The FMS (Factory Management System) is a real-time automation orchestrator for the entire plant, including production equipment, intra-plant logistics, warehouses, utilities and clean rooms. Through the integration with Level 2, 3 and 4 applications, the FMS delivers a layer of abstraction and a single-point-of-access that allows operations to monitor and control all processes in real time. FMS’ abstraction layer is based on the concept of the Unified Namespace where all applications can exchange the required information for process automation in a decoupled architecture and where Cinfa has set the information semantic model of its business.

Problem:

Cinfa was undergoing a major project of a new greenfield plant, called the NEO project, to manufacture pharmaceutical products. The plant was designed with a high level of automation, where all production and logistics operations, including internal material handling and warehousing of raw materials, WIP and finished goods were managed by a plethora of isolated applications.

The problem was that Cinfa needed to create a hub where all applications could interface without changing their native communication protocol, and at the same time could monitor and dispatch messages according to the production plan. The solution envisioned by Cinfa was not a simple communication middleware, but rather an orchestrator of all messages that each step of the production required at specific moments, based on the overall status of the plant.

By the time Cinfa realized that without such an application they could not successfully run their new plant, there was less than a year remaining before the first batch of product was supposed to launch production. Additionally in the pharmaceutical sector, every application, before going live, has to pass a thorough and lengthy validation process.

Solution:

IDOM implemented a solution that now plays a fundamental role in the orchestration of plant operations, acting as a link between previously isolated or disconnected systems and subsystems. The SCADA/MES capabilities of Ignition have made it possible to serve, with a single solution, different needs for both supervision and production management.

IDOM built the overall solution architecture around the following main modules:

1. Inbound/Outbound interfaces
2. System operations
3. Use case development
4. User interface
5. System alarms and events
6. System administration

Operational use cases include:

1. Production Process Control: nine use cases, including work order sequencing, clean room conditions to launch production, work order setup procedures, batch dispatching to all plant equipment, access and door control, real-time dashboard of all in-progress production, alarms and notifications.
2. Production recipes management: select and dispatch product recipes to equipment.
3. Production container management: 12 use cases, from object creation and management, real-time visualization of current location, container cleaning, etc.
4. Pallet management: three use cases, including raw material (RM) movement to consumption points in the plant, RM returns to the warehouse, and finished good movements.
5. Traceability: two use cases, with full traceability for all raw materials and all WIP.
6. Data acquisition: four use cases, including production equipment efficiency, production step progress, machine status and signal historian, production reports for all rooms.

List of interfaces with legacy applications:

• ERP/SAP
• APS/Asporva
• LIMS/STARLIMS
• CMMS/ORMA
• WMS/ULMA automated warehouse (AS/RS)
• AGVs/ABB-ASTI
• RFID/Cima
• BMS/Desigo
• Production process 1/Syntegon
• Production process 2/Fette
• Production process 3/Bosch
• Production process 4/SERVOLiFT
• Clean rooms/Vidmar
• Doors/STE

List of technologies/protocols used for the interfaces:

• WebServices/API REST/IDOCS
• DB connectivity (Oracle 19c)
• OPC UA server/client
• S7 driver
• Modbus TCP

IDOM built a Unified Namespace according to ISA-95 structure. The UNS is structured using UDTs and a specific tag provider named “UNS” created in Ignition to provide the semantic model for all data interchanged in the plant, independently from the connectivity protocol utilized. A total of about 4,000 different messages have been configured to be managed by Ignition during operations.

List of UI developed, using both Perspective and Vision:

1. Perspective workstations: 14 UIs to monitor all production processes in the different areas. SCADA functionalities and clean room conditions using tables.
2. Vision clients: six UIs to monitor all climate conditions of clean rooms and eight UIs to monitor the different production areas.

Ignition Modules used:

1. SQL Bridge
2. Perspective
3. Vision
4. Alarm Notification
5. OPC-UA
6. Web Dev

Results:

The Ignition solution has delivered the following benefits to Cinfa:

- Real-time visibility of the state of the business, providing elements of supervision of the production process "end-to-end" both to people, through graphical interfaces, and to other systems through the UNS.

- Reduction of human error, integrating the electronic batch record (EBR) with plant equipment. Equipment recipes are automatically transferred, and process values and controls are integrated without the need for manual intervention.

- Reduction and optimization of internal material movements, managing the transfers of pallets and product containers through AGVs between storage points and consumption points.

- Regulatory compliance, immutably recording the details of all production process operations so that the evolution of a batch can be faithfully reconstructed, storing all the transformations from raw materials to the final product throughout the different processes.

As an unexpected or unanticipated benefit, thanks to an exhaustive traceability of production operations, the system has made it possible to analyze the causes of deviations between theoretical and actual production times, which has led to a substantial increase in productivity.

Start Date: September 2022

Deploy Date: September 2023

Project Scope:

Tags: 8,414

Screens: 41

Clients: 38

Alarms: 230

Devices used: 1 Siemens S71500, 1 Modbus TCP, 19 OPC Connections

Architectures used: Standard

Databases used: Oracle

Historical data logged: 9 tags

Project Summary: 

The objective of this project was to develop the Automation Infrastructure and Data Management System backbone for the world’s largest Cell and Gene Therapy Pharmaceutical (CDMO). Center for Breakthrough Medicines’ (CBM) primary aim was to automate data collection and contextualization while ensuring logical controls are in place to protect client data and proprietary information. The system will provide a platform for equipment deployment and integration, maximizes flexibility and redundancy while minimizing upkeep and maintenance, and incorporates cutting-edge automation and IT technology for expansion and future growth.

Problem:

Development of the automation backbone is essential for efficient management of data, which, in turn, is crucial for business and organizational success across industries. CBM’s goal was to develop a robust Data Management System (DMS) and automation infrastructure to provide time-series and relational data archives as well as deploy process control capabilities for future needs.

The primary objective of the project was to automate data collection and contextualization as much as possible and make data readily available to all end users, including clients, enabling scientists to focus on science and business developers to focus on business development by eliminating manual data collection activities. In line with this objective, the DMS aims to ensure logical controls are in place to protect client data and proprietary information while still providing a high-availability interface.

Solution:

To facilitate quick and easy equipment deployment and integration, the system provides a platform from both functional and validation perspectives. The system's flexibility and redundancy are maximized while minimizing system upkeep and maintenance to ensure maximum efficiency.

The DMS is designed utilizing cutting-edge automation open-source technologies, including Opto 22 hardware, Inductive Automation’s Ignition SCADA platform, Canary Historian, and utilizing MQTT protocol, providing CBM with the tools and platforms for expansion and future growth. This architecture was designed with scalability in mind, ensuring that it can accommodate the growing needs of CBM as the company expands not just site-wide, but globally. The system's capabilities also allow CBM to collect, record, and distribute data more efficiently, reducing the potential for human error and freeing up scientists to focus on higher-level analysis and scientific research. By incorporating all data repository and collection systems under one global strategy based on their data model, CBM has a streamlined approach to data management, enabling efficient and effective data analysis, quicker decision-making, and minimizing the effort of system expansion.

To ensure that the system is accessible to all end users, the DMS is designed to provide a high-availability interface, enabling all authorized personnel to access data readily. The system also features logical controls to protect client data and proprietary information, reducing the risk of data breaches and ensuring that CBM remains compliant with regulatory requirements. Utilizing these logical controls, a client portal has been incorporated into the DMS using Ignition to enable clients to securely access their data and reports in real time. The client portal is independent of the core DMS infrastructure, ensuring both cyber and data security as well as data integrity. This level of client access is unprecedented in the CDMO space.

The DMS also provides CBM with a platform to streamline equipment deployment and integration, reducing the time and resources required for installation and setup. By maximizing system flexibility and redundancy while minimizing system upkeep and maintenance, the DMS enables CBM to allocate its resources more efficiently and focus on core business activities. CBM has estimated the use of MQTT and Sparkplug B, in conjunction with Opto22 groov EPICs, Ignition, and Canary, has reduced tag deployment and verification efforts by 95%. Additionally, the use of MQTT enables CBM to contextualize data and add required metadata at the source of generation, thereby eliminating the need for a data lake, as batch records can be extracted utilizing the raw data directly.

Result:

This approach to development of automation infrastructure and data management, the development of a robust architecture utilizing open-source technologies, incorporating data repositories, and collection systems, has been instrumental in enhancing data management efficiency, data analysis, EBR development, and decision-making process for CBM based on its data model. The system's automation, scalability, flexibility, and redundancy, combined with the high-availability interface, logical controls, and equipment deployment platform, ensures CBM can operate efficiently, expand rapidly, and meet the demands of a rapidly evolving industry. CBM’s DMS architecture and automation infrastructure are the most advanced in the pharmaceutical industry, enabling CBM to provide more services and data availability to its clients and giving it a major advantage over its competitors.

Deploy Date: August 2022

Project Scope:

Tags: Initial installation: 6,000. Final estimate: 150,000-200,000

Screens: Currently 30-50 independent views, which will quickly become innumerable based on requests from end users and planned development.

Clients: 10+

Alarms: Ignition is the platform we used to build our Alarm Management System, a sub-functionality of the DMS. As this is a pharmaceutical production facility specification, we have all the alarms.

Devices used: Thousands

Architectures used: Hub & Spoke, Wheel & Spoke

Databases used: Ignition; In addition to Ignition, CBM is using Canary Historian, SQL, Mongo, and file share.

Historical data logged: Canary Historian is historicizing all tags.

Project Summary:

Merck & Co., Inc., the premier research-intensive biopharmaceutical company in the world, requested Grantek’s assistance in building an alarm management system with the Ignition SCADA platform for its facility in West Point, PA. This solution provides alarm monitoring, historization, and a management interface for 10,000+ points while also delivering ad hoc and scheduled reporting tools to aid in the rationalization of up to 30,000 alarm events per day. Grantek’s solution, built with Ignition Perspective, also provides point change management and tracking tools to assist site administrators in maintaining the associated point metadata. These results would be difficult to achieve without Ignition’s versatile capabilities.

Problem:

Merck reached out to Grantek for help providing an alarm management system with the use of the Ignition SCADA system software at their facility in West Point, Pennsylvania. Managing and monitoring alarms is vital in the highly regulated pharmaceutical manufacturing industry, and the U.S. Food & Drug Administration continues to show an interest in alarming management. This project involved installing a new Ignition system to handle alarming at the facility.

The system needed the ability to add various alarm data from existing Siemens PLCs and through OPC tags for different systems on site. To keep the configuration setup as simple as possible, all data was mapped through the existing Matrikon OPC Server. This eliminated the need to verify all tags back to the PLCs or other systems while also simplifying the overall tag verification for the existing system. Tags connecting to Siemens controllers were connected through the Ignition OPC Server.

The alarming system had to provide all levels of alarm management for the configured tags, and a variety of metadata type options. As part of the alarm system, the user needed the ability to enter comments for each active alarm that would be historized and available for review via reports and the alarm’s history. The alarm system will also work in conjunction with Active Directory to access a list of designated users and populate the given parameters that are made available. User profiles and characteristics also had to be created. Additionally, reports needed to be configured to be filterable by time (day/week/month), severity, category, and users.

Solution:

Merck selected Ignition by Inductive Automation as the SCADA system for this solution. Grantek recommended Ignition Perspective, as it is uniquely positioned to deliver the results Merck wanted to achieve. The solution had a connection to 22,000 tags and 11,000 alarm points through an OPC/DA server architecture, while also providing a future avenue to connect 96 new Allen Bradley PLCs directly to the gateway. To ensure a user-friendly experience for Merck operators and managers, a real-time alarm management workflow was developed to allow operators to view, dispatch, acknowledge, and annotate unique alarm events through the life cycle of the event. Perspective views for high-density alarm summary and alarm history were also developed to let users perform advanced filtering of the alarm tables as needed. This feature also allows operators and management to view all relative metadata and response status in a conveniently docked pane.

Results:

This solution provided Merck with an application that enables well-defined management (dispatching, answering, annotating, and monitoring) of all alarms (up to 10,000 alarm events per day). Furthermore, the solution lets application administrators perform in-depth configuration updates to existing alarms points (and associated metadata including setpoints, sensor details, troubleshooting steps, owners, and various other details) and the ability to add and configure new alarms through the application’s frontend. Additional features included automatic notifications and report distribution, and a high-availability DB architecture allowing for historical retention of an immense amount of alarm information.

The system delivered custom data archiving to Merck, which automatically creates archive tables for the ALARM_EVENTS and ALARM_EVENT_DATA tables and moves events to the archive tables at the defined interval. A report schedule builder was also developed, letting management and administrators schedule any of the 10 pre-built reports on daily, weekly, or monthly periods and specify the recipient, report parameters, and report format from the UI.

To provide more flexibility, the ad hoc and filterable reports let users view historical events in highly customized formats. This feature can also be helpful in the event of a regulatory review or audit. The UI tools allow authorized users to manage points from the system’s frontend. They will be able to change point metadata, alarm parameters, point owners, and specify users who will be notified by email or text if the point enters an alarm state. Alternatively, users can export point data to a CSV file from the Ignition Perspective application, make necessary edits, and re-import the CSV file into the application to modify points. 

UI tools were incorporated into this solution to provide workflow for users to request changes to points while allowing system administrators to track the progress of the changes and record annotation. Grantek’s solution also provides AD integration and workflow so that system administrators can approve or deny new users or role changes.

Grantek’s Ignition-based solution equips Merck with a reliable, adaptable, and consistent alarm management system that can be used by operators and trusted by management to display dependable insights.

Deploy Date: May 2023

Project Scope:

Tags: 22,000

Screens: 32 pages, 26 main views, 67 pop-up views, 40 template/embedded views

Clients: 6 dedicated clients plus an indeterminate amount of enterprise web clients

Alarms: 11,000

Devices used: Currently using Matrikon Funnel as a data aggregator, Ignition is connected to this aggregate data source only; however, the system has been designed to allow for ease of migration to Rockwell-based controllers which will total 96 at a future state.

Architectures used: Ignition Perspective redundant gateways, secondary Ignition gateway for reporting from web clients, and a development/engineering Workstation gateway

Databases used: Redundant Postgres DBs using Pgpool for replication and high availability

Historical data logged: 22,000 tags historized with pruning set to 30 days, 30,000 alarm events per day for life of system with no alarm data pruning, alarm tables are automatically archived by Ignition.