Creating a High-Capacity Lab for COVID Testing in Just Nine Months

Project Scope

• Tags: 100,000
• Screens: 50
• Clients: 50
• Alarms: 2,500
• Devices used: More than 40 Rockwell PLCs, 30 Fanuc robots, 20 Keyence vision controllers, numerous types of lab equipment
• Architectures used: Standard
• Databases used: PostgreSQL (2)
• Historical data logged: 100,000 rows/day in production

Project Overview

Ginkgo Bioworks expanded its automation capabilities in response to the COVID-19 pandemic and the need for increased nationwide diagnostic testing capacity. Ginkgo designed and built a new sample-processing lab very quickly. The lab is highly automated; once a rack of samples is placed on the line by an operator, it makes its way through a series of work cells with no human intervention. Ultimately the goal of the system was to prepare COVID-19 test samples for DNA sequencing & PCR testing for the virus. At the heart of this new lab is the Ignition SCADA platform.

 The automation system in this lab is responsible for:

• Transferring samples from station to station with MagneMotion
• Handling samples with robots
• Processing samples with sophisticated lab equipment
• Quality-control vision inspections of samples throughout the process

Problem

Sample traceability is of utmost importance. With more than 40 separate PLCs all triggering actions that impacted the samples, Ignition was critical to unifying the process data and allowing Ginkgo to extract meaningful testing results.

All of this occurred while the biological process, physical lab layout, and system requirements were changing daily. Ignition is the only SCADA platform that could have allowed DMC to hit a moving target at this scale and pace.

The most important challenges Ignition solved were:

1. Providing event logs and sample traceability. Events occur multiple times per second throughout the lab across many PLCs. Ignition processes them and sends them all to a database. This ensures all steps of the process can be traced without any potential for lost data. 

2. Acting as an information broker between PLCs and lab equipment. The lab equipment is not able to communicate directly with PLCs, so Ignition relays commands from PLCs to the lab equipment and statuses from the lab equipment to the PLCs.

3. Providing intuitive operator interfaces and dashboards to control, monitor, and support performance of the lab. Ignition facilitated multiple tiers of users and associated permissions.

4. Supporting rapid development and shifting requirements.

Solution

Traceability

To achieve robust traceability as samples moved throughout the lab, DMC created the following architecture within Ignition:

• Handshake-based event-logging framework between Ignition and more than 40 Rockwell PLCs.
  • This consisted of multiple event queues on the PLC, and a trigger tag for the PLC to set off transaction groups within Ignition.
  • The transaction group items were set to OPC Read Data Mode to ensure that the data was consistent, and time-synchronized.
• Ignition logged the relevant data to a PostgreSQL table.
  • From there, Ginkgo’s internal software team built a “continuous event bus” which consumed the new events in PostgreSQL and distributed them to downstream software.

§  This downstream software utilized the data produced by Ignition to parse the events that occurred in the lab and ultimately match COVID test results from pooled genetic sequences with a particular sample barcode and thus, a particular patient receives their test result based on data created by Ignition.

• At full capacity, the architecture processed hundreds of thousands of events per day.
  • Data loss was not acceptable, so Ignition’s store-and-forward capability was critical to creating a robust system.
• This architecture required hundreds of transaction groups to be maintained on a regular basis. DMC created an automated Python tool in Ignition to import/modify transaction groups using XML files.

Lab Equipment Communication

Most lab-grade machines are not built to interface via standard automation protocols. Instead, DMC used a combination of custom-developed applications and Ignition to implement a robust control solution. DMC successfully consolidated a diverse array of lab instruments, all using wildly different communication protocols, and created a consistent, clean interface to present to the system’s PLCs to control.

To enable PLC control of the lab equipment, DMC created a custom REST API interface backed by individual custom applications to talk to each device’s specific protocol. Setup and configuration parameters for each device were stored in a database and maintained from an Ignition user interface, which provided users with the ability to rapidly and flexibly install, commission, or modify lab devices throughout the system.

This setup provided a robust two-way communication architecture between PLCs, Ignition, and devices/drivers. Custom handshaking allowed reliable tracking of command progress and status, as well as multiple levels of equipment feedback and process data. The interface included considerations for parameterized commands to allow highly custom, on-the-fly changes to operating parameters of the lab equipment at runtime. This entire implementation leveraged Ignition UDTs and script libraries to make maintenance of this code seamless.

Operator Interface and Control

Each work cell within the lab has a stationary touchscreen HMI which displayed an Ignition Vision client. DMC worked with its internal UX/UI designer and Ginkgo’s internal UX team to craft an HMI that allowed operators to interact with the work cells quickly and intuitively, and to visualize and control the overall line.

Each work cell within the lab adhered to a strict PackML state machine architecture. Ignition then provided a consistent interface between all work cells to manage state and functionality.

Rapid Development

With the Ignition platform, DMC developed a highly advanced SCADA application and built tools to automate time-consuming manual development processes. All of this occurred with constantly changing system requirements. The key features of Ignition that enabled success are:

• Seamless deployment of UI updates across many clients
• Reusable and organized data types, templates, and script libraries
• Flexible scripting and automated development tools

Result

This automated lab went from conception to bio-validation in nine months, so it was important to have a SCADA platform that could scale quickly. Commissioning occurred seven days per week, so redundant Ignition servers allowed the facility to achieve nearly 100% SCADA uptime. Ignition’s responsibilities included connecting more than 40 work cells (each with their own PLC, lab equipment, robot, and vision inspection systems), providing real-time dashboards for production data, connecting PLCs to lab equipment with a consistent interface, and reliably storing event data for the lab.