Large Data Center, Manageable System

Many large organizations rely on data centers to backup and protect their most sensitive information. To accomplish this, data centers must manage their own operational data precisely. As a data center grows more complex, how does it ensure that no detail falls through the cracks and causes a downtime disaster?

That was the challenge facing American Internet Services (AIS), an enterprise-class data center services company which operates five facilities in San Diego, Calif. The largest of the five facilities is the 80,000 square foot Lightwave Data Center. Like many large data centers, Lightwave uses several critical systems for building management, network management, data center monitoring and power distribution. AIS needed to streamline these separate systems into a single monitoring system.

After a number of vendors failed to show AIS a system versatile enough for its needs, the company turned to Brown Engineers, a mechanical and electrical consulting firm it had worked with on other projects. Brown Engineers suggested a modular software platform called Ignition by Inductive Automation®.

“We had been working with Inductive Automation for a while on some of the water / waste water, plant specific and even SCADA-type systems,” said Dee Brown, PE, principal of Brown Engineers. “So we thought: why not use a software system that is already built for drawing any screen you can imagine, that has an alerting engine built into it, and has OPC drivers to talk to anything you want? Why not use that as an overarching monitoring system that can talk to everything? So that’s what we presented to them and that’s how this project really got started.”

Practical Downtime Prevention

Brown Engineers specializes in reducing downtime for mission-critical systems, and the Lightwave Data Center presented a complex challenge. Its electrical system has several layers of redundancy at the Uninterruptable Power Supply (UPS) level, making it much more complicated than a system in a noncritical building. Almost every server in the building has at least two power supplies, more than one utility service, multiple Internet connectivity services and more. One of Brown Engineers’ critical tasks was working on UPS system monitoring. In the event of a power interruption that causes the servers to go down, the UPS acts as a super fast but short duration bridge while the emergency standby generators get up to speed and then take back the load.

In a system this large and complex, any number of downtime situations could occur and when they do, corrective action must be started in no more than a few minutes.

“There’s no margin for error in these systems,” said Sam Vandiver, PE, electrical engineer for Brown Engineers. “Those servers have to be up and running, and a mistake as simple as plugging too many of those servers into the wrong power strip can overload the circuit and no amount of emergency generators or UPS’s is going to change that,” he said.

Before Ignition was installed at Lightwave, in order to check if any circuits were overloaded, staff would have to walk to each of the many power distribution units spread across the building and read a small display on each one. Now, Ignition tracks the information from all of the power distribution units, reading power usage on a circuit-by-circuit basis. Ignition can generate a report of the most overloaded or close-to-overloaded circuits in the entire data center. Overloaded circuits and tripped circuit breakers can now be easily detected and fixed before they cause an outage.
 

Bridging the Gap Between IT and Facilities

“We’ve found that in the data center world, there are usually a lot of IT guys who understand IT systems and servers but who may not be keeping an eye on the UPS’s or generators, incoming utility power and the kilowatt information. So when a utility fails, generators don’t start or the transfer switch doesn’t switch over, it can be difficult for the facility guys and the IT guys to tell what went wrong and keep those types of things from happening again,” said Brown.

With Ignition gathering the information from Lightwave’s various control systems, both the IT and facility workers can monitor what’s going on. Ignition now allows the facility staff to look at all their control systems data from a large-format monitor on their conference room wall. They check regularly for alarms and any abnormalities, and use this information for planning. Also, some of the IT staff use Ignition for monitoring purposes through their network operations center.

By bringing all of this information together, Ignition also allows a data center to examine its circuits and see if a customer is paying for all of the amps they are actually using. This gives data centers a clear way to measure production against revenue.
 

Making a New Framework

Brown Engineers performed the initial implementation of the Lightwave project in 2012 using Ignition v7.5. After Ignition v7.6 was released in May 2013, Brown’s team did a second implementation that utilized the software’s updated features. Their goal was to find a new framework for alarming, templating and visual aspects of the system.

They originally implemented the project using the standard one-tag-at-a-time method, but on the second implementation they utilized Ignition’s user defined types (UDTs) to build reusable templates based on the common data for all asset types in the facility. To make templates for another asset class, they can simply add the data that is different about that particular class while keeping the rest. When change the master template, it automatically updates all of its instances. They set up color coding for the alarms (green for OK, red for bad, orange for unknown, and gray for disabled) at the asset level so they only had to define colors once for any type of asset. By disabling alarms that were dependent on other alarms, they reduced the number of false alarms.

Using UDTs, Brown Engineers was able to standardize the process of setting up and managing tags. They also stored the tags by asset type rather than by location as they had previously been, which makes it much less time-consuming when changes need to be made.

With the new asset groupings, the team configured a simplified dashboard screen that gives an overview of all of the assets in the facility. This helps operators spot problem areas more quickly. Rather than scrolling through hundreds of buttons to check for red alerts, operators can now look at one screen with about a dozen buttons which each represent a different asset type. The operator can then click down into any asset type that is colored red to see which specific asset is having a problem. The exact same dashboard can also be viewed on the Ignition Mobile Module.

Now, the Brown Engineers team says their eventual goal is to get the entire data center onto one screen – an objective that would have been unthinkable without Ignition.

“Being with the Inductive Automation family for about the last four years has really been incredible,” Brown said onstage at the 2013 Ignition Community Conference. “Something I really appreciate about the tool,” he said, “is that it is more than just a SCADA and HMI product. Depending on what your customer or client needs, you can tailor it a lot of different ways, which is fantastic. I think that is really broadening what we’re going to be doing with the product.”

Brown Engineers, LLC, is an electrical, mechanical and automation engineering firm based in Little Rock, Ark. The firm was first inducted into the Professional Services Management Journal’s Circle of Excellence in 2010 in recognition of its sound business practices.
www.brownengineers.net