Recently, the Internet went into high alert over a newly discovered vulnerability in a logging library called Log4j that affected many systems that use Java. Inductive Automation was fortunate to not be affected by this vulnerability, but the truth is that we were lucky enough to dodge a bullet. Many software vendors who weren’t as lucky had to scramble to come up with a quick fix.
While we were relieved that Ignition was not affected, it did give us a moment to reflect on what could have been. Like many software vendors, we do our best to make sure our platforms are as secure as possible but, to be honest, there are times when events like Log4j can happen despite our best efforts. When they do happen, it is not only a vendor’s duty to focus on security, it’s the end user’s duty as well.
In this blog post, we’d like to explain the Log4j vulnerability and how we responded to the security announcement. We’ll also highlight the actions we took to address the issue, both with Ignition and our internal systems, and how this event illuminates how important it is for end users to employ a strong security strategy.
What is Log4j and How Did We Approach It?
On December 9, the National Institute of Standards and Technology (NIST) put out a notice on their National Vulnerability Database (NVD) about a zero-day vulnerability regarding Log4j, (CVE-2021-44228). Log4j is a common Java logging library used in many of today’s software platforms. The vulnerability notice specifically calls out a remote execution attack that exploits the Log4j library and carries a severity score of 10. Basically, the Log4j vulnerability would have allowed attackers to gain access and control of connected devices.
A Software Development Perspective
The moment we were aware of the issue, our Software Engineering team verified that Ignition was unaffected by this vulnerability. In this case, verification was fairly straightforward since neither of the current LTS versions of Ignition (7.9 and 8.1) use Log4j for their logging; we use a different library called “Logback.” We also analyzed our transitive dependencies to be sure that we didn’t incidentally package Log4j with Ignition as a dependency of any other library that we use, and were pleased to find that we did not. It’s also important to note that we use an automatic dependency analysis tool which will flag any dependency (direct or transitive) of Ignition that has a published CVE disclosure so that we can take countermeasures as soon as possible.
Older versions of Ignition (7.8 and older) did use the Log4j library, but they used an older version (1.2) that was not affected by this vulnerability.
The IT/Cyber Security Perspective
Our Cyber Security team launched a parallel investigation across all company assets to uncover any internal instances of Log4j. Using our broad stack of security tools, we searched Github for any versions in use, and used network/vulnerability scanners to identify any potentially affected services. From there, we further dug in with host-based software inventory tools to find hidden dependencies embedded within otherwise benign software packages. After concluding our investigation, we leveraged prevention rules in our network and web application firewalls to thwart the mass scanning that began taking place.
Immediately after our security verification and remediation steps were put in place, we quickly posted a technical advisory on our website, sent notifications to our mailing list, and included the notification in our weekly News Feed.
It is important for us to be diligent about security and to be transparent with our customers of any issues that can be incredibly damaging to their business. Luckily, we were able to assure Ignition users around the world that the Log4j vulnerability is not an issue and did not require a patch for the Ignition platform.
What Would Happen if the Story Was Different?
For most people, the story ends there and all is well. But security never rests and there will always be a vulnerability lurking in the shadows. Given that we lucked out in dodging this particular issue, it’s worth stepping back and asking an important question: What if Log4j was heavily embedded in Ignition?
It really brings us back to our main message on security, that it is important to take advantage of the security tools that are available. While the software vendor can provide a high level of security, there are many things that are in control of the end user, such as networks, devices, terminals, and making sure software updates are regularly being made. The last point is something we want to stress.
Due to our quick release trains, we likely could have had a same-day update out on the website in the form of a nightly release. As great as a same-day patch may be, releasing a patch doesn’t protect customers who are unable to install that patch, so this begs the question: Are customers in a position to deploy an emergency Ignition patch for a heavily exploited, in-the-wild vulnerability?
Working Together to Make Security Better
To make sure our customers are quickly informed of high-priority issues, we regularly post critical announcements to our Support Portal on our website. Customers can find up-to-date information on any security issues, available updates, and critical patches. In addition, customers can sign up for our Ignition Support and Product Update mailing list. This list is only used to inform customers of critical issues that may impact their Ignition systems. These notifications are sent out immediately once a technical advisory has been posted.
Knowing that there is a security issue and that a fix is available is one way to strengthen your security strategy. It’s also important to make sure that these fixes are applied on a regular basis.
The Patch Gap
While we as an organization can strive to provide excellence in security, we also must encourage and educate our users to take on security measures themselves. For example, while we are able to put out patches quickly, some of our customers may find themselves in a “patch gap.”
The “patch gap” is the time between when a vulnerability is discovered to the time the patch is deployed. Our responsibility ends at publishing the patch in a timely manner — it’s on the customer to deploy it to all of their systems. Being in a position to quickly respond to situations like these is of critical importance.
One way to achieve this is to maintain a regular patch cadence, which ensures that patches are applied in a timely manner for maximum protection. In addition, having a regular patch cadence helps customers to avoid a situation where they’re forced to make a major upgrade, which can be rather risky. Frequently updating Ignition installs will smooth out any potential issues and guarantee that an organization will be in the best possible position to respond to emergency patches.
This is obviously easier said than done and there will be cases where patching in a timely manner may not be feasible. This is where a defense-in-depth approach shines, as compensating controls can buy you valuable time and ensure that you stay safe while waiting to patch.
Application Control
When it comes to stopping emerging threats and zero-day vulnerabilities, two of the most important controls that can be deployed are strict outbound/egress filtering on your network and strict application control on your servers. Both of these controls rely on an allowlisting approach where only pre-approved network connections can leave your network and only pre-approved software can run on your servers.
In the case of Log4j, many of the initial exploits relied on the affected server reaching out to attacker-controlled infrastructure in order to pull down a malicious payload. If your firewall only allowed requisite outbound connections, this initial lookup would fail and the malicious payload would never be downloaded to your server. If the attacker somehow bypassed your network filtering, application control/allowlisting on the server could prevent the payload from executing. Additionally, strong network segmentation from your firewall would reduce the blast radius of any compromise and prevent the attacker from pivoting deeper into your network.
Finally, ensuring that the user running the Ignition Java process has as few privileges as possible to do their job would reduce the capabilities of what the attacker can do once the process is breached. At a minimum, the user should not run as a system or root user or a user with sudo, if at all possible.
Having a solid security strategy is essential and it should always be evolving as times change. While the Ignition platform was spared from the Log4j vulnerability, it gave us the opportunity to visit a topic that is important to us. Vulnerabilities and attacks will always be on the horizon, and we will continue to address them as they arrive. Ultimately, there is no better way to approach security than to have a solid security strategy.
Key Takeaways:
- Sign up for the Ignition Support and Product Update list to stay up-to-date.
- Patch your Ignition installs regularly to stay agile.
- Deploy compensating security controls as a safety net.
