Infrastructure Is A Disaster – The Lessons From Log4j Vulnerability

New day. New threat. New technology to combat said threat. Sound familiar?

The threat landscape is continually evolving and getting more sophisticated, and, in an attempt to keep up, many organizations are quick to adopt the latest buzz-worthy product. This is a recipe for disaster. When the security responsibility is distributed across cloud teams, application owners and various silos, it inevitably means inconsistent security – with different IT assets having different security controls and configurations.

Like many technology leaders, we quickly responded to the vulnerability in Log4Shell. The Log4Shell vulnerability is a remote code execution, or RCE, which allows any bad actor behind a computer to run code on a server. This is one of the worst things that can happen, even causing CISA Director Jen Easterly to note it as one of the most serious exploits.

Proactive monitoring by our global operations team, which is what we call the SOC — raised the issue to our key stakeholders, which kicked off a channel for collaborative internal work. When the vulnerability was disclosed, we worked through the night attempting to validate and then provided information describing the nature and severity of the exploit itself. Within a day of discovery, we gathered a list of potential points of compromise: internal, collectors, open-source, and determined that at no time was Sumo Logic exploited.

Our system security team conducted pen testing and then disseminated information internally, that identified where the library is in use and then updated those versions. For Sumo Logic, the collector is not designed such that it invokes anything that it is receiving on the internet. The logging that we use for Log4j on the collector is for internal audit purposes. This helps us understand what organizations are using, where the traffic is coming from, and then work on blocking its impact.

Security has to be managed from a defense-in-depth perspective, which means that just because we don't think we're vulnerable doesn't mean we won't update. For this specific vulnerability, new ways to exploit it were found almost immediately that rendered previous protections—like having a recent JDK—considered to be good enough, are not good enough anymore. Things change and one security vulnerability can leverage the next, so it is imperative that we respond as aggressively as possible. The Sumo Logic team realized this within 24 hours of the discovery of the exploit and began developing a proactive update.

In spite of the ubiquitousness of this issue on the internet at large, at Sumo Logic, we were able to prevent it. We have built-in compliance measures, and have done all this work proactively, such that even though this vulnerability impacted some, you're not vulnerable.

Don’t wait for an issue like this to ruin your weekend. Proactive monitoring is key to immediately identify and understand what is happening so you can stop outbound traffic and isolate it. At Sumo Logic, we continue to monitor for any change in the nature of the vulnerability, methods of compromise, and methods of bypassing detection. We have searches set up to identify how/if it was used prior and how people are attempting to leverage this going forward.

Here is a common search that you can use to find current versions of the exploit that bad actors may be attempting to abuse, which may help you identify cases:

We have much more insight to share. read the Log4Shell CVE-2021-44228 Situational Awareness Brief for a deeper technical dive and context around the searches.

George Gerchow is Chief Security Officer (CSO) at Sumo Logic. Read more Sumo Logic guest blogs here. Regularly contributed guest blogs are part of MSSP Alert’s sponsorship program.