MSSPs Help Organizations Through CrowdStrike IT Outage

Organizations around the world were grappling with massive IT outages on July 19 caused by a routine software update to CrowdStrike technology that resulted in the Blue Screen of Death for users. CrowdStrike, in its company blog, said it is aware of reports of crashes on Windows hosts related to the Falcon Sensor and had issued a workaround.

Blackswan Cybersecurity, one of many MSSPs dealing with the aftermath of the incident, experienced an increase in calls for support related to it, according to Mike Saylor, CEO, who provided insight to MSSP Alert in an email. Those calls came in even though Blackswan Cybersecurity is not a CrowdStrike partner.

Saylor said that the calls for support involved users looking to understand the situation and how or if it impacts them. The company also received calls from current clients looking for guidance and support for addressing their environment. Saylor said there are a couple of different impacts to MSSPs dealing with this crisis.

Read our full coverage across our entire network:

MSSPs and the CrowdStrike IT Outage

“If an MSSP is monitoring a client environment that includes CrowdStrike, they will likely observe outages, errors and alerts coming from the impacted end points,” Saylor said. “In this case, the MSSP is mostly focused on communicating what they see and determining if it is related to the CrowdStrike incident or something else, then updating their tickets accordingly — mostly assess, report, document.”

MSSPs that partner with CrowdStrike or co-manage the solution will likely be even more involved in assisting with response and remediation activities, according to Saylor.

“Note that in most cases an MSSP does not typically have additional resources (people) to deploy for incident response or remediation assistance,” he said. “This situation will require some level of hands at the keyboard for remediation support of each physical device impacted.”

Saylor added that his company is supporting others in the service provider community as they work through the crisis.

“Blackswan is actively supporting other MSSPs, MSPs and current clients, providing remediation guidance and communications consolidated from various sources including CrowdStrike,” he said.

Trusted Internet CEO’s Advice for MSSPs

As bugs in software updates are not uncommon, and despite the widespread IT outages involving air travel, banks, retail or more, Jeff Stutzman, CEO of MSSP Trusted Internet, cautioned against an overreaction.

“While this is a very bad one, I’d hate to throw the baby out with the bath water,” he said. “CrowdStrike is an amazing tool, hence the market share. It’s unfortunate that this has occurred, and there is a manual component to the workaround/restoration.”

Stutzman advised MSSPs to be on alert — “border protection is a must until clients know they’re protected post-fix.”

“Even if users decide to continue with CrowdStrike (I’m sure most will), there will be a gap,” he said. “Clients might want to hold on restoring CrowdStrike if they want to wait for stability. MSSPs should be prepared to offer an alternative.”

Stutzman emphasized that MSSPs using CrowdStrike should test for stability before deploying.

“This should be the norm, but in smaller shops it may not be,” he said. “This is, in my mind, one of the most important issues as CrowdStrike MSSPs move to restoration.”

Huntress Helps Spread the Word

John Hammond, principal security researcher at Huntress, advised that MSSPs should be concerned with their own clients and customers, taking inventory of who is impacted and being able to help in any way possible. He believes recovery will be a slow effort.

“Huntress is working to help spread the word and get the messaging out for mitigation. I am helping shine the spotlight on this issue and explain the intricacies," he said. "Anecdotally, I am aware of multiple organizations working on this, with tens or hundreds of thousands of affected endpoints.”

A Lesson About Rigorous Testing: Acronis

Kevin Reed, chief information security officer for Acronis, a cybersecurity vendor that partners with MSSPs, said that the outages may cause MSSPs to lose visibility over their customers’ systems. That could leave them unable to respond to attacks or even see them with their endpoint detection and response (EDR) capabilities disabled.

Reed said that Acronis systems are not affected and are fully operational. But for any Acronis customers who may have been affected, Reed recommends they restore their systems using boot media to the last known good state.

“This incident highlights the importance of rigorous testing and staged updates for EDR agents,” he said. “Normally, testing is done with every release and can take days to weeks, depending on the size of the update or changes.”

How MSSPs Can Protect Against Future Outages

Neatsun Ziv, CEO and co-founder of OX Security, a provider of software to help secure the application development process, said a lesson that can be taken from the outage is the importance of choosing a vendor who can protect your server as a distinct and valuable portion of the network, separate from endpoints.

“Moving forward, a system of agentless updates as opposed to automatically updating agents on the endpoint servers could help alleviate issues like this,” he said. “The associated convenience of automating these updates creates more potential for outages and security incidents, and this kind of event could happen to any vendor that uses agent technology."

Martin Jartelius, chief security officer at Outpost24, a cybersecurity vendor focused on minimizing risks, said for any MSSP basing their services on the solutions that introduced the issues, a fair bit of work with customers will remain.

“Depending on their agreements, who assumes responsibility for costs of remediation due to the offered services remain a potential long term legal issue,” he said. “So, both MSSPs directly impacted as well as MSSPs who help with operations the issue can have caused substantial costs and efforts."

Intezer Triages Alerts for MSSP Partners

Itai Tevet, CEO for cybersecurity vendor Intezer, told MSSP Alert that is still ingesting and actively triaging alerts from CrowdStrike for its MSSP partners. Intezer is ensuring that when availability is restored to any impacted machines they will have the information they need to respond in the event of an incident.

“We’re here to support our MSSP customers, ensuring that they aren’t overloaded with security alerts after impacted Windows machines get back online and any critical threats get escalated for fast response,” Tevet said. “It is still a good idea to assure your clients that your security platforms remain fully operational and unaffected, communicating promptly to let them know of any steps you’re taking to manage the situation.”