A massive IT outage has caused multiple services to be taken offline, while planes and trains were grounded as a result.
This story first appeared at SC Media. See the original story here.
Potentially linked to an update from CrowdStrike to Microsoft users, the issue is apparently due to a misconfigured update which is causing users globally to hit a "blue screen of death" (BSOD).
In a Reddit update, a post — apparently by a member of the CrowdStrike team — said they are aware of "widespread reports of BSODs on Windows hosts, occurring on multiple sensor versions. Investigating cause."
Read our full coverage across the CRA network:
- MSSPs Help Organizations Through CrowdStrike IT Outage
- MSPs Come Together to Hasten CrowdStrike Outage Remediation
- Analyzing the CrowdStrike Incident and Its Ripple Effects
- Seven tips that offer short-term and long-term fixes following the CrowdStrike outage
- CrowdStrike confirms faulty update is tied to massive global IT outage: ‘Fix has been deployed’
- Security pros brace for manual system-by-system fix to CrowdStrike outage
- What the CrowdStrike update outage means for cybersecurity
- CrowdStrike Update Causes Global Outages: Analysis
Defect in single content update
CrowdStrike CEO George Kurtz issued a statement saying the company is actively working with impacted customers, and the issue was caused by a defect found in a single content update for Windows hosts.
Confirming the outage was not “a security incident or cyberattack,” Kurtz said: "The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.
“We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers."
Reports of crashes
The disruptive file was identified as an update for CrowdStrike's Falcon sensor, which was released on July 9. A workaround was published, with CrowdStrike saying it is "aware of reports of crashes on Windows hosts related to the Falcon sensor."
CrowdStrike has confirmed that it is no longer pushing the update, “so you only have to fix the machines that were already stuck in a BSOD loop: anything that isn't impacted now shouldn't be impacted.”
Writing on X, CrowdStrike's chief threat hunter Brody Nisbet offered a workaround, recommending users:
1. Boot Windows into Safe Mode or WRE.
2. Go to C:WindowsSystem32driversCrowdStrike
3. Locate and delete file matching "C-00000291*.sys"
4. Boot normally.
Cybersecurity researcher Kevin Beaumont wrote on X that the global IT outage "is CrowdStrike as cause, not Microsoft" and that two different outages were linked together. While the Microsoft one was solved a while ago, it may be the CrowdStrike update that is causing the issue.
Infinite loop
Ilkka Turunen, field CTO at Sonatype said in an email to SC UK that the update caused a BSOD loop on any Windows machine, essentially making it boot and crash on an infinite loop.
“Making it worse is the fact that there are a significant number of Windows machines that the update was auto-installed on overnight,” he said. “There are workarounds that customers of theirs will apply, but it seems to be very manual.
"It’s definitely a supply chain style incident — what it shows is that one popular vendor botching an update can have a huge impact on its customers and how far a single well-orchestrated update can spread in a single night."
