One of the major challenges for companies recovering from the Crowdstrike IT outage is that each endpoint must be remediated individually by a technician on site. Endpoints typically cannot be remediated remotely, and that's made the recovery process tedious and time consuming. MSP and MSSP business models call for remote and automated remediations.
But cyber disaster recovery firm Fenix24 says it has issued recovery scripts that can remediate many of the end points remotely. The company created the scripts to assist companies attempting to restore IT capabilities in the aftermath of the CrowdStrike update that caused the Blue Screen of Death for an estimated 8.5 million PCs and servers.
“Since we developed these scripts on Friday, there have been hundreds of companies downloading them, and our understanding is that the automatic scripts have taken on 95% of the workload without hands on the keyboard going to each individual system,” Heath Renfrow, co-founder of Fenix24 told MSSP Alert on Monday. “The one challenge is BitLocker, of course, and there were no capabilities around that even with our scripts, except for some manual work. However, CrowdStrike has a new ability through the Falcon platform to help try to resolve this and success has been good thus far.”
Fenix24 said the scripts are free of charge and publicly available. Access the scripts here.
Read MSSP Alert’s comprehensive coverage of the event.
Manual vs. Remote vs. Automated Fixes
Fenix24 said that CrowdStrike’s fix requires manual intervention for computers already in blue screen mode — a manual process that is slowing the recovery process for those affected. Fenix24 said its automated fix will be critical to restoring operations at-scale not only for desktops but also for virtual servers that IT managers would otherwise need to patch manually.
The scripts were created for Windows and VMware using public information and the Fenix24 team's internal expertise. The Windows scripts force the reboot of machines into safe mode and then remove the problematic file.
If the drive is secured with BitLocker, users will need to enter the BitLocker key manually and then proceed to safe mode, Fenix24 said. The VMware scripts use a working server to detach the virtual disk, mount it, remove the problematic file, dismount it, reattach it to the problem VM and then reboot it.
Most Scripts Automate Recovery
Renfrow said the company's scripts do not need to be run manually on each computer. They will automatically apply to:
- Azure. The script will mount the virtual hard disks (VHD) to a helper virtual machine (VM), then remediate the file. The IT admin will need to press enter after each VM is finished to move onto the next.
- AWS. Documentation for AWS has been provided to quickly roll back all VMs.
- ESXi. After a helper VM is created, the script will automatically mount each Windows VMDK (virtual machine disk) to the helper machine, delete the file, and then power the VM back up. This works for both bootable and blue-screened VMs.
- Hyper-V. The script will mount the VHD to the Hyper-V host, remove the file, and then power the VM back up. This works for both bootable and blue-screened VMs.
- GPO. The removal script will automatically be pushed out to all bootable machines or devices in safe mode with networking. GPO is a collection of group policy settings that defines what a system will look like and how it will behave for a defined group of users.
- PDQ (parallel data query for bootable devices). The removal script will automatically be pushed out to all bootable machines or devices in safe mode with networking.
Manual Recovery Required in Some Cases
Fenix24 said the following solutions will require touching the individual machine to either boot into safe mode or PXE.
- Offline USB Copy
- System Center Configuration Manager (SCCM)
- Microsoft Deployment Toolkit (MDT) / Preboot Execution Environment (PXE still working on this script)
- PDQ (for devices that are blue screened)
“The manual work would be done by internal IT, possibly MSSPs, or companies like Fenix24 that have the manpower to send engineers onsite,” Renfrow said.
Fenix24 works with MSSPs but said that it has not had any interactions over this issue thus far.
Microsoft has released an updated recovery tool with two repair options to help IT admins expedite the repair process. The signed Microsoft Recovery Tool can be found in the Microsoft Download Center: https://go.microsoft.com/fwlink/?linkid=2280386.
