Powered by MOMENTUM MEDIA
cyber daily logo
Breaking news and updates daily. Subscribe to our Newsletter

CrowdStrike reveals details of incident behind global PC outage

The cyber security firm says a misconfigured Rapid Response Content update to its Falcon platform caused PCs to crash worldwide.

user icon David Hollingworth
Wed, 24 Jul 2024
CrowdStrike reveals details of incident behind global PC outage
expand image

Cyber security company CrowdStrike has released an update on what caused a global wave of Windows PCs to crash to the notorious blue screen of death.

As we know, the issue was an update sent out to the company’s Falcon endpoint detection and response platform – specifically, a single misconfigured Rapid Response Content update sent out to the Falcon Sensor on those platforms.

The patch impacted version 7.11 and above of the sensor and was released on 19 July at 04:09 UTC. Every machine running that sensor version and online until 05:27 UTC – when the update was reverted – was affected.

============
============

Unlike a Sensor Content release, which a customer has control over in terms of rolling it out onto their fleet of machines, Rapid Response Content is deployed automatically to effectively track and identify new threats.

“This capability is used by threat detection engineers to gather telemetry, identify indicators of adversary behaviour and perform detections and preventions,” CrowdStrike said in a 24 July update to its Remediation and Guidance Hub for the incident.

“Rapid Response Content is behavioural heuristics, separate and distinct from CrowdStrike’s on-sensor AI prevention and detection capabilities.”

Rapid Response Content is released as “Template Instances”, which map to “specific behaviours for the sensor to observe, detect or prevent. Template Instances have a set of fields that can be configured to match the desired behaviour.”

In this case, it was an InterProcessCommunication ( or IPC) Template Type, which had been first tested and validated on 5 March and rolled out the same day via Channel File 291.

“Subsequently, three additional IPC Template Instances were deployed between April 8, 2024 and April 24, 2024,” CrowdStrike said.

“These Template Instances performed as expected in production.”

But on 19 July, “two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data.”

“Based on the testing performed before the initial deployment of the Template Type (on March 05, 2024), trust in the checks performed in the Content Validator, and previous successful IPC Template Instance deployments, these instances were deployed into production,” CrowdStrike said.

When received and loaded by the Falcon Sensor, the “problematic content in Channel File 291 resulted in an out-of-bounds memory read triggering an exception”.

“This unexpected exception could not be gracefully handled, resulting in a Windows operating system crash (BSOD),” CrowdStrike said.

In response to the incident, which affected more than 8 million Windows 10 PCs, CrowdStrike has said it will improve its Rapid Response Content testing and add further validation checks to specifically guard against such content from deploying.

In addition, the company will improve the error handling that already exists in its deployments, and – perhaps most key – stagger the deployment of Rapid Response Content and give customers control of the process “by allowing granular selection of when and where these updates are deployed”.

“In addition to this preliminary Post Incident Review, CrowdStrike is committed to publicly releasing the full Root Cause Analysis once the investigation is complete,” CrowdStrike said.

David Hollingworth

David Hollingworth

David Hollingworth has been writing about technology for over 20 years, and has worked for a range of print and online titles in his career. He is enjoying getting to grips with cyber security, especially when it lets him talk about Lego.

newsletter
cyber daily subscribe
Be the first to hear the latest developments in the cyber industry.