When the UK awoke on the morning of Friday the 19th of July, a stir of a major outage at Microsoft was brewing. News reports had stated that “Microsoft was down”, and this was true. An outage in the North America Microsoft Cloud region affected many Microsoft 365 services, however, as we later find out, this was a result of a failed cyber security attack Microsoft had thwarted. Something additionally as serious was also happening – people’s computers were “blue screening” and would not start correctly…

As the morning went on and Microsoft 365 services largely came back online, it had become obvious through the error messages being presented on the blue screen of death (BSOD) that some people were encountering when trying to use their computers, a component of CrowdStrike, a popular cyber threat detection and response software, was causing this major issue.

An estimated 8.5 million computers crashed as a result of a CrowdStrike software update, so how did this happen and what have we learned?

CrowdStrike’s hidden bug

CrowdStrike, like many other security vendors, utilise a special area of the Windows operating system called the kernel. The kernel is useful for security software as it provides broad access to the device and a powerful ability to block certain operations such as writing and modifying files that can’t be circumvented by a virus or attacker. But with this level of access, comes great responsibility as the kernel is unforgiving when it comes to software bugs. Any bugs generally cause the whole system to crash, and these types of crashes prevent startup and this is why it took so long to recover.

Having reviewed the root cause analysis of the CrowdStrike incident, it reads like a series of unfortunate events. A new feature was introduced back in February of 2024 which included the software bug which sat dormant. Testing plans to identify these types of bugs effectively masked the bugs’ true behaviour. Confidence was gained, and a full deployment for a new update which was expected to not be of any particular impact, in fact, had the right parameters in which the bug would manifest.

Where is the real problem – CrowdStrike or Windows?

There have been two lines of thinking; was the issue only with CrowdStrike and so we need to rely on CrowdStrike’s own processes to improve, or, given there are many other security products with this level of access to the Windows Operating System, could this happen again with someone else?

This, as ever, is a complex answer. Looking at CrowdStrike first, there was a process and testing failure – accepting that bugs can occur, and serious bugs can also occur, deploying updates to everyone, all in one go, at the same time, even if it is to address a security threat, seemed unwise. CrowdStrike is now adding more controls to users to allow granularity on when these are now received.

Should security products have kernel access to Windows if it could cause a huge impact if security vendors get something wrong? If we take a look at Apple’s MacOS, they do not permit this level of access to anyone, only themselves. For security vendors, MacOS provides special abilities, but they do not operate in the MacOS kernel. This approach however can limit the ability of security vendors to innovate their security products with any new feature they may come up with to thwart attacks, potentially requiring the functionality to be provided by MacOS first.

If we take a look back to Windows Vista days, the successor to Windows XP, this was the path Microsoft wanted to take with no access to the kernel. But they faced opposition from security vendors and raised eyebrows from regulators as it was perceived that Microsoft was curtailing other vendors’ ability to carry out effective security services on the Windows operating system.
As we look forward, and given the CrowdStrike event, there is potential now for this to move forward again.

What can you do to protect yourself?

Diversify

Having multiple security products would not have helped in the CrowdStrike scenario as it would have taken out Windows machines regardless of whether you had two security products or 50 installed on your machine.

However, you should not rely on a single security product to address all of your security needs. Layering security products is always good practice, but you will always be likely to have only one endpoint detection and response (EDR) product on your Windows desktops.

Business Continuity Planning

For those using CrowdStrike on July 19th, business continuity plans were likely dusted off, and probably did not include this particular scenario. But planning is important for such events and it is helpful to ask some key questions on how you would respond if this happens again. How would you fair in the event of a non-threat related disruption to your business if we were to encounter something like CrowdStrike again? For scenario planning, if you needed physical access to every desktop device, how would this be achieved? If a vendor had no fix to a breaking software update, how would you recover operating systems?

Vendor Due Diligence

What we have learnt, is that you should check with your security vendor the release process for new updates, including those deemed so important they need to be distributed quickly. A few machines being affected are better than all of your computers.

For Microsoft Defender users, you benefit from a gradual rollout programme with capabilities to delay updates beyond this if required.

In summary, for those using CrowdStrike as their main security product, it was an incredibly difficult scenario to anticipate. There was unlikely anything they could have done to prevent it, and anything they could have done to recover quickly from the issue. Their options are to stick with CrowdStrike, accepting they’ve improved their processes and leverage the new capabilities to control how they receive updates, or, to move to an alternative security product.