More Zonkyness

This is a bit of rant poking fun at the sheer quantity of misinformation about CrowdStrike’s little issue yesterday (to clarify when this post was written – more information will come out).

Microsoft

Some of the earliest symptoms of the issue were some Microsoft services having issues. Oddly enough I wasn’t using many of those yesterday (I usually do) except for Teams which didn’t seem to suffer … at least not as much.

It appears that Microsoft may run CrowdStrike Falcon on at least some of their servers (although the jury is still out on this one – some are saying it was an independent outage). Despite Microsoft having their own security tools (Defender), this isn’t quite as unlikely as it may seem – particularly safety conscious organisations may well run two of more anti-malware products.

And CrowdStrike is more mature than Defender at least in the fancy “behavioural analytics” area.

The Internet

… wasn’t broken at all. Many services were broken true enough, but probably more were working just as well as normal. Microsoft’s platforms are very widely used, and CrowdStrike is a big name in cybersecurity, so it is hardly surprising that there was so much disruption.

But to say this broke the Internet is a bit of an exaggeration. Kind of what you would expect from mainstream media.

Who Are CrowdStrike?

Not surprisingly, many people just haven’t heard this name before. It is very widely known in the cybersecurity community with a wide variety of security focused services, including top-flight anti-malware products.

But they don’t sell to individuals so they are not well known amongst the general community.

The product at the centre of all this is CrowdStrike Falcon, an anti-malware agent that goes a bit beyond “anti-virus” in that it attempts to go beyond blocking known viruses and attempts to block behaviours known to be malicious.

As such, it receives very frequent updates – up to every hour (although probably many hours) which puts this sort of catastrophic failure at a rate of somewhere in the order of 0.001%.

What Went Wrong?

This starts to get a bit technical …

Some of this was informed by CrowdStrike’s update; some by educated (I work in this field although I’m not familiar with CrowdStrike’s product) common sense.

First of all, this was not a kernel driver update (although the relevant filename made it appear so) but a content update. As previously mentioned, these are sent out very frequently. The content update triggered a bug in the kernel driver and caused a “blue screen of death“. This would repeat after every reboot until the relevant update was removed or updated (the crash doesn’t occur immediately which sometimes allows the agent to download a fixed update).

Secondly this update was tested before being released (do you really believe that an approximately 0.001% failure rate is achieved without testing?), but something went wrong with the testing process. We don’t know what, and CrowdStrike don’t either. Yet.

Why Was It So Widespread?

Simply because although generally unknown to the general public, CrowdStrike Falcon is generally regarded as an excellent security product and is very widely used. Perhaps more widely used than previously suspected.

But the whole Internet? Clearly not, but it’s in the mainstream’s media to be a bit ‘click-baity’ in their reports.