The Importance of Robust IT Systems in Light of Recent Global Outages

The clock strikes around 6 am GMT on Friday, and the world comes to a grinding halt due to an unprecedented IT outage that single-handedly disrupted airlines, stopped trains in their tracks, and delayed critical healthcare services.

So, the real question is: why did this happen?

The blame seems to be being placed at the door of CrowdStrike, a cybersecurity firm that offers cloud-based online security solutions to tech giants like Amazon Web Services (AWS), Microsoft, and some of the world’s leading banks and airlines.

The issue arose when a seemingly routine piece of antivirus software proved defective. As a result, it became the root cause of widespread chaos, as major companies and services worldwide were brought to a blue-screened halt.

CrowdStrike was quick to react to this news by posting a reassuring blog post hours later, in which George Kutz, CrowdStrike Founder and CEO, stated, "The outage was caused by a defect found in a Falcon content update for Windows... This was not a cyber attack"  and goes on to offer an apology and that they understood the gravity and impact of the situation.

Industries in Turmoil: Who Was Affected?

Microsoft Vice President David Weston estimates that “CrowdStrike’s update affected 8.5 million Windows devices or less than one per cent of all Windows machines.” However, he recognises that while the percentage of devices affected is small, the broad economic and societal impact is huge.

The Airline Industry

One sector that was arguably the most badly affected was the airline industry. CrowdStrike’s defective software update had significant repercussions for major aviation carriers such as American Airlines, British Airways, and Virgin Australia, as passengers arriving at airports across the globe could not board their flights.

The entire airport experience has become much more seamless in recent years as technological advancements have led to online check-ins, digital boarding passes, and even digital queuing processes in some cases. However, the global IT outage resulted in airline staff having to go back to basics as they resorted to manual check-ins, which are tried and tested.

While this may have provided a solution for some who could board their flights successfully, many travellers and flight catchers were laboured by long queues, flight delays, and cancellations. Sky News reported that just over 5,000 flights were grounded on July 18th, which equates to nearly 5% of all scheduled flights globally.

The Healthcare System

The mayhem that ensued from CrowdStrike’s ineffectual software update inconvenienced more than just the travel industry. Healthcare professionals also had a torrid time as they experienced difficulties with communication systems, patient records, and administrative processes. Similar to the airline workers, some hospitals had to resort to a back-to-basics mindset as they opted for manual record-keeping and communication methods in response to the crisis.

Patient care had to be put on hold as appointment scheduling and access to critical medical records were removed due to the outage, resulting in significant delays and operational challenges for hospitals and general practices (GPs). Emergency services were also impacted globally as 911 services were temporarily shut down in the USA.

Time will tell how significant the global IT outage was for healthcare systems worldwide, as surgeries, critical procedures, and routine appointments were delayed or cancelled.

The Railway Industry

The railways were also equally affected. Commuters and travellers had their plans turned upside down as they faced failures regarding ticket dispensing machines. UK railway operators such as South Western Railway and Gatwick Express were forced to warn customers that they were experiencing widespread IT issues. They advised them to purchase their tickets online instead of buying them on-site.  

Uncovering the Costs and Repercussions of the Global IT Outage

At this stage, a couple of weeks after the event, we cannot pinpoint the costs and repercussions incurred due to this intentionally felt IT failure, as the true fallout may take months to calculate. However, we can estimate that the following implications would have been felt by the companies involved:

Reputational Impact

• CrowdStrike: This incident has undeniably damaged CrowdStrike’s software's reputation. The root cause of all this upheaval is their faulty antivirus software. While they acted promptly to rectify the issue, the outage will still dent their reputation as a cybersecurity provider.
• Microsoft: Although technically, Microsoft wasn’t directly responsible, their association with the outage may have affected people’s perceptions of the brand. They are a globally recognised tech behemoth, and customers and stakeholders would have expected the brand to have more stringent and robust systems in place to prevent the scale of the disruption that was caused. Microsoft, in turn, pointed to EU competition laws that require them to give this high level of access to companies like CrowdStrike.

Trust Erosion

• CrowdStrike: Organisations relying on CrowdStrike’s solutions may already be considering an alternative cybersecurity provider. The incident highlights the need for a competent provider that customers can trust so that their business can continue to run smoothly at all costs.
• Microsoft: Users will undoubtedly question Microsoft’s ability to prevent such incidents from happening in the future and may opt instead for a competitor for their cloud-based productivity, communication, and collaboration tools.

Budgetary Impact

• CrowdStrike: Mashable states, “CrowdStrike’s stock has plummeted almost 22 per cent since the outage, wiping around $16 billion of the company’s value.” However, it’s not just their share price that is set to plummet, with experts claiming the cost of CrowdStrike’s outage could “amount to billions of dollars.” Who’s to say at this stage whether the cybersecurity firm will ever fully recover from this lapse financially, especially if they have to pay out compensatory costs?
• Microsoft: Microsoft deployed hundreds of engineers to work directly with customers to restore critical services across the globe. This resource-intensive and expensive effort will no doubt have had a budgetary impact on the tech giant. Regarding economic impact, we know the percentage of affected devices was relatively small (less than 1%). However, the full extent of the broader economic and societal impact will no doubt be concluded in due course.

Lessons Learned: Why IT Investment is Crucial in Today’s Digital Age

Last week’s global IT outage triggered by a flawed software update from cybersecurity firm CrowdStrike sent shockwaves that reverberated across industries worldwide. Organisations grappled with disruptions that spanned from airline companies to healthcare providers. In many cases, they had to resort to manual, back-to-basics processes to solve the outage.

Here are the key takeaways and lessons we should take from the extraordinarily disruptive event:

1. Interconnectedness: In a society that has witnessed such drastic technological advancement in recent years, last week’s global IT outage highlighted how interconnected our digital ecosystem is and how potentially vulnerable that interconnectedness can make the systems we use day in and day out.
2. Risk Mitigation: If this unprecedented fault of global IT systems proved anything, investing in IT infrastructure in 2024 isn’t a luxury; it’s a necessity. Large and small organisations across industries must allocate appropriate resources to prevent, detect and respond to potential disruptions or risk suffering the consequences.
3. Expertise Matters: CrowdStrike’s flawed update seemingly slipped through the cracks on this occasion. Partnering with experts specialising in cybersecurity, software deployment, and disaster recovery can prevent such incidents from happening.
4. Invest in Managed IT Services: A competent Managed IT Service Provider (MSP) will offer tailored solutions to your business problems. Partnering with one would mean your systems would be monitored around the clock, routine patches and updates would be installed regularly, and robust backups and contingencies would be implemented to overcome any worst-case scenarios.

Expert Opinions and Final Thoughts

Michael Hamer (Virtual IT Director): The Netitude Take

CrowdStrike’s update testing was flawed. Their preliminary write-up identifies the need for all updates to pass through internal testing, ‘canary’ testing, and then a wider rolling deployment. This is already standard software practice, but there’s pressure to get protections out as soon as possible in security software.

Lessons will be learned across the industry, and ultimately, many IT systems will be made more reliable. However, it’s not a solved problem. Software is complicated; mistakes can and will happen again.

Despite CrowdStrike cancelling many IT engineers’ weekends, people are coming forward to defend their track record. CrowdStrike has saved them from many potential ransomware attacks that would have been just as disruptive at a company level.

The key takeaway is that society and businesses receive huge benefits from using technology: saving time and money far outweighs even this ‘biggest ever’ outage.

For management teams, work out your key business functions, identify dependencies (technology and otherwise), plan for outages, work out alternatives that let you keep working in some capacity and train staff. You never know when you need to fall back on pen and paper for a day.

How Netitude’s Clients Were Affected

While our clients were not directly affected, key software providers or partners they work with may have been. We work with market leader SentinelOne (CrowdStrike is their closest competitor). They take a different approach to updates without compromising security, allowing for deeper testing before updates go out. “Bad updates” is a scenario we consider and prepare for in our internal Incident Response planning. Planning ahead is the best way of minimising the potential for disruption.

Final Thoughts

One lesson from this global affair can be learned: investing in your IT is critical. It’s a stark reminder of the severe repercussions and consequences that widespread technological failures can feel.

Partnering with experts – such as globally recognised Managed IT Service Providers like Netitude – can provide proactive risk management to ensure potential faults and failures are located before they cause outages and downtime.

If you’re worried about cybersecurity processes or have doubts about your general IT infrastructure, contact our friendly team of experts today. They’re happy to alleviate any IT-related concerns or queries you might have.