Featured image

Table of Contents Link to heading

The most instructive outages are rarely caused by exotic failure modes. The cases below are representative of the incidents that define real-world IT risk: software updates that bypassed staging validation, human errors in high-stakes operational tooling, and configuration changes that hadn’t been tested against the failure scenarios they were meant to prevent. Each one is instructive not because it was unusual, but because the conditions that caused it are common.

The CrowdStrike Outage – A Software Update Gone Wrong Link to heading

What Happened? Link to heading

Example
On July 19, 2024, CrowdStrike pushed a content configuration update to its Falcon sensor — a file intended to update threat detection logic, not OS-level code. The update contained a logic error that caused the Windows kernel driver component to dereference an invalid pointer on boot, triggering a stop error (BSOD) that put 8.5 million Windows endpoints into an unrecoverable boot loop.

Impact Link to heading

  • Airports, hospitals, financial institutions, and broadcasters were among the affected organisations.
  • Fortune 500 companies reported an estimated $5.4 billion in aggregate losses from downtime.
  • Recovery required manual intervention at each affected machine — remote management tools were inaccessible because the machines wouldn’t boot.

How It Was Fixed Link to heading

CrowdStrike identified the faulty content file within hours and pushed a corrected version, but the fix was irrelevant to machines already in a boot loop. Recovery required:

  • Booting affected machines into Windows Recovery Environment (WinRE) or Safe Mode
  • Manually deleting the specific CrowdStrike content file from C:\Windows\System32\drivers\CrowdStrike\
  • Restarting to allow a clean boot and automatic update to the corrected content

At scale, across BitLocker-encrypted endpoints requiring recovery keys, this became a multi-day manual operation for large enterprise IT teams.

Lesson Learned Link to heading

Warning
Content updates delivered through security agents bypass the change management controls that govern software deployments — they’re designed to be fast, automatic, and frequent. That speed is a feature for threat detection, but it also means a faulty content update reaches production endpoints within minutes of release. Staged rollouts with canary deployments and automatic rollback criteria should apply to content updates, not just software versions.

The AWS Outage – A Typo That Cost Millions Link to heading

What Happened? Link to heading

Example
In February 2017, an AWS engineer executing a debugging procedure for a billing system issue entered an incorrect parameter in a command intended to remove a small number of servers from an S3 subsystem. The error caused a much larger set of S3 index servers to be removed — specifically the servers responsible for managing object storage metadata across a significant portion of the US-EAST-1 region.

Impact Link to heading

  • S3 returned errors across the US-EAST-1 region for approximately four hours.
  • Downstream services including Slack, Quora, Medium, and many others reliant on S3 for static assets or data storage were affected.
  • S&P 500 companies reported approximately $150 million in revenue impact.

How It Was Fixed Link to heading

AWS engineers restarted the affected S3 subsystems. The restart process itself took longer than expected because many of the index and placement subsystems hadn’t been restarted in years — their startup time at scale hadn’t been tested recently and exceeded operational expectations.

Lesson Learned Link to heading

Note
Two separate issues contributed to this incident: the absence of a minimum capacity guardrail on the operational tooling (which should have rejected the command before executing), and the undiscovered brittleness of the restart procedure for large-scale subsystems. Both are common in mature systems that have been running stably for long periods — operational paths that are never exercised accumulate hidden assumptions about state that only surface under stress. Regular disaster recovery drills, including actual restarts of critical subsystems in production-equivalent environments, catch these assumptions before incidents do.

The Facebook Outage – A System Bug That Took Down Social Media Link to heading

What Happened? Link to heading

Example
In October 2021, a configuration change to Facebook’s backbone routers — intended to audit capacity — inadvertently withdrew all BGP routes advertised by Facebook’s autonomous system. Facebook’s DNS servers, which were only reachable via those routes, became unreachable from the global internet. The result was a complete internet-facing outage for Facebook, Instagram, and WhatsApp lasting approximately six hours.

Impact Link to heading

  • Approximately 3.5 billion users lost access to Facebook, WhatsApp, and Instagram.
  • Meta’s market capitalisation dropped by approximately $47 billion during the outage.
  • Internal recovery was complicated because the tools Facebook engineers used to access internal systems were also dependent on the affected infrastructure.

How It Was Fixed Link to heading

Engineers had to physically access data centre hardware to restore the BGP configuration, bypassing remote access tooling that was itself unreachable. Internal communications also relied on infrastructure that was down, requiring engineers to coordinate through external channels while working on-site.

Lesson Learned Link to heading

Warning
This incident demonstrated a failure mode that’s easy to overlook: the management plane and the data plane shared the same failure domain. When the routing change took the data plane offline, it also took the tools used to fix it offline. Out-of-band management access — physically separate from the production network — is the control that ensures operators can always reach their systems regardless of the state of the network those systems run on. For any infrastructure that could take its own management tooling offline, OOB access is not optional.

The AT&T Outage – A Configuration Error That Blocked 92 Million Calls Link to heading

What Happened? Link to heading

Example
In February 2024, AT&T Mobility experienced a 12-hour nationwide outage caused by a misconfigured network change that was part of a network expansion. The error caused a subset of routing configurations to become inconsistent, degrading connectivity for mobile devices and blocking call establishment at scale.

Impact Link to heading

  • Approximately 125 million mobile devices were affected during peak impact.
  • An estimated 92 million calls failed to connect, including approximately 25,000 attempted calls to emergency services.
  • AT&T faced regulatory scrutiny from the FCC following the outage.

How It Was Fixed Link to heading

AT&T engineers identified the misconfigured parameters and rolled back the network change. Service was restored gradually over several hours as the corrected configuration propagated.

Lesson Learned Link to heading

Note
Configuration changes to carrier-scale networks require a different validation discipline than enterprise networks. The blast radius of a misconfiguration is orders of magnitude larger, and recovery time scales with network size. Staged rollouts — applying changes to a small subset of infrastructure first, validating, then expanding — are the standard approach in carriers precisely because full-network validation before deployment is operationally impractical. The absence of a staged rollout in this incident compressed the failure from a small-scale misconfiguration to a nationwide outage.

Preventive Architecture: How to Reduce Blast Radius Before the Next Incident Link to heading

Best Practices for IT Resilience Link to heading

  1. Test updates in controlled environments before production rollout. This applies to software deployments, content updates, and configuration changes. The staging environment should be representative of production — scaled-down but architecturally identical.
  2. Automate safeguards against human error in operational tooling. Minimum capacity guardrails, confirmation prompts for destructive operations, and mandatory dry-run modes are standard safeguards. Implement them before incidents reveal their absence.
  3. Maintain out-of-band management access. Management tooling that runs on the same infrastructure it manages is a single-failure-domain risk. OOB access — dedicated management networks, console servers, lights-out management — ensures operators can always reach their systems.
  4. Test your recovery procedures, not just your backups. A backup you’ve never restored is an assumption. A restart procedure you’ve never tested at scale is an unknown. Regular recovery drills surface the hidden brittleness that only appears under stress.
  5. Implement staged rollouts with automatic rollback criteria. Changes that reach all production systems simultaneously have a blast radius that matches your entire production footprint. Staged rollouts with defined success criteria and automated rollback bounds the impact of a bad change.

Investing in IT Stability Link to heading

  1. Redundant infrastructure across independent failure domains prevents single points of failure from becoming total outages.
  2. Automated rollback systems that can detect and reverse a bad change faster than a human can diagnose it are particularly valuable for high-frequency changes like security content updates.
  3. Monitoring and alerting that spans all layers — application, infrastructure, network, and dependency — gives teams the signal they need to detect and scope an outage in minutes rather than hours.

Prevention Over Response: The Common Thread Across All Four Incidents Link to heading

The incidents above were not caused by advanced persistent threats or novel failure modes. They were caused by known-category risks — untested updates, human error in operational tooling, absent rollback procedures, and management plane dependencies on the data plane — operating without the controls that would have contained them.

The investment required to prevent these incidents is real but bounded: staging environments, operational guardrails, OOB management, staged rollouts, and recovery drills. The cost of the incidents themselves — in revenue, in regulatory scrutiny, in recovery hours, and in some cases in public safety — is unbounded.

Note
IT resilience is not built during incidents — it’s built in the maintenance windows, architecture reviews, and runbook development that happen before incidents. The teams that handle outages well are the ones that have thought carefully about how they fail, not just how they work.