Featured image

Table of Contents Link to heading

What Happened Link to heading

Info
The Optus 2023 outage was a major network failure affecting approximately 10 million customers on November 8, 2023, disrupting mobile voice, internet access, and emergency services for the better part of a day.

Root Cause Link to heading

Note
According to Optus’s post-incident disclosure, at approximately 4:05 AM, a routine software upgrade triggered a change in routing information received from an international peering network. These BGP updates propagated through multiple layers of the Optus IP core, exceeding preset safety thresholds on key routers. Unable to process the load, those routers withdrew from the IP core to protect themselves — a self-preservation mechanism that cascaded into a nationwide outage.

The underlying failure mode is a well-known class of BGP incident: a peer advertises an excessive number of prefixes or malformed route attributes, and the receiving network lacks adequate filtering or dampening to absorb the impact. In this case, the routing update volume wasn’t anomalous on its own — what failed was the chain of safeguards that should have caught it before it reached production infrastructure.

A comparable failure can occur in smaller-scale environments: a misconfigured route-map redistributing routes between IGP domains can inject thousands of unexpected prefixes into the routing table, exhausting router memory or CPU and triggering process restarts.

Similar Incidents Link to heading

  1. In 2012, a small Indonesian ISP inadvertently announced itself as the optimal path to millions of global IP prefixes. Upstream providers propagated the update without validation, flooding the ISP’s routers with traffic they couldn’t handle. The resulting overload caused widespread internet disruption for several hours — a textbook BGP hijack through misconfiguration rather than malicious intent.

  2. In 2014, a Chinese ISP erroneously originated over 50,000 prefixes belonging to other networks. Several peer networks accepted and propagated these routes, creating a routing loop that caused congestion and packet loss across North America and Europe — demonstrating how a single AS’s bad route announcement can have hemispheric impact.

  3. In 2018, a Nigerian ISP announced over 200,000 spurious prefixes. Upstream providers that accepted these routes forwarded them to their peers, redirecting a significant portion of African internet traffic to an infrastructure incapable of handling it. The outage lasted over an hour and affected connectivity across much of the continent.

Each of these incidents shares the same structural failure: insufficient BGP policy enforcement at peering boundaries, and an implicit trust in route announcements that should have been validated before acceptance.

BGP Hardening: Best Practices to Prevent Recurrence Link to heading

Tip

  1. Enforce strict BGP prefix filtering at all peering and transit boundaries.

    • Use prefix lists to permit only the specific IP prefixes a peer is authorised to originate — reject anything outside those bounds at the BGP policy level, not after the fact.
    • Validate AS_PATH to ensure routes are not traversing unexpected autonomous systems, which can indicate hijacking or redistribution errors.
    • Use BGP communities and route maps to apply consistent policy across all peers rather than managing per-peer exceptions manually.

  2. Apply BGP maximum-prefix limits with appropriate thresholds.

    • Configure maximum-prefix on all BGP sessions to trigger warnings or tear down sessions when a peer announces an abnormal number of prefixes. This is the most direct mechanism to prevent a flood of routing updates from reaching your core.
    • Tune thresholds based on the expected prefix count for each peer — a transit provider might legitimately announce a full table, while a customer peer should never exceed a small, well-defined set.

  3. Implement route dampening selectively to suppress flapping routes.

    • Route dampening penalises routes that repeatedly change state, preventing instability from propagating. However, apply it judiciously — overly aggressive dampening can suppress legitimate recovery paths during an outage, extending recovery time.

  4. Monitor BGP session state and routing table anomalies in real time.

    • Use SNMP, NetFlow, or streaming telemetry to track BGP prefix counts per peer and alert on sudden changes. A 10x increase in received prefixes from a peer is an operational signal worth acting on immediately.
    • Tools like RIPE NCC’s BGPmon, RouteViews, or open-source alternatives can provide visibility into prefix announcements from your AS’s perspective on the global routing table.

Network Outage Taxonomy Link to heading

Understanding the root cause taxonomy of outages matters because each category has a different detection and prevention strategy:

  • Maintenance and upgrades — Planned changes that introduce unintended side effects. Mitigated by change management processes, staging environments, and rollback procedures.
  • Hardware failure — Physical component failure in routers, switches, or line cards. Mitigated by redundant hardware, hot-spare strategies, and hardware lifecycle management.
  • Resource exhaustion — CPU, memory, or routing table limits reached on network devices. Often precedes a cascade if not caught by monitoring.
  • Physical layer damage — Cable cuts, connector failures, or fibre breaks. Often unpredictable; mitigated by physical path diversity and redundant uplinks.
  • Network congestion — Bandwidth saturation causing queueing, packet drop, and latency spikes. Requires QoS policy, capacity planning, and traffic engineering.
  • Power failure — Loss of power to network infrastructure. Mitigated by UPS, generator backup, and dual-feed power distribution.
  • Security attacks — DDoS, BGP hijacking, or exploitation of device vulnerabilities. Requires layered defences including BCP38, RTBH, and scrubbing centres.

These incidents collectively reinforce that rigorous testing, disciplined change management, and operational monitoring are not optional in networks that underpin critical services. The Optus 2023 outage, and the similar incidents that preceded it, are reminders that the BGP trust model — built on the assumption that peers announce only what they’re authorised to — remains one of the most consequential unresolved design constraints in the global internet.