Featured image

Table of Contents Link to heading

The most instructive network incidents are rarely the ones caused by hardware failure or DDoS attacks — the root cause is usually obvious. The difficult ones are caused by misconfigurations, overlooked dependencies, or single points of failure that nobody thought to protect. The cases below are representative of that category: failures that looked complex on the surface but resolved to a single traceable root cause.

Incident 1: DNS Single Point of Failure — Total Loss of External Connectivity Link to heading

What Happened? Link to heading

Example
A mid-sized company’s employees lost internet access across the board. Hardware was up, cables were connected, and the ISP reported no outage on their end. Devices could communicate with each other on the LAN but couldn’t reach anything external.

Troubleshooting Steps Link to heading

  1. Hardware verification — No physical layer issues: ports were up, cables were intact, power was stable.
  2. Layer 3 diagnostics — ipconfig and internal ping tests confirmed LAN connectivity was intact. The failure was isolated to external resolution and routing.
  3. DNS isolation — Testing with a hardcoded external IP (ping 8.8.8.8) succeeded; DNS resolution failed. The internal DNS server had crashed, taking name resolution for all external destinations with it.

How It Was Fixed Link to heading

  • Restarted the DNS server and validated its configuration.
  • Deployed a secondary DNS server as a failover target.
  • Added monitoring alerts for DNS service availability, so future failures would surface before users started calling.

Lesson Learned Link to heading

Note
DNS failure is one of the highest-impact, lowest-visibility single points of failure in most networks. Because it affects all external access simultaneously, it’s often misdiagnosed as a WAN or routing issue before the DNS service itself is examined. Running two DNS servers in separate failure domains eliminates this class of outage. Monitor DNS response time and availability as a first-class service metric, not as an afterthought.

Incident 2: Rogue DHCP Server — IP Addressing Chaos Across the LAN Link to heading

What Happened? Link to heading

Example
Devices across a large office started receiving incorrect IP addresses — wrong subnets, missing default gateways, or addresses that conflicted with existing static assignments. Connectivity to shared resources, printers, and the internet degraded across the entire office simultaneously.

Troubleshooting Steps Link to heading

  1. DHCP scope inspection — The authorised DHCP server showed an unusually low lease count; something else was handling the address assignments.
  2. Network scanning — A second DHCP server was responding to discover requests on the same broadcast domain, handing out addresses from an overlapping, incorrectly configured pool.
  3. MAC address tracking — The rogue server traced back to a consumer Wi-Fi router that had been plugged into a wall port with its DHCP service enabled, intended as an access point but misconfigured.

How It Was Fixed Link to heading

  • Disconnected and properly reconfigured the rogue device (DHCP disabled, configured as an access point only).
  • Renewed DHCP leases across affected devices to restore correct addressing.
  • Implemented DHCP Snooping on managed switches — an 802.1Q feature that blocks DHCP server responses on untrusted ports, ensuring only the authorised DHCP server can respond to client requests.
  • Implemented network access control to prevent unmanaged devices from connecting to production network ports.

Lesson Learned Link to heading

Note
DHCP Snooping is the correct architectural defence against rogue DHCP servers — it enforces at the switch level that DHCP offers can only originate from trusted uplink ports. It should be standard configuration on all access-layer switches. The administrative overhead of enabling it is minimal; the cost of not having it is an outage of the type described above.

Incident 3: WAN Bandwidth Saturation from a Single Host Link to heading

What Happened? Link to heading

Example
Network performance degraded progressively across the organisation — video calls dropped, file transfers slowed to a crawl, and latency on interactive applications spiked. The degradation was consistent and widespread rather than affecting specific users or destinations.

Troubleshooting Steps Link to heading

  1. Bandwidth utilisation review — SNMP polling of WAN interface counters showed near-saturation on uplink and downlink simultaneously. A single source IP was responsible for a disproportionate share of the traffic.
  2. Flow analysis — NetFlow data confirmed the source: a company server was sustaining high-volume outbound transfer, consistent with a large download or backup job.
  3. Root cause — An employee had initiated a 4K video download directly to a company server, consuming the majority of available WAN bandwidth with no rate limiting applied.

How It Was Fixed Link to heading

  • Applied a traffic policy to block non-business traffic categories (streaming, large personal downloads) at the firewall.
  • Deployed a bandwidth monitoring tool with threshold-based alerting to flag unusual consumption patterns proactively.
  • Implemented QoS policies to prioritise latency-sensitive traffic (voice, video conferencing, interactive applications) over bulk data transfer, ensuring that even when WAN capacity is saturated, critical traffic degrades last.

Lesson Learned Link to heading

Note
QoS cannot create bandwidth that doesn’t exist, but it can ensure that the bandwidth available is allocated in order of business priority. Without QoS, a single high-throughput transfer can degrade the entire user experience. With QoS, that transfer gets queued behind critical applications and the impact is contained.

Incident 4: Firewall Misconfiguration Blocking Legitimate Business Traffic Link to heading

What Happened? Link to heading

Example
A new firewall was deployed to replace an aging device. The migration was treated as a hardware swap — same logical topology, new hardware. Within hours of cutover, email delivery failed, cloud service connectivity dropped, and remote workers couldn’t establish VPN sessions.

Troubleshooting Steps Link to heading

  1. Rule audit — The new firewall’s default-deny posture was more restrictive than the previous device’s configuration. Rules that had existed as implicit permits on the old platform were not replicated.
  2. Log analysis — Firewall deny logs confirmed that legitimate outbound traffic to SaaS platforms, mail relays, and VPN endpoints was being blocked.
  3. Traffic profiling — Systematically identified which traffic flows were being dropped and mapped them back to required business services before adjusting rules.

How It Was Fixed Link to heading

  • Created explicit permit rules for each identified traffic category, validated against the deny log before production cutover.
  • Implemented logging and alerting for deny events, making misconfigurations visible immediately rather than discovered through user complaints.
  • Established a policy requiring firewall changes to be tested in a staging environment and validated against a defined traffic matrix before production deployment.

Lesson Learned Link to heading

Warning
Firewall migrations are not hardware swaps. Each platform has different default policies, different implicit permit behaviours, and different ways of handling stateful inspection. Any firewall replacement should involve a full audit of the current rule set, translation to the new platform’s syntax, and validation against a known-good traffic baseline — before, not after, cutover.

Preventive Controls: From Reactive Troubleshooting to Proactive Operations Link to heading

Best Practices for Network Troubleshooting Link to heading

  1. Monitor at the service level, not just the interface level. Interface up/down tells you the link exists; monitoring DNS resolution, DHCP lease counts, and application response times tells you whether the network is actually working.
  2. Use structured escalation. Work the OSI model from the bottom up: physical, data link, network, transport, application. Jumping to application-layer conclusions before confirming lower-layer health wastes time.
  3. Implement DHCP Snooping and Dynamic ARP Inspection on all managed access-layer switches. These are standard features with minimal overhead that eliminate entire categories of attack and misconfiguration.
  4. Maintain a current network diagram and IP address register. During an incident, time spent reconstructing what the network looks like is time not spent fixing it.
  5. Establish change management discipline. The most common network incidents are caused by changes — planned changes, emergency changes, and undocumented changes made outside of process. A change log with rollback procedures is foundational to operational resilience.

Investing in Network Stability Link to heading

  1. Redundant DNS and DHCP in separate failure domains eliminates the most common cause of widespread, simultaneous connectivity failures.
  2. NetFlow and bandwidth monitoring give you the data to identify saturation before it becomes an outage and to diagnose the source when it does.
  3. Network monitoring with service-level alerting surfaces problems before users notice them and provides the historical data needed to identify patterns and capacity trends.

Troubleshooting vs. Prevention: Where the Real Value Is Link to heading

The incidents described above share a common thread: each one was preventable with controls that are well-understood, widely available, and not particularly expensive to implement. DHCP Snooping, redundant DNS, QoS policies, and firewall staging environments are standard practices in mature network operations.

The gap between organisations that experience these incidents repeatedly and those that don’t is rarely technical capability — it’s operational discipline: the willingness to invest in monitoring, redundancy, and change management before an incident forces the issue.

Note
Reactive troubleshooting is a necessary skill, but it’s not where the value in network operations lies. The value is in building systems that fail gracefully, surface problems early, and recover automatically — so that the troubleshooting cases that do require human intervention are genuinely hard, not preventable through basic operational hygiene.