Featured image

Table of Contents Link to heading

The Initial Picture: Symptoms That Do Not Point to Root Cause Link to heading

The security team described the symptoms accurately: cameras lose signal for a few minutes, consistently around 2 PM, then recover on their own. Not every day — only on hot, sunny ones.

Info
Intermittent failures that self-recover, follow a time-of-day pattern, and correlate with ambient conditions are the textbook signature of a thermal event or resource contention — not physical hardware failure. A genuinely faulty device fails consistently; it does not recover on its own.

What the previous technicians had already tried:

Action Estimated Cost Result
Replace all cameras with new units ~$320 No change
Replace suspect network cable runs ~$120 No change
Restart the Switch every morning Labour cost Improved for a few days, then recurred
Total >$600 Problem unchanged

Every one of these actions was wrong because they were treating symptoms rather than diagnosing the cause. The morning Switch restart actually worked by accident — it reset the Switch’s thermal state after a cool night — but the effect only lasted until the next heat cycle.

When our team arrived, the first thing we looked at was not the cameras or the cables. It was the Switch CLI:

show power inline

The number on the screen explained everything.

What PoE Is and Why It Matters More Than Most People Think Link to heading

Info
PoE (Power over Ethernet) delivers electrical power over Cat5e/Cat6 network cable alongside data, allowing endpoint devices — cameras, wireless access points, IP phones — to operate without a separate power supply. A PoE switch acts simultaneously as a network device and a centralised power source.

PoE simplifies installation — one cable, two functions. But that convenience conceals an important technical constraint: a PoE switch has a fixed total power budget, and the combined draw of all connected devices cannot exceed it.

What most people miss: the PoE budget is not a hard brick wall that cuts everything off at once. Many switches, when overloaded, shed load progressively by cutting power to the ports with the lowest priority — typically the ports that were connected last or that have no explicit priority configuration.

That is exactly what was happening to this camera system.

Root Cause: PoE Budget Overload Combined with Thermal Throttling Link to heading

The system as deployed:

  • 24-port PoE switch, maximum budget: 250W
  • Active cameras connected: 22
  • Power draw per camera (2MP IR dome): average 11–12W (IEEE 802.3af, Class 3)
  • Total actual load: ~240–264W — at or beyond the rated budget

But that is only part of the story.

The Failure Mechanism Link to heading

Warning
Thermal throttling in PoE switches is a hardware self-protection mechanism. When the internal temperature exceeds a safe threshold, the PoE controller automatically reduces the available PoE budget to lower heat dissipation — even if the current electrical load has not exceeded the nameplate rating.

The sequence of events on every hot afternoon:

13:30 — Ambient temperature begins rising (network cabinet has no cooling)
        Switch drawing 240W, PoE controller chip temperature climbing

14:00 — Chip temperature crosses thermal throttle threshold (~70°C)
        Controller reduces available PoE budget to ~200W
        Switch needs to shed ~40W from its ports

14:00–14:03 — Switch cuts PoE power to lowest-priority ports
             (cameras connected last, no priority config applied)
             → Cameras lose power → lose signal
             → NVR records "camera offline"
             → Security team concludes "camera fault" ← WRONG DIAGNOSIS

14:03–14:05 — Camera disconnections reduce total load
             → Chip temperature drops slightly
             → Controller restores budget
             → Switch re-powers the cameras
             → Cameras boot (~60–90 seconds) → back online

14:05 — System appears to "recover" — cycle repeats next hot day

The reason it does not happen every day: on overcast, cooler days, the ambient temperature never pushes the chip past its throttle threshold.

Traffic Flow Diagram Link to heading

                        ┌─────────────────────────────────────┐
                        │         INTERNET / WAN              │
                        └──────────────────┬──────────────────┘
                                           │
                        ┌──────────────────▼──────────────────┐
                        │           Core Router               │
                        └──────────────────┬──────────────────┘
                                           │ Uplink (1 Gbps)
                        ┌──────────────────▼──────────────────┐
                        │   PoE Switch 24-port, 250W budget   │◄── [CABINET, NO COOLING]
                        │                                     │
                        │  Total PoE Load: ~240W (96% budget) │
                        │  Ambient Temp: 35–42°C at 14:00     │
                        │  Chip Temp: ~72°C → THROTTLING      │
                        └──┬──┬──┬──┬──┬──┬──┬──┬──┬──┬──┬──┘
                           │  │  │  │  │  │  │  │  │  │  │
                    [P1][P2][P3]...[P18] ← Cameras 1–18 (ONLINE)
                                   [P19][P20][P21][P22] ← Cameras 19–22
                                         ↑
                               CUT ON THERMAL THROTTLE
                               (connected last, no priority)

NORMAL DATA FLOW:
Camera → [PoE Port] → Switch → [Uplink] → NVR Server → Storage

DATA FLOW DURING THROTTLE:
Camera → [PoE Cut] → Camera loses power
                   → Switch log: "PoE overload port X"
                   → NVR log: "Camera X offline"
                   → Security team: "Camera is broken" ← MISDIAGNOSIS

Current Topology vs Correct Topology Link to heading

Old Topology: Single Point of Failure Link to heading

                    [ISP / Router]
                          │
              ┌───────────▼──────────────┐
              │   PoE Switch 250W        │
              │   22 cameras × 12W       │
              │   = 264W OVER BUDGET     │
              │                          │
              │   Missing:               │
              │   • PoE priority config  │
              │   • Cabinet cooling      │
              │   • PoE load monitoring  │
              └──────────────────────────┘
                 │││││││││││││││││││││││
            Cam1  Cam2  ...  Cam18  Cam19 Cam20 Cam21 Cam22
                                    └────────────────────┘
                                     Cut during throttle

New Topology: Proper Load Distribution Link to heading

                    [ISP / Router]
                          │
              ┌───────────▼──────────────┐
              │   Core Switch (non-PoE)  │
              │   Layer 3                │
              └───────┬──────────┬───────┘
                      │          │
         ┌────────────▼──┐  ┌───▼────────────┐
         │ PoE Switch A  │  │  PoE Switch B  │
         │ 24-port, 250W │  │  24-port, 250W │
         │ 11 cameras    │  │  11 cameras    │
         │ Load: ~132W   │  │  Load: ~132W   │
         │ = 53% budget  │  │  = 53% budget  │
         └───────────────┘  └────────────────┘
              │││││││││││        │││││││││││
           Cam1..Cam11          Cam12..Cam22

         [Cabinet with ventilation fan or mini AC unit]
         [PoE Priority: critical cameras = HIGH]
         [SNMP monitoring of PoE load in real time]
Tip
Design rule: Never let PoE load exceed 70–75% of total budget. The remaining headroom accounts for thermal derating, cable loss, and future expansion. On a 250W switch, the practical ceiling is 175W.

Technical Solution: Three Layers of Intervention Link to heading

Layer 1 — Immediate Fix: PoE Priority Configuration Link to heading

Before purchasing any new equipment, configure PoE port priority to control which cameras are protected when overload occurs.

Cisco Catalyst / IOS:

! Check current PoE state
show power inline
show power inline detail

! Example output that signals a problem:
! Available:250.0(w)  Used:238.4(w)  Remaining:11.6(w)
! ← This number is a red flag

! Set priority HIGH for critical cameras (entrance, cash register)
interface GigabitEthernet1/0/1
 power inline port priority high
 description "CAM-ENTRANCE-01 - HIGH PRIORITY"

interface GigabitEthernet1/0/2
 power inline port priority high
 description "CAM-CASHIER-01 - HIGH PRIORITY"

! Set priority LOW for less critical cameras (parking lot, warehouse)
interface GigabitEthernet1/0/20
 power inline port priority low
 description "CAM-PARKING-01 - LOW PRIORITY"

! Cap maximum per-port wattage (if camera spec is known)
interface GigabitEthernet1/0/1
 power inline port maximum 15400
! 15400 mW = 15.4W IEEE 802.3af max — prevents unknown devices from drawing excess

! Verify the result
show power inline GigabitEthernet1/0/1

HP / Aruba ProCurve:

show power-over-ethernet
power-over-ethernet 1 priority critical
power-over-ethernet 2 priority high
power-over-ethernet 20 priority low

UniFi (UI):

Navigate to Switch → Port Settings → Power Management → Set PoE Mode to PoE+ and Priority to “High” for critical camera ports.

Layer 2 — Permanent Fix: Switch Load Distribution Link to heading

This is the correct long-term solution. Distribute the camera load across two or more PoE switches.

Warning
Do not cascade a second PoE switch off the first. Daisy-chaining a PoE switch as a downstream device of another PoE switch does not reduce PoE load on the upstream device — it only adds data traffic. Each PoE switch must have its own independent uplink to a core switch or router.
❌ WRONG:
ISP → Router → PoE Switch A (250W) → PoE Switch B (cascaded)
                Cameras 1–11          Cameras 12–22
[Switch A still carries the uplink power draw for Switch B]

✅ CORRECT:
ISP → Router → Core Switch (non-PoE)
                    ├── PoE Switch A (250W) → Cameras 1–11
                    └── PoE Switch B (250W) → Cameras 12–22

Recalculated load after distribution:

Each switch: 11 cameras × 12W = 132W
Switch budget: 250W
Utilisation: 132 / 250 = 52.8% ← safe, with 47% headroom remaining
Capacity for expansion: ~9 more cameras per switch before hitting the 75% threshold

Layer 3 — Prevention: Thermal Management and Monitoring Link to heading

! Enable SNMP to monitor PoE load in real time
snmp-server community public RO
snmp-server enable traps power-ethernet

! Configure syslog to record PoE events
logging on
logging buffered 4096
logging trap warnings

! Key SNMP OIDs for PoE monitoring:
! 1.3.6.1.2.1.105.1.3.1.4.x  — pethPsePortActualPower (per-port wattage)
! 1.3.6.1.2.1.105.1.1.1.7.x  — pethMainPseConsumptionPower (total draw)
! 1.3.6.1.2.1.105.1.1.1.6.x  — pethMainPseOperStatus (overload status)

! Monitor Switch temperature (if supported)
show environment temperature
show environment all

Thermal management for the network cabinet:

  • Install a 12V ventilation fan inside the cabinet if no AC is present
  • Maintain at least 1U of clearance between stacked devices for airflow
  • Add an IoT temperature sensor inside the cabinet — alert when it exceeds 40°C
  • Consider a dedicated mini AC unit for server rooms where PoE load is substantial

PoE Standards Reference: Choosing the Right Switch from the Start Link to heading

Standard Common Name Max Power / Port Delivered to Device Typical Use Case
IEEE 802.3af PoE 15.4W 12.95W SD cameras, IP phones, basic APs
IEEE 802.3at PoE+ 30W 25.5W PTZ cameras, dual-band APs
IEEE 802.3bt Type 3 PoE++ 60W 51W AI cameras, tri-band APs, thin clients
IEEE 802.3bt Type 4 UPoE 100W 71.3W High-end displays, advanced endpoints
Cisco UPOE Cisco proprietary 60W ~51W Cisco high-end IP phones
Note
Modern AI cameras (face recognition, licence plate recognition) typically consume 15–25W, placing them in the PoE+ category. If the switch only supports IEEE 802.3af (15.4W maximum), the camera will either fail to power on or operate with features disabled. Always verify the camera’s power class against the switch’s per-port capability before purchasing either device.

Calculating PoE Budget for a Camera System Link to heading

Step 1: Determine total actual power consumption
─────────────────────────────────────────────────
P_actual = Σ (number of cameras × watts per camera)
         + Σ (other PoE devices: APs, phones, etc.)

Step 2: Account for cable loss (significant on long runs)
─────────────────────────────────────────────────────────
Cable loss ≈ 10–15% for Cat5e runs of 50–80 metres
P_with_loss = P_actual × 1.15

Step 3: Apply headroom factor (minimum 25%)
────────────────────────────────────────────
P_required = P_with_loss / 0.75

Step 4: Select a switch with PoE budget ≥ P_required
─────────────────────────────────────────────────────
If P_required > 250W → use two switches or a 370W/740W model

Example — the 22-camera system at 12W per camera:

P_actual    = 22 × 12W = 264W
P_with_loss = 264W × 1.15 = 303.6W
P_required  = 303.6W / 0.75 = 404.8W

→ Minimum 405W of PoE budget required
→ Solution A: 2× 250W switches (11 cameras each, 132W = 53% per switch)
→ Solution B: 1× 740W switch (264W load = 36%, significant expansion headroom)

Questions to Ask Before Signing Off on Any PoE System Link to heading

This incident changed the first question asked during every PoE system survey:

Design questions:

  • What is the switch’s total PoE budget in watts? (Not the port count)
  • What is the combined wattage of all connected PoE devices?
  • Is the current utilisation below 75% of budget?
  • What is the longest cable run in the system, in metres?

Expansion questions:

  • Are there plans to add cameras, APs, or other PoE devices in the next 1–2 years?
  • If so, does the current headroom accommodate that growth?

Environment questions:

  • Is the network cabinet ventilated or cooled?
  • What is the peak cabinet temperature in summer?
  • Is the switch located near other heat sources (servers, large UPS units)?

Operations questions:

  • Is PoE load being monitored in real time?
  • Are alerts configured when load exceeds a threshold?
  • Has PoE priority been configured based on camera criticality?
Tip
The number to remember: keep PoE utilisation at ≤ 70% of rated budget. On a 250W switch, the practical ceiling is 175W. That 30% margin is not waste — it is insurance against thermal derating, cable loss, and the devices that get added six months after commissioning. The cost of a second small PoE switch is a fraction of the cost of three rounds of incorrect repairs plus the erosion of customer trust that comes with a system that cannot be relied on.

A surveillance system that stays online is not the product of expensive hardware. It is the product of load distributed correctly, operating environment controlled, and someone asking the right questions at the design stage rather than after the first failure.