Brownout, Not Blackout: Multi-WAN Failover Tests That Matter
Tech GuruQuick Answer
Test WAN failure while the link remains up: latency, loss, DNS failure and selected unreachable destinations reveal problems a cable-pull misses. Measure detection, route change, new connections, existing sessions and failback separately. Keep independent management access and a metered-backup budget. A green gateway probe does not prove an application works, and a successful route change does not guarantee that existing sessions survive.
Evidence status: OPNsense and UniFi multi-WAN guidance was checked. No impairment, packet capture, detection timing, session-survival or LTE-cost trial was run. Fault levels and acceptance thresholds are deliberately not invented.
What To Check
A health check can reach one destination while the user cannot resolve a name or reach a required application. Write the expected path for each test flow, including local DNS and guest restrictions, before changing gateway settings.
The OPNsense guide documents distinct loss/latency triggers and warns that its failover/failback state options kill firewall states. Session reconnection in that configuration is expected behavior to measure, not automatic proof the failover failed.
Key Concepts In Plain Language
- Brownout: degraded or selective connectivity while the physical WAN remains up.
- Detection time: when the configured health logic declares failure, distinct from when a user transaction recovers.
- Failback: moving traffic back after recovery, including what happens to sessions still using a metered link.
Technical Checklist
- Record each WAN health target, interval, threshold, routing policy, NAT behavior, failback rule, and LTE cost or quota boundary.
- Inject latency, loss, DNS-only failure, route blackhole, and selected-destination failure while physical link state remains up.
- Measure detection, route change, active-session survival, guest-policy behavior, LTE usage, alert delivery, and failback stability.
- Use independent targets and user-journey checks so one healthy ping endpoint cannot mask a broken provider path.
Interactive Operating Model
Use the planned checks below with this overview. No test result is implied by the diagram.
Before You Start: Safe Defaults
Use a controlled test network or canary path, independent management access and a reviewed method to remove the impairment. Record health targets, intervals, thresholds, NAT/routing, DNS, failback and LTE quotas/cost limits. Do not impair an ISP or another operator's network.
- Export the current router, firewall, resolver, wireless, and switching configuration before changing policy.
- Keep direct management access that does not depend on the service or route being tested.
- Use one canary client or site and record the expected path before applying a fleet-wide setting.
- Test a negative case as well as a successful case; a green dashboard alone is not path evidence.
Go, Hold, Recover, Or Escalate
Proceed only within the approved test path and stop budget. Hold broad policy changes when traffic takes an unexplained route or management depends on the impaired link. Restore the test path promptly when safety, data integrity or the backup quota is at risk; keep the diagnostic evidence.
Planned Checks And Recovery
For OPNsense, inspect System > Gateways > Configuration for monitoring and thresholds, System > Gateways > Group for tier/trigger choices, and Firewall > Diagnostics > States for state evidence. The vendor guide also separates firewall-originated traffic/default-gateway switching from client policy routing. Do not apply its sample LAN rule without preserving local DNS and internal-network exceptions.
Step 1: Record normal paths
Capture health results and ordinary DNS/application transactions through each intended WAN. Record the client source, destination, egress and expected guest restriction. Include the path used by alerts and management, not just customer traffic.
Step 2: Inject one bounded impairment
Using an approved test setup, introduce latency, loss, DNS-only failure, a route blackhole or a selected-destination failure separately. Keep link state up and record fault start/removal. Partial packet-corruption tests belong only in a controlled lab with a defined recovery method.
Step 3: Observe control and user outcomes
Measure gateway detection, routing transition and successful new transactions. Observe existing sessions separately and correlate logs/captures with client times. Check guest policy and alert delivery on the backup link rather than assuming the primary-link rules still describe the outcome.
Step 4: Recover and watch failback
Remove the impairment and observe route recovery, session reconnection or persistence, and LTE usage. Check for repeated switching near thresholds. Restore the original settings and confirm no test routes, impairment or broad exceptions remain.
Validation Checklist
- Every fault has an identified test path, start/removal times and independently reachable management.
- Detection, route transition and client recovery are separate measurements; existing sessions are not silently omitted.
- DNS/application checks expose failures a single healthy probe can miss.
- Failback respects guest restrictions, monitoring and metered-link limits without unexplained oscillation.
Troubleshooting
- Gateway stays green during a user outage: Compare the probe destination with DNS and application dependencies; add independent journey evidence.
- Route changes but sessions drop: Check source/NAT changes and deliberate state-kill settings before promising seamless continuity.
- Traffic remains on LTE after recovery: Inspect existing states and failback policy; do not clear all production sessions without accounting for impact.
Security, Privacy, Legal, And Recovery Boundaries
Security
Keep management interfaces off untrusted networks, sanitize captures before sharing, and avoid weakening authentication merely to make a test pass.
Privacy
DNS logs, packet captures, client addresses, and wireless telemetry can reveal people, sites, and browsing patterns; collect the smallest useful sample and control retention.
Legal
Only impair or capture traffic on systems you own or are authorized to administer, and follow local communications and privacy requirements.
Recovery
Preserve a direct management path, a configuration export, and the exact commands or UI changes required to restore the last known working route or policy.
Record any local exception, its owner, its expiration condition, and the evidence required to remove it.
What The Evidence Does Not Prove
No universal latency/loss threshold or session-survival promise follows from these docs. UniFi and OPNsense behavior depends on gateway model, release, policy and topology. Provider diversity and a working LTE link also do not prove capacity or acceptable cost during a long outage.
Related TechGeeks Resources
- Networking Field Notes: Start Here
- 15 Router Security Settings to Audit Before Something Breaks
- What Should You Monitor in a Homelab?
References
Next Action
Start with one DNS or selected-destination failure that leaves the link up. Keep gateway observations and client transaction results on the same timeline, then test failback before changing wider policy.
