Linux Failure Injection Lab: Test systemd, Disk, DNS, and Recovery

The safe way to break a Linux lab is to make every failure smaller than the recovery boundary. Use a disposable VM or a canary with no production traffic, inject one fault at a time, stop automatically on unexpected impact, prove the alert from outside the target, recover through a prepared path, and verify the user-facing service before declaring success. Never fill a production disk, remount a production root filesystem, disable a shared resolver, or pause a production process just to see what happens.

Hard boundary: The commands below are for a disposable, console-accessible Linux VM. Names beginning with tg- and paths below /var/tmp/tg-failure-lab are deliberate guardrails. Do not substitute a production unit, mount, disk, resolver, or host.

The Short Answer

• Start from a known-good snapshot or backup and prove that the recovery copy can be read.
• Record normal service, storage, resolver, application, and alert state before injecting anything.
• Use a synthetic systemd unit, a fixed 256 MiB loop-backed filesystem, and a service-private network namespace. Do not damage the host's real resources.
• Define the stop condition and recovery command before the fault command.
• Require four kinds of evidence: injector state, operating-system state, application behavior, and independently delivered alerting.
• Recover, remove the fixture, rerun the ordinary user workflow, and check for data or configuration drift.

What Is Documented, Recommended, And Not Yet Tested

Documented facts: Upstream systemd documentation defines service failure states, restart policies, start-rate limits, OnFailure=, PrivateNetwork=, and the service watchdog protocol. Linux and util-linux documentation define loop devices, mount state, and quota mechanisms. The filesystem projects document important snapshot and rollback limits.

TechGeeks recommendation: The ordering, stop gates, evidence matrix, bounded 256 MiB image, and decision criteria in this article are an editorial test design. They are intentionally more conservative than the minimum syntax in the manuals.

Unperformed lab work: TechGeeks did not execute these commands, collect screenshots, time alerts, or restore a VM for this draft. Code blocks are documentation-backed examples, not observed output. Validate package paths and behavior on the exact distribution and systemd version before publication or use.

Production Is Not The First Test Target

A production experiment is not made safe by calling it chaos engineering. Shared storage, hidden DNS dependencies, automatic failover, backup jobs, and management access can expand a small action into a site outage. Google SRE's published experience includes a controlled test that spread farther than expected and took longer to recover because rollback had not been tested in a test environment. AWS Fault Injection Service similarly makes a declared steady state and stop condition first-class controls. The transferable lesson is simple: containment and abort logic are part of the experiment, not paperwork around it.

For a home lab, use a cloned VM with a console in the hypervisor. For a small office, use a staging instance with synthetic data and no production credentials. For a production service, first prove the same fault and recovery in a representative canary. A later production exercise needs its own approval, impact budget, maintenance window, observer, communications path, and automatic stop conditions; this article does not authorize that step.

Preflight: Define Steady State And Stop Conditions

Write down what healthy means before creating a failure. A green systemd unit alone is weak evidence. Include an ordinary client request, a write-and-read check where appropriate, queue or database health, current free blocks and inodes, resolver behavior, backup age, and the alert channel's last successful heartbeat.

• Target: exact VM, unit, data path, filesystem, dependency, and owner.
• Blast radius: no production traffic, data, credentials, mounts, or shared network services.
• Time box: maximum injection duration and maximum recovery duration.
• Stop conditions: lost console or SSH access, an unexpected mount or unit, production alarms, unrelated service degradation, data-integrity uncertainty, or a missed recovery deadline.
• Recovery: exact reversal command, restart order, snapshot or restore point, alternate restore target, and person allowed to escalate.
• Evidence: UTC timestamps, command output, journal slice, application result, alert receipt, acknowledgment, recovery notification, and final diff.

Capture the platform before the drill. Replace tg-app-canary.service only with a non-production clone you own.

date -u +%FT%TZ
uname -a
systemd --version
findmnt -no TARGET,SOURCE,FSTYPE,OPTIONS /
df -hT
df -i
systemctl show tg-app-canary.service \
  -p LoadState -p ActiveState -p SubState -p Result -p NRestarts
journalctl -u tg-app-canary.service --since "-10 min" --no-pager

Interactive Failure And Recovery Model

Open each failure card before scheduling the lab. Every card pairs an injection with the observation and recovery that make it useful.

Test 1: A systemd Service Fails Repeatedly

Stopping a service with systemctl stop is not the same as a crash. The systemd service documentation says a stop performed by systemd does not trigger the configured restart policy. To test Restart=on-failure, create a synthetic transient unit that exits unsuccessfully. The following unit has no application data and is rate-limited to three starts in ten seconds.

sudo systemd-run \
  --unit=tg-fail-exit \
  --property=Type=exec \
  --property=Restart=on-failure \
  --property=RestartSec=1s \
  --property=StartLimitIntervalSec=10s \
  --property=StartLimitBurst=3 \
  /bin/sh -c 'exit 42'

Do not expect one universal status string across every systemd release and timing window. Inspect the properties and journal together. A useful result shows the nonzero exit, attempted restarts, rate limiting or a final failed state, and the monitoring event. The app-level monitor should also fail if this synthetic unit is standing in for a canary application.

systemctl show tg-fail-exit.service \
  -p ActiveState -p SubState -p Result -p ExecMainStatus -p NRestarts
journalctl -u tg-fail-exit.service --since "-5 min" --no-pager
systemctl --failed

OnFailure= can activate another unit when a unit enters the failed state, but that only proves local dispatch. It does not prove that an email, webhook, or pager escaped a failed host. Validate both the local handler and an external service probe. Also inspect restart loops directly: a service that restarts quickly may disappear from systemctl --failed while still being unusable.

Recover by stopping the fixture and resetting its failed state. Upstream systemctl documentation says reset-failed also clears the start-rate and restart counters. Confirm that the transient unit is no longer failed; it may be unloaded completely.

sudo systemctl stop tg-fail-exit.service
sudo systemctl reset-failed tg-fail-exit.service
systemctl is-failed tg-fail-exit.service

Test 2: Disk Full Without Filling A Real Disk

Never create a giant filler file on /, /var, a production database volume, or a thin pool. A loop device maps a regular file to a block device, so a fixed-size image creates a small failure domain. This example preallocates 256 MiB and caps the attempted write at 300 MiB. Even if the mount check is missed, the write count is bounded.

Run this only after checking that the disposable VM has more than 512 MiB free under /var/tmp. Keep the loop-device value in the same shell until cleanup.

LAB_ROOT=/var/tmp/tg-failure-lab
LAB_MNT="$LAB_ROOT/mnt"
LAB_IMG="$LAB_ROOT/ext4.img"

df -h /var/tmp
sudo install -d -m 0700 "$LAB_ROOT" "$LAB_MNT"
sudo fallocate -l 256M "$LAB_IMG"
LOOP=$(sudo losetup --find --show --nooverlap "$LAB_IMG")
sudo mkfs.ext4 -q -L TGFILL "$LOOP"
sudo mount -o nosuid,nodev,noexec "$LOOP" "$LAB_MNT"
sudo chown "$(id -u):$(id -g)" "$LAB_MNT"
findmnt -no TARGET,SOURCE,FSTYPE,SIZE,OPTIONS --target "$LAB_MNT"

Before filling it, point only a canary application's scratch, upload, queue, or data path at this mount. Do not bind a production data directory into it. Prove the source is the expected loop device, then write the bounded payload. The expected injection result is a nonzero write with No space left on device or equivalent application evidence. Exact free space differs because of filesystem metadata and reserved blocks.

test "$(findmnt -no SOURCE --target "$LAB_MNT")" = "$LOOP" || exit 1
dd if=/dev/zero of="$LAB_MNT/payload.bin" \
  bs=1M count=300 conv=fsync status=progress
df -h "$LAB_MNT"
df -i "$LAB_MNT"
printf 'write probe\n' > "$LAB_MNT/probe.txt"

Check more than the error message. Did the service stay up but reject writes? Did it crash and restart? Did it corrupt a partial upload, wedge a queue, or continue reporting healthy? Did the disk alert arrive before the final write failed? Did the alert identify the lab filesystem rather than the host's root disk? Record block use and inode use separately because a directory can run out of inodes while byte capacity remains.

Recover by removing only the bounded payload, then make and read a fresh probe. If the application owns a database or queue on the mount, stop it cleanly first and follow its documented consistency check before restarting. Deleting the filler file restores capacity; it does not prove application state is consistent.

rm -f "$LAB_MNT/payload.bin" "$LAB_MNT/probe.txt"
df -h "$LAB_MNT"
printf 'recovery probe\n' > "$LAB_MNT/recovered.txt"
cat "$LAB_MNT/recovered.txt"

If a representative test must share a larger lab filesystem, a project quota is another bounded option. Linux quotas can limit consumed blocks and inodes; XFS and ext4 support project-oriented controls through different setup procedures. Use a dedicated test filesystem and its current vendor documentation. Enabling or changing quotas on an existing production filesystem introduces a new mount and policy change, so it is not the safe shortcut for this drill.

Test 3: The Application Data Path Becomes Read-Only

Reuse the loop-backed filesystem after the full-disk payload has been removed. Flush pending writes, remount only that lab mount read-only, and verify the effective mount options with findmnt. This tests how the application handles a read-only data path. It does not reproduce a read-only root filesystem, a storage-controller fault, journal corruption, or intermittent I/O errors.

sync
sudo mount -o remount,ro "$LAB_MNT"
findmnt -no TARGET,SOURCE,FSTYPE,OPTIONS --target "$LAB_MNT"
printf 'this must fail\n' > "$LAB_MNT/should-not-exist.txt"

The failed shell write proves the mount rejects this write. It does not prove the application noticed. Exercise the real canary operation: accept a synthetic upload, rotate a test log, persist a queue item, or commit a throwaway record. A safe application may reject the operation clearly, switch to a documented degraded mode, or stop. An unsafe result is a false success, silent data loss, an endless retry storm, or a health endpoint that stays green while writes fail.

Recover by remounting the lab filesystem read-write and verifying both the mount state and an application-level write/read. If the filesystem changed to read-only because of a real metadata or device error, do not blindly remount it read-write. Preserve logs, reduce writes, follow the filesystem and storage vendor's diagnostic procedure, and restore elsewhere when integrity is uncertain.

sudo mount -o remount,rw "$LAB_MNT"
findmnt -no TARGET,SOURCE,FSTYPE,OPTIONS --target "$LAB_MNT"
printf 'read-write restored\n' > "$LAB_MNT/rw-check.txt"
cat "$LAB_MNT/rw-check.txt"

Test 4: Separate DNS Failure From Network Failure

DNS loss and network loss are different incidents. Test them separately or the result will not tell you whether the application cached an address, bypassed the configured resolver, retried another resolver, or had no route.

For total network loss, run a duplicate client command or a canary service with systemd's PrivateNetwork=yes. Upstream documentation says this creates a network namespace with only its own loopback interface. Do not add this property to the production unit. A transient example can display the namespace and run a synthetic dependency check:

sudo systemd-run \
  --unit=tg-netloss \
  --wait --pipe \
  --property=Type=exec \
  --property=PrivateNetwork=yes \
  /bin/sh -c 'cat /proc/net/route; getent ahosts dependency.example'
sudo systemctl reset-failed tg-netloss.service

Replace dependency.example with a lab-only name. A hostname can still resolve from /etc/hosts, a cache, or a filesystem-accessible local resolver socket even though the namespace has no external interface. Therefore, route inspection plus the application's actual connection failure is the acceptance evidence. This is also why a successful getent result does not disprove network isolation.

For DNS-only loss, keep IP connectivity to a controlled lab endpoint while making the canary's resolver unreachable inside its own VM, container, or network namespace. Do not edit the host's shared /etc/resolv.conf or block port 53 globally. First prove a normal name-based request. During the injection, require the name lookup to fail while a pinned-address request to the same controlled endpoint still succeeds with the expected host name and TLS identity. Applications that use DNS over HTTPS, embedded resolvers, service discovery, or long-lived connections need an injection at their actual resolver path.

Measure timeout and retry behavior. A 30-second resolver delay multiplied across worker threads can be more damaging than a fast failure. Check whether queues grow, circuit breakers open, caches serve stale data intentionally, and alerts identify DNS rather than merely reporting a generic application timeout. Recover by removing the canary-only resolver or namespace override, then repeat both the lookup and the end-to-end request.

Test 5: Prove A systemd Watchdog Actually Recovers

WatchdogSec= is not a generic setting that makes any service self-healing. The daemon must implement systemd's notification protocol and send WATCHDOG=1 keep-alives. If the interval is exceeded, systemd places the service in a failed state, terminates it, and can restart it when the restart policy includes watchdog failures.

Use only a watchdog-capable canary whose state can survive an abrupt termination. Confirm the effective timeout and notification behavior from the unit and application documentation. Then pause only the main process long enough to miss the heartbeat:

systemctl show tg-watchdog-canary.service \
  -p MainPID -p WatchdogUSec -p WatchdogTimestampMonotonic -p NRestarts
sudo systemctl kill \
  --kill-whom=main \
  --signal=SIGSTOP \
  tg-watchdog-canary.service

Wait only for the declared timeout plus a small observation margin. The pass condition is not merely a higher restart counter. Require journal evidence of a watchdog result, a bounded restart, successful application initialization, an ordinary client request, and alert delivery. If the service never times out, the watchdog may be disabled, the wrong process may be sending notifications, or the unit may not attribute notifications as expected.

If the watchdog does not terminate the paused process, immediately continue it before diagnosis. Otherwise restart the canary through its normal recovery path. A watchdog proves that one liveness signal stopped; it does not prove the service was functionally healthy while heartbeats were arriving.

sudo systemctl kill \
  --kill-whom=main \
  --signal=SIGCONT \
  tg-watchdog-canary.service
sudo systemctl restart tg-watchdog-canary.service
systemctl status tg-watchdog-canary.service --no-pager

Alert Validation: A Page Must Escape The Failure Domain

A local journal entry is evidence, not an alert. For each injection, verify the complete alert path: signal collection, rule evaluation, notification delivery, human acknowledgment, and recovery closure. At least one heartbeat or service probe should originate outside the VM and outside the host being tested. Otherwise a total host, network, or storage failure can silence both the service and its observer.

Inspect alert text for secrets, customer names, file paths, query content, and tokens before sending test evidence to a shared channel. Use synthetic accounts and data. Record detection time from the injection timestamp, notification time, acknowledgment time, recovery time, duplicates, and false clears. One fast page does not establish long-term alert reliability, but a missing or misleading page is an immediate failure.

Snapshots, Backups, And Rollback Are Different Tools

Take a platform-native VM or volume snapshot before the drill when the platform supports a documented, tested rollback. Also export the service configuration and keep a separate backup of authoritative data, credentials, certificates, encryption keys, and dependency information. The snapshot accelerates local reversal; the backup covers a broader failure domain and should be restorable without the original host.

Filesystem details matter. Btrfs documentation explicitly warns that a snapshot is not a backup because it initially shares data blocks with the original, and nested subvolumes are not included recursively. OpenZFS documents that rollback discards changes after the selected snapshot and that some options also destroy newer snapshots, bookmarks, or clones. LVM, hypervisor, and cloud snapshots have their own consistency, quiescing, space, and dependency rules. Use the exact current documentation for the target platform rather than translating a command from another storage stack.

Before injection, restore a representative file and, for stateful services, restore the application to an alternate isolated target. Confirm permissions, ownership, database or queue consistency, keys, version compatibility, and an ordinary read. CISA recommends regular testing of backup availability and integrity; a completed backup job is not that test.

A Recovery Sequence For Every Test

1. Stop the injector. Do not troubleshoot while the filler, namespace override, read-only remount, or paused process is still changing the system.
2. Preserve evidence. Capture UTC time, unit properties, relevant journal window, mount state, capacity, resolver state, alert timeline, and client result. Prefer reads over exploratory writes.
3. Apply the prepared reversal. Reset the synthetic unit, remove the bounded payload, remount only the lab filesystem, remove the canary namespace or resolver override, or resume/restart the watchdog canary.
4. Validate storage and state. Run the application's documented integrity or recovery check. If integrity is uncertain, restore to an alternate target rather than overwriting the failed state.
5. Verify from the client. Repeat the same normal workflow captured at baseline. A green process and open port are not enough.
6. Verify monitoring. Confirm the alert closes for the right reason and that independent heartbeats resume. Watch for a false clear during a restart loop.
7. Compare and clean up. Diff configuration, remove fixtures, unmount and detach the loop image, delete synthetic accounts or tokens, and record unresolved drift.

After the full-disk and read-only tests are complete, clean up the loop-backed lab. The source guard prevents detaching an unexpected device.

LAB_ROOT=/var/tmp/tg-failure-lab
LAB_MNT="$LAB_ROOT/mnt"
LAB_IMG="$LAB_ROOT/ext4.img"
LOOP=$(sudo losetup --associated "$LAB_IMG" --noheadings --output NAME)

test -n "$LOOP" || exit 1
test "$(findmnt -no SOURCE --target "$LAB_MNT")" = "$LOOP" || exit 1
sudo mount -o remount,rw "$LAB_MNT" 2>/dev/null || true
rm -f "$LAB_MNT/recovered.txt" "$LAB_MNT/rw-check.txt"
sudo umount "$LAB_MNT"
sudo losetup --detach "$LOOP"
sudo rm -f "$LAB_IMG"
sudo rmdir "$LAB_MNT" "$LAB_ROOT"
sudo udevadm settle

When Recovery Fails

Stop the exercise and invoke incident handling when the target is not the one expected, management access is lost, the recovery deadline expires, production or unrelated systems react, the filesystem will not return to its expected state, the service enters a persistent restart loop, or application integrity is uncertain. Do not widen permissions, force a filesystem read-write, delete snapshots, reset a database, or repeatedly restart a stateful service just to make the dashboard green.

Use the hypervisor console or other independent management path, preserve the failed VM or volume when practical, and restore a known-good copy to an alternate isolated target. Rotate test credentials if they entered logs or artifacts. If the drill exposed real compromise, customer data, regulated records, or a contractual outage, switch from exercise notes to the organization's security, privacy, legal, and communications procedures.

What This Evidence Does Not Prove

• A synthetic exit does not prove a stateful application survives a crash without data loss.
• A 256 MiB ext4 image does not reproduce thin-pool exhaustion, inode exhaustion, quota behavior, NFS failure, cloud-volume throttling, device errors, or a full production database volume.
• A read-only application mount does not reproduce a read-only root filesystem, ext4 or XFS error handling, or corrupt media.
• PrivateNetwork=yes models complete interface isolation for the test process. It does not model packet loss, latency, partial routing, firewall asymmetry, or every DNS path.
• A watchdog timeout proves a heartbeat stopped. It does not prove the heartbeat represented end-to-end service health.
• A local OnFailure= handler does not prove an alert escaped a dead host or reached a person.
• A snapshot rollback does not prove backup isolation, clean data, acceptable recovery-point objective, or an alternate-host restore.
• Documentation review does not prove that the commands, package paths, properties, and outputs match every distribution or systemd release.
• No original TechGeeks lab was performed for this draft, so it reports no measured detection or recovery times.

Publication-Day Rechecks

• Reopen the current upstream systemd.service, systemd.unit, systemd.exec, systemd-run, systemctl, and sd_notify manuals.
• Confirm transient support for every property in the synthetic unit on the distribution used for command validation.
• Recheck current util-linux losetup, mount, and findmnt behavior, including loop cleanup and remount syntax.
• Recheck ext4/XFS quota setup against the exact distribution and filesystem version if a quota example is added.
• Reopen Btrfs, OpenZFS, hypervisor, or storage-platform snapshot and rollback documentation used by the final example.
• Recheck current CISA backup guidance and every external and TechGeeks link.
• Confirm all planned tests remain labeled unperformed unless timestamped artifacts are actually collected.
• Preview the Gutenberg model on narrow mobile and desktop widths, verify all details controls open, and confirm no downloadable Quick Reference card or affiliate block was added.

Related TechGeeks Reading

• Linux and Homelab Notes: Start Here
• Homelab Backup Strategy: Restore Tests, NAS, and Offsite Copies
• RAID, ZFS, Snapshots, Sync, and Backup: What Each Layer Protects
• Monitoring and Health Checks for a Plex and Arr Homelab

References

• systemd.service: restart policy and watchdog behavior
• systemd.unit: OnFailure and start-rate limits
• systemd.exec: PrivateNetwork and execution isolation
• systemd-run: transient service properties and wait behavior
• systemctl: status inspection and reset-failed
• sd_notify: watchdog notification protocol
• util-linux losetup manual
• util-linux mount manual
• util-linux findmnt manual
• Linux kernel quota subsystem documentation
• Red Hat Enterprise Linux 9: filesystems, read-only roots, and quotas
• Btrfs subvolume and snapshot documentation
• OpenZFS snapshots, clones, and rollback behavior
• CISA StopRansomware Guide: backup isolation and restore testing
• Google SRE: test-induced emergencies and rollback lessons
• AWS Fault Injection Service: steady state and stop conditions

Last technical source review: August 15, 2026. This draft remains documentation-backed; no TechGeeks failure-injection lab or timed recovery exercise has been performed.