Pool Recovery Runbook

Purpose: Step-by-step procedures for recovering from pool disconnection, failure, or data corruption. Last Updated: 2026-01-04

Quick Reference

Scenario	First Action	Recovery Time
Clean hot-unplug (drives removed)	Reconnect drives, wait for auto-upgrade	2-3 min
Stale mount (I/O errors)	`sudo systemctl restart mergerfs-pool.service`	1-2 min
SAB final-dir `ENOSPC` with free pool	Check mergerfs category schema before retrying	5-15 min
Partial disconnect (some drives)	Check connections, restart mergerfs	2-3 min
Corrupt files after crash	Run integrity checks, restore from backup	15-60 min

Scenario 1: Clean Hot-Unplug Recovery

Symptoms: Drives were disconnected while system was idle or after graceful drain.

Steps

Reconnect drive bays (USB dock/enclosure)

Wait for auto-detection (30-60 seconds)

bash

# Check if drives are detected
ls /dev/disk/by-label/Fast*

Verify pool mount

bash

# Check mergerfs status
sudo systemctl status mergerfs-pool.service

# If not started, start it
sudo systemctl start mergerfs-pool.service

Check pool health

bash

./scripts/pool-health-monitor.sh --status

Auto-upgrade should trigger (if enabled)
- Watch for desktop notification: "Media Stack Upgraded"
- Check logs: tail -f /tmp/media-stack/pool-health.log
Manual upgrade if needed
bash
```
./scripts/smart-start.sh restart
```
1

Scenario 2: Stale FUSE Mount

Symptoms: Containers show I/O errors, ls /var/mnt/pool hangs, but drives are connected.

Steps

Force unmount stale mergerfs
bash
```
sudo umount -l /var/mnt/pool
```
1

Restart mergerfs service

bash

sudo systemctl restart mergerfs-pool.service

Restart pool-dependent containers
bash
```
./scripts/restart-pool-containers.sh
```
1

Verify container health

bash

docker compose ps
docker logs sonarr --tail 20
docker logs plex --tail 20

Scenario 3: SAB final-dir `ENOSPC` under mergerfs `epmfs`

Symptoms: SAB post-processing fails with OSError: [Errno 28] No space left on device while df -h /var/mnt/pool still shows aggregate free space. Example: Gachiakuta failed creating /pool-downloads/tv/... even though the pool had several TB free.

Root cause to check: the deployed pool uses category.create=epmfs, moveonenospc=true, and minfreespace=50G. With existing-path create policy, a sparse category schema can strand a category on one branch; minfreespace then becomes the co-factor that excludes that branch from new writes. If downloads/tv exists only on /var/mnt/Fast_4TB_2 and that branch has less than minfreespace, SAB can fail even when other branches have room.

Evidence capture

bash

# Capture deployed policy from source; mountinfo may hide mergerfs extended options
rg -n "category.create|minfreespace|moveonenospc" deploy/systemd/mergerfs-pool.service
findmnt -T /var/mnt/pool/downloads/tv -o TARGET,SOURCE,FSTYPE,AVAIL,OPTIONS

# Build a branch/category presence matrix for the affected category
for d in /var/mnt/Fast_*/downloads/tv; do
  [ -d "$d" ] && printf '%s\t' "$d" && df -h "$d" | tail -1
done

Recovery rule

Do not run SAB retry_all first. A retry can fail again or churn the same full branch.
Validate the payloads before deciding they are junk. For RAR remnants, prefer SAB retry/post-processing if unrar t succeeds.
Standardize the SAB category directory across eligible pool branches (downloads/tv, downloads/movies, downloads/anime, etc.) with matching owner/mode, or intentionally change the mergerfs create policy. Do this as a documented schema change, not ad hoc per release.
Retry one history item through SAB as a pilot and confirm history.status=Completed plus output under /var/mnt/pool/downloads/<category>/... before retrying a batch.

Reference artifacts and baseline policy docs:

Storage and Remote Access — baseline mergerfs policy and runtime introspection caveat.
/home/deck/.local/share/stash-curation/mergerfs-sab-category-schema-matrix-2026-07-02T2210Z.json — category × branch schema evidence.
/home/deck/.local/share/stash-curation/mergerfs-schema-policy-review-2026-07-02T2214Z.json — policy review and Gachiakuta pilot outcome.

Scenario 4: Partial Drive Failure

Symptoms: mergerfs reports degraded state, some drives missing.

Steps

Identify missing drives

bash

./scripts/pool-health-monitor.sh --status

# Check physical connections
ls /dev/disk/by-label/Fast*
lsblk

Check drive health

bash

# For each connected drive
sudo smartctl -a /dev/sdX

If drive is physically disconnected
- Reconnect and restart mergerfs
- Pool will resync automatically
If drive has SMART errors
- Back up data from that drive immediately
- Replace drive
- Run btrfs scrub on replacement

Scenario 5: Data Corruption After Crash

Symptoms: Files won't open, services report database errors, unexpected EOF.

Immediate Actions

Stop all services

bash

docker compose down
docker compose -f docker-compose.reading.yml down

Check filesystem integrity

bash

# For each btrfs drive in the pool
sudo btrfs check --readonly /dev/disk/by-label/Fast_4TB_1
# Repeat for all drives

Check for orphaned/corrupt downloads

bash

# Find incomplete downloads
find /var/mnt/pool/downloads/incomplete -type f -mmin +60

# Find zero-byte files (corruption indicator)
find /var/mnt/pool -type f -size 0 -name "*.mkv" -o -name "*.mp4"

Database Recovery

SQLite databases (Sonarr, Radarr, etc.) are stored on native btrfs (/var/mnt/fast8tb/config/), NOT on mergerfs. They should be safe, but verify:

bash

# Check database integrity
sqlite3 /var/mnt/fast8tb/config/sonarr/sonarr.db "PRAGMA integrity_check;"
sqlite3 /var/mnt/fast8tb/config/radarr/radarr.db "PRAGMA integrity_check;"
sqlite3 /var/mnt/fast8tb/config/komga/database.sqlite "PRAGMA integrity_check;"

If database is corrupt:

bash

# Restore from backup (configs are backed up to OneDrive)
cp /path/to/backup/sonarr.db /var/mnt/fast8tb/config/sonarr/sonarr.db

Media File Recovery

Check Plex for missing files
- Open Plex web UI
- Settings → Troubleshooting → "Optimize Database"
- Settings → Manage → Libraries → "Empty Trash" (don't auto-empty!)

Rescan libraries

bash

# Plex full rescan
curl -X POST "http://localhost:32400/library/sections/all/refresh?X-Plex-Token=$PLEX_TOKEN"

# Komga rescan
curl -X POST "http://localhost:8081/api/v1/libraries/scan" \
  -H "Authorization: Bearer $KOMGA_TOKEN"

Check for partial transcodes

bash

# Tdarr leaves temp files
find /var/mnt/pool -name "*.tmp" -o -name "*_TDARR_*"

# Delete if corrupt
rm -f /var/mnt/pool/media/movies/*_TDARR_*.mkv

Scenario 6: Complete Pool Loss

Symptoms: All drives failed or catastrophic data loss.

Immediate Actions

Stop panicking - configs are on internal NVMe, not pool
Stop all services to prevent further issues
Assess damage:
- Are drives physically damaged or just disconnected?
- Is data recoverable?

If Drives Are Recoverable

Reconnect one drive at a time
Check each with btrfs check --readonly
Mount read-only to assess data
Reconstruct pool gradually

If Data Is Lost

Configs are safe - stored on /var/mnt/fast8tb/config/
Watch history is safe - in Plex/Komga databases (also on internal drive)
Media must be re-acquired - use Sonarr/Radarr to search again

Restoring From Backup

bash

# If you have restic/borg backups
restic -r /path/to/backup restore latest --target /var/mnt/pool

# If restoring from OneDrive
rclone copy onedrive:Backups/media-stack /var/mnt/pool

Prevention Checklist

[ ] Pool health monitor running (systemctl --user status pool-health-monitor)
[ ] Graceful drain enabled (check graceful stop on disconnect)
[ ] Configs on internal NVMe, not pool
[ ] Regular config backups (to OneDrive or external)
[ ] SMART monitoring for drive health
[ ] UPS for clean shutdown during power loss

State Files Reference

Location: /tmp/media-stack/

File	Purpose
`stack-mode`	Current mode: `full`, `local`, `pool-degraded`
`pool-state`	Pool health: `healthy`, `degraded`, `stale`, `unmounted`
`pool-health.log`	Event log for debugging
`pool-degraded`	Timestamp of last degradation
`stack-started`	Timestamp of last stack start
`start-method`	How stack was started: `autostart`, `manual`, `auto-upgrade`

Boot and Launchers - Mode detection and startup
Storage Architecture - mergerfs pool design
Reading Stack - Portable mode services

Pool Recovery Runbook ​

Quick Reference ​

Scenario 1: Clean Hot-Unplug Recovery ​

Steps ​

Scenario 2: Stale FUSE Mount ​

Steps ​

Scenario 3: SAB final-dir ENOSPC under mergerfs epmfs ​

Evidence capture ​

Recovery rule ​

Scenario 4: Partial Drive Failure ​

Steps ​

Scenario 5: Data Corruption After Crash ​

Immediate Actions ​

Database Recovery ​

Media File Recovery ​

Scenario 6: Complete Pool Loss ​

Immediate Actions ​

If Drives Are Recoverable ​

If Data Is Lost ​

Restoring From Backup ​

Prevention Checklist ​

State Files Reference ​

Related Documentation ​

Pool Recovery Runbook

Quick Reference

Scenario 1: Clean Hot-Unplug Recovery

Steps

Scenario 2: Stale FUSE Mount

Steps

Scenario 3: SAB final-dir `ENOSPC` under mergerfs `epmfs`

Evidence capture

Recovery rule

Scenario 4: Partial Drive Failure

Steps

Scenario 5: Data Corruption After Crash

Immediate Actions

Database Recovery

Media File Recovery

Scenario 6: Complete Pool Loss

Immediate Actions

If Drives Are Recoverable

If Data Is Lost

Restoring From Backup

Prevention Checklist

State Files Reference

Related Documentation