Storage Architecture & Remote Access

Last Updated: 2025-12-29 Status: Production configuration, boot-persistent

This document covers the complete storage setup and remote access configuration for the media stack.

Storage Overview

Storage Tier	Mount Point	Size	Purpose
Primary NVMe	`/var/mnt/fast8tb`	8TB	Config, OneDrive sync, high-speed access
MergerFS Pool	`/var/mnt/pool`	41TB	Media library (movies, TV, anime)
Individual NVMe	`/var/mnt/Fast_*`	4-8TB each	Pool members (8 drives)

MergerFS Configuration

What is MergerFS?

MergerFS creates a union filesystem that presents multiple drives as a single mount point. Files are distributed across drives based on policies, but appear as one unified directory.

/var/mnt/pool/           <-- Single unified view (41TB)
    ├── movies/
    ├── tv/
    └── anime-tv/
         │
         ├── Actually on /var/mnt/Fast_4TB_1
         ├── Actually on /var/mnt/Fast_8TB_2
         └── Actually on /var/mnt/Fast_4TB_3

Pool Members (8 Drives)

Drive	Size	Mount Point
Fast_4TB_1	4TB	`/var/mnt/Fast_4TB_1`
Fast_4TB_2	4TB	`/var/mnt/Fast_4TB_2`
Fast_4TB_3	4TB	`/var/mnt/Fast_4TB_3`
Fast_4TB_4	4TB	`/var/mnt/Fast_4TB_4`
Fast_4TB_5	4TB	`/var/mnt/Fast_4TB_5`
Fast_8TB_1	8TB	`/var/mnt/Fast_8TB_1`
Fast_8TB_2	8TB	`/var/mnt/Fast_8TB_2`
Fast_8TB_3	8TB	`/var/mnt/Fast_8TB_3`

Mount Command

bash

mergerfs -o defaults,allow_other,use_ino,cache.files=auto-full,dropcacheonclose=false,category.create=epmfs,moveonenospc=true,minfreespace=50G,fsname=mergerfs-pool \
  /var/mnt/Fast_4TB_1:/var/mnt/Fast_4TB_2:/var/mnt/Fast_4TB_3:/var/mnt/Fast_4TB_4:/var/mnt/Fast_4TB_5:/var/mnt/Fast_8TB_1:/var/mnt/Fast_8TB_2:/var/mnt/Fast_8TB_3 \
  /var/mnt/pool

Mount Options Explained

Option	Value	Why It Matters
`cache.files`	`auto-full`	CRITICAL: Uses Linux page cache for file data. Without this, every read hits disk, causing massive CPU overhead.
`dropcacheonclose`	`false`	CRITICAL: Keeps cached data after file close. Set to `true` caused 286% CPU usage!
`category.create`	`epmfs`	Existing-path, most-free-space policy. Keeps release/category trees together for hardlink/import locality, but category directories must exist on eligible branches.
`moveonenospc`	`true`	Auto-migrate files if destination drive is full during write.
`minfreespace`	`50G`	Reserve 50GB per drive before considering it "full".
`allow_other`	-	Docker containers (running as different users) can access the mount.
`use_ino`	-	Preserve inode numbers for hardlink support.
`fsname`	`mergerfs-pool`	Friendly name in `df` and `mount` output.

Runtime caveat: mountinfo/findmnt may show only the generic FUSE options (rw,user_id=0,group_id=0,default_permissions,allow_other) and omit mergerfs extended options. Verify intended policy from deploy/systemd/mergerfs-pool.service; if behavior contradicts the unit, rebuild the runtime evidence before changing categories or retrying failed downloads.

Performance Impact of Caching

Setting	CPU Usage	I/O Latency	Notes
`cache.files=off`	286%	High	Every operation hits disk
`cache.files=auto-full`	0-5%	Low	Linux page cache handles reads

On a 96GB RAM system, the page cache can hold 40-60GB of frequently accessed file data, eliminating most disk reads.

Boot Persistence (Systemd Service)

The pool auto-mounts on boot via systemd:

bash

# Service file location
/etc/systemd/system/mergerfs-pool.service

# Check status
systemctl status mergerfs-pool.service

# Manual control
sudo systemctl start mergerfs-pool.service
sudo systemctl stop mergerfs-pool.service

Service Dependencies: Requires all 8 drive mounts to be ready before starting.

Docker Volume Mapping

All containers access media through the pool:

yaml

# In docker-compose.yml
volumes:
  - /var/mnt/pool/movies:/movies
  - /var/mnt/pool/tv:/tv
  - /var/mnt/pool/anime-tv:/anime-tv
  - /var/mnt/pool/downloads:/downloads

Why /pool not /var/mnt/pool?

Container paths are simplified. The host path /var/mnt/pool/movies maps to /movies inside containers. This makes *arr app configuration cleaner.

Remote Access with Tailscale

The Problem

ISP (Live Oak Fiber) rotates external IP addresses aggressively. Port forwarding and DDNS solutions break frequently.

The Solution

Tailscale provides a mesh VPN with stable IPs:

Server gets fixed IP: 100.115.21.9
Works regardless of ISP IP changes
More secure than port forwarding (no public exposure)
WireGuard encryption end-to-end

How It Works

┌─────────────────────────────────────────────────────────────┐
│                     Tailscale Network                       │
│                                                             │
│   ┌──────────────┐  WireGuard   ┌──────────────┐           │
│   │ Media Server │◄────────────►│ Your Phone   │           │
│   │ 100.115.21.9 │   (encrypted)│ 100.x.x.x    │           │
│   └──────────────┘              └──────────────┘           │
└─────────────────────────────────────────────────────────────┘

Accessing Services

From any Tailscale-connected device, use the landing pages first:

Tier	URL
Public/friends	https://joe-steambox.alai-opaleye.ts.net
Admin/adult	https://joe-steambox.alai-opaleye.ts.net:8443

Direct service links generally use Tailscale Serve HTTPS:

Service	URL
Plex	https://100-115-21-9.a45a678d4702421bbeb6c7623deb3eff.plex.direct:32400/web
Sonarr	https://joe-steambox.alai-opaleye.ts.net:8989
Radarr	https://joe-steambox.alai-opaleye.ts.net:7878
SABnzbd	https://joe-steambox.alai-opaleye.ts.net:8180
Prowlarr	https://joe-steambox.alai-opaleye.ts.net:9696
Tdarr	https://joe-steambox.alai-opaleye.ts.net:8265
Overseerr	https://joe-steambox.alai-opaleye.ts.net:5055

Exception: Transmission/Gluetun currently remains direct-tailnet only at http://100.115.21.9:9092 because port 9092 has no Tailscale Serve mapping.

Plex Configuration

In Plex Settings → Network:

Custom server access URLs: https://100-115-21-9.a45a678d4702421bbeb6c7623deb3eff.plex.direct:32400
LAN Networks: 100.64.0.0/10 (Tailscale CGNAT range)
Secure connections: "Preferred" or "Required" once clients are confirmed compatible
Remote access: Can disable traditional port forwarding when using Tailscale exclusively

Security Comparison

Aspect	Port Forwarding	Tailscale
Internet exposure	Yes (public)	No (mesh only)
Brute force risk	High	Very low
ISP dependency	Static IP/DDNS	None
Encryption	Per-service TLS	Always WireGuard
Setup complexity	Router config	Install & login

Adding Devices to Tailscale

bash

# On any device
tailscale up

# Check network status
tailscale status

Troubleshooting

MergerFS High CPU

Symptom: mergerfs process using 200%+ CPU

Cause: Caching disabled (cache.files=off or dropcacheonclose=true)

Fix:

bash

# Unmount and remount with correct options
sudo umount /var/mnt/pool
sudo systemctl start mergerfs-pool.service

Pool Not Mounting on Boot

Check: Are the underlying drives mounted first?

bash

systemctl list-units --type=mount | grep Fast

Fix: The systemd service requires all drive mounts. Ensure drives are in fstab or auto-mounted.

Containers Can't Access Pool

Symptom: Permission denied or empty directories in containers

Check:

bash

# Verify pool is mounted
mount | grep mergerfs

# Check permissions
ls -la /var/mnt/pool/

Fix: Ensure allow_other is in mount options.

Stale FUSE Mounts After Pool Remount

Symptom: ENOTCONN: socket is not connected or Transport endpoint is not connected errors

Cause: Containers started before MergerFS was mounted, or pool was remounted while containers were running. The containers have "zombie" references to the old mount point.

Fix: Use the restart script to fix all affected containers:

bash

# Check which containers have stale mounts
./scripts/restart-pool-containers.sh --check

# Restart all pool-dependent containers
./scripts/restart-pool-containers.sh

The script will:

Verify MergerFS is mounted
Restart SABnzbd, Transmission, Tdarr, *arr apps, Plex, Komga, etc.
Verify all mounts are healthy after restart

Prevention: The systemd service (systemd/mergerfs-pool.service) includes an ExecStartPost that automatically restarts containers after MergerFS mounts.

Verification Commands

bash

# Pool status
df -h /var/mnt/pool

# Cache settings
getfattr -n user.mergerfs.cache.files /var/mnt/pool/.mergerfs

# MergerFS process (should be ONE, low CPU)
ps aux | grep mergerfs | grep -v grep

# RAM cache usage
free -h

# Tailscale status
tailscale status

Configuration Files

File	Purpose
`systemd/mergerfs-pool.service`	Boot persistence (copy to `/etc/systemd/system/`)
`scripts/restart-pool-containers.sh`	Fix stale mounts after pool remount
`docker-compose.yml`	Container volume mappings
`.env`	Pool path variables

Storage Architecture & Remote Access ​

Storage Overview ​

MergerFS Configuration ​

What is MergerFS? ​

Pool Members (8 Drives) ​

Mount Command ​

Mount Options Explained ​

Performance Impact of Caching ​

Boot Persistence (Systemd Service) ​

Docker Volume Mapping ​

Remote Access with Tailscale ​

The Problem ​

The Solution ​

How It Works ​

Accessing Services ​

Plex Configuration ​

Security Comparison ​

Adding Devices to Tailscale ​

Troubleshooting ​

MergerFS High CPU ​

Pool Not Mounting on Boot ​

Containers Can't Access Pool ​

Stale FUSE Mounts After Pool Remount ​

Verification Commands ​

Configuration Files ​

Storage Architecture & Remote Access

Storage Overview

MergerFS Configuration

What is MergerFS?

Pool Members (8 Drives)

Mount Command

Mount Options Explained

Performance Impact of Caching

Boot Persistence (Systemd Service)

Docker Volume Mapping

Remote Access with Tailscale

The Problem

The Solution

How It Works

Accessing Services

Plex Configuration

Security Comparison

Adding Devices to Tailscale

Troubleshooting

MergerFS High CPU

Pool Not Mounting on Boot

Containers Can't Access Pool

Stale FUSE Mounts After Pool Remount

Verification Commands

Configuration Files