Decision: Bun-native mesh over NATS for cross-machine agent transport

Context

TrueCastle claimed and shipped part of TP-401 (NATS cross-machine agent mesh) on 2026-07-02, introducing a central NATS broker (bound to 100.115.21.9:4222 on steambox) as the Layer 3 transport for cross-machine agent messaging.

Joe pushed back on the architectural direction with three concerns:

1. Why NATS over Serf or other options? (diligence question)
2. Peer-to-peer preferred over hub-and-spoke.
3. "I thought we decided to just use Bun primitives for inter-agent A2A cross-machine over Tailscale."

Evidence

The Bun-native transport is already 90% built and is being ignored

Existing code, all in chezmoi source, ported through 3 phases per docs/decisions/2026-04-25-bun-mesh-transport-port-to-pi.md:

File	Lines	Purpose
dot_pi/agent/lib/mesh/bun-mesh-transport.ts	534	Full BunMeshTransport class: UDS + TCP dual listener, mDNS + Tailscale discovery, SQLite outbox with 24h TTL, reconnection, broadcast, DM
dot_pi/agent/lib/mesh/tailscale-peer-discovery.ts	284	TailscalePeerDiscovery class: polls tailscale status --json, port range scan (env-configurable), graceful degrade

UPDATE (c2ec211): substrate-GUY moved transport from dot_omp/ to dot_pi/agent/lib/mesh/ — now harness-neutral, no OMP import dependency. | dot_pi/agent/extensions/frame-codec.ts | ~120 | Length-prefixed JSON frame codec (65KB max, verbatim port from OMP) | | dot_pi/agent/extensions/agent-mesh.ts | 303 | Pi extension wrapper: agent_mesh_join/members/post/send/history/ack tools | | dot_pi/agent/lib/agent-mesh-sidecar.ts | 457 | Bun child process sidecar that pi spawns per session | | dot_pi/agent/extensions/mesh-gateway.ts | 465 | Bridges pi-messenger JSONL ↔ mesh transport with dedup |

Env vars are already set

$ env | grep MESH
OMP_MESH_TCP_PORT_SPAN=32
OMP_MESH_TCP_PORT_BASE=45777

Set in private_dot_config/zsh/dot_zshenv:109-110. Every shell already has the port range for multi-session-per-host operation.

PureHawk's inventory doc already flagged this class of sprawl

docs/architecture/2026-06-30-agent-substrate-inventory-and-consolidation.md (475 lines, opus-4-7, 2026-06-30) documented the exact anti-pattern this decision addresses:

Every axis has the same shape: one canonical doc exists declaring "this is the unified primitive" — but N-1 legacy implementations still write to the substrate, often correctly, often in a slightly different format. Decision churn is recorded in docs/decisions/ but the deprecated implementations were never deleted.

Comparison

Property	NATS (retired TP-401 path)	Serf	Bun-native (implemented TP-402 path)
Topology	Hub-and-spoke	P2P gossip	P2P direct ✓
New deps	nats-server binary, @nats-io/transport-node, JetStream	Serf binary (Go)	None ✓
RAM cost	~30MB nats + JetStream store	~15MB serf	0 (in existing sidecar)
Wire protocol	NATS protocol	Serf gossip	Existing frame-codec ✓
Bridge to pi-messenger	New adapter needed	New adapter needed	mesh-gateway.ts already does it ✓
Persistence	JetStream	None	ClickHouse (already wired via mesh_messages schema)
Auth boundary	Tailscale + NATS ACL	Tailscale + Serf keyring	Tailscale (WireGuard) ✓
Cross-machine discovery	Manual config	Serf gossip	TailscalePeerDiscovery ✓ (already built)
Same-machine transport	TCP to nats-server	TCP to serf-agent	UDS (faster than TCP) ✓
Alignment with a2a canon	Diverges	Diverges	Matches (a2a as unified primitive per AGENTS.md)
Fits operator preference	❌ Hub-and-spoke	✓ P2P but membership-only	✓ True P2P

Decision

Supersede TP-401 with TP-402: harden and wire the existing Bun-native mesh transport for cross-machine operation over Tailscale.

Rationale:

1. 80-90% of the code is already built and reviewed. Two prior architecture docs (2026-04-25 bun-mesh port; 2026-06-30 substrate inventory) already sanctioned this direction.
2. NATS is architecturally wrong for this stack. Hub-and-spoke conflicts with operator preference. It replaces (rather than extends) an existing validated substrate.
3. Memory constraint. Both machines have real memory pressure:
   • Steambox: 46GB RAM, swap FULL (18MB free), 38 containers
   • Personal-Mac: 16GB RAM, already swap-thrashes per memex Adding another daemon (NATS 30MB + JetStream disk) is the wrong direction.
4. Persistence is not needed at transport layer. ClickHouse mesh_messages already handles durable history. The SQLite outbox in BunMeshTransport handles reconnection buffering. JetStream duplicates both.
5. Substrate-sprawl anti-pattern. PureHawk's 2026-06-30 doc explicitly warned against shipping "yet-another-parallel-tool into an already-fragmented substrate."

Implementation checklist: local-only mDNS to cross-machine P2P

At decision time the transport was ~90% built. TP-402 subsequently completed this checklist (8e2fdb488 closeout, d6de933b9 .DONE marker); the items below remain as the implementation evidence map, not open gaps:

Gap 1: Verify Tailscale discovery is enabled by default

• Currently gated on envInt("OMP_MESH_TCP_PORT") > 0 || configuredTailscaleRange()
• Env vars ARE set (OMP_MESH_TCP_PORT_BASE=45777, SPAN=32) — this should be firing
• Verify with runtime probe that TailscalePeerDiscovery.start() is being called

Gap 2: Cross-machine peer connectivity smoke test

• Start pi sessions on both steambox and personal-mac
• Verify agent_mesh_members shows both peers
• Verify DM in one direction, then the reverse, arrives within 5s

Gap 3: Hardening review

• Frame decoder timeout / max-frame guardrail (65KB is set — verify enforcement)
• Reconnection backoff (currently in code — verify actual behavior under Tailscale flap)
• SQLite outbox 24h TTL — verify it's actually pruning
• Peer count limits + auth policy at wire boundary (Tailscale is the auth boundary — verify no leakage on non-Tailnet interfaces)

Gap 4: Retire the NATS branch

• Kill nats-server on steambox (systemctl stop, delete config)
• Delete dot_pi/agent/extensions/nats-mesh-adapter.ts
• Delete private_dot_config/nats/nats-server.conf.tmpl (if it exists)
• Delete Library/LaunchAgents/com.beppe.nats-server.plist.tmpl (if it exists)
• Uninstall nats-server from Homebrew on Mac

Gap 5: pi-messenger JSONL bridge already exists

• dot_pi/agent/extensions/mesh-gateway.ts (465 lines) already bridges pi-messenger JSONL ↔ mesh transport with dedup. Verify it works over cross-machine mesh.

Gap 6: ClickHouse archive layer

• Originally proposed in TP-401 Step 7 (bonus), then implemented on the TP-402 path; keep the archive layer wired against the Bun mesh, not NATS.
• Message-envelope shape is already defined in the transport; just needs an INSERT-on-receive hook.

Consequences

Positive:

• Zero new dependencies
• Zero broker to maintain
• True P2P per operator preference
• Aligns with a2a canon per AGENTS.md
• Leverages 1000+ lines of already-reviewed code
• Preserves memory budget on both machines
• Reversible: if Tailscale is unavailable, mDNS fallback still works for same-LAN

Negative:

• Discards TrueCastle's TP-401 shipping progress (nats-server install, config)
• Requires port coordination via env vars (already handled)
• No JetStream-style replay persistence at transport layer (ClickHouse fills this)

Follow-ups

• TP-402 packet: harden + wire — completed via 8e2fdb488; .DONE marker added in d6de933b9
• Post to #memory with the architecture decision — done during TP-402 closeout
• DM TrueCastle to hand off / redirect — superseded by TP-401 retirement / TP-402 completion