The complete feature inventory.

Why this article exists

After several research passes, the gmux ecosystem has grown faster than any single document captures. The PyPI package, the Tauri app, the Cloudflare Worker, the gesture engine, the voice daemon, the phone app prototypes, the desktop integration bridges, the MCP brain layer — they're all real, they all interact, and they all live in different repos.

This article is the complete map. Read it before adding a feature; you'll likely find it already exists. Read it before pitching the product; you'll find more is built than you remember.

The three repos

The "gmux project" is actually a federation of repositories that mirror each other through a manual sync pipeline:

Plus three satellite repos:

Part 1 · Shipped & working features

These features work today. If you launch gmux right now, you get all of these.

1.1 Terminal & tmux integration

The base layer. Everything else assumes this works.

Session management

• Creates a named tmux session from a generated gmuxtest.tmux.conf (or gmux.tmux.conf)
• Auto-sources your existing ~/.tmux.conf first — your TPM plugins, your colours, your bindings all preserved
• Status bar overrides applied after TPM has loaded — no plugin clobbering
• automatic-rename off and allow-rename off enforced — custom project names persist forever
• Session name configurable via $GMUX_SESSION or --session CLI flag

Status bar — the main visible feature

Called by tmux every 1 second via pane_status.py. Three calling conventions handle different parts of the bar:

• right [session] — the right-side compact attention summary
• session window_index — inactive tab format
• session window_index active — active tab format (brighter highlight)

Status-left: A purple gmux brand badge with session name in bg=#6c5ce7.

Status-right (attention-only): !3 ●6 ✋ 📷 🎤 — window numbers showing permission, waiting, error states; plus service indicators (camera ✋ gesture 📷 voice 🎤).

Window tabs: Each tmux window renders as N:ICON tool_suffix name todo_sfx. Real example:

6:◉:bash ai diary 3/5

That tab is window 6, agent is working, running bash, project name is "ai diary", 3 of 5 todos done.

Seven state icons with strict colour rules

Multi-pane priority

A tmux window can hold multiple panes (a fish shell next to a qalcode2 next to a logs tail). The tab shows the most urgent state across all panes. Priority order: error → permission → sub_permission → waiting → working → done → not_started → idle. This prevents the bug where a fish pane shows "idle" and hides the qalcode2 next to it that's waiting for permission.

tmux keybindings (all use Ctrl+A prefix)

1.2 AI state detection — the killer feature

This is what makes the status bar meaningful. gmux doesn't pattern-match terminal output. It reads each AI agent's own state directly from its HTTP API.

SSE-based real-time event stream

One SSE listener thread per running qalcode2/opencode instance. Connects to http://127.0.0.1:{port}/event?directory={cwd} and reconnects every 5 seconds on disconnect.

Events monitored

Agents supported

• QalCode2 / OpenCode — full SSE support, all 7 states
• Claude Code — detected, but no SSE (working / idle only)
• Aider — detected, no SSE
• Any bun-based process on a local port — basic detection

Port discovery

Bun starts opencode on a random local port. Discovery:

1. pgrep -P <shell_pid> bun → bun process ID
2. ss -tlnp | grep pid={bpid} → extract 127.0.0.1:PORT
3. Two-level tree walk handles shell → bun (direct) and shell → sh → bun (fish shell)
4. Cached 30 seconds per pane (handles bun restarts)

OpenCode 1.4 compatibility

All endpoints now require ?directory=<cwd> query parameter. monitor.py handles both old format (/session/status returns list) and new format (returns {sessionID: {type}} record).

Sub-agent (Task tool child) detection

When permission.updated fires, the monitor does an async HTTP GET /session/{sessionID} to check the parentID field. If present, this is a sub-agent and the tab displays ^! instead of !. You can see at a glance: is the main agent waiting for permission, or a child it spawned?

Aggregator thread (every 10 seconds per pane)

• Hits /session/:id/message and sums: token_in, token_out, token_reasoning, cache_read, cache_write, cost_usd
• Captures model name from latest assistant message
• Refreshes todos from /session/:id/todo

1.3 Gesture control

The browser-side gesture engine is fully working. Backend desktop bridges (KDE/GNOME/Hyprland) work but are opt-in.

MediaPipe hand landmarker

• Local model first — 7.5MB hand_landmarker.task ships with the project, no CDN required
• CDN fallback if local missing
• Up to 2 hands tracked simultaneously
• 21 landmarks per hand
• ~60 fps in browser

Eight static gestures (geometric classification)

Four motion gestures

Detected from wrist velocity averaged over 5 frames:

• SWIPE_RIGHT / SWIPE_LEFT (horizontal velocity dominant)
• SWIPE_UP / SWIPE_DOWN (vertical velocity dominant)

Hand role assignment

• Right hand = navigation (swipe to switch panes, pinch+drag to scroll)
• Left hand = commands (point = voice toggle, thumbs up = approve, three fingers = jump red)

Confirmation model (anti-twitch)

• Same gesture must be detected 3 consecutive frames before action fires
• 300ms cooldown after firing
• 800ms cooldown between same gesture re-firing
• Eliminates single-frame false positives

Two modes

• Passive (default while typing) — threshold 0.90 confidence, swipes blocked
• Active (deliberate navigation) — threshold 0.72, all gestures accepted

Mode switches via OPEN_PALM held 1.5 seconds.

Visual feedback

• Purple border around terminal in gesture mode
• Hand skeleton overlay — right hand cyan, left hand purple
• Gesture pill at top-center: → win 3 or ✋ SWIPE_RIGHT with slide-in animation
• Gesture HUD bottom-left: gesture name + confidence ring + hand position
• Live webcam preview bottom-right with skeleton drawn on it
• All toggleable in Settings panel

Dwell selection

Session pills register as dwell targets. Hold a fingertip over a pill for ~1 second; a fill bar animates from 0% to 100% width across the bottom of the pill, then selects. Useful for projector mode where you can't easily pinch.

Calibration

• Live webcam feed + skeleton displayed in calibration view
• Per-gesture confidence bars updating in real time
• Threshold sliders
• Settings saved to ~/.config/gmuxtest/gesture_calibration.json
• Projector mode: 4-corner homography calibration with CalibrationTransform class

1.4 Voice control

Two voice paths, auto-selected based on what's available.

Path 1 · faster-whisper daemon (local, offline)

• gmux_voice_daemon.py running on port 8770
• Model options: tiny / base / small / medium
• Pipeline: mic → energy-based VAD → 200ms chunk buffer → faster-whisper → WebSocket → UI
• 16 kHz sample rate, 0.008 RMS silence threshold, 0.8s silence gap before transcribe
• Max buffer 15 seconds before forced transcription
• Whisper settings: beam_size 5, vad_filter True, word_timestamps False
• CLI flags: --model, --device, --port, --lang, --continuous, --compute, --list-devices

Path 2 · Web Speech API (browser native)

• Chrome / Brave / Chromium native speech recognition
• interimResults = true, continuous = false
• Auto-fallback if voice daemon WebSocket fails
• Zero setup required

Wake words

kalarc, kal arc, cal arc, qalarc, qal arc, kalarck, kalark, calarc, kal-arc — multiple spellings handle STT variation. Longest match wins.

After a wake word: command routes to the active AI pane (priority: WAITING > WORKING > DONE > any AI).

Voice → tmux command map

Voice commands intercepted before reaching the AI:

Anything not matching the command map gets routed as a chat message to the focused AI agent.

Voice toggle methods

• Left-hand POINT gesture
• Both index finger tips touching (planned)
• Ctrl+Shift+Space global Tauri shortcut
• prefix + V tmux keybinding
• Push-to-talk button in chat panel

1.5 Phone remote

The HTTP server on :8768 serves a mobile-optimised PWA. The WebSocket on :8767 handles real-time control.

HTTP endpoints

WebSocket protocol

Phone → Desktop:

• voice_command — {text, window?}
• gesture_event — {static, motion, confidence, index_x, index_y, mode}
• request_layout — get pixel rectangles of all tmux panes
• select_agent — {pane_id, window_index} switches active tmux window
• status / ping

Desktop → Phone: welcome, status, pane_layout, ack, gesture_ack, pong

Pane layout API

Returns pixel rectangles for every visible tmux pane. Uses tmux list-panes for character positions plus xdotool search --name to find the terminal window's pixel dimensions. Falls back to 1920×1080 if xdotool unavailable.

Phone HTML prototypes

Three mobile design versions exist (v1 Terminal Dark, v2 Glass Cards, v3 Command Bridge). A v4 unified version is planned. None have been built as APKs yet — Capacitor is the planned build path but CLI not installed.

Volume key plan (for APK)

• Volume DOWN = cycle active agents (green border tracks selection)
• Volume UP = activate voice mode for focused agent (border turns red, STT starts)

1.6 The Tauri desktop app

The desktop window that owns the terminal. Working in dev mode; production packaging paused.

Window architecture

Dev URL: http://localhost:1421. App identifier: dev.gmux.test. Version: 0.1.0.

PTY implementation

• portable-pty Rust crate v0.8
• 40 rows × 200 cols default
• Runs tmux new-session -A -s <session> inside the PTY
• Auto-detects which session to attach to (env var or most-recent timestamp)
• PTY output streams to xterm.js via pty-data Tauri events
• Resize via pty_resize(cols, rows) Tauri command

Auto-started sidecars

The Tauri app spawns three Python processes on launch:

If a port is already bound, the sidecar is skipped (don't fight existing instances). start_with_retry() waits 1.5s, checks if the process died, then polls the port for up to 4 seconds.

State polling thread (1 Hz)

Reads multiple JSON files and emits events to all three windows simultaneously:

• /tmp/gmuxtest-pane-state.json → gmux-state event
• /tmp/gmuxtest-services.json → gmux-services event
• /tmp/ram_tracker_agents.json → gmux-ram event
• /tmp/gmuxtest-memory.json → memory-update event
• /tmp/gmuxtest-activity.json → activity-tick event
• /tmp/gmuxtest-files.json → files-update event

Wayland workaround

GDK_BACKEND=x11 required for Tauri on Wayland (WebKitGTK rendering issue). Set automatically in launch.sh.

1.7 The web UI (v3 — 6,700+ lines, single HTML file)

The interface that runs inside the Tauri WebView (or as a standalone browser app). Source of truth: gmuxtest/UI_creation_independent/v2/index.html.

Status bar source indicator

A small coloured dot at the bottom corner shows which data source is feeding the UI:

• ● tauri live — Tauri events (best path)
• ● live :8769 — HTTP+SSE from monitor
• ● live (poll) — HTTP polling fallback
• ● mock — no backend, simulated data

Agent grid

• Live pane cards with real agent names, state dots, todo progress bars
• Hardware tab per agent showing: RAM MB, CPU%, uptime, model, token_in/out, child processes, tool timeline
• Column count adjustable by two-hand spread/pinch gestures
• Drag-and-drop to reorder cards (currently swaps; insert-above planned)

Chat panel (right sidebar)

• Real conversation history fetched from OpenCode /session/:id/message
• Per-pane lazy fetch (only when panel opens)
• 3-second throttle on refresh
• sendChat() → POST to /session/{id}/prompt_async via Tauri or direct fetch
• Chat summary line shows: model + RAM + CPU + tokens
• Voice input strip with microphone waveform + level meter

Data source priority

1. Tauri events (if window.__TAURI_INTERNALS__ present)
2. HTTP+SSE on :8769 via EventSource
3. HTTP polling every 2 seconds if SSE fails
4. Mock evolution fallback (UI never looks broken even offline)

1.8 Camera architecture (v4l2loopback)

A real webcam can only be opened by one process at a time. gmux solves this with a virtual camera.

Physical webcam (/dev/video0)
       ↓
   cam-broker (ffmpeg mirror)
       ↓
Virtual cam (/dev/video2, v4l2loopback)
       ├── Browser gesture engine (getUserMedia)
       ├── Tauri calibration view
       └── Python gesture engine (if used)

Ownership rules (strictly enforced)

• gmux-ui running → Tauri WebView owns camera, Python gesture engine skipped entirely
• gmux-ui not running → Python gesture engine can read /dev/video2
• Neither ever opens /dev/video0 directly — broker's exclusive domain

1.9 Multi-agent session management

Built on tmux-resurrect with substantial extensions.

Session save/restore

tmux-resurrect saves window layout and CWDs. gmux extends with:

• qalcode2 agent re-launch — regex-detect bun run.*opencode.*src/index\.ts\s+(\S+?), extract --cwd and --agent, restart on resurrect
• Window name persistence — saved every 30s atomically to /tmp/gmuxtest-window-names.json
• Startup stagger — 0.3s between agent re-launches (prevents bun port contention)

Per-pane OpenCode session data

The aggregator captures, per pane, every 10 seconds:

• model + provider name
• token_in, token_out, token_reasoning, cache_read, cache_write
• cost_usd (currently always 0 due to OpenCode 1.1.x bug)
• msg_count
• todos
• tool_history (last 30 SSE tool events)
• current_tool while WORKING

1.10 The landing page (gmux.ai)

A static site plus a Cloudflare Worker handling votes, emails, and stats.

Worker URL

https://gmux-ai.fivelidz.workers.dev

Display counter formula

Real vote count is multiplied by ~3.2 with non-clean wobble + jitter:

display = real * 3 + floor(log(real+1) * 1.1) + (real*11+3) % 4

Drips toward target at 2 increments/hour via 15-minute cron. Always strictly increasing. Public API field named emails (not real) to obscure the formula.

Current stats (May 12, 2026)

• 26 real clicks → 82 displayed
• 12 email signups (11 real + 1 test)
• 24/27 visitors Australian
• Mostly macOS + Safari/Chrome
• ~50% mobile

Part 2 · Working but opt-in

These features work but are disabled by default. Enable them deliberately.

2.1 Desktop environment gesture bridges

The gesture engine emits events on a Unix socket (/tmp/gmux-gesture.sock). Three desktop bridges subscribe to it and translate gestures into native desktop actions.

KDE Plasma (via D-Bus)

Hyprland (via hyprctl)

Cooldown: 800ms between any two desktop gesture actions (configurable). Prevents rapid double-fires.

2.2 Projector mode

Designed for use 2-3 metres from a wall projector. Demo exists; full integration in progress.

• 2xl typography, ultra-high contrast
• Single agent full-screen at a time
• Swipe to cycle through agents
• Shows agent name, state, current tool, last few lines of output
• 4-corner homography calibration via InteractiveCalibrator

Activation: gmux start --projector

2.3 The aquarium window

A second Tauri window for decorative crab avatars (one per agent), animated to reflect each agent's state. Currently hidden by default — moved to "later" per user feedback ("drop the avatar system... I want clear documentation"). Code preserved in extras/avatar_system/.

Part 3 · The memory layer (gmux-brain)

Built. Documented. Not yet wired into production opencode.json. Highest-value 30-minute task in the entire portfolio.

3.1 The three-layer memory architecture

3.2 MCP router

Stdio MCP server speaking protocol 2024-11-05. Routes queries to the right memory layer using keyword classification (no LLM call needed):

3.3 Eight MCP tools

3.4 Context assembler

Generates a ~500-800 token context block that gets injected into each new agent pane via CLAUDE.md or AGENTS.md:

• Section 1: Graphify GRAPH_REPORT.md summary (truncated 1500 chars)
• Section 2: mempalace wake-up episodic summary (truncated 800 chars)
• Section 3: gmux workspace state (other panes, what they're working on)
• Section 4: Setup hints (what's missing, how to wire it)

Written with <!-- gmux-brain:start/end --> markers for safe replace-in-place.

3.5 MCP configuration (when wired)

{
  "mcpServers": {
    "gmux-brain": {
      "type": "stdio",
      "command": "python3",
      "args": ["/home/fivelidz/projects/gmux-brain/src/router.py"]
    }
  }
}

Part 4 · Active development

Work in progress as of May 12, 2026.

4.1 Completed this session

• Full UI/backend audit
• Backend: psutil collector (RAM/CPU/uptime/children per pane)
• Backend: OpenCode aggregator thread (model, tokens, cost, msg_count)
• Backend: real todos from /session/:id/todo
• UI: replaced all MOCK_* constants with real data
• UI: real chat history from /session/:id/message
• UI: media queries for 1024 / 768 / 480 / 2560 pixels
• Self-launch test passed: Tauri auto-starts all 3 sidecars cold
• Tauri: restart_backend, backend_health, open_dashboard, hide_dashboard commands added
• Global shortcut Ctrl+Alt+D for dashboard toggle
• #sb-backend health indicator with restart button
• gmux-brain: MCP router + assembler written, fully documented

4.2 High-priority pending

1. Token rate sparklines — verify real data feeds into sparkline infrastructure
2. Tool history — pushToolEvent() not called from real SSE path, only mock; add to applyRealState
3. Cost USD = 0 — decide path: wait for OpenCode fix, compute from token totals, or keep calcCostUsd() fallback
4. Chat panel SSE subscription — currently refetches every render (rate-limited 3s), should subscribe to stream
5. Live messages streaming — SSE for streaming chat responses into UI

Part 5 · Paused

These exist but are deliberately frozen.

5.1 Installer / packaging work — paused 2026-05-12

Frozen until five end-to-end criteria pass:

1. ./scripts/launch.sh opens Tauri app cleanly on fresh shell
2. Status sidebar shows live pane state from :8769
3. Spawning an agent via UI creates a new tmux window + opencode
4. Permission approve/reject from UI works against a real OpenCode session
5. Voice (ws://localhost:8770) connects and transcribes into UI

Specifically not to be worked on:

• install.sh (drafted but frozen)
• systemd unit file
• .desktop entry
• One-command install UX
• AUR packaging
• Release artefacts

An installer that installs something that doesn't run cleanly is worse than no installer. The installer is the first impression.

5.2 APK / Capacitor build

Three mobile HTML prototypes exist (v1 Terminal Dark, v2 Glass Cards, v3 Command Bridge). v4 unified version planned. Capacitor not installed.

5.3 WezTerm migration — research only

wezterm_investigate/ is a proof-of-concept folder. WeztermController class drafted but not integrated.

Part 6 · Abandoned

These were tried and explicitly rejected.

6.1 Option A — gmux-ghost (transparent overlay)

Abandoned in favour of Option B (full Tauri app). Reason: DOM-based pane borders couldn't align reliably with terminal pane positions; Wayland gives no API for finding another window's pixel coordinates; Web Speech API doesn't work in webkit2gtk on Linux.

6.2 Electron

Considered as desktop framework. Not pursued. Tauri + Rust is already the stack; Electron would duplicate Chromium + Node bundling.

6.3 THUMBS_DOWN gesture

The classifier rule was never implemented. The gesture literally never fires. Decision: remove from gesture map. The reject button stays clickable/pinchable in the permission UI, but the gesture binding is removed.

6.4 Fake canned replies

Removed in v3.1 refactor. Previously: fake "Processing your message..." and "Got it" responses. Now: real errors shown, no fabricated AI responses.

6.5 Avatar system (from main UI)

Extracted and archived. Reason: competed with state dot, status chip, and pane border glow for visual attention. Code preserved in extras/avatar_system/ for potential revival in the aquarium window.

Part 7 · Service topology reference

Every port, every IPC channel, every JSON file.

7.1 Network ports

7.2 IPC files

Part 8 · The honest position

A clear-eyed summary for anyone considering gmux's place in the ecosystem.

8.1 What's genuinely unique

Combined, gmux has features that no other terminal tool has:

• Live AI state aggregation across many parallel agents (not screen scraping — actual API events)
• Gesture control of terminal panes with hand role separation
• Voice command routing with wake words and pane targeting
• Phone remote with pixel-accurate pane layout
• Cross-pane state priority resolution (handles fish + qalcode2 sibling panes correctly)
• v4l2loopback camera sharing so multiple apps can use the same webcam
• Three-layer memory architecture (structural / episodic / workspace) via gmux-brain

8.2 The qalcode2 overlap (honest)

Many "headline" gmux features are surfacing data that qalcode2 generates:

What gmux uniquely adds:

• Multi-pane view — qalcode2 is single-session; seeing 10 agents at once is gmux-only
• Gesture / voice / phone — qalcode2 has none of these
• Cross-agent status bar — only meaningful across multiple sessions
• Camera virtualisation — outside qalcode2's scope entirely

qalcode2 is the single-agent executor; gmux is the multi-agent interaction layer.

8.3 The distribution gap

Everything is built. The codebase is large and capable. But:

• The Tauri app is not yet packaged (.deb / .AppImage / .msi all deferred)
• AUR PKGBUILD doesn't exist yet
• The mobile APK is three HTML prototypes deep
• gmux-brain is unwired

The 30-minute action that would unlock the most value: register gmux-brain in opencode.json, wire graphify + kalarc-memory, restart the agent sessions, watch every pane get ~600 tokens of smart context for free. The 5-criterion checklist for installer resumption is well-defined and three of five are already green.

Article generated: May 12, 2026 from full codebase audit across 6 repositories. Source counts: 12 internal Markdown documents, 6 Python entry points, 1 Rust lib.rs (~1500 lines), 1 monolithic HTML UI (~6700 lines), 2 Cloudflare Worker source files. Every feature claim is traceable to source.