Thine · Backend Voice Dictation · Handoff
Mic tap → Soniox streaming STT biased with the user's own people → one GPT-5.6-terra cleanup pass → cleaned text in the composer. This page is the operating manual: architecture, the invariants the code guarantees, the review process that hardened it, and the runbook for whoever touches it next.
01 · Architecture
Everything rides the app's existing WebSocket. The client speaks msgpack events (dictation.audio.start / chunk / end / cancel, each carrying a session_id); the backend answers with status → delta → final or a typed error. Two Durable Objects sit between phone and Soniox:
stt-rt-v5
Streaming STT with context.terms = the bias hot-set (≤ 8,000 serialized chars). Endpoint detection on; manual finalize on mic release.
reasoningEffort: "low", store: false, streamed deltas to the client. Prompt carries "Thine" + the top of the hot-set (≤ 1,200 chars) so homophones get fixed.The client discards the final 15 s after mic release. The backend therefore bounds the whole post-END path with FINALIZE_TOTAL_BUDGET_MS = 12_000: open-wait (≤5s) + Soniox finalize-wait (≤5s) + cleanup (≤6s, shrunk by whatever the waits consumed, skipped entirely at zero). If cleanup is aborted mid-stream it returns the raw transcript — never a truncated cleaned prefix. Other knobs: connect timeout 7s, idle timeout 120s (30s alarm), run-on finalize every 15s of endpoint-less speech.
02 · Invariants
These five properties are what ~19 review rounds converged on. If you change the relay or the session machine, keep them true — each one exists because its absence was a shipped bug.
DICTATION_TRANSCRIPT_FINAL is only sent when Soniox actually returned finished: true with no tokens. Never-opened, mid-finalize errors, EOF-send failures, close-without-finished, and finalize timeouts all emit typed errors instead (single guard on hasReceivedFinished).no_session, answered locally) included.busy-rejected device can't steal a live session's stream.hasSentEof makes finalize+EOF single-shot; every terminal path (success and all error codes) logs session metrics exactly once via teardown(reason) — failure dashboards see the failure population.Old preview builds send no session_id and camelCase samplingRate. The server synthesizes a UUID at START, an undefined incoming id adopts the single active session, and the ignored camelCase rate defaults to the only supported value (16 kHz). Strict typing stays the go-forward contract; the server merely tolerates old peers.
03 · Process
Every round: two independent adversarial reviewers (Claude Fable 5 + GPT-5.6-Sol, high reasoning) → the orchestrator triages each finding (over-defensive proposals are rejected, not softened) → GPT-5.6-Sol fixers apply the accepted set → validation → commit+push → re-review. The loop ends only after two consecutive rounds with zero new accepted fixes.
| Phase | Rounds | Fixes | Headline catches |
|---|---|---|---|
| Main loop | 1–19 | ≈55 | 2× P0 (wire-contract break; mid-finalize teardown losing transcripts), the empty-final family, the busy-rejection routing steal, deploy-window data loss, the iOS 15s timing-contract drift, EU-jurisdiction & Atlas-gate violations |
| Fresh dual review | +1 | 6 | 2× P1/P2 — stale isRecording bricking dictation after abnormal voice-chat death; same-socket displaced STARTs stranded. Net −48 lines (dedup + guard removal) |
| Confirmation | +1 | 0 | DUAL CLEAN — line-level verification of all six, fresh sweep empty |
| Rejected along the way | — | ≈20 | Compat shims, byte caps, zod relayering, generation machinery, speculative UI rework, and several factually-wrong findings — recorded in the PR so they aren't re-litigated |
Convergence by round: 20 → 12 → 6 → 6 → 3 → 4 → 5 → 1 → 0 → 1 → 4 → 1 → 0 → 1 → 1 → 0 → 1 → 0 → 0. Several later fixes were net deletions — the reviews made the code smaller, not more defensive. The proto/dictation-phase0 bench harness that seeded the design was deliberately dropped from the PR once its bake-off job was done (retrievable from branch history).
04 · Current state
| Surface | State | Notes |
|---|---|---|
| Backend sid/dictation-phase0 | 0f5c3a874 · MERGEABLE | PR #1278 → feat/atlas-unified. Base merged in (their lockfile + TS 7.0.2 taken; our forwarder and test wiring kept). The 61 lint errors on CI are base-branch debt — verified identical on a pristine base checkout. |
| Mobile sid/dictation-phase0 | 4f8318ce · clean | PR #516. Based purely on feature/wikiViews (develop deliberately removed by rebase). Local-only, uncommitted: AppConstants.baseUrl → preview URL. |
| Preview env | 4c3a0204f deployed | All 11 workers at *-dictation-preview.siddhartha-5c5.workers.dev. Functionally identical to the merge head for dictation. |
| Cleanup model | gpt-5.6-terra · low | Verified live against the OpenAI API before switching. store: false everywhere. |
| Greptile | 2/2 resolved | Both findings fixed with commit references; re-review requested on the new head. |
05 · Runbook
wrangler.jsonc has a top-level routes: [{ pattern: "api.thine.com", custom_domain: true }]. A named env that doesn't override routes inherits it — one deploy silently moved api.thine.com onto the preview worker (dev outage; prod untouched; rebound via the Cloudflare domains API). Every dictation-preview env now carries an explicit "routes": []. Never remove it, and verify domain ownership after every deploy.~/thine/wt-deploy-dictation — detached checkout of the branch head with the 11 env.dictation-preview sections as uncommitted wrangler edits (kept off the PR on purpose). To bump code: git stash push apps/*/wrangler.jsonc → git checkout --detach <sha> → git stash pop.pnpm install --frozen-lockfile && pnpm exports:build (force HTTPS for the git dep via GIT_CONFIG_* env vars if SSH auth is absent).npx wrangler deploy --env dictation-preview — order doesn't matter anymore; all preview scripts exist, so cross-script DO bindings resolve. (First-ever bootstrap needed a two-phase strip-bindings deploy; that's done.)workers.dev URL per worker, and the domains API must show no preview worker holding a real domain: GET /accounts/<acct>/workers/domains → api.thine.com → control-plane-dev, ga → control-plane-atlas, prod → control-plane-prod../node_modules/.bin/vitest run apps/voicechat/src/dictation.test.ts — 17 tests covering the finalize/teardown/error matrix, budget math, cancel semantics, and the cleanup-vocabulary prompt. Mobile: RoutedSTTServiceIntegrationTests plus a reusable XCUITest E2E harness (drives the sim by bundle id) built during the stuck-composer investigation.
06 · Decisions & gotchas
Each of these was proposed as a "fix" by at least one reviewer and rejected on purpose. They're recorded here (and in the PR) so the next review cycle doesn't re-open them.
| Decision | Rationale |
|---|---|
No samplingRate alias, no zod at the relay boundary, no audio byte caps | The ignored camelCase key defaults to the only supported rate (16 kHz); validation lives in the DO matching existing voicechat precedent; the connect timeout already bounds the buffering window. All three "fixes" were defensive bloat. |
| Timer-starve over generation machinery | A stale voice-mode 500 ms close timer is defeated by clearing it on every dictation CHUNK (~250 ms cadence) — two lines instead of cross-mode lifecycle state. |
| No terminal-reply rebinding for half-open reconnects after END | Owner-socket routing (multi-device correctness) was chosen over recovering a seconds-wide reconnect window; the client's 15 s timeout + delta salvage degrades gracefully. |
undefined session_id = wildcard | Legacy peers carry no ids; it's a per-user DO with one active session, so the wildcard is the old protocol's exact semantics, not a hole. |
records-off interopPackr for VoiceChat-originated frames only | msgpackr's default record extension broke the iOS decoder in live testing (keyless objects). ControlPlane's own client frames keep plain pack — that's the pre-existing CP wire format. A reviewer's claim that the custom packer was redundant was refuted empirically. |
Cleanup at low reasoning | Time-to-first-token must fit the 12 s finalize budget; the task is mechanical. Higher effort buys little accuracy at real latency cost. |
voicechat's typecheck script removed | Composite project references made bare per-package tsc emit cross-project rootDir noise; CI never ran it. Kept removed through the base merge. |
Mobile branch based purely on feature/wikiViews | develop was merged then deliberately rebased out — the branch must carry no develop-side changes. Consequence: it inherits wikiViews' DZNetworking WebSocket swap (#528); landing the branch lands that swap app-wide. |
| Bench harness dropped from the PR | Its bake-off job was done and it measured a non-production cleanup arm; six review fixes had gone into keeping a dead spike presentable. Branch history retains it for future STT A/Bs. |
| Gotcha | What to do about it |
|---|---|
wrangler named envs inherit top-level routes | The api.thine.com incident (§05). Every preview env must carry an explicit "routes": []; verify domain ownership after every deploy. |
pnpm install regeneration churns the whole lockfile | dedupe-peer-dependents re-resolves unrelated subtrees (effect-beta drift rode in this way once — via pnpm install run inside the non-workspace proto dir). To add one dep: hand-edit the importer entry against an existing snapshot key, then validate with --frozen-lockfile. Post-base-merge the lockfile is raw pnpm format (no longer prettier-formatted) — match it. |
ai@6.0.79 aborts close the stream normally | No throw on AbortController.abort() — an aborted cleanup loop ends cleanly with a partial prefix. Always check signal.aborted after consuming textStream; return raw. |
| Soniox protocol shape | Normal close is preceded by finished: true — close-without-finished is abnormal. Post-finalize error events are expected teardown, not failures. <end> tokens are endpoint boundaries (flush + reset run-on limiter); <fin> is manual-finalize completion (reset limiter, don't flush). |
| DO facts that shaped the design | WebSockets terminate on deploys/restarts (hence relay-close verdicts); in-memory flags reset on hibernation (hence the liveness-based busy gate); the open Soniox client socket pins the DO in memory; non-storage awaits allow event interleaving (hence pending-stash-before-connect). |
| Base branch carries its own debt | feat/atlas-unified has ~61 lint errors and a root-tsc TS18003 of its own. Before blaming the PR, baseline a pristine base checkout — that comparison has settled it twice. |
| TypeScript narrows through const boolean aliases | Two reviewer "compile error" claims died on this: an aliased condition (const isX = a?.b === y) narrows a at the use site. Conversely, mock casts to complex SDK types need as unknown as T under TS 5.9+/7. |
| Simulator STT ≠ device STT | The sim may lack local speech assets (fast .notAvailable) or have them (real backend round-trips). Don't infer device behavior from either; the XCUITest harness in the mobile repo drives real flows by bundle id. |
07 · Open items
| Item | Why it matters |
|---|---|
| Committed secrets rotation | OpenAI key in openai.client.ts, Soniox key in voicechat/wrangler.jsonc — pre-existing, tracked in the rotation workstream; dictation depends on both. |
| WebSocket transport swap rides this branch | Mobile PR replaces the app-wide hand-rolled socket client with DZNetworking (#528, authored on wikiViews by its maintainer). Landing the branch lands the swap for everyone — review it as core infra, not a dictation detail. |
| Silent-stop error toast | Stopping a dictation with zero speech sometimes shows "Voice input stopped" on the sim. Suspected empty-audio finalize path; needs one on-device check with real speech, then wrangler tail on voicechat if it reproduces. |
| Header "Liquid Glass" glitch | One-off device screenshot of white toolbar pills + smeared logo. Two audits cleared our commits; consistent with an iOS 26 glass backdrop-snapshot failure. If it recurs: scroll (forces re-snapshot) — if that clears it, wrap the header glass in one GlassEffectContainer. |
| wikiViews is 18 commits ahead | The mobile branch pins an older wikiViews point; a same-lineage merge is clean whenever it's wanted. |
| Preview env config is uncommitted | The dictation-preview wrangler sections live only in the deploy worktree. Fine while it exists; commit them somewhere durable if the env should outlive this machine. |
| STT "Thine" detection | "Thine" heads the Soniox bias list and the cleanup prompt now corrects thine/dine/tine/shine when addressing the assistant. If it still misses: add bias phrases ("ask Thine…") and check Soniox's free-text context field — the deleted bench harness is the right A/B tool and lives in branch history. |