Preview Fabric

The problem it solves

• The real app (mission-control-workspace) is 2 repos, 34 services, 11 volumes, 2 PostgreSQL databases — far too big to run on a laptop.
• A single shared staging env means devs overwrite each other; VMs per dev rot and drift.
• Preview Fabric: every task branch gets its own complete, isolated, disposable copy of the whole stack on one beefy shared host, kept in sync with your local uncommitted edits in near-real-time.
• It is a control plane: a CLI + local daemons on your laptop, and two small programs on the cluster (an operator and a source agent).

What using it feels like

./preview-local bootstrap                      # one-time: builds its own venv
./preview-local cluster configure --host … --user …
./preview-local project configure mission-control-workspace \
    --definition …/mission-control-workspace.json \
    --component mission-control=…/mission-control …

./preview-local dev session my-feature \
    --project mission-control-workspace \
    --component mission-control=…/task-worktree --background
# … edit code locally; changes stream to the running env …
./preview-local dev urls my-feature            # http://…:8080/… via SSH tunnel
./preview-local dev smoke my-feature           # scripted HTTP checks
./preview-local dev down  my-feature           # stop (state kept)
./preview-local dev purge my-feature           # destroy everything, with receipts
    

Everything below this slide explains what those commands actually do.

Kubernetes in ten minutes K8S

Just enough Kubernetes to read the rest of the deck. If you know K8s, swipe right.

• You never "run a command" to start things; you write an object (YAML/JSON) and a controller notices.
• Objects have metadata (name, labels = indexed key/value tags, annotations = unindexed notes), a spec (desired) and a status (observed).

Workload objects K8S

Storage, networking, isolation K8S

The CRD + operator pattern K8S

• Level-triggered, not edge-triggered: it acts on current state, never on "the event", so missed/duplicated events are harmless. Think fixpoint iteration, or React's render loop, not a message handler.
• Reconciles must be idempotent — they rerun constantly, on any change, on restart, on a timer.
• Leader election: multiple copies may run for availability; a lock object (Lease) ensures only one reconciles at a time.

Talking to a cluster K8S

• In this system the laptop has no kubeconfig and no network route to the API server at all. Every kubectl runs on the shared host, over SSH. One credential (your SSH key) is the entire security model.

ssh -F /dev/null -o BatchMode=yes -o StrictHostKeyChecking=yes user@host \
  '/usr/bin/env KUBECONFIG=… kubectl get previewenvironments …'
    

Ownership & garbage collection K8S

• Why: cascading GC deletes by name-based linkage; this system wants deletes to require proof (exact UID + resourceVersion), so a stale or colliding actor can never destroy the wrong thing.
• UID — every object gets a unique immutable ID at creation; a re-created same-name object has a different UID. Preview Fabric leans on this constantly.

The big picture

Everything user-initiated flows left→right. The operator alone touches runtime objects; the client never creates a pod.

Who owns what — the seven authorities

Every design decision below is some form of: keep these authorities from stepping on each other without introducing new identity or auth systems.

One trust domain, zero new auth

• There is exactly one security boundary: SSH to the shared host (and the kubeconfig it grants). Preview Fabric adds no HTTP endpoints, users, tenants, OAuth/OIDC, bearer tokens, or app-level authorization.
• installationID (e.g. preview-fabric-system) is a collision scope, not identity — it namespaces object names so two installations can share a cluster.
• Inside the domain, safety is logical, not authn: exact labels, recorded UIDs, writer fences, receipts, and compare-and-swap everywhere.

The seven state spaces

Client anatomy

• ./preview-local is a 96-line self-bootstrapping Python launcher: creates .preview-local-venv/, installs pinned deps (requirements/runtime.lock), then os.execv-re-executes itself inside the venv.
• Deliberately tiny runtime deps: anyio, pydantic, watchfiles, tzlocal, idna. ~27 k lines of Python across 34 modules.
• Three kinds of local process, all supervised:
  • Environment Manager — one daemon per alias; the only thing that mutates cluster state.
  • Watcher session — feeds source observations to the manager.
  • Tunnel — one shared ssh port-forward per installation.

CLI map

• Global: --json for machine output, --state-root, --config-root. Mutating commands accept --detach; default wait timeout 120 s.
• An alias (e.g. my-feature) is the developer-facing name of one environment; everything else derives from it.

Configuration layers

• State root (.preview-local-state/): client-owner ID, per-alias binding (bound paths, last receipts, watcher identity), flock files, manager journal + IPC sockets.
• Bindings never store cluster truth — the CR is the source of truth for lifecycle; local state only holds what the cluster cannot reconstruct.

The Environment Manager

• One supervised daemon per alias; a flock singleton guarantees uniqueness. CLI talks to it over a Unix domain socket (preview-fabric-manager-ipc/v2, length-prefixed canonical JSON ≤ 1 MiB, peer-uid checked, 8 worker threads).
• It owns all cluster mutation: CR writes, secret application, source-agent calls. The CLI itself never touches the cluster.
• Its SQLite journal (WAL, FULL sync) records request/plan/receipt identities before any side effect — enough to resume an interrupted operation exactly, never a workflow engine.

A timeout never retries by itself; an interrupted Ensure is resumed under its original identity, not re-planned. This is the idempotency spine of the whole client.

Journal phases & recovery

Prepared
  -> FenceClaimed                 # writer fence current on source agent
  -> SourcePartiallyAccepted | SourceAccepted
  -> APIAccepted                  # CR write accepted by API server
  -> OperatorAcknowledged         # operator echoed the proposal
  -> Terminal
    

• On restart the manager always finishes the journaled request first — replaying the stored Plan under its original request ID — before planning anything new.
• Every remote effect is keyed: request ID + payload digest + dedupeUntil. A lost response is resolved by querying the retained outcome, adopting only the exact recorded result.
• Receipts flow back to the CLI so wait can distinguish my convergence from someone else's (Succeeded / Superseded / Blocked / PLAN_STALE / Deleted…).

Watching your worktree

• Built on watchfiles (Rust notify backend → inotify on Linux: the kernel pushes file-change events; no scanning).
• The watch set is non-recursive and policy-pruned: the client walks each component root once, registers one native watch per retained directory, and skips .git, git-ignored trees (node_modules…), declared watch_excludes, secret paths, .previewignore matches. Budget: 100 000 watches.
• New paths are classified live with git check-ignore; creating a directory or editing .gitignore/.previewignore triggers a full watch-set rebuild.
• WSL nuance: watchfiles force-polls on WSL by default; the client parses /proc/self/mountinfo and re-enables inotify when all roots are on native Linux filesystems.

Computing the dirty overlay

• On each trigger, the client asks git what is actually dirty:
  • git diff --name-only HEAD → tracked edits
  • git ls-files --others --exclude-standard → untracked files
  • plus explicit include_ignored force-includes
• Existing paths → overlay_paths (content to ship); missing → deleted_paths; committed drift vs the last-shipped HEAD → changed_paths (observational).
• A deterministic fingerprint = sha256 over HEAD + each path's (mode, size, content-hash) — lets everything downstream detect "same state" cheaply.

The latest-state coalescer

• Problem: shipping source takes seconds; you keep typing. Naive queues grow unboundedly and replay stale states.
• Solution: a thread-safe slot holding exactly one pending observation. New events merge into the slot (latest state wins); submit() never blocks the watcher.
• Per publish cycle: ship current state, then at most one follow-up for whatever accumulated meanwhile. Sequence reservation stops an old slow validation from regressing a newer state.
• Overflow of the slot's bounds sets full_reconciliation_required instead of growing memory.

The session loop

The watcher is a pure input producer: it can keep ingesting while the operator is slow, because the source lane and the runtime lane are independent.

Shipping source: the exec-only agent

• preview-source-agent is a Go binary in a 1-replica Deployment. Its long-running container just sleeps (daemon verb). It mounts one hostPath: /var/lib/preview-fabric/source.
• It has no Service, no port, no listener, no service-account token. There is nothing to scan, nothing to authenticate, no API surface to version behind a load balancer.
• Every request is one fresh process:

  
  ssh user@host -- 'kubectl exec -i -n preview-fabric-system \
    deployment/preview-source-agent -- /preview-source-agent serve'
        

• One process = one framed request on stdin, one framed response on stdout, exit. Client-side timeout 900 s. The SSH/kubeconfig trust domain is the auth.

Wire format: preview-fabric-source/v2

PFSOURCE/2\n                      # magic
[u32 big-endian header length]
{ canonical JSON header, ≤ 1 MiB, unknown fields rejected }
[bundle bytes  … declared size + sha256]
[overlay bytes … declared size + sha256]
    

• Header carries: operation, transaction ID, request ID + dedupeUntil, writer fence, component, repository key, monotonic sequence, expectedReceipt (CAS), commit, sparse patterns, overlay/deleted/required paths, semantic inputs.
• Operations: ingest · status · publish · remove · claim · takeover · writer-status · outcome · purge · purge-status.
• Bounds: bundle/overlay ≤ 8 GiB each, ≤ 20 000 paths, receipts ≤ 4 MiB. v1 magic is recognized only to say unsupported-protocol — before any store initialization.

Payload = git bundle + tar overlay

• Bundles are built deterministically (pack.window=0 pack.depth=0 pack.threads=1) — the bytes are hashed into the receipt, so they must be reproducible.
• Incremental bundles (the shipped transfer speedup): writer-status advertises up to 32 repository tips the agent already has; the client builds git bundle create <rev> --not <tips> — a new env at the baseline tip ships a ~642-byte bundle instead of full history.
• The agent parses the bundle header's prerequisites itself and fails bundle-basis-unknown before touching anything; the client retries the same request ID with a full bundle. Tip presence doubles as the capability signal — old agents reject the unknown header field.

Server pipeline: stage → validate → finalize

Concurrency safety, in layers

Store layout & the reverse path

/var/lib/preview-fabric/source/
├── repositories/INSTALLATION/REPOKEY.git    # one shared bare repo per repo
├── worktrees/INSTALLATION/ENV_UID/COMPONENT # ← pods mount exactly this
├── receipts/  writers/  retired/  purges/
├── outcomes/          # 256-shard append-only idempotency log
└── .staging/  .finalizing/  locks/
    

• Remote commits flow back too: if an agent (human or AI) working inside the env's dev-shell commits to the server worktree, the next ingest preserves that commit under an unpublished ref before overwriting.
• dev remote publish ALIAS COMPONENT asks the agent for a bundle of it and imports it locally at refs/preview-fabric/<alias>/<component> — no checkout, no merge, no branch change. Normal git workflow takes over from there.

The PreviewEnvironment CRD

• Group preview.preview-fabric.io/v1alpha1, kind PreviewEnvironment, cluster-scoped (lives outside any namespace) — so the operator can derive and own the environment's namespace and detect collisions before creating anything.
• Deterministic identity chain:

  
  CR name:    pe-<alias>-<sha256(installation┃project┃alias)[:10]>
  namespace:  pf-<inst≤11>-<proj≤16>-<alias≤20>-<hash(…+CR-UID)[:5]>
          

  The namespace hash includes the CR UID → a re-created same-name environment gets a different namespace; the old one is left untouched.
• The spec is the complete runtime graph — the operator needs no other input: sources, services, PVCs, routes, network policies, database clones, checks, lifecycle protocol.

Lifecycle v2: birth of an environment

• Creation starts as a Claiming skeleton: protocolVersion: v2, initialization phase Claiming + a random client-binding nonce + claimExpiresAt (15 min), desired state Stopped, and — enforced by CEL — zero sources/services/PVCs/routes/clones.
• A skeleton creates no namespace and no runtime child. It exists only to reserve identity and let the writer fence be established on both planes (source agent + CR).
• The manager then flips Claiming → Active (one-way) together with the first complete runtime proposal.
• Recovery of an interrupted create requires the exact nonce + environment UID; a same-name leftover or expired skeleton is never adopted.

Spec tour — the parts that matter

Status: bounded observation, never history

• status.observedGeneration == metadata.generation means "the operator has seen this spec" — not that it succeeded. (generation increments on every spec change.)
• Phases (summary only): Pending → Reconciling → Ready / Stopped, plus Degraded, Blocked (ownership conflict), GarbageCollected, Terminating.
• Conditions (the real signal): ProposalAccepted · Ready · Progressing · Degraded · OwnershipConflict · LeaseExpired · DatabaseClonesReady · RoutesReady · Validated.
• lifecycle.currentTarget / lastAppliedTarget: the activation being converged, with expected child revisions — what a precise wait correlates against.
• Terminal outcomes ledger: ≤ 128 unexpired results keyed by activation/action ID — the cluster-side half of exactly-once semantics.

Schema-enforced protocol (CEL)

• ~20 validation rules run inside the API server on every write — even a buggy client cannot corrupt protocol state:

Analogy: database CHECK constraints for a distributed handshake.

The operator

• Go, built on controller-runtime (the standard operator framework): a shared watch-cache of API objects, a work queue of "reconcile requests", and your Reconcile(ctx, req) function. 1 replica + Lease-based leader election; 64 Mi RAM requested.
• Watch topology: it watches PreviewEnvironment (only spec-generation / finalizer / deletion changes — status writes don't self-trigger), and watches every child kind (Deployment, StatefulSet, Job, PVC, Service, Secret, NetworkPolicy, HTTPRoute), mapping child events back to the owning CR via labels, since ownerReferences don't exist here.
• Ignores any CR whose installationID isn't its own → two installations coexist.

Reducer / executor split

• Effect order: namespace → PVCs → NetworkPolicies → Services → workloads (topological dependency order, gated) → clone Jobs → routes → validation Job → prune orphans.
• Dominance order in the reducer: deletion > writer-fence rejection > Claiming > lease expiry > proposal validation > actions > runtime plan.
• Requeue classes: event-driven (no timer) · immediate-after-effect (1 s) · deadline (e.g. lease expiry, claim expiry) · rate-limited error backoff · terminal.
• Rejections are first-class: RequestIDReused · OutcomeExpired · EffectPlanUnbounded · OutcomeCapacityExhausted · WriterFenceRejected · InvalidDeclaration.

What rolls a pod — the revision hash

The operator also computes one runtimeIntentID — a sha256 over the canonical JSON of the whole runtime spec — the sole authority for "did the intent change?" (the client's copy is advisory; golden tests pin Go and Python to byte-identical encoding).

Projected objects

Sources and the state cache enter pods as operator-derived hostPath volumes; raw hostPath in user PodTemplates is rejected outright.

Exact-identity ownership

• Every managed object must have: empty ownerReferences + exact managed-by / installation / project / environment-name / environment-UID labels + installation / environment-UID annotations + (for namespaced children) the status-recorded namespace UID.
• Any mismatch — including someone adding an ownerReference — is a foreign resource: reconcile stops, phase Blocked, condition OwnershipConflict, retry every 30 s. The operator repairs nothing it cannot prove.
• All deletes carry UID + resourceVersion preconditions — the K8s equivalent of an atomic compare-and-delete.
• Orphan pruning: list by owned labels, delete what the desired set doesn't contain, ≤ 64 per pass, keep failed Jobs referenced in status.
• On CR deletion a finalizer (a "you may not fully delete until I've cleaned up" marker) deletes proven children in bounded batches — and intentionally retains the namespace: a namespace may contain foreign resources the operator cannot enumerate or prove.

Operator constants worth remembering

Every list is bounded, every retry is bounded, every message is truncated (512 B). Nothing in the operator can grow without limit — a deliberate posture.

Secrets: client-owned, content-blind

• Secret values live only on your machine (e.g. .env files, TLS dirs — declared in the project definition, paths overridable per machine).
• The manager loads them (≤ 1 MiB each / 4 MiB total), and applies them directly as ownerless, environment-labelled K8s Secrets into the env namespace — after verifying the namespace's exact UID and identity metadata.
• Each Secret gets a random 32-hex revision annotation — content-independent. Unchanged content (compared only in memory) reuses the revision; changed content gets a fresh unrelated one.
• Workloads carry a per-consumer fingerprint = sha256 of their (secret-name : revision) list → changing a secret rolls exactly its consumers.

Secret flow

• dev secrets apply ALIAS — the explicit path after editing a secret: in-memory no-op detection; a change journals new revisions and submits one freshly recomputed complete runtime proposal.
• Replaying an interrupted Ensure reuses its journaled revisions — so replays are invisible, and a different Ensure can't infer content changes.
• Derivations (≤ 32, bounded set: postgres-password, postgres-dsn, neo4j-auth) compute connection materials from declared inputs — still client-side.

Databases: baselines & clones

• dev main ensure — one routed command: create if absent, exactly resume an interrupted create, activate a Stopped one, then submit a single sync. A merge to main reaches the baseline with no destructive step; untouched inputs classify no-op and write nothing.
• dev main status — read-only freshness: per component, the commit the baseline was ensured at vs the canonical checkout's HEAD now. Fails closed (unreadable checkout ⇒ "not fresh").
• Creating a database-backed feature first Ensures the baseline and requires that Ensure to be accepted — but no longer waits for it to be Ready (ADR 0010): the operator gates in-cluster instead.

The clone gate (ADR 0010)

• A feature's clone declares project.baselineRef → the exact baseline CR (same installation + project).
• The operator creates no clone/seed Job until it observes the baseline: undeleted, generation-current (has observed its own latest spec), and Ready=True for that same generation. A mid-convergence or stale baseline creates nothing.
• Each clone is an immutable revisioned Job: clone-<name>-<revision[:12]>. Failure stays visible as evidence; the operator never auto-retries.
• Recovery = dev clone retry ALIAS CLONE: a target-scoped action naming the exact failed revision; the action ID becomes part of the new Job's revision. Replaying the same action attaches; changed payloads are rejected.
• While any clone is unfinished, a clone barrier holds back all services outside the clones' prerequisite closure and withholds route programming — you cannot reach a half-cloned environment.

Mode 1: logical clone (default)

• Prerequisite: the feature's own database service must be Ready first (declared prerequisite).
• K8s Jobs are at-least-once → two pods may run the clone concurrently. A PostgreSQL advisory lock (an app-defined DB mutex) makes the restore single-writer.
• Declared external-effect contract: Idempotent — rerunning yields the same result.

Mode 2: PhysicalSeed

• Inverts the ordering: copy the files of the whole instance before first start — no dump/restore CPU.
• No running server to lock in ⇒ duplicate pods serialize on one atomic mkdir: same identity + complete stamp ⇒ attach and exit; anything else (different revision, in-progress stamp, populated dir without stamp) ⇒ visible refusal. A seed never wipes a data directory.
• Honest engineering coda: measured for mission-control and rejected — its logical dump is 8 MiB vs 47–64 MiB PGDATA, below the fixed-cost crossover. The mechanism ships for bigger databases; rabbitmq/neo4j stay cold-path (hostname-coupled state / no safe online clone).

Why all this ceremony?

• At-least-once is the enemy. Kubernetes may run any Job pod twice; nodes crash mid-restore. Every clone path must be safe under duplication and interruption — hence advisory locks, mkdir markers, immutable revisions, and explicit retries.
• Failure as evidence. A failed Job is kept, inspectable via dev status / dev logs; silent retries would destroy the trail and can mask data corruption.
• The baseline is protected. Clones only ever read from it (dump / replication connection); no feature can write baseline state. Refreshing it is an explicit, non-destructive dev main ensure.
• Umbrella condition DatabaseClonesReady + per-clone status (mode, revision, phase) tell you exactly where things stand.

Reaching your environment

• Cluster side: one shared Traefik behind a Gateway API Gateway (preview-system/preview-fabric, listener web). Each environment publishes HTTPRoute objects with hostname templates like {alias}.mc.preview → hostname-based routing to that env's Services.
• Laptop side: nothing is exposed to any network. One supervised SSH tunnel:

  
  ssh -N -o ExitOnForwardFailure=yes \
      -L 127.0.0.1:8080:127.0.0.1:30080  user@host
        

  loopback→loopback only, local 8080 → the ingress NodePort 30080.
• cluster tunnel --background/--status/--stop; health = unit alive and the port actually bound. One tunnel serves all environments (routing is by Host header).

urls, smoke, forward, shell, logs

• dev urls ALIAS — prints each route as http://<hostname>:8080/… for use through the tunnel.
• dev smoke ALIAS — GETs each route's declared smoke checks against 127.0.0.1:8080 with a spoofed Host: header; exit codes distinguish check-mismatch / operational failure / tunnel-unavailable.
• dev forward ALIAS SVC — for non-HTTP ports (e.g. postgres): one command that runs ssh -L and a remote kubectl port-forward chained; lives while the process lives.
• dev shell ALIAS — kubectl exec -it into the tier's dedicated dev-shell workload (a container with your source mounted — where an in-env agent can work and commit; cf. dev remote publish).
• dev logs ALIAS SVC — label-selected pod logs, remote kubectl.

Application checks

• Beyond "pods are Ready", the project can declare application checks: HTTP-JSON probes (and Job checks) run inside the cluster by one operator-created validation Job after convergence.
• Policy Optional | Required; correlation RuntimeOnly | ExactSource (the check proves it saw this exact source receipt, or reports BehaviorPassedSourceUnproven).
• Bounded like everything else: ≤ 32 checks, ≤ 3 attempts, 120 s backoff, 300 s deadline; on timeout the real failure is recovered from the pod's termination message.
• Result lands in the Validated condition — the strongest "your env truly works" signal, consumed by dev diagnose --gate validated.

Create, end to end

Down, leases, retention, GC

• dev down → desiredState: Stopped: workloads scale to 0, Jobs/routes go away, PVCs and Secrets stay — dev up is cheap.
• Leases: features carry an absolute lease.expiresAt (renewal computes a business-day deadline in your current timezone, then stores the absolute instant — changing timezone later can't move it). Baselines, pinned envs, and no-expiry envs are protected.
• Expiry is reconciled in-cluster — works with every client offline:
  1. at expiry: stop workloads, condition LeaseExpired
  2. deleteAfter = expiresAt + retention (default 72 h)
  3. after that: delete only exact-identity objects, with UID/RV preconditions → phase GarbageCollected
• The namespace survives GC (unprovable contents); dev gc is a read-only view — there is no client-side cleanup daemon.

Purge: destruction with receipts

• A lost response is resolved via purge-status; a same-name successor has a different UID and can never satisfy the old tombstone. A stale manager can never recreate source for a retired UID.
• Broad "rm -rf the source root" does not exist by design — the routed purge is the only deletion path, and it refuses rather than widening its target.

The no-adoption principle

• Nothing in the system ever "takes over" an object it didn't create with proof: not a same-name CR, not an expired skeleton, not a half-deleted namespace, not a foreign Secret, not a warm pod.
• Instead: fresh UID → fresh namespace → fresh objects; the old generation is fenced, retained, or explicitly purged.
• What this buys, concretely:
  • a crashed client can always resume or abandon — never corrupt;
  • two devs colliding on a name get a visible Blocked, not silent mixing;
  • deletion can never cascade into someone else's resources;
  • every mutation is attributable to exactly one fenced writer.
• The cost — rebuilding state from scratch each time — is then bought back with caches, not shared identity. Which is Module 12.

Making creation fast

Measured cost order for a cold create (descending):

The obvious alternative — a pool of pre-warmed claimable environments — was explicitly rejected: claiming is adoption with a protocol; the alias is baked into immutable names; secret revisions would roll every warm pod anyway; idle spares burn RAM on one shared host.

The state cache (ADR 0009)

/var/lib/preview-fabric/state-cache/
└── INSTALLATION/PROJECT/deps/<DEP_FINGERPRINT>/<VOLUME>/
    ├── .preview-fabric-complete     # marker written before publish rename
    └── node_modules/ …              # finished dependency tree
    

• Content-addressed by the dependency fingerprint (lockfile hash) that already names the Job's revision. States: absent · staging (.staging.* sibling) · published. No partial published state (marker exists before the atomic rename(2)); concurrent publishers resolve via kernel ENOTEMPTY.
• The baseline's dependency Job publishes; a feature's Job imports (only into an empty volume, marker-verified) — a feature whose lockfiles match the baseline runs no package manager at all.
• cp --reflink=auto: metadata-cost copy-on-write on capable filesystems (XFS/btrfs), silently a plain copy on ext4. An accelerator, never a dependency.
• Nobody reconciles this space: no K8s objects, no owner, no finalizers. The operator only derives the mount path; the Jobs the project declares do all copying. Any anomaly ⇒ fall through to the ordinary cold build. Wrong is impossible; slow is the worst case.

The rest of the toolkit

Results (live, office-2 host)

• k3s on ext4/NVMe (~3.1 GB/s) — so all cache copies are full copies; reflink/XFS is an untapped follow-up.
• Baseline refresh through an additive definition rollout: 70 s.
• Full-tier mission-control + workspace feature create: Ready in 1 m 50 s — all three dependency volumes imported from cache (zero package-manager work), logical clones green, smoke 8/8.
• Every fast path is gated on a fingerprint or emptiness check with a cold fallback — a stale cache can only produce a slower environment, never a wrong one.

Also fixed along the way: the additive-rollout defect (a service added to the project definition previously couldn't converge onto a live environment through any non-destructive operation), and stale receiptless syncs now terminalize as PLAN_STALE.

How it's tested

The client-path e2e trick

• Problem: the client's only transport is SSH — how do you e2e-test it against a container cluster?
• Answer: put the whole trust domain in a box. Dockerfile.e2e-sshd builds an Alpine sidecar running sshd (per-build host keys, pubkey-only) with kubectl + a rewritten kubeconfig inside; cluster configure points at it. The client can't tell it from a real host.
• All client state is sandboxed (HOME, config/state roots, process supervisor); phases P0–P8 cover transport, bootstrap, create, fact assertions, hot path returns without pod replacement, watcher-observed edit sync, receipt-CAS purge, supervision teardown — with wall-clock timings printed per phase.
• Assertions mirror a recorded live-host run, so CI and reality can't quietly drift apart.

Installing & operating

• make verify && make build && make images → push both images → record digests → ./scripts/operator-install.sh with digest-pinned image env vars (…@sha256:… enforced; tags refused).
• Install preflight checks the Gateway API version and that the shared Gateway is Accepted+Programmed with an HTTP listener named web; scripts refuse fixed-name resources that exist without exact installation labels; reset/uninstall demand exact confirmation + precondition deletions and never touch foreign resources.
• What lands: namespace preview-fabric-system, the CRD (with a schema-floor annotation the client checks), RBAC, operator Deployment, source-agent Deployment. Uninstall retains the CRD, namespace, and any pull Secret.
• Admin without a laptop kubeconfig: the ops scripts can run through preview-local cluster _kubectl — the client's own SSH bridge.
• Day-2 truth lives in docs/operations/operator-runbook.md: status interpretation, source/database recovery, limitations.

The invariants, one last time

1. Never adopt. Ownerless exact identity; empty ownerReferences; UID-pinned everything.
2. Plan → Ensure once → Wait → Diagnose. Identities journaled before side effects; receipts compare-and-swapped; exact replay.
3. The operator is level-triggered and bounded — bounded API work, bounded lists, bounded retries, no blocking waits, validation-only knowledge of host roots.
4. Secret values and content hashes never leave the laptop — opaque revisions only.
5. Jobs are immutable revisions; failure is evidence; retry is explicit.
6. Golden vectors pin cross-language contracts.
7. Every fast path has a gate and a cold fallback — stale caches make slower environments, never wrong ones.

Further reading, in order: docs/architecture/overview.md → runtime-lifecycle.md → source-authority.md → docs/state-spaces.md → ADRs 0006–0010 → the operator runbook.