The gap list was the honest document
The best planning artifact this month was not a roadmap. It was a coverage gap list. Each row named one thing the platform claimed to do and one way that claim could break in production.
That changed the work. Instead of asking whether a feature exists, we asked whether a live system can prove it still works: OAuth refresh and revocation, calendar idempotency, KB rebuilds, tenant teardown, webhook retries, rate limits, trace truncation, and more.
A useful live test has a shape
The useful tests are small, named, and close to real behavior. They create the minimum data needed, run the mechanism, assert the thing that would hurt, and clean up. They are not generic end-to-end tours. They are targeted probes.
- P0 tests protect contracts that would break tenants or data safety.
- P1 tests protect operational behavior that fails under concurrency, retries, or external dependencies.
- P2 tests protect admin workflows, traceability, and long-term maintainability.
The most valuable checks were unglamorous
The memorable work was not the spectacular demo. It was proving that duplicate calendar webhooks do not create duplicate runs, that connection-pool exhaustion degrades instead of collapsing, that a tenant secret never crosses isolation, that a tenant's search index is strictly its own, and that a long run can pause, resume, and still know which graph version it came from.
Those checks rarely look good in a launch video. They are exactly what lets a product survive contact with enterprise operations.
A mature AI product is not the one with the longest feature list. It is the one that can tell you which assumptions it tests every week.
Verification belongs in the admin story
Live tests should not be a private engineering ritual. They create evidence that operators care about: when a deployment happened, what version is running, which checks passed, which systems are degraded, and whether critical background loops are still alive.
That is why the deployment, operations, release, and project-history surfaces matter. A platform admin should not have to trust our confidence. They should see the current state and the history behind it.
- Named checks map to product claims
- Failures point to the responsible mechanism
- Admin surfaces show current deployment state
- A green build as the only evidence