Load, Capacity and Resilience Verification
Adapted from Shadow textbook chapter 13. Performance tests run candidate code against real isolated dependencies for the measured region. Recorded or simulated dependencies are useful for orchestration tests but cannot establish their real capacity, latency or failure behavior.
Declare the performance profile
Pin the artifact vector, graph/configuration revisions, sanitized base, resource topology, region, quotas, runtime/device, build mode and external-effect policy. Record the load generator's capacity and network path. Shared application processes or stores can introduce interference; report it and use a separate process/resource allocation when the claim requires isolation.
Production-shaped inputs preserve route mix, arrival bursts, concurrent sessions, payload distribution, hot keys, relationship/cardinality skew, cache conditions and long-lived connections. Recent traffic alone may miss old entities needed by those requests. Keep representative cohorts and a curated rare-case corpus alongside generated scale.
Test families
| Test | Question | Required evidence |
|---|---|---|
| Smoke | Does the complete measured journey execute correctly? | Assertions, trace/coverage and generator health |
| Baseline/load | Does expected demand meet the SLO? | Throughput, latency distribution, errors and resource saturation |
| Stress/capacity | Where does demand exceed sustainable capacity? | Offered/admitted/completed work, backlog, rejection and recovery |
| Spike | Can the system handle a burst and settle? | Admission behavior, backlog drain and post-burst health |
| Soak/leak | Does resource use remain bounded over repeated lifecycles? | Post-quiescence memory/resource slope, restarts, retained ownership |
| Resilience/fault | Does a defined fault preserve correctness and recover? | Fault boundary/time, retries, invariant checks, data/effect state |
| Cold/startup | What happens before caches/JIT/pools warm? | Startup and first-operation distributions separately from steady state |
Arrival models and overload
An open arrival model schedules external demand independently of response time. A closed model holds a user population whose next work waits for prior completion; it can self-throttle as a service slows. Choose the model from the production question and report dropped iterations plus generator saturation. Otherwise an apparently stable latency chart may conceal unoffered load. k6 open and closed models.
Report offered, admitted, successful and failed work with denominators. Include timeout/cancelled work in outcomes, not only the fast successes. Observe p50/p95/p99 where sample volume supports them; rare-tail claims require adequate samples. Preserve repetitions, time series and confidence/variability rather than treating one mean as capacity.
Compare fairly and gate explicitly
Warmup, steady measurement and recovery are separate intervals. Baseline and candidate use equivalent pre-state and capacity; alternate/repeat runs where drift or shared-host interference matters. Test both cold and warm caches deliberately. Virtual-clock replay duration is not a production latency measurement.
Use an absolute SLO/error budget plus a baseline-relative regression rule. A relative percentage around an almost-zero baseline is unstable; report absolute delta too. Required measurements include target, revision, units, interval and collector status. Missing input or failed capture is a measurement failure, not a passed budget.
Capture instance metrics and targeted profiles alongside throughput. CPU, memory, GC, threads, pools, queue lag, storage I/O, Worker usage and cost explain the limiting resource. Instrumentation overhead and sample loss belong in the result.
Faults and stateful correctness
Select one effect boundary and introduce timeout, post-commit acknowledgement loss, duplicate delivery, retry, out-of-order input, disconnect/reconnect or dependency failure there. Do not inject a queued event while also executing its producer. Check durable state, user-visible output and external-effect intent after recovery.
For at-least-once delivery, validate product-level deduplication and stable logical operation IDs rather than asserting transport-level exactly once. A successful HTTP response alone does not prove one durable row or one notification. The send/retry/reconnect example provides a concrete correctness case. Cloudflare Queue guarantees.
AI and corpus scope
Recorded model answers test downstream orchestration. Live model-quality evaluation pins prompts, model/version, retrieval data, criteria and cost budget, with repeated/semantic judgments appropriate to nondeterminism. Keep that report separate from replay fidelity and latency gates.
Run a small stable corpus during editing, representative workloads at checkpoints and scale/fault/soak profiles before relevant releases. Deduplicate routine cases without discarding rare failures. Graph impact selects an initial region; unknown dynamic paths, shared schemas, auth and configuration changes expand the scope. Agent changes use fixed acceptance rules as described in Clients and agents.