Playbook: Load-Testing Wealth-Tech APIs at 10,000 Synthetic Households

Sizing: why 10K and not 1K or 100K

1K is too small to exercise query-plan transitions, index-vs-scan thresholds, or the long-tail customer behaviors that surface at 1-in-1000 frequency. 100K stresses the test infrastructure beyond what most pre-prod environments can sustain and produces test runs that take days. 10K is the sweet spot: large enough to exercise realistic behavior distribution, small enough that the test itself doesn't become an operational burden.

If your production scale is materially above 100K, run the 10K test as the standard nightly and a 100K test pre-release. The 10K test catches 80%+ of production-readiness issues; the 100K test catches the remainder.

Corpus structure: what the synthetic dataset must contain

A 10K corpus dropped onto an API doesn't load-test the API — it load-tests an unrealistic uniform distribution. The corpus must reflect the production behavior distribution.

Concurrency profile: shape the load like production traffic

Constant-rate load tests are the easy case to pass and the unrealistic case for prod. Real production load has concurrency that bursts on quarter-end, market-close, and 9:30am ET market-open. The load profile should reflect this.

The profile we recommend: a baseline rate that ramps over 10 minutes (mimics a workday warm-up), holds for 30 minutes, transitions to a 5x burst for 5 minutes (mimics market open or quarterly close), drops back to baseline for 20 minutes, then ramps down. Total run is 75 minutes, which is enough to surface query-plan transitions, GC pauses on JVM-based services, and connection-pool saturation.

// k6 load profile pseudocode
export const options = {
  stages: [
    { duration: '10m', target: 100 },  // ramp up
    { duration: '30m', target: 100 },  // hold baseline
    { duration: '2m',  target: 500 },  // burst ramp
    { duration: '5m',  target: 500 },  // burst hold
    { duration: '3m',  target: 100 },  // burst tail
    { duration: '20m', target: 100 },  // post-burst hold
    { duration: '5m',  target: 0 }     // ramp down
  ],
  thresholds: {
    http_req_duration: [{
      threshold: 'p(99) < 500',
      abortOnFail: true
    }]
  }
};

Hot-account simulation: the failure mode you're testing for

In production, some accounts get queried more than others — high-AUM clients with active-management, accounts that just had a major event, accounts being viewed by a CSR investigating a ticket. Real hot-account distribution is roughly Pareto: 10% of accounts get 50% of the traffic.

Synthetic load that's uniformly distributed across the 10K-household corpus misses this. Configure the load to follow a Pareto distribution: 10% of households get 50% of requests. This produces hot-account contention that surfaces caching layer issues, lock contention on per-account state, and connection-pool starvation when the same hot account holds connections.

Read-vs-write mix

Wealth-tech APIs are typically read-heavy — 90%+ reads is common. But the writes are where most production incidents start. Configure the load to mirror your production read/write mix, then run a separate write-heavy test to specifically stress the write paths.

Write operations to include: account updates, transaction posting, document uploads, profile changes. Each of these has different downstream effects (cache invalidation, webhook fan-out, audit-log generation) that need to be exercised.

What to measure: the metrics the post-test review consumes

The post-test review wants four metric groups: latency profile, throughput sustained, error rate by error class, and resource utilization at the bottleneck. Don't just report aggregate latency — report by endpoint, by account-segment (hot vs. cold), and by request size.

Hot-key avoidance: the synthetic-data-specific risk

Synthetic corpora generated naively can have hot-keys — IDs that are sequential, timestamps that cluster, hash distributions that aren't uniform. Hot-keys in your test data produce hot-keys in your test load that are artifacts of generation, not realistic patterns.

Verify: ID distribution is uniform across the keyspace, timestamps are spread realistically across the longitudinal window, secondary keys (account types, product mixes) match production's distribution. WealthSchema corpora are generated with these properties; if you're using a custom-generated corpus, validate before load-testing.

Post-test: artifact for the production-readiness review

The artifact the review consumes is a 1-page summary plus a detailed appendix. Summary: 'sustained N RPS for X minutes at p99 < Y ms with Z% error rate.' Appendix: per-endpoint detail, hot-account vs. cold-account separation, error class breakdown, bottleneck identification, and the recommended remediation if any thresholds were exceeded.

File the artifact in the same place your other production-readiness reviews live. Re-run the test on a regular cadence (typically nightly during active development, weekly during stable maintenance, before every release).

Key takeaways

• 10K is the right scale for a daily load test — large enough to exercise realistic behavior distribution, small enough that the test isn't its own operational burden.
• Uniform-distributed load against a synthetic corpus is unrealistic. Pareto-shaped hot-account access is what catches the cache, lock, and connection-pool issues that bite in production.
• Burst profiles (5x for 5 minutes) catch query-plan transitions and GC pauses that constant-rate loads miss. The bursts are where production incidents actually originate.
• Hot-key artifacts in synthetic data corrupt load tests if not addressed. Verify ID, timestamp, and secondary-key distributions before relying on the test.

FAQ