Phase 6: Testing at Scale

In one line: The testing pyramid expands dramatically — tens of thousands of unit tests, contract tests between services, chaos engineering against production, plus dedicated security and compliance testing pipelines.

In plain English

At a startup, testing is "Vitest, Playwright for critical paths, maybe a Sentry alert." At an enterprise, testing is a whole org of disciplines: unit tests with coverage requirements, contract tests that verify services keep their promises, chaos engineering that deliberately breaks production to make sure failover works, plus security tooling that scans every dependency.

Why the expansion? Because at this scale, you can't keep the whole system in one engineer's head. The tests are how the system explains itself back to you.

The expanded pyramid

Unit tests: Tens of thousands. Strict coverage requirements (often 80%+ for critical services).

Integration tests: Per-service, against test instances of dependencies (or contract tests).

Contract tests: Pact or similar. Verify services adhere to their published API contracts. Critical in microservices.

End-to-end tests: Limited and carefully maintained. Flaky E2E tests are a known plague at scale; teams invest heavily in fixing or removing them.

Load tests: Tools like k6, Gatling, JMeter, or in-house. Simulate production traffic patterns. Run before major releases.

Chaos engineering

• Tools like Chaos Monkey, Gremlin, LitmusChaos deliberately break things in production (or staging) to verify resilience.
• "Do failovers actually work?" The answer is often "no" until tested.

Game days: Scheduled exercises where teams simulate failures and rehearse responses.

Highlight: untested failovers don't work

Every system claims to have automatic failover. At any scale, the actual fraction of failovers that work the first time you trigger them in anger is much lower than the fraction that worked on the design doc.

The only way to find out is to deliberately break things in a controlled setting. That's what chaos engineering and game days are for: stress-test your assumptions before reality stress-tests them at 3 AM on Black Friday.

Security testing

A full security testing stack:

• SAST (Static) — Semgrep, CodeQL, Snyk Code.
• DAST (Dynamic) — OWASP ZAP, Burp Suite Enterprise.
• SCA (Dependencies) — Snyk, Dependabot, custom scanners.
• Container scanning — Trivy, Clair.
• Infrastructure scanning — Checkov, tfsec.
• Penetration testing — Quarterly or continuous via bug bounty.
• Red team exercises — Periodic.

Each tool covers a different layer. SAST sees the source code, DAST sees the running app, SCA sees the dependency tree, container scanners see the image, IaC scanners see the Terraform. You need all of them because attacks come from all of those layers.

Compliance testing

• Automated checks for HIPAA, PCI-DSS, SOX, GDPR requirements.
• Audit trail verification.
• Data retention enforcement.

Compliance testing is how you prove to auditors that the controls aren't just written down — they're actually enforced. Every quarter, the audit team runs the test suite and the output is the evidence that, for instance, all production database access was logged.

Worked example: how Netflix discovered Chaos Engineering

Netflix's "Chaos Monkey" started as a tool that randomly terminated production EC2 instances during business hours. The hypothesis: if our system is truly resilient, killing a random instance shouldn't matter.

The first few months they ran it, things broke constantly. Each break uncovered a hidden assumption: "this service can't tolerate a dependency restart," "this retry logic has a bug," "this load balancer doesn't drain connections correctly."

Once they fixed every break Chaos Monkey found, they had genuinely resilient systems — not because they'd designed for resilience, but because they'd been forced to. That's the core idea of chaos engineering: never trust a recovery path that hasn't been exercised under stress.

Common mistakes

Where people commonly trip up

• Letting a flaky E2E suite rot in CI. A 5% flake rate on 200 tests means roughly every run fails for "reasons." Engineers learn to re-run until green, which trains everyone to ignore real failures. Quarantine flaky tests immediately and either fix or delete them — don't let "it's just flaky" become acceptable.
• Chasing a coverage percentage instead of meaningful tests. 90% line coverage with shallow tests catches less than 60% with tests that exercise real edge cases. Don't set the coverage gate so high that teams game it with expect(true).toBe(true).
• Running chaos engineering without an error budget to absorb it. If breaking a service in staging would derail the team's launch, nobody will let you do it. Chaos only works when the org genuinely accepts that finding a problem now is cheaper than finding it at 3 AM.
• Bolting on contract tests after the fact. Pact or similar contract tests work when producers and consumers both opt in from day one. Retrofitting them across an existing 50-service mesh is months of work and usually ends with half-finished adoption — start contracts when you start splitting services.
• Treating security scanners as "noise from the AppSec team." A SAST finding ignored for six months is a CVE waiting for an auditor. Triage on a schedule, suppress with a reason (not silently), and treat the dashboards like a queue, not a wall.

Page checkpoint

Checkpoint Quiz

Did enterprise testing stick?

Required

What's next

→ Continue to Phase 7: CI/CD at Scale — how all these tests actually run, in parallel, across tens of thousands of builds a day.