Chaos Engineering for Kubernetes, built into the same CLI

Kubernetes is designed to self-heal, but those mechanisms only work when everything is configured correctly. Most clusters carry hidden fragility - single-replica deployments, missing pod disruption budgets, services with no circuit breakers - that traditional monitoring only surfaces after an incident. kubeqa chaos engineering injects controlled failures into your cluster so you discover those weaknesses before they cause an outage, not at 2 a.m. during a production incident.

How it works

Every kubeqa experiment follows the same structured workflow: validate steady state before injecting failure, apply the disruption within your blast-radius controls, observe the impact on the target and its dependencies, verify recovery, then assign a resilience score. If the experiment breaches your safety thresholds, kubeqa automatically rolls back the injection.

$ kubeqa chaos run pod-failure \
    --namespace production \
    --deployment api-gateway \
    --count 1 \
    --duration 60s \
    --abort-on error-rate>50%

  [1/5] Validating steady state...
  ✓ api-gateway: 3/3 pods ready, 0 restarts, p99 latency 42ms

  [2/5] Injecting failure: killing 1 pod...
  ✓ Pod api-gateway-7b9f4 terminated

  [3/5] Observing impact (60s window)...
    - Error rate: 0.3% (within threshold)
    - p99 latency: 89ms (degraded but acceptable)
    - Dependent services: no cascading failures

  [4/5] Verifying recovery...
  ✓ Replacement pod scheduled in 1.2s
  ✓ Pod ready in 6.8s
  ✓ Steady state restored in 8.0s

  [5/5] Resilience score: 4/5
    - Recovery time: 8.0s (good, <15s target)
    - Error rate during failure: 0.3% (excellent)
    - No cascading impact detected

What it covers

kubeqa ships with the failure injection primitives every cluster should test against:

• Pod kill - terminate pods and measure how fast Kubernetes replaces them. Tests replica count, readiness probes, and deployment config.
• Network partition - simulate a split between two services to reveal whether they degrade gracefully with circuit breakers and timeouts, or cascade into a broader failure.
• CPU stress - inject CPU pressure to test resource limits, HPA configuration, and priority classes under contention.
• Node drain - simulate a node going down for maintenance to test pod disruption budgets, anti-affinity rules, and scheduling constraints.

Each primitive runs with configurable count, duration, and abort conditions, plus a maximum duration ceiling so an experiment can never run away.

Why it matters

• Catch fragility before production. Chaos testing proves the real impact of single-replica deployments, missing PDBs, and absent circuit breakers - issues monitoring only reports after they cause downtime.
• One tool, not four to six. Resilience testing lives in the same CLI as health scanning, compliance auditing, and deployment gates, so chaos findings correlate directly with configuration and compliance findings in one unified score.
• Safe by default. Steady-state validation before and after every run, configurable abort thresholds, and automatic rollback mean you can graduate experiments from staging to production with confidence.
• Continuous, not one-off. Wire chaos into CI/CD as a resilience gate and track your resilience score over time instead of running occasional manual game days.

Steady state, blast radius, and scoring

The whole method rests on three controls. Steady-state validation confirms the target workload is genuinely healthy before you touch it, and re-confirms it recovered afterward - so a “passing” experiment means something. Blast-radius controls (count, namespace scope, duration, and abort conditions like --abort-on error-rate>50%) bound exactly how much damage an experiment can do. And the resilience score turns each run into a number based on recovery time, error rate, and degradation, so you can compare experiments and watch the trend move.

Compare that to running a standalone chaos platform. Tools like Litmus or Gremlin are separate products with their own setup, their own dashboards, and their own reports - and commercial chaos tooling typically starts around $1,500 per month. kubeqa bundles chaos with health, compliance, and gates in one free CLI, so resilience is part of the same quality pipeline rather than a tool you have to integrate and pay for separately.

Open source and pricing

The kubeqa CLI is Apache 2.0 licensed and free forever - chaos engineering included, with no feature gated behind a paywall. Paid kubeqa Cloud (coming soon) adds multi-cluster dashboards, historical resilience trends, team collaboration, SSO, audit logs, and compliance PDF exports, but it never restricts the core engine. See pricing for details.

Related

• Kubernetes Chaos Engineering with kubeqa: Breaking Clusters to Build Resilience - the deep-dive walkthrough of all five experiment types
• Cluster Health Scanner
• Compliance Automation
• Deployment Gates
• Kubernetes QA Tools Comparison

Test your cluster’s resilience in under five minutes: brew install nomadx-ae/tap/kubeqa. Star kubeqa on GitHub and see pricing.