KubernetesTerraformAWSDockerAnsiblePrometheusGrafanaHelmArgoCDVault
Logo
47 Labs Online3 Active Incidents

Break infrastructure.
On purpose.
Before production breaks it for you.

Spin up isolated environments that mirror your stack. Inject failures, trace blast radius, and ship fixes — all without a post-mortem to write afterward.

47labs
Live Environments
2.1kengineers
Active This Week
99.2%
Experiment Success
< 90sec
Avg Spin-up Time
// 01INCIDENT · CASCADING TIMEOUTIntermediate45 min

ALERT: upstream service /api/orders exceeded 30s threshold — 14 dependent services now queued behind a dead lock.

When one slow service takes down twelve healthy ones.

Timeouts don't fail fast — they queue. And queued threads hold connections. And held connections starve your database pool. You've seen this at 2 a.m. Now you can reproduce it in 90 seconds, trace the blast radius, and wire in circuit breakers before it ever touches production.

Lab Environment

Timeout Cascade Lab — v2.3

Kubernetes cluster with 6 microservices. Inject latency into the orders service via tc-netem, watch the dependency graph collapse in Grafana, then implement bulkhead + timeout patterns to contain the failure.

KubernetesIstioGrafanatc-netemCircuit Breaker
infralab — bash — 80×24
infra@lab:~$infralab create --scenario cascading-timeout --nodes 6
✓ Provisioning k8s cluster (3 control, 3 worker)...
✓ Deploying microservice mesh (orders, inventory, billing, notify, auth, gateway)
✓ Grafana dashboard ready at :3000
infra@lab:~$infralab inject latency --service orders --delay 31s --jitter 5s
⚠ Injecting 31s latency into orders:8080
✗ gateway/thread-pool: 100% utilization (512/512 threads)
✗ billing: connection pool exhausted — 0/20 available
✗ inventory: upstream timeout after 30004ms
✗ notify: queue depth 4800 — processing halted
# Blast radius: 4 of 6 services degraded. Fix: circuit breaker.
infra@lab:~$infralab apply-fix --pattern circuit-breaker --threshold 5s
✓ Circuit breaker armed — fallback responses enabled
✓ Downstream services recovering (billing: 18/20 conns)
infra@lab:~$
// 02INCIDENT · DISK PRESSUREBeginner30 min

CRITICAL: node worker-03 disk at 98.4% — kubelet evicting pods, PVCs detaching, logs vanishing mid-write.

Disk fills. Pods die. Logs disappear at the worst moment.

Disk pressure is invisible until it's catastrophic. Your node hits 95%, kubelet starts evicting low-priority pods, then medium-priority, then your database sidecar. By the time you notice, the logs that would have told you why are gone. This lab teaches you to detect pressure early, set eviction thresholds, and configure log rotation that survives the incident.

Lab Environment

Disk Pressure Eviction Lab — v1.8

Single-node k8s with constrained 2GB disk. Fill disk with a log-flooding deployment, observe kubelet eviction order, then implement proper resource quotas, log rotation, and PVC size limits.

KuberneteskubeletPVClogrotateResource Quotas
infralab — bash — 80×24
infra@lab:~$infralab create --scenario disk-pressure --disk-limit 2Gi
✓ Node provisioned: worker-lab (2 CPU, 4GB RAM, 2GB disk)
✓ Deploying flood-logger (writes 100MB/s to /var/log)
infra@lab:~$kubectl get nodes -w
NAME STATUS ROLES AGE VERSION
worker-lab Ready <none> 2m v1.28.4
worker-lab Ready,DiskPressure <none> 3m v1.28.4
worker-lab NotReady,DiskPressure <none> 3m30s v1.28.4
✗ pod/api-server-7d9f8 evicted — DiskPressure
✗ pod/postgres-0 evicted — DiskPressure
# Disk: 1.98Gi/2Gi (99.1%). PVCs detached.
infra@lab:~$infralab apply-fix --pattern log-rotation --eviction-threshold 80%
✓ logrotate configured: 50MB max, 7 rotations
✓ kubelet eviction.hard: nodefs.available<20%
✓ Node recovered — disk at 61.4%
infra@lab:~$
// 03INCIDENT · DNS POISONINGAdvanced60 min

ALERT: internal DNS resolving payments.internal to 10.0.4.99 — unknown host, traffic silently redirected for 18 minutes.

Your DNS lied. Traffic went somewhere it shouldn't.

DNS failures are the cruelest kind — they're silent, they're intermittent, and by the time you check DNS you've already ruled it out twice. This lab walks you through a poisoned CoreDNS ConfigMap, traces the packet path with tcpdump, and shows you how to lock down DNS with DNSSEC validation and network policies before your next deploy.

Lab Environment

DNS Redirect & Poisoning Lab — v3.1

CoreDNS cluster with manipulated ConfigMap rewriting payments.internal to a honeypot IP. Use dig, tcpdump, and k8s audit logs to detect the redirect, then harden with DNSSEC and admission webhooks.

CoreDNSDNSSECtcpdumpOPANetwork Policy
infralab — bash — 80×24
infra@lab:~$dig payments.internal @10.96.0.10
;; ANSWER SECTION:
payments.internal. 30 IN A 10.0.4.99 ← WRONG
;; Expected: 10.0.2.15 (payments-svc ClusterIP)
infra@lab:~$kubectl get configmap coredns -n kube-system -o yaml | grep rewrite
rewrite name payments.internal honeypot.internal
infra@lab:~$tcpdump -i eth0 dst 10.0.4.99 -c 20
18:23:41 IP 10.0.2.8.45231 > 10.0.4.99.443: Flags [S]
18:23:41 IP 10.0.2.9.51204 > 10.0.4.99.443: Flags [S]
# 847 packets captured — 18 min exposure window
infra@lab:~$infralab apply-fix --pattern dnssec --admission-webhook dns-policy
✓ DNSSEC validation enabled on CoreDNS
✓ OPA webhook blocking ConfigMap mutations in kube-system
✓ Audit log alert: any A-record rewrite → PagerDuty
infra@lab:~$
// 04INCIDENT · CERTIFICATE EXPIRYIntermediate50 min

CRITICAL: TLS cert for api.internal expired 4 hours ago — all HTTPS handshakes failing, health checks timing out, on-call paged at 03:47.

The cert expired. Nobody set a reminder. You know the rest.

Certificate expiry is the most embarrassing production incident because it's 100% preventable. But cert-manager misconfiguration, auto-renewal failures, and Let's Encrypt rate limits have humbled teams at every scale. This lab lets you reproduce a cert expiry mid-traffic, diagnose the TLS handshake failure chain, and build the monitoring that would have paged you 30 days earlier.

Lab Environment

TLS Cert Expiry & Rotation Lab — v2.0

NGINX ingress with cert-manager. Artificially expire the TLS certificate, observe the handshake failure cascade, then implement cert-manager ClusterIssuer with 30-day pre-expiry alerts and automated rotation.

cert-managerNGINX IngressLet's EncryptPrometheusACME
infralab — bash — 80×24
infra@lab:~$openssl s_client -connect api.internal:443 2>&1 | grep -E "verify|expire"
verify error:num=10:certificate has expired
notAfter=Feb 23 03:47:01 2026 GMT ← EXPIRED 4h ago
infra@lab:~$kubectl get certificate -n production
NAME READY SECRET AGE
api-tls False api-tls-secret 94d
# cert-manager renewal failed: ACME rate limit hit
infra@lab:~$kubectl describe certificaterequest api-tls-xxxxx
Message: 429 urn:ietf:params:acme:error:rateLimited
Retry after: 2026-02-28T03:47:00Z (23h remaining)
infra@lab:~$infralab apply-fix --pattern cert-rotation --notify-days 30,14,7
✓ ClusterIssuer configured with ZeroSSL fallback
✓ PrometheusRule: alert if cert_expiry_seconds < 2592000
✓ Cert renewed — valid until 2026-05-28T03:47:00Z
infra@lab:~$
You've recognized your own 3 a.m. incident.

The lab that would have
prevented it is live now.

No credit card. No setup. GitHub SSO gets you into a running environment in under 90 seconds. 47 labs online right now.

github.com SSO · spins up in <90s