Failure Lab

Break systems on purpose. Learn forever.

Each lab is a uniquely engineered broken system. Read the problem, tune the knobs, watch the simulation react, and unlock the cause & solution.

Easy

Duplicate orders

An e-commerce checkout service is charging customers twice for the same order. The queue, the consumer, and the payment provider all log success — yet two charges go through.

checkout-api → orders

Enter lab

Medium

DLQ overflow

A single 'poison' message is being received over and over. Every retry burns consumer time and the DLQ never receives it. Healthy traffic is being starved.

event-ingest → events

Enter lab

Medium

Latency spike

P99 receive latency spikes every few seconds. The consumer is healthy and the queue isn't deep — yet messages sit visible for surprisingly long.

api-gateway → notifications

Enter lab

Hard

Slow consumer trap

A video transcoding worker takes 25 ticks per message, but visibility timeout is 10. The queue is small but workers report processing the same job dozens of times.

upload-service → transcode-jobs

Enter lab

Hard

Lost batches

A producer sends batches of 10 messages. Some messages from each batch vanish — they're never received by a consumer and never reach the DLQ.

analytics-ingest → metrics-batch

Enter lab

Expert

Out-of-order FIFO

A FIFO queue is processing messages out of order despite being… FIFO. Customer 'A' sees their 'shipped' event before their 'paid' event.

order-events → orders.fifo

Enter lab