A circuit breaker sits between your code and a service that you call. When the service keeps failing, the breaker stops the calls for a while and answers with an error at once.
- A solid dot is a call that got an answer.
- An orange ring is a call that failed.
- A grey dash is a call that the breaker did not send.
- Every figure below uses the same breaker. It opens when 5 of the last 10 calls fail, and it stays open for 3 seconds.
Without a breaker
Your service has 5 workers, and every request needs one call to another service. The call gives up after 3 seconds.
- Press Make the service slow. Every worker now waits 3 seconds for an answer that does not come.
- You get 2 requests per second, so all 5 workers are busy after about 2 seconds and the next request finds no free worker.
- Now your service is down too, and the service that calls you is next. Michael Nygard wrote about this in Release It, the book that made the circuit breaker popular.
Closed
A closed breaker passes every call and remembers how the last 10 calls ended.
- At 40% the breaker counts 4 failures in 10 calls and stays closed, because a few bad calls are not an outage.
- Move the first slider to 50% and the fifth failure opens the breaker.
- Set the second slider to 2 and the same 40% opens it. A low number reacts fast, and it also opens on a bit of noise.
- A timeout counts as a failure too, so a slow service opens the breaker the same way as a dead one.
Open
An open breaker does not call the service. It answers at once, so no worker waits.
- The first calls still time out, because the breaker needs 5 failures before it opens. A shorter timeout makes this part shorter.
- After that only the trial call takes a worker, and every other request gets its answer at once.
- You decide what the caller gets while the breaker is open. An error is the simple choice, and a cached copy or a default value is often better.
Half-open
After 3 seconds the breaker lets one trial call through. It closes when that call works, and it opens for another 3 seconds when the call fails.
- A broken service gets one call every 3 seconds and not the full traffic, so it has room to recover.
- A long open time costs you after the fix, because the service works again and the breaker still refuses calls. Set the slider to 8 seconds and fix the service to see it. With a short open time you send more trial calls to a service that is still down.
Library defaults
You do not write a breaker yourself. Every library has the same three settings, with different defaults.
- Opens when
- 50% of the last 100 calls fail
- Stays open for
- 60 seconds
- Half-open
- 10 trial calls
- Resilience4j and Polly look at the share of calls that fail, and gobreaker counts failures in a row.
- Resilience4j and Polly wait for 100 calls before they decide. A service with little traffic takes a long time to get there, so set the numbers for your own traffic.
- Hystrix from Netflix is in maintenance mode, and its README sends new projects to Resilience4j.
Where to use it
Use a breaker on a call that crosses a network and can stay broken for minutes. Pick a place to see what the breaker counts there.
- You call
- Another service
- Counts as a failure
- Timeouts and 5xx answers
- While it is open
- A cached copy or a default value
- The Claude API answers
529when it is overloaded. A breaker that counts those can send the next requests to a second model. - Envoy and Istio put the breaker in the proxy, so your code does not change. Envoy marks a rejected request with the
x-envoy-overloadedheader. - Do not put a breaker on a call inside one process. Microsoft's pattern guide says that it only adds overhead there.
- Count only the failures of the service. A
400is a mistake of the caller, and it says nothing about the health of the service.
Sources
Newsletter
Email from me
Occasional notes on agentic development — new posts, videos, and tools I'm actually using.