Circuit Breaker
A resilience pattern that monitors for failures in downstream service calls and temporarily stops making requests when a failure threshold is exceeded, preventing cascade failures across the system.
The circuit breaker pattern works like an electrical circuit breaker. In the closed state, requests flow normally. When failures exceed a threshold (e.g., 50% error rate over 10 seconds), the circuit opens, and subsequent requests fail immediately without calling the downstream service. After a cooldown period, the circuit enters a half-open state, allowing a few test requests to determine if the service has recovered.
This pattern prevents a failing downstream service from taking down the entire system. Without circuit breakers, a slow or failing dependency causes requests to queue up, consuming thread pools and memory, eventually cascading the failure to upstream services and creating system-wide outages.
For AI systems that depend on external model APIs, circuit breakers are essential. If an LLM provider experiences degraded performance or outages, a circuit breaker can immediately fail over to a cached response, a simpler model, or a graceful fallback experience rather than letting requests pile up and degrade the entire application.
Related Terms
A/B Testing
A controlled experiment comparing two or more variants to determine which performs better on a defined metric, using statistical methods to ensure reliable results.
Feature Flag
A software mechanism that enables or disables features at runtime without deploying new code, used for gradual rollouts, A/B testing, and targeting specific user segments.
MLOps
The set of practices combining machine learning, DevOps, and data engineering to reliably deploy, monitor, and maintain ML models in production.
Model Serving
The infrastructure and systems that host trained ML models and handle inference requests in production, optimizing for latency, throughput, and cost.
Semantic Search
Search that understands the meaning and intent behind a query rather than just matching keywords, typically powered by embedding-based similarity comparison.
CI/CD (Continuous Integration / Continuous Deployment)
An automated software practice where code changes are continuously integrated into a shared repository, tested, and deployed to production, reducing manual intervention and accelerating delivery cycles.