Database Sharding
A horizontal scaling technique that partitions data across multiple database instances based on a shard key, distributing load and enabling datasets to grow beyond single-server limits.
Sharding splits a large database into smaller, more manageable pieces called shards. Each shard holds a subset of the data, determined by a shard key (like user ID or tenant ID). Queries are routed to the appropriate shard based on this key, and each shard operates as an independent database instance.
The primary benefit is horizontal scalability. When a single database server can no longer handle the load, sharding distributes both data and query traffic across multiple servers. This allows the system to scale linearly by adding more shards as data grows.
Sharding introduces significant complexity: cross-shard queries are expensive, rebalancing shards when data distribution is uneven is operationally challenging, and application logic must be shard-aware. Many teams find that read replicas, caching layers, and vertical scaling defer the need for sharding. When sharding becomes necessary, choosing the right shard key is the most critical decision, as it determines data distribution and query routing efficiency.
Related Terms
A/B Testing
A controlled experiment comparing two or more variants to determine which performs better on a defined metric, using statistical methods to ensure reliable results.
Feature Flag
A software mechanism that enables or disables features at runtime without deploying new code, used for gradual rollouts, A/B testing, and targeting specific user segments.
MLOps
The set of practices combining machine learning, DevOps, and data engineering to reliably deploy, monitor, and maintain ML models in production.
Model Serving
The infrastructure and systems that host trained ML models and handle inference requests in production, optimizing for latency, throughput, and cost.
Semantic Search
Search that understands the meaning and intent behind a query rather than just matching keywords, typically powered by embedding-based similarity comparison.
CI/CD (Continuous Integration / Continuous Deployment)
An automated software practice where code changes are continuously integrated into a shared repository, tested, and deployed to production, reducing manual intervention and accelerating delivery cycles.