System design patterns for scalable APIs

Your shiny new API just launched, and you’re already hearing chirps about slow responses from your biggest customer. The P99 latencies are creeping up. You thought you built a good system, but now you’re debugging connection pool exhaustion at 3 AM. Sound familiar? We’ve all been there. Building a truly scalable API isn't just about throwing more instances at the problem; it's about understanding fundamental system design patterns that prevent these fires before they even start.

The Foundation: Statelessness and Idempotency

Let's start with the absolute basics, stuff you'd get grilled on in any serious interview. If your API isn't stateless, you're tying yourself in knots before you even ship. Stateless services don't store client-specific data between requests. This makes scaling trivial—just add more instances behind a load balancer. Each instance is interchangeable. You don’t have to worry about session stickiness or replicating session state across a cluster, which is a total nightmare to manage at scale. Seriously, don't do it.

Idempotency is the cousin to statelessness, and just as critical. An idempotent operation produces the same result whether you call it once or a hundred times. Think of a DELETE request: deleting a resource once or ten times should still result in that resource being gone. This is huge for fault tolerance. If a client times out after sending a request, they can safely retry it without causing unintended side effects. For writes, this often means including a unique client-generated request ID, allowing your backend to detect and ignore duplicate submissions. Stripe does this brilliantly with their Idempotency-Key header. You should too.

Asynchronous Processing: When Real-time Isn't Real-time

Not every request needs an immediate, synchronous response. In fact, most don't. Trying to process every complex operation synchronously will crater your API's performance under load. This is where asynchronous processing comes in, and it's a cornerstone of scalable design.

Imagine a user uploading a large video file. You don't want their HTTP request hanging open for five minutes while you transcode it. Instead, accept the upload, return a 202 Accepted status with a job ID, and then hand off the transcoding to a background worker. The client can poll a separate endpoint with that job ID to check the status, or even better, you can use webhooks or WebSockets to notify them when it's done.

Message queues are your best friends here. Kafka, RabbitMQ, SQS—pick your poison. When a request comes in that requires heavy lifting, your API service publishes a message to a queue. A separate pool of worker services consumes messages from that queue, processes them, and updates the state. This decouples your frontend API from your backend processing, making each component independently scalable and resilient. If a worker dies, the message stays in the queue for another worker to pick up. If your API service is slammed, it can still quickly accept requests and offload work.

Caching Strategies: Your First Line of Defense

Caching is probably the single most effective way to improve API performance and reduce database load for read-heavy workloads. But it's not a magic bullet; you need a strategy.

You've got a few layers for caching:

• Client-side caching: HTTP caching headers (Cache-Control, ETag, Last-Modified) tell browsers and CDNs how long they can cache responses. This is the cheapest cache hit you can get.
• CDN caching: For static content and even dynamic API responses that don't change frequently, a CDN like Cloudflare or Akamai can serve content from an edge location closer to your users, drastically cutting latency.
• API Gateway caching: Some API gateways offer caching capabilities, letting you cache responses before they even hit your backend services.
• In-memory caching: Within your service, libraries like Guava Cache (Java) or simple hash maps can store frequently accessed data. Fast, but volatile and limited to a single instance.
• Distributed caching: Redis or Memcached are the go-to solutions here. They run as separate services, allowing multiple API instances to share a cache. This is where you store data that's expensive to compute or fetch from the database. Think user profiles, product catalogs, or API rate limits.

The trick with caching is invalidation. "Cache invalidation is one of the two hard problems in computer science," right? You need to decide:

1. Time-based invalidation (TTL): Data expires after a certain period. Simple, but can lead to stale data if updates happen before expiry.
2. Event-driven invalidation: When data changes in your source of truth (e.g., your database), you publish an event that tells your caches to invalidate or update that specific entry. More complex, but ensures freshness.

Don't just cache everything. Profile your API, find your hot spots, and cache only what makes sense. Over-caching can introduce more complexity than it solves. For instance, caching highly dynamic, personalized user dashboards might lead to more headaches with invalidation than the performance gain is worth.

Rate Limiting and Throttling: Protecting Your API

You built a great API, people want to use it. That's fantastic! Until someone decides to hammer your endpoints with 10,000 requests per second, either maliciously or accidentally. Without rate limiting, your API will fall over.

Rate limiting restricts the number of requests a user or client can make within a given time window. Throttling is similar, often implying a more dynamic adjustment of limits. Implement this early. Seriously.

Common algorithms include:

• Fixed Window Counter: Simplest. Track requests in a fixed time window (e.g., 100 requests per minute). Resets abruptly at the window's end. Can suffer from a "burst" problem right at the window boundary.
• Sliding Window Log: More accurate. Store timestamps of each request. When a new request arrives, count how many timestamps fall within the last N seconds/minutes. Requires more memory.
• Sliding Window Counter: A hybrid. Divides the time into smaller windows, and uses the average of the current and previous window's counts, weighted by how much of the current window has passed. Good balance of accuracy and memory.
• Leaky Bucket: Models a bucket with a fixed capacity that leaks at a constant rate. Requests add water to the bucket. If the bucket overflows, requests are dropped or queued. Smooths out bursts.
• Token Bucket: Similar to leaky bucket, but tokens are added to a bucket at a fixed rate. A request consumes a token. If no tokens are available, the request is denied. Allows for bursts up to the bucket's capacity.

You can implement rate limiting at various layers:

• API Gateway: Tools like AWS API Gateway, Nginx, or Kong can handle this out of the box. This is often the easiest and most effective place to do it.
• Service Layer: If you need more fine-grained, business-logic-driven rate limits, you might implement it within your services using a distributed cache (like Redis) to store and increment counters.

Don't forget to communicate your rate limits clearly in your API documentation. Use standard HTTP status codes like 429 Too Many Requests and include Retry-After headers.

Database Scaling Strategies: Beyond Just Vertical Scaling

Your database is almost always the bottleneck in a scalable API. You can horizontally scale your application servers all day long, but if they're all hitting a single database instance, you're toast.

• Read Replicas: The simplest scaling for read-heavy workloads. Spin up multiple read-only copies of your database. Your API can then distribute read queries across these replicas. Writes still go to the primary. This is a common pattern for relational databases like PostgreSQL or MySQL.

• Sharding (Horizontal Partitioning): When a single database instance can't handle the load, even with read replicas, you partition your data across multiple independent database instances (shards). Each shard holds a subset of your data. For example, users could be sharded by their ID. This significantly increases your database's capacity for both reads and writes. It also adds complexity:

  • Sharding Key: Choosing a good sharding key is crucial. It needs to distribute data evenly and minimize cross-shard queries.
  • Rebalancing: What happens when one shard gets too big or too hot? You'll need a strategy to rebalance data.
  • Distributed Transactions: Transactions spanning multiple shards become much harder.
  • Joins: Joins across shards are either impossible or incredibly inefficient.

• Denormalization: Sometimes, replicating data or pre-calculating aggregates can save expensive joins or queries. Instead of always joining users and orders tables, you might store a user's last_order_date directly in the users table. This trades off storage space and write consistency for read performance. It's a pragmatic choice for many high-traffic APIs.

• Polyglot Persistence: Don't be afraid to use different types of databases for different data needs.

  • Relational databases (PostgreSQL, MySQL) for structured, transactional data.
  • NoSQL document stores (MongoDB, DynamoDB) for flexible, semi-structured data.
  • Key-value stores (Redis, Cassandra) for high-speed lookups and caching.
  • Graph databases (Neo4j) for highly connected data.

The point is, don't force a square peg into a round hole. Pick the right tool for the job. You wouldn't use Redis as your primary transactional database for financial records, just like you wouldn't use PostgreSQL to store a billion user sessions for real-time analytics.

Observability: See What's Happening

You can build the most scalable API in the world, but if you don't know what's happening when things go wrong, you're flying blind. Observability isn't just about logs; it encompasses metrics, traces, and logging.

• Metrics: Collect quantitative data about your system's performance. Response times (P50, P90, P99), error rates, request counts, CPU utilization, memory usage, database connection pool size—these are your vitals. Prometheus, Datadog, New Relic are common tools. Set up dashboards and alerts. If your P99 latency jumps by 200ms, you want to know immediately.

• Logging: Detailed records of events within your application. Don't just dump raw messages; structure your logs (JSON is great) so they're easily searchable and parsable. Include request IDs, correlation IDs, and relevant context in every log entry. This allows you to trace a single request through multiple services. Tools like ELK stack (Elasticsearch, Logstash, Kibana), Splunk, or Datadog Logs are essential here.

• Distributed Tracing: When a request goes through multiple microservices, it's incredibly hard to pinpoint where latency is introduced or where an error originated. Distributed tracing systems (OpenTelemetry, Jaeger, Zipkin) assign a unique trace ID to each request as it enters your system. This ID is then propagated across all services involved in processing that request. You can then visualize the entire request flow, seeing the time spent in each service and identifying bottlenecks. This is a must-have for complex microservice architectures.

Invest in observability early. Trying to add it when your system is already on fire is like trying to install seatbelts during a crash. It's too late.

API Gateway & Microservices: Managing Complexity at Scale

As your API grows, a single monolithic service becomes harder to manage, deploy, and scale. This is where the microservices pattern often comes into play, managed by an API Gateway.

• API Gateway: This acts as a single entry point for all client requests. It handles:

  • Routing: Directing requests to the appropriate backend service.
  • Authentication/Authorization: Validating tokens, applying access policies.
  • Rate Limiting: As discussed earlier.
  • Caching: Some basic caching.
  • Request/Response Transformation: Modifying payloads.
  • Monitoring/Logging: Centralized collection.
  • Circuit Breaking: Preventing cascading failures.

  Tools like AWS API Gateway, Kong, Nginx, or even a custom-built service can serve this role. It decouples clients from individual microservices, providing a stable interface.

• Microservices: Breaking down a large application into smaller, independently deployable services. Each service owns its data, has a clear responsibility, and can be scaled independently.

  • Benefits: Independent deployment, technology diversity (different languages/frameworks per service), improved fault isolation, easier to scale individual components.
  • Drawbacks: Increased operational complexity, distributed transactions are a nightmare, data consistency across services is harder (eventual consistency often required), inter-service communication overhead.

Don't jump to microservices just because it's trendy. A well-designed monolith is often simpler and more efficient for early-stage products. The complexity of microservices is a tax you pay for the flexibility and independent scaling they offer. Start with a modular monolith and extract services when the pain points become undeniable. For example, if your notification service is causing outages for your core product, maybe that's a good candidate for extraction.

Designing for Failure: Resilience Patterns

Scalable systems will fail. Components will go down. Networks will partition. You need to design for these eventualities.

• Circuit Breakers: Imagine a client service repeatedly trying to call a backend service that's currently unhealthy. This can exhaust resources on both ends. A circuit breaker pattern wraps calls to external services. If a certain number of calls fail within a threshold, the circuit "trips," preventing further calls to that service for a period. Instead of making the call, it immediately returns an error or a fallback response. After a timeout, it tries a single call to see if the service has recovered (half-open state). Hystrix (though in maintenance mode) popularized this, but many libraries exist in various languages.

• Bulkheads: Isolate components so that a failure in one doesn't bring down the entire system. Think of a ship with watertight compartments. If one compartment floods, the ship doesn't sink. In software, this means isolating resource pools (thread pools, connection pools) for different services or request types. If your payment processing service starts hogging all database connections, it shouldn't starve your user profile service.

• Retries with Exponential Backoff: When a transient error occurs (e.g., a network glitch, a temporary service unavailability), simply retrying the request immediately might just exacerbate the problem. Instead, retry after increasing delays (e.g., 1s, 2s, 4s, 8s). Add some jitter (randomness) to the backoff to prevent all clients from retrying at the exact same moment, which could create a "thundering herd" problem.

• Timeouts: Every external call you make—database queries, HTTP requests to other services, message queue operations—should have a timeout. Services that don't respond within an expected timeframe are usually deadlocked or overloaded. You don't want your service waiting indefinitely, tying up resources.

• Graceful Degradation: What if a non-critical service fails? Can your API still function, albeit with reduced features? If your recommendation engine is down, can you still serve product pages without recommendations? Design your API so critical functionality can operate even if secondary services are unavailable.

Building resilient systems requires a mindset shift: assume failure, don't just hope for success.

Putting It All Together: A Mental Model for Interviews

When you're in a system design interview, they're not looking for you to perfectly recall every single tool or algorithm. They want to see your mental model for building scalable systems.

Here's how I approach it, and how you should too:

1. Clarify Requirements: Always start here. What's the scale? QPS? Data size? Latency requirements? Read-heavy or write-heavy? What are the consistency needs? What's the budget? What's the team size? This shapes everything.
2. Estimate Scale: Back-of-the-envelope calculations. 100 QPS? 10,000 QPS? This determines if you need sharding, how big your caches need to be, etc.
3. High-Level Architecture: Draw a box diagram. Client -> API Gateway -> Services -> Database.
4. Deep Dive into Bottlenecks:
   • Reads: Caching (client, CDN, distributed), read replicas, denormalization.
   • Writes: Asynchronous processing (queues), sharding, batching.
   • Service Communication: API Gateway, microservices vs. monolith.
   • Failure: Circuit breakers, retries, timeouts, bulkheads.
5. Data Modeling: How would you store the data? Relational? NoSQL? Why?
6. API Design: RESTful? gRPC? Idempotency? Error handling?
7. Monitoring & Alerts: How do you know if it's working? Logs, metrics, traces.

Don't just list patterns. Explain why you'd use them, what problem they solve, and what trade-offs they introduce. For example, "I'd use Redis for distributed caching because it's fast, in-memory, and supports various data structures, but I'd need to consider cache invalidation strategies like TTLs or event-driven updates to prevent stale data." That's a much stronger answer than "I'd use Redis."

Remember, it's a conversation. Engage with your interviewer. Ask clarifying questions. Defend your choices, but be open to alternatives. There's rarely a single "right" answer in system design, only trade-offs. The best solution depends entirely on the specific constraints and requirements.

Ready to Ace Your Next Interview?

Practice with AI-powered mock interviews tailored to your target role and company. Start Practicing for Free | Explore Interview Prep