Latency and Performance | Learn System Design
Latency and Performance
Latency and performance decide what using the system actually feels like, and they decide how much data and traffic it can carry before it starts to buckle.
Three levers control them: where the data sits, how work is spread, and what gets remembered.
Data locality shortens the distance work has to travel, load balancing spreads the work out, and caching removes work altogether
Data Locality
Data locality is the organization and distribution of data within a system so that as little data as possible has to be moved between nodes.
Store related data together, and store it near the nodes that access it most often. Retrieval gets faster because the data has less distance to travel, and the system as a whole gets faster because the network is doing less.
The techniques that achieve it are data partitioning, sharding, and data replication.
Load Balancing
Load balancing distributes incoming network traffic or computational work across multiple nodes so that no single node is overwhelmed.
That does three things: resources get used more evenly, response times drop, and the system stops being able to overload one machine while others sit idle.
Several algorithms do the distributing:
- Round-robin, which hands each request to the next node in turn.
- Least connections, which sends each request to whichever node is currently handling the fewest.
- Consistent hashing, which maps each request to a node in a way that survives nodes being added or removed.
Round-robin is blind but cheap. Least connections reacts to actual load. Consistent hashing keeps requests for the same key landing on the same node, which is what makes caching at the node level worth anything.
Caching
Caching stores frequently accessed data or computed results temporarily, so the system can retrieve it quickly instead of recalculating it or fetching it from the primary data source again.
The saving is not the copy, it is the work you no longer do.
Picture a service that keeps recomputing the same expensive result and refetching the same rows from its primary data source. Every request pays the full cost, and latency climbs with traffic. A cache holds those frequently accessed results temporarily, so they are read from cache instead of recomputed or refetched, and the primary data source stops being the bottleneck.
Common caching strategies are in-memory caching, distributed caching, and content delivery networks (CDNs).
Without a cache every request reaches the primary data source, and with one the repeated work is answered from a temporary copy
How They Fit Together
| Lever | What it changes | What it saves |
|---|---|---|
| Data locality | Where data sits relative to the nodes using it | Network transfer between nodes |
| Load balancing | Which node handles each request | Time lost to one overloaded node |
| Caching | Whether the work is done at all | The work itself |
Read the right-hand column downwards. Locality shortens the trip, load balancing keeps any one machine from becoming the queue, and caching removes the work. The third is the largest saving, which is why caching is usually the first thing to reach for and the first thing to get wrong.
💡 When you are asked to make something faster, say which of the three you are pulling and why. "The read is slow because it is a cross-region call, so this is a locality problem, not a caching one" is a diagnosis. Reaching for a cache before knowing where the time goes is guessing.
Key takeaway: Latency and performance come down to three levers. Data locality places data near the nodes that use it, achieved through partitioning, sharding and replication. Load balancing spreads work across nodes using round-robin, least connections or consistent hashing. Caching stores frequently accessed data or results temporarily so the same work is not repeated, through in-memory caches, distributed caches and CDNs.