Introduction to Scaling
Introduction to scaling concepts in distributed systems.
Scaling is the ability of a system to handle increasing load by adding or upgrading resources while maintaining acceptable performance and reliability. Scaling means growing your system so it can handle more users, more requests, and more data.
Latency Latency is the total time it takes for a request to go from the client to the server and back with a response.
Latency is made up of multiple parts: Network latency (travel time over network) Queueing latency (waiting in line) Processing latency (CPU work) I/O latency (disk or DB access)
As load increases: Requests start queueing CPU becomes saturated Memory pressure increases Disk I/O becomes slow Database connections get exhausted Results: Latency increases Timeouts start happening Error rates increase System becomes unstable
Core Concept
CPU Time CPU time is the amount of time the CPU actively spends executing code for a single request. Examples of CPU-heavy work: JSON parsing Encryption/decryption Image processing Sorting large data Business logic calculations Example: If one request needs 20 ms of CPU time: That means the CPU must work for 20 milliseconds to finish that request. If you get 100 requests in one second: Total CPU needed = 100 × 20 ms = 2000 ms of CPU
CPU Cores (Parallelism) A CPU core is a unit that can execute one thread at a time. Total CPU capacity per second = Number of cores × 1000 ms Example: 4-core machine: Total CPU per second = 4 × 1000 = 4000 ms If each request needs 20 ms CPU: Max RPS per server = 4000 / 20 = 200 RPS
I/O Time (Waiting Time) I/O time is the time spent waiting for external systems or devices. This includes waiting for: Disk reads/writes Database queries Network calls to other services File system access One request: CPU work = 10 ms DB query = 40 ms Total request latency ≈ 50 ms Even though CPU only worked for 10 ms, the user still waits 50 ms.
Memory Memory is RAM used to store data while the application is running. Memory is used for: Caches In-memory objects Request buffers Session data Connection pools If your service caches 5 GB of data in memory, and the machine has only 4 GB RAM: System will swap to disk Latency will explode Performance will collapse
Disk I/O Disk I/O is reading or writing data to storage. Disk is much slower than memory. Typical speeds (rough): RAM: nanoseconds SSD: microseconds to milliseconds HDD: milliseconds to tens of ms Example: If each request writes a log entry to disk: Disk becomes busy Requests start waiting Latency increases
Network Network is data sent between: Client and server App server and DB App server and other services Network has: Latency (time) Bandwidth (data per second) Example: If each request sends 1 MB response and you handle 500 RPS: Bandwidth = 500 MB/sec If NIC can only handle 200 MB/sec: Packets queue Latency increases Requests fail