Little's Law and stability

Little's Law

For any system in steady state, the time-average number of items in the system equals the arrival rate times the average time each item spends in the system.

L is the mean number in system, λ is the arrival rate, and W is the mean time in system. The law holds regardless of the arrival distribution, service distribution, number of servers, or scheduling discipline — which is what makes it the most general result in the field.

Worked example

A web service handles λ = 200 req/s with a mean response time of W = 80 ms = 0.08 s. By Little's Law, the average number of requests being processed concurrently is:

That bare number is one of the most important you can know about a system: it tells you how many workers / threads / connections your service needs to provision before the queue starts growing.

Stability

A queueing system is stable when its arrival rate stays below its total service rate. For c servers each serving μ requests per second, the utilisation is:

Stability requires ρ < 1. At ρ = 1the queue grows unboundedly; at ρ > 1 work arrives faster than it can be served and latency goes to infinity in finite time.

High utilisation is much worse than it intuitively feels

For the simplest stable system, M/M/1, the mean wait time is:

That denominator (1 − ρ) goes to zero as utilisation approaches 1, so wait time explodes. Comparing equal-step increases in utilisation: