ENGINEERING NOTES
Limiting Async Concurrency: Workers and Backpressure
Build a bounded worker loop and understand the difference between concurrency limits, rate limits and queue capacity.
What problem does this solve?
A bulk import that issues thousands of requests at once can exhaust sockets, database connections and upstream quotas. A fully sequential loop protects resources but may leave useful I/O concurrency idle.
What is it?
A concurrency limit caps in-flight tasks. A rate limit caps starts per time window. Backpressure makes producers slow down when consumers cannot keep up. These are related controls with different guarantees.
Mental model
Use a fixed number of checkout counters. Each worker takes the next item only after finishing its current item. A bounded waiting area is still needed if new items arrive faster than all counters can process them.
How it works internally
In the example, workers claim an index synchronously before awaiting. On one JavaScript thread, those claims do not interleave midway. Work completes out of order, but results are written at the original index. This is I/O concurrency, not parallel CPU execution.
Production example
For a notification batch, choose a small limit based on provider quotas and observed latency. Do not automatically retry ambiguous send timeouts without a delivery identifier or deduplication rule.
Node.js / TypeScript implementation
This in-memory teaching example accepts a finite array. It waits for all workers to settle before reporting a failure; already-running work is not cancelled. Streaming inputs need a bounded queue as well.
async function mapLimit<T, R>(
items: T[], limit: number, work: (item: T) => Promise<R>
): Promise<R[]> {
if (!Number.isInteger(limit) || limit < 1) throw new Error("Invalid limit");
const results: R[] = new Array(items.length);
let next = 0;
let stopped = false;
async function worker() {
while (!stopped && next < items.length) {
const index = next++;
try { results[index] = await work(items[index]); }
catch (error) { stopped = true; throw error; }
}
}
const outcomes = await Promise.allSettled(
Array.from({ length: Math.min(limit, items.length) }, worker)
);
for (const outcome of outcomes)
if (outcome.status === "rejected") throw outcome.reason;
return results;
}
mapLimit([1, 2, 3], 2, async n => n * 2).then(console.log);Common mistakes
A semaphore around already-started promises cannot undo the load. Promise.race-based limiters often leave rejected tasks unobserved. A process-local limit is multiplied by replicas and is not a global provider quota.
Performance / reliability considerations
Bound queue depth, task duration and retry count as well as concurrency. Watch memory and pool wait time. Add cancellation and per-item result tracking when partial completion must be recoverable. Persistent jobs need durable ownership and visibility after a process crash.
Interview questions
Why does concurrency differ from requests per second? What prevents a queue growing forever? What happens when one task fails? How does adding replicas change the effective limit?
Quick Revision
Claim work before starting it. Cap in-flight tasks and queued tasks separately. Concurrency is not a rate limit or CPU parallelism. Preserve result indexes deliberately. Observe every rejection. Define cancellation, retries and partial completion. Local limits do not automatically coordinate across instances.