Building throttleGate — A Production-Grade API Gateway From Scratch

Every backend system eventually needs an API gateway. Most teams reach for Kong, Envoy, or AWS API Gateway. But building one from scratch taught me more about distributed systems than any off-the-shelf solution ever could. This is the story of throttleGate — a Node.js gateway with adaptive rate limiting, circuit breaking, and observability built in.

Why Build vs Buy?

The honest answer: learning. But there's a practical argument too. Off-the-shelf gateways are general-purpose. They handle the 80% case. The remaining 20% — custom rate limit algorithms, tenant-specific circuit breaker thresholds, hot-reloadable config per route — requires either expensive enterprise tiers or working around someone else's abstractions.

Building your own gateway is the right call when you need:

• Custom rate limiting algorithms (token-bucket + sliding-window-log, not just fixed-window)
• Tenant-aware stacked limits (IP + API key + endpoint simultaneously)
• Circuit breaker thresholds that differ per endpoint, not per service
• Hot-reloadable config without restarting the process

Architecture Overview

The gateway follows a layered middleware architecture on top of Express. Each layer handles a cross-cutting concern:

IP Filter → CORS → Input Sanitizer → Request Size Limit → Tracing → Rate Limiter → Auth → Coalescer → Circuit Breaker → Router → Proxy

The order matters. Security middleware runs before tracing to reject obvious threats without creating spans. Rate limiting runs before auth so unauthenticated requests still consume rate limit budget. The coalescer runs after auth so dedup keys include tenant context.

Redis Lua Scripts for Atomic Rate Limiting

The centerpiece of throttleGate is two rate-limiting algorithms implemented as Lua scripts executed atomically in Redis. The critical insight: check-then-increment in application code creates race conditions under concurrent requests. By doing both operations inside a Lua script that Redis executes atomically, we guarantee correctness without locks.

-- Token Bucket (lua/token_bucket.lua)
-- KEYS[1] = bucket key
-- ARGV: capacity, refillRate, now, cost

local tokens = redis.call("HMGET", KEYS[1], "tokens", "lastRefillTime")
local elapsed = math.max(0, now - lastRefillTime)
tokens = math.min(capacity, tokens + elapsed * refillRate)

if tokens >= cost then
    tokens = tokens - cost
    allowed = 1
end

redis.call("HMSET", KEYS[1], "tokens", tokens, "lastRefillTime", now)
return {allowed, remainingTokens, retryAfter, resetTimestamp}

The sliding-window-log variant uses a Redis sorted set to track request timestamps within the window, pruning expired entries before counting.

Fail-Open vs Fail-Closed

This is the production decision interviewers ask about. When Redis is unreachable, should the gateway block all traffic (fail-closed) or allow all traffic (fail-open)? I chose fail-open by default because rate limiting is a QoS mechanism, not a security boundary. A Redis outage shouldn't cascade into a full application outage. The security layer (auth, IP filtering) runs in-process and doesn't depend on Redis.

What's Next

The gateway is running in my local Docker stack with Prometheus metrics and Grafana dashboards. The full source code, load test results, and architecture documentation are on GitHub.