FlowForge — Workflow Automation Platform

Workflow automation sits at the intersection of distributed systems and application integration. The core problem is reliable execution: if a workflow has 10 steps and step 5 fails, the entire workflow shouldn't crash — it should retry, log, and eventually move to a dead-letter queue for investigation. FlowForge treats every workflow as a Directed Acyclic Graph (DAG) and executes nodes based on topological ordering.

FlowForge uses an event-driven architecture. When a workflow triggers, the engine parses the DAG, resolves dependencies via topological sort, and enqueues each executable node into a Redis-backed job queue. Workers pick up jobs, execute the corresponding action (HTTP request, email, database query, etc.), and emit completion events that trigger downstream nodes. Failed jobs retry with exponential backoff (1min → 5min → 30min) before moving to a dead-letter queue for manual investigation.

Workflows are stored as DAG definitions (nodes + edges) in PostgreSQL. Node types include Trigger, Action, Condition, and Wait. Each execution creates an immutable run record with per-node status (pending, running, succeeded, failed, skipped). Queue state lives in Redis for performance, with periodic snapshots to Postgres for durability.

Ensuring exactly-once execution semantics was the hardest part — workers can crash after completing an action but before recording success. Solved with idempotency keys on each job: if a worker picks up a job with the same idempotency key, it checks if the action was already performed before re-executing. Another challenge was DAG cycle detection — workflows must be validated as acyclic before execution. Implemented Kahn's algorithm for topological sort with cycle detection.

Workers scale horizontally — add more containers to process more workflows. The queue (BullMQ/Redis) is the bottleneck, so Redis runs in cluster mode for high-throughput deployments. Workflow definitions are cached in Redis after first load. Long-running workflows checkpoint progress to Postgres so they can resume after worker crashes.

Three-tier retry: immediate retry for transient failures, exponential backoff (1m/5m/30m) for intermittent failures, dead-letter queue for permanent failures. DLQ alerts trigger manual review. Failed webhook deliveries retry with idempotency to prevent duplicate processing.

Every workflow execution emits structured logs with a run ID. Per-node timing metrics help identify bottlenecks. Dashboard shows active runs, queue depth, failure rate, and average execution time. Failed nodes surface the exact error and input context for debugging.

Redis-backed queues were chosen over Kafka for simplicity (FlowForge doesn't need Kafka's log retention or replay capabilities). PostgreSQL stores workflow definitions because they're relational and change infrequently. BullMQ was chosen over RabbitMQ for its Redis-native integration and job scheduling features.