FlowForge — Distributed Workflow Automation

Workflow automation turns sequential logic into a directed acyclic graph problem: parse the graph, topologically order it, detect cycles, and execute with parallelism. FlowForge models workflows as versioned DAGs, plans parallel branches, and runs each node as a durable BullMQ job so a crash never loses an in-flight execution.

Five microservices: an API server (Fastify with plugin architecture, 22 source files), a BullMQ worker with retry and dead-letter queue, a cron scheduler that fires time-based triggers, a Socket.IO realtime gateway, and a Next.js web app. A shared workflow-engine package provides DAG parsing, topological sort, cycle detection, and parallel branch planning. 13 shared packages cover auth, contracts, db, logger, nodes, observability, queue, redis, triggers, and workspace.

17 Prisma models: users, sessions, refresh tokens, workspaces, workspace members, workflows, workflow versions, workflow triggers, executions, execution nodes, execution logs, encrypted secrets, and audit logs.

The execution engine had to be durable and resumable — a workflow might have thousands of nodes. Cycle detection prevents infinite loops, topological sort guarantees dependency order, and BullMQ jobs make each node restart-safe. Real-time updates required a dedicated gateway decoupled from the API to avoid blocking request handling.

Independent microservices scale horizontally. BullMQ queues each node execution as a job with exponential backoff retry and dead-letter queue. The scheduler scans for due cron triggers independently. Socket.IO scales via the realtime gateway. Secrets are encrypted with per-workspace keys.

Node execution retries with exponential backoff; failed jobs move to a dead-letter queue. Execution state is persisted so workflows resume after crashes. WebSocket disconnects pause real-time views while the durable queue keeps processing.

Pino structured logging, Prometheus metrics, real-time execution logs and node events streamed over Socket.IO, and per-execution status tracking in the UI.

Fastify over Express for plugin architecture and validation. React Flow over a custom editor for battle-tested interaction. BullMQ over Redis-only pub/sub for durable job semantics. Microservices over monolith to isolate the worker, scheduler, and realtime planes.