Troubleshooting

A symptom-first field guide for startup, authentication, stuck workflows, idle workers, system tasks, Conductor clients, and overloaded installations.

Start with the boundary that stopped moving. A workflow system has several independent loops; restarting all of them may hide the evidence and does not tell you which loop failed.

Follow the first answer that is no

Capture evidence before changing anything

For one affected execution, keep:

request id          response header or structured error
execution id        UUID returned by start
workflow name/ver   the immutable definition actually used
task id + ref name  distinguish definition name from taskReferenceName
principal           GET /v1/auth/whoami
server roles        NODE_FLOW_ROLES for each replica
time window         UTC, including timezone offset at the caller

Then collect the execution, history and queue view:

nf executions get "$EXECUTION_ID" --json > execution.json
curl -sS -H "Authorization: Bearer $NF_API_KEY" \
  "$NF_URL/v1/ns/$NF_NAMESPACE/executions/$EXECUTION_ID/history"
curl -sS -H "Authorization: Bearer $NF_API_KEY" \
  "$NF_URL/v1/ns/$NF_NAMESPACE/queues"

Do not start by editing the definition version used by the run: versions are immutable, and a new version cannot explain the state of an old execution.

The server does not start

`Invalid environment configuration`

Boot validation reports every bad value in one error. Fix the whole list before restarting. Common cases:

Message or symptom	Meaning	Fix
`DATABASE_URL is required`	No database was selected	Supply an explicit Postgres URL; there is no safe default
JWT secret must be at least 32 characters	Signing key is missing or weak	Generate a deployment-specific random value
unknown role `decidr`	No process would own the intended loop	Use `api`, `decider`, `poller`, comma-separated
malformed JSON on a connection map	Shell quoting changed the JSON	Validate locally with `jq`; quote the entire value once
S3 needs a bucket	Blob store selected without complete config	Add `NODE_FLOW_BLOB_S3` or return to `fs` for one-node use
secret storage refuses writes	No master key is configured	Set `NODE_FLOW_SECRET_KEYS`; plaintext fallback is intentionally absent

Database connection succeeds, readiness fails

Compare the three health surfaces:

curl -i http://localhost:3000/v1/health/live
curl -i http://localhost:3000/v1/health/ready
curl -i http://localhost:3000/v1/health

Live fails: the process is not serving; inspect the container exit and startup log.
Live passes, ready fails: a required dependency or migration state is not ready. Do not route traffic yet.
Both pass, detailed health degrades: the API can serve, but a background loop or queue is unhealthy. Inspect the named check rather than restarting the API role.

Postgres must be 18 or newer. In Docker, make sure host tools use port 5433 for the repository compose file; connecting to a different local server on 5432 often looks like a password error.

`401` and `403`

401 means no acceptable identity was established. 403 means the identity was established and lacks authority.

curl -i -H "Authorization: Bearer $NF_API_KEY" \
  "$NF_URL/v1/auth/whoami"

Check, in order:

Header spelling and scheme. Native API keys use Authorization: Bearer ….
Base URL. Native endpoints begin at /v1; the Conductor JavaScript SDK uses <origin>/conductor. Clients expecting an API-root URL use <origin>/conductor/api. See connection settings.
Namespace. The credential namespace must match /ns/{namespace}.
Scope. Reading definitions does not imply starting executions or polling.
Resource and tag grants. A general scope can still be narrowed by a grant.
Expiry and revocation for JWTs and sessions.

For Conductor clients, X-Authorization is a raw token—no Bearer prefix. The compatibility guard accepts the SDK's wire shape; sending that header to a native /v1 endpoint is the wrong protocol.

Start returned no execution id

`404`

Check the workflow name and version in the same namespace. Omitting a version selects the latest registered version, not a draft in the visual editor.

`409`

The usual causes are:

an idempotency key already belongs to another input;
maxConcurrentExecutions is full;
an immutable workflow version is being registered again;
a task report carries an expired or replaced lease token.

Keep the structured error code and details. Do not branch application logic on the English message.

`429`

Admission or a namespace quota refused the request. A workflow rate limit may queue a run instead, so distinguish a rejected HTTP request from an execution whose admission state is waiting.

The workflow stays `RUNNING`

Read history before forcing a decision:

nf executions get "$EXECUTION_ID" --json
curl -sS -X POST -H "Authorization: Bearer $NF_API_KEY" \
  "$NF_URL/v1/ns/$NF_NAMESPACE/executions/$EXECUTION_ID/decide"

If forcing a decision unblocks the run, preserve the original time window and inspect the decide-queue and decider logs. The manual action is a recovery, not the root-cause fix.

Last visible state	Likely boundary
No tasks exist	Decider role absent, decide queue not consumed, or definition failed before scheduling
Terminal task exists, downstream absent	Completion did not enqueue evaluation, or decider is behind
`WAIT` remains beyond its time	Poller role/timer sweeper absent or overloaded
`JOIN` waits forever	A branch head did not reach a terminal state; inspect the named refs and iterations
Sub-workflow terminal, parent waiting	Child outcome was not applied to the parent; inspect parent and child history together
Workflow paused	Pause is non-terminal and intentionally suppresses progress until resume
Failure workflow absent	Check the failure workflow name/version and the original failure history

Repeated evaluations are safe. A missing evaluation is not. Queueing one manually is appropriate recovery after collecting evidence.

A `SIMPLE` task never reaches the worker

Confirm the exact queue

The queue is the task definition name, optionally suffixed by its domain. It is not the task reference name.

name              charge_card
taskReferenceName charge_primary
domain            eu-west
queue             charge_card:eu-west

Domain precedence is: explicit task route, * route, inherited sub-workflow route, then no domain. Compare the execution's resolved route with the worker's configured queue.

Separate backlog from no poller

Depth above zero + no recent poller: worker process, URL, domain or auth.
Depth above zero + active poller: concurrency, semaphore, rate limit, quota, or lease count.
Depth zero + task scheduled: it may already be leased. Inspect task status, worker id and lease expiry.
Depth zero + no task: go back to the decider and workflow history.

Check server-side gates

All execution controls are enforced at dequeue; changing SDK settings cannot bypass them. Look for:

concurrentExecLimit on the task definition;
a named semaphore with every permit held;
task rate-limit tokens exhausted;
namespace quota or workflow maxConcurrentTasks;
scheduleToStartTimeout already expired the task.

The worker receives work but cannot complete it

Response	Interpretation
`400`	Report shape, status or output violates the contract/schema
`401`	Worker credential is missing, expired or malformed
`403`	Credential cannot update this queue/task/namespace
`404`	Wrong task id or namespace; do not treat as retryable success
`409`	Lease fencing rejected a stale worker or duplicate transition

Heartbeat before the lease expires when work can run longer than leaseSeconds. A heartbeat extends ownership; it does not complete the task. After a network timeout on report, read the task before repeating a non-idempotent side effect. Reporting the same terminal result can be safe; performing the business action twice may not be.

Use the task id as the idempotency key in the external system when possible. An attempt number distinguishes retries when each attempt must be recorded.

A system task fails

System tasks require the poller role. A definition may validate even when the operator has deliberately not configured its destination; execution then fails visibly rather than using a hidden default.

HTTP and HTTP_POLL

destination refused or private-address errors: SSRF protection blocked the resolved address or a redirect. Add the precise host to the allowlist only after verifying it.
Retry on a 4xx is usually wrong; inspect the task's retry classification.
HTTP_POLL yields between attempts. A long gap may be its next poll time, not a consumed runner slot.
Circuit open means recent calls crossed the configured failure ratio. Fix the destination before forcing more traffic.

JDBC, gRPC, email, brokers and AI

Confirm that the name in the task input exists in the corresponding environment map. Then check that the poller replica has that configuration; configuring only the API role does not configure the process that runs the task.

For broker event handlers, use a real broker when diagnosing consumer-group, acknowledgement or redelivery behaviour. An HTTP stub cannot reproduce those semantics.

INLINE and jq

Timeout and memory/output limits are hard boundaries. Increasing them expands the work one task may monopolize. First confirm the program terminates and its output is bounded; then adjust the specific limit, not global task concurrency.

A task is going wrong and you want it stopped

Pausing the workflow stops the decider scheduling anything new; it does not recall a task already queued or leased. To stop a specific one:

nf executions cancel-task <id> <task-ref>

That marks it CANCELED, releases its permits and timers, and pauses the run around it so nothing else moves while you look.

It is not recorded as a failure, which matters if you are watching failure rates: no retry is spent and the failureWorkflow does not run. An operator intervening should not appear in your reports as the workflow failing.

Your worker keeps running. node-flow releases the lease, so the result is refused when the worker reports — correctness holds — but the process itself is untouched and the work continues to completion. If a task can cause damage that must be stopped mid-flight, the worker has to check for that itself; there is no server-side kill.

Only some tasks need to run again

After fixing a worker, rerun is usually too blunt — it discards everything from a point onward. To re-run specific tasks:

nf executions pause <id>          # required; see below
nf executions rerun-tasks <id> transcribe

By default only the named tasks run again. Anything that consumed their output keeps the result it already has, which means that result was computed from an output that no longer exists. The command prints which tasks those are:

stale — still holding output from the run being replaced:
  transcreate, speech_generate

If that is wrong for your case — usually it is, when the downstream work is derived rather than independent — use --cascade, which re-runs the dependents too.

You want	Use
One artifact regenerated, nothing else touched	`rerun-tasks <ref>`
The task and everything derived from it	`rerun-tasks <ref> --cascade`
Everything from a point onward, regardless of dependency	`rerun --fromTaskRef <ref>`

The workflow must be paused or finished. A 409 saying to pause first is not a quirk: re-running a task while its downstream is still executing races the decider for the task frontier and a worker for a row about to be deleted.

Dependency is computed from the compiled definition, not from timing — a parallel branch that merely started later is not swept up, and a task reading another's output across a fork is, even with no control-flow edge between them.

Payload and storage failures

Payloads above NODE_FLOW_PAYLOAD_THRESHOLD_BYTES are offloaded. In a multi-replica deployment, fs is only correct when every replica sees the same durable mount. Prefer S3-compatible storage otherwise.

Typical pattern: the API accepts a run, then a decider or worker-facing replica cannot read its payload. If small payloads succeed and large ones fail, compare blob-store configuration, bucket permissions, endpoint reachability and prefix across every role.

Postgres rejects NUL bytes in JSON strings. node-flow sanitizes supported payload paths; if an external integration returns binary data, store or encode it rather than forcing raw bytes into JSON.

Conductor SDK compatibility

For a complete walkthrough, see Conductor compatibility. For the JavaScript SDK, use this base URL:

const config = {
  serverUrl: 'https://node-flow.example.com/conductor',
  keyId: process.env.CONDUCTOR_KEY_ID,
  keySecret: process.env.CONDUCTOR_KEY_SECRET,
};

The JavaScript SDK builds /api/token and other /api/* paths itself. Use the prefix above rather than depending on a version's trailing-/api normalization. Python and Java clients expecting an API-root URL use …/conductor/api instead. The credential selects the namespace.

When registration fails, read the SDK error message: compatibility validation flattens offending schema paths into it because several Conductor SDKs ignore structured error details. Test migration with the real builder and client—not hand-written JSON that approximates what the SDK emits.

Search compatibility is intentionally partial because Conductor exposes an Elasticsearch query dialect while node-flow searches Postgres. Prefer native search when porting a query that is not in the supported subset.

The installation is slow

Measure before adding infrastructure:

Signal	What it usually means
Decide queue grows; task queue stays small	Decider CPU or expensive evaluations
Task queue grows; workers all busy	Worker fleet capacity
Task queue grows; no pollers	Deployment/configuration fault, not capacity
Timers overdue grows	Poller role overloaded or unavailable
Outbox backlog grows	Destination/relay problem
Postgres WAL rises with throughput	Expected queue churn; check storage bandwidth and vacuum
Table bloat and dequeue latency rise	Autovacuum is not keeping up with hot queue tables
Only large workflows slow	Inspect expression refs/frontier size and payload offload

Scale roles independently: add API replicas for connections, deciders for evaluation work, and pollers for timers/runners/outbox. Do not add Redis merely because the task queue is slow; first prove the database queue is the limiting component rather than a CPU-bound Node process or an absent poller.

Ask for help with a safe diagnostic bundle

Include:

node-flow version and deployment topology;
redacted environment names and non-secret values;
health response and relevant metrics window;
one execution and history response with business payloads removed;
queue depth, active worker and role information;
structured error code/details and request id;
the minimal immutable workflow definition that reproduces it.

Never include API keys, service-account secrets, session cookies, webhook URLs, master keys, database URLs with passwords, or resolved secret values.

Execution controls explains every gate that can intentionally delay work.
Workers covers leases, heartbeats and idempotency.
Self-hosting covers health, metrics and scaling.
Security model explains authentication and refusal boundaries.