Execution controls

Everything that decides whether a task runs, when it runs again, and how many run at once. All of it is enforced server-side, at dequeue or in the decider — anything enforced in an SDK is advisory and the first worker written in another language bypasses it.

Where each control lives

Retries

The policy

Set on a task definition:

curl -X POST "$NF_URL/v1/ns/default/metadata/task-definitions" \
  -H "x-api-key: $NF_API_KEY" -H 'content-type: application/json' \
  -d '{
    "name": "charge",
    "retryCount": 5,
    "retryLogic": "EXPONENTIAL_BACKOFF",
    "retryDelaySeconds": 2,
    "backoffScaleFactor": 2,
    "maxRetryDelaySeconds": 300,
    "jitter": 0.2,
    "retryBudget": 0.3,
    "nonRetryableErrors": ["card_declined", "INVALID_ARGUMENT"]
  }'

The delay formula

FIXED                delay = retryDelaySeconds
LINEAR_BACKOFF       delay = retryDelaySeconds * backoffScaleFactor * attempt
EXPONENTIAL_BACKOFF  delay = retryDelaySeconds * backoffScaleFactor ^ (attempt - 1)

delay = min(delay, maxRetryDelaySeconds)
if jitter > 0:  delay = delay ± (delay * jitter)
delay = clamp(delay, 0, maxRetryDelaySeconds)

attempt is 1-based: 1 is the delay before the first retry.

The cap is applied twice — before and after jitter. Capping only beforehand lets upward jitter push the delay back over the ceiling, which defeats the point of having one.

With the defaults (retryDelaySeconds: 1, factor 2, cap 3600):

Jitter

On by default, because synchronised retries from many workers are a self-inflicted thundering herd. jitter: 0.2 randomises the delay by ±20%. Set it to 0 only if you genuinely need deterministic timing and have one client.

Retry budget

The control that stops a degraded dependency being held down by the retry traffic its own degradation caused — the classic way a partial outage becomes a total one.

• The window is 300 seconds, so a service that recovers is not punished for a burst an hour ago.
• Fewer than 20 executions in the window never trips it, because a tiny sample would trip on the first retry before there is enough traffic to mean anything.
• retryBudget: 1 disables it entirely.

When the budget is spent, a failure is not retried even though attempts remain. The task fails, and the workflow fails with it.

nonRetryableErrors

Each entry is checked as a substring of the task's reasonForIncompletion. So "card_declined" matches Stripe error: card_declined (do_not_honor).

What is retryable at all

A worker sends FAILED_WITH_TERMINAL_ERROR to say "this input will never succeed". The system tasks classify their own failures — a 4xx from HTTP is terminal, a 503 is not; INVALID_ARGUMENT from gRPC is terminal, UNAVAILABLE is not; an INLINE script that throws is terminal, because it will throw again on identical input.

optional

Orthogonal to retries. A task marked "optional": true still exhausts its retries; once it has, the failure is absorbed, the task ends COMPLETED_WITH_ERRORS, and the workflow carries on.

{ "name": "log_to_analytics", "taskReferenceName": "analytics", "type": "SIMPLE", "optional": true, "retryCount": 1 }

Timeouts

Five classes, each answering a different question. That is why they are separate numbers and not one.

All default to 0, which means unbounded, except the last two. A policy of 0 arms no timer at all.

curl -X POST "$NF_URL/v1/ns/default/metadata/task-definitions" \
  -H "x-api-key: $NF_API_KEY" -H 'content-type: application/json' \
  -d '{
    "name": "charge",
    "scheduleToStartTimeout": 120,
    "startToCloseTimeout": 60,
    "timeoutSeconds": 300,
    "heartbeatTimeout": 30,
    "timeoutPolicy": "TIME_OUT_WF"
  }'

Waiting tasks get only the total budget

WAIT, WAIT_FOR_WEBHOOK, HUMAN and PULL_WORKFLOW_MESSAGES are never dispatched, so the two deadlines that measure dispatch and execution do not apply to them — arming scheduleToStart on a WAIT would fire while it is legitimately waiting and mark it TIMED_OUT, turning the feature into a bug that looks like one of the engine's own guarantees.

Only timeoutSeconds is armed. For WAIT_FOR_WEBHOOK it is the only thing that ends a callback which never arrives.

timeoutPolicy

What a missed deadline means. Set on the task definition (and, at workflow level, on the definition).

Workflow timeout

{ "timeoutSeconds": 3600 }

Armed on the one evaluation guaranteed to happen exactly once per execution — the first. When it fires the run ends TIMED_OUT and its failure workflow is started, the same as any other failure.

Concurrency

concurrentExecLimit — per task definition

A global cap on in-flight tasks of this type, across every workflow and every run.

{ "name": "charge", "concurrentExecLimit": 2 }

Enforced at dequeue, under a transaction-scoped advisory lock keyed on the queue. A worker asking for ten when the cap is two gets two — and a second worker asking while those two are held gets zero. When a holder finishes, the next lease flows.

maxConcurrentTasks — per execution

{ "name": "fan_out", "maxConcurrentTasks": 2, "tasks": [ /* a fork over 8 branches */ ] }

Bounds fan-out blast radius within one run. A dynamic fork over ten thousand items would otherwise schedule all of them at once and bury the queue.

Operators and waiting tasks are exempt: they perform no external work, and blocking them would stall the control flow that decides what runs next — the cap would prevent the workflow making progress at all.

maxConcurrentExecutions — per definition

{ "name": "nightly_reconcile", "maxConcurrentExecutions": 1 }

Checked before the row is created, and a start over the cap is refused with LIMIT_EXCEEDED. Admitting a workflow and then stalling every one of its tasks against a concurrency cap looks identical to a broken worker pool from the outside; rejecting the start says exactly what happened.

Named semaphores

A shared ceiling across unrelated task types — the thing a per-definition cap cannot express: "these six different tasks must not exceed four concurrent hits on one fragile legacy API".

curl -X PUT "$NF_URL/v1/ns/default/semaphores/legacy_erp" \
  -H "x-api-key: $NF_API_KEY" -H 'content-type: application/json' \
  -d '{"permits":4}'

curl -X POST "$NF_URL/v1/ns/default/metadata/task-definitions" \
  -H "x-api-key: $NF_API_KEY" -H 'content-type: application/json' \
  -d '{"name":"export_report","semaphores":["legacy_erp"]}'

curl -s -H "x-api-key: $NF_API_KEY" "$NF_URL/v1/ns/default/semaphores"
# [{ "name": "legacy_erp", "permits": 4, "held": 4 }]

Two properties worth knowing:

• All or nothing. A task acquires every semaphore it needs or none of them. Taking a subset and waiting for the rest is how two tasks needing the same two permits deadlock, each holding one.
• Permits are leased, not held. A worker that crashes releases its permits on expiry instead of blocking the semaphore forever; a poller sweeps stale holders.

An unconfigured semaphore does not gate anything — declaring semaphores: ["not_yet_created"] is a no-op until permits are set.

Rate limits

Two different things share the phrase. They solve different problems.

Task rate limit — throughput

{ "name": "partner_api", "rateLimitPerFrequency": 100, "rateLimitFrequencySeconds": 60 }

At most 100 dispatches per 60-second window, whatever the concurrency. This is what most third-party APIs actually meter, and it is not a concurrency cap: it bounds throughput rather than parallelism, and completing a task does not refill the budget.

The window is a fixed bucket — floor(now / size) — rather than a sliding one. Fixed windows admit a burst at a boundary, but the alternative needs per-event timestamps, and a burst of at most 2× for one window is a fair trade for a single upsert on the dispatch path.

Workflow rate limit — per-key admission

{
  "name": "tenant_sync",
  "rateLimitConfig": { "rateLimitKey": "${workflow.input.tenantId}", "concurrentExecLimit": 1 }
}

At most concurrentExecLimit executions sharing the resolved key run at once. Later starts are queued, not refused: the run is created RUNNING with awaitingAdmission: true, schedules nothing, and is admitted when a holder finishes.

curl -s -H "x-api-key: $NF_API_KEY" "$NF_URL/v1/ns/default/executions/$ID" | jq '.awaitingAdmission'

Different keys never contend: three runs for acme queue behind each other while a run for globex starts immediately.

A key that resolves to nothing shares a bucket named by the expression itself, rather than escaping the limit — an unkeyed run is still a run against whatever the limit protects.

Combining them

When several gates apply, the most restrictive wins: the allowance is the minimum across concurrentExecLimit, the rate-limit tokens and the semaphores.

Namespace quotas

Platform-level limits, set per namespace by an administrator.

curl -X PUT "$NF_URL/v1/ns/default/quotas" \
  -H "x-api-key: $NF_API_KEY" -H 'content-type: application/json' \
  -d '{
    "maxConcurrentExecutions": 5000,
    "maxExecutionsPerMinute": 2000,
    "maxWorkflowDefinitions": 500,
    "maxSchedules": 100
  }'

Exceeding one is a 429 carrying { quota, limit, current, retryAfterSeconds }. Execution quotas are checked in the same transaction as the insert, so two concurrent starts cannot both see room for one more.

An absent quota is unlimited.

Circuit breakers

Shared by every system task that leaves the process — HTTP, HTTP_POLL, WEBHOOK, GRPC. Off by default, because a breaker changes how failures behave and that should be a decision, not a surprise after an upgrade.

NODE_FLOW_CIRCUIT_BREAKER='{"enabled":true}'
# or, with everything spelled out:
NODE_FLOW_CIRCUIT_BREAKER='{"enabled":true,"failureRatio":0.5,"minimumRequests":10,"windowMs":30000,"openMs":5000,"maxOpenMs":60000}'

The failure this exists for: a dependency goes down, every task calling it takes its full timeout, and those tasks occupy the system-task runner for thirty seconds each. One dead host then consumes the capacity every other workflow needs.

Four decisions worth knowing:

• Per process, not cluster-wide. A shared breaker would need shared state — a second dependency — to protect against a dependency being down, and would make the breaker itself a thing that can fail. Envoy and Hystrix are per-instance for the same reason.
• Only server-side failures count. A 404 or a 422 is the caller's problem and says nothing about the dependency's health; counting it would let one workflow with a bad URL open the breaker for every other workflow calling the same host. Timeouts, connection errors, 5xx and 429 count.
• An open breaker fails the task, but never terminally. The retry policy still owns what happens next. A terminal failure here would turn a transient outage into permanently dead workflows, which is precisely what the breaker is supposed to prevent.
• Half-open admits exactly one probe. Letting the whole backlog through the moment the window expires is how a recovering service is knocked over again.

Breakers are keyed by target: HTTP and HTTP_POLL calling the same host share one, so a host known to be down is known to both. gRPC is keyed by the configured service name rather than its address, because that is the name an operator knows and two services on one host should fail apart.

Diagnosing "why is my task not running?"

Every control reports itself. LimitExceededError carries a control field naming which one tripped, because that question has to be answerable from the execution view rather than from server logs.

curl -s -H "x-api-key: $NF_API_KEY" "$NF_URL/v1/ns/default/executions/$ID" \
  | jq '{status, awaitingAdmission, reasonForIncompletion}'

curl -s -H "x-api-key: $NF_API_KEY" "$NF_URL/v1/ns/default/queues/charge/depth"
curl -s -H "x-api-key: $NF_API_KEY" "$NF_URL/v1/ns/default/queues"

curl -s -H "x-api-key: $NF_API_KEY" "$NF_URL/v1/ns/default/queues/workers"

curl -s -H "x-api-key: $NF_API_KEY" "$NF_URL/v1/ns/default/semaphores"
curl -s -H "x-api-key: $NF_API_KEY" "$NF_URL/v1/ns/default/metadata/task-definitions/charge"

curl -X POST -H "x-api-key: $NF_API_KEY" "$NF_URL/v1/ns/default/executions/$ID/decide"

Metrics that move first

See Self-hosting → observability.

Choosing values

A rough starting point, to be argued with:

Two rules that are not negotiable:

1. Set scheduleToStartTimeout on anything a worker runs. It is the only control that distinguishes "no workers on this queue" from "the worker is slow", and without it a starved queue hangs silently.
2. Set a downstream idempotency key in any worker with side effects. Every task may be delivered more than once — see Workers → idempotency.

Next

• Workers — the other side of these controls.
• Configuration — the server-level settings named here.
• Self-hosting — metrics, roles and scaling.