Sebastian KaiserA personal engineering notebook

The notebook

Durable distributed jobs and the design of djob

A worker sends a command to a gateway. The gateway executes it. Before the worker records the response, its process disappears. What should the replacement worker do?

Retrying might repeat the command. Moving on might leave the job unfinished. Keeping the job in a database preserves what the system knows, but cannot recover a response nobody recorded.

I have had to answer these questions while building djob, a Go engine for durable distributed jobs. Its architecture keeps recorded progress connected to permission to use a shared target.

If you are choosing between scheduled HTTP delivery, a durable workflow and a conversation with a shared device, this article examines how much recovery each approach carries for you. I use a research-based vocabulary to compare djob with Temporal, Dapr and two smaller HTTP schedulers, then follow one gateway job through a lost result and the decisions needed to continue safely.

I maintain djob, which remains a public preview.

Use a shared vocabulary before comparing features

A callback scheduler and a workflow engine can both promise to retain accepted work while preserving very different kinds of progress. Counting their features obscures that difference. We need to ask what each system considers a job, how it runs that job, and what survives a failure.

The Workflows Community Initiative provides a useful starting point. In A Terminology for Scientific Workflow Systems, Suter and colleagues identify five axes for describing workflow systems. System builders and practitioners developed the terminology and applied it to 23 systems. I use it to describe responsibilities; it supplies no universal ranking.fn:1

Five research axes
Research axis Question for a job system
Workflow characteristics Is the unit of work a callback, a function, or a process with related steps and waits?
Composition How do we define that work, connect its steps and select the code that implements it?
Orchestration What activates work, chooses an executor and controls access to limited resources?
Data management Which inputs, intermediate state and results are stored, moved or reconstructed?
Metadata capture Which execution history, provenance and observations help us explain what happened?

The axes come from the paper; the questions translate them to background jobs and gateway conversations. Scientific workflows often emphasize data movement and compute placement. Here, a small command can be more consequential than a large dataset, so recovery and external effects need closer examination.

There is precedent for examining those separately. Yu and Buyya's workflow taxonomy distinguishes task-level recovery, such as retry and checkpoint/restart, from recovery of the larger workflow.fn:2 Reliable State Machines research makes another useful boundary explicit: the state and messages covered by a persistence protocol determine what that protocol can recover.fn:3

Language choice, deployment dependencies, licensing and operational experience still affect a selection. These are practical selection criteria, separate from the research vocabulary and its description of system behavior.

Recovery does not settle whether an effect happened

Return to the gateway command. There are at least two useful records: the request accepted by the scheduler and the command authorized for one execution attempt. Neither proves that the gateway completed the operation.

This is an old distributed-systems problem. Birrell and Nelson's RPC paper describes the ambiguity after a failed call: the procedure may have run, even though the caller received no result.fn:4 A timeout gives us a deadline observation. Treating it as evidence that nothing happened would be an application decision with consequences.

For an HTTP service, a stable operation identity can help. The receiver records that identity together with the business change, then recognizes another delivery as the same operation. This requires cooperation at the place where the effect happens. A database entry saying "payment processed" cannot make a separate, non-idempotent payment API call transactional.

A device may offer no such mechanism. We then need a way to observe its state, reconcile the result, or stop until someone can establish what happened. Persisting an uncertain outcome can be the correct recovery behavior.

For each project, I therefore ask what its workers retry, which identity survives that retry, and how the recipient recognizes an operation it has already performed.

Two ways to deliver an HTTP job

Two public repositories I came across offer concrete examples of callback scheduling. I use their source to examine implementation choices; the comparison criteria come from the research above.

Job Scheduler Platform uses PostgreSQL for job and execution records, with Redis for dispatch. The application submits a callback payload; workers call the endpoint and record the attempt. Delayed work, cron, retries and dead-letter inspection surround that basic unit of execution. A transactional outbox bridges committed database changes to Redis, allowing the dispatch path to be repaired after a crash. Architecture.

Job Scheduler Platform architecture with PostgreSQL, transactional outbox, Redis dispatch, Go scheduler and workers
Figure 1: Job Scheduler Platform architecture. Diagram by Burhaan Ashraf, reproduced unchanged.

distributed-scheduler keeps active jobs and their queue in MySQL. Creating a scheduled occurrence and advancing its schedule happen in one transaction. Executors claim rows with FOR UPDATE SKIP LOCKED, record a lease and increasing attempt number, and commit before making the HTTP call. Cassandra holds archived execution history. This avoids a database-to-broker handoff for active work, while putting claim traffic on MySQL. Schedule materialization, claims and result updates.

Its result update checks that the row is still running under the same worker and attempt. A reconciler returns expired claims to pending work, or marks them dead when attempts are exhausted. The attempt guard prevents an old executor from overwriting a replacement's database result. It cannot retract an HTTP request already sent to the recipient. Its architecture illustration makes these service and storage boundaries easy to follow. Lease reconciliation.

The receiver contracts differ in a detail that matters. distributed-scheduler sends the same job ID as Idempotency-Key on every attempt. Job Scheduler Platform sends <jobID>:<attempt> and also supplies X-Job-ID. A receiver deduplicating across attempts must therefore use an identity stable across those attempts; the latter header can supply the job identity, with suitable tenant and operation scope. Neither header enforces deduplication by itself. distributed-scheduler receiver contract, Job Scheduler Platform request headers.

Both projects leave a multi-step business process inside the recipient unless the application adds orchestration. That is a reasonable boundary when the desired service is "call this endpoint at this time." It becomes important when the endpoint must remember several exchanges across failures.

Workflow engines retain progress between operations

Temporal records workflow history and reconstructs workflow progress through replay. Activities perform external operations; durable timers and messages let the workflow wait across process failures. Activity retries still require suitable effect semantics. Temporal also supports Standalone Activities, so a background job need not be wrapped in a multi-step workflow. Workflows, activities.

Dapr separates scheduling from orchestration. Its Jobs API arranges callbacks; Workflow provides durable orchestration, activities, timers and external events. Jobs scheduling uses the Scheduler service and its embedded etcd, while Workflow persists through the configured actor state store. Choosing Dapr Jobs alone does not preserve arbitrary progress inside a callback. Jobs, Scheduler, Workflow architecture.

For a process that waits for approval, calls several services and later compensates for a failed step, this wider execution model can save substantial application code. Its price includes learning the workflow programming model, retaining compatible code and operating the required persistence and runtime services. Managed offerings change who carries some of that operational work.

An HTTP callback can also start a workflow. In that arrangement, the scheduler records delivery while the recipient's workflow engine owns progress from there. The application combines the two responsibilities.

Comparing the five systems

Each column describes the mechanisms supplied by one system across the same five research axes. Dapr Jobs and Workflow remain separate building blocks within its column. Multi-step logic inside a callback recipient belongs to that application. Scroll horizontally to read the full comparison.

Five systems across five axes
Research axis Temporal Dapr Jobs + Workflow Job Scheduler Platform distributed-scheduler djob
Workflow characteristics Multi-step workflows and activities; Standalone Activities for individual jobs Jobs: scheduled callbacks. Workflow: related steps, timers and event waits Scheduled HTTP callbacks; longer processes belong to the recipient Scheduled HTTP requests; longer processes belong to the recipient Resource-free functions and stateful conversations that can retain shared resources
Composition Workflow and activity code registered with SDK workers Jobs API configuration; workflow and activity code through Workflow SDKs Submission API configures callback payloads, schedules and retry policy Task API configures endpoints, schedules, priorities and retry policy Versioned Go handlers and drivers; definitions and bindings; optional worker-release pinning
Orchestration Task queues and workers; schedules, durable timers and workflow messages Scheduler invokes jobs; Workflow coordinates activities, timers and external events Scheduler and transactional outbox feed Redis dispatch; workers deliver callbacks MySQL row claims with leases and attempt guards; priority ordering and expiry reconciliation Raft-backed claims check capabilities, quotas, grants and resource reservations; workers execute authorized actions
Data management Persisted workflow history and recorded results reconstruct workflow progress through replay Jobs data in Scheduler's embedded etcd; workflow history in the configured actor state store PostgreSQL job, execution and outbox records; Redis carries dispatch notifications MySQL task and active-job state; terminal execution history archived to Cassandra Replicated bbolt state, WAL and snapshots retain pinned inputs, checkpoints, actions, waits and reservations
Metadata capture Execution histories, workflow visibility and Web UI inspection Workflow instance status and history through APIs and CLI Job and attempt records, structured logs and operational metrics Job status, execution history and dead-letter inspection through the Job Service Console inspection of runs, attempts, actions and events; resolution audits and operational metrics

Shared targets require more than exclusive job claims

My starting problem was communication with gateways. Two meters behind one half-duplex bus cannot necessarily be addressed concurrently, even when their jobs have different IDs and run in different worker processes. Both jobs must resolve to the same canonical resource before admission.

That resource stays occupied through the conversation: commands, reply waits and cleanup. Releasing it while waiting for a response would allow another job to enter an exchange that is still in progress. Adding workers can improve concurrency across independent buses; it does not increase the capacity of one serial bus.

A lease helps the controller decide who is authorized. Expiry does not prove that the old process stopped or that its messages can no longer arrive. Chubby's sequencer mechanism addresses this by having the recipient validate the generation associated with an operation. The enforcement point is at the protected service.fn:5

A legacy gateway may not understand such a token. In that case, djob cannot manufacture physical fencing through Raft. Its conservative policy blocks reassignment when previous access is uncertain, until an integration or operator establishes safe release. That can stop every queued job using the bus, including jobs for other meters. Recovery time becomes part of the availability cost. A general workflow platform can implement a similar policy. djob places the reservation and recovery rules directly in its execution model.

Controllers, workers and console have different responsibilities

djob architecture with a management console, replicated Raft controllers, versioned Go SDK workers, optional Kubernetes pools and shared gateway resources
Figure 2: djob's current architecture. Controllers retain accepted work and authority; independent workers execute application code and integrations. The console uses controller APIs. Managed Kubernetes workers are optional, and broker acknowledgement remains separate from device completion.

Controllers retain job definitions, triggers, runs and execution records through etcd Raft, with bbolt state, a write-ahead log and snapshots on each member. They decide eligibility and resource admission. Ready indexes and timers are rebuilt from durable records; internal dispatch needs no external message broker. Project overview, dispatch state.

The management console is a separate authenticated client. It provides job authoring, run inspection and worker management. It neither executes jobs nor opens controller databases. An accepted mutation belongs to the controller's durable state, regardless of which console replica submitted it. Console implementation.

Independent workers connect outward using authenticated bidirectional gRPC over TLS. Their Go registry supplies exact handler and driver versions. Submitting a definition selects deployed code; it does not upload that code to the controller. Workers contain company business rules and integrations without becoming Raft members. Go SDK, worker transport.

Handlers make bounded, pure decisions from recorded state. A decision can request a checkpoint, an action, a wait or completion. Drivers perform the effects and retain temporary protocol sessions within an attempt. This gives application code a place to express "read a measurement" while a driver handles MQTT, HTTP, Modbus or another transport. The controller remains responsible for authorizing the action and recording its result. Handler and driver contracts.

Follow a conversation through failure

Consider a job that selects a meter, requests a reading and finishes the exchange. I will call its run R and the shared bus B. These labels belong to an illustrative example, not a report of a physical deployment or an API naming convention.

The run is one accepted execution of the job. An attempt is one worker's turn executing that run; a later attempt has its own identity. Within attempt A, the request for a reading is action 2, identified by its run and action sequence. The resource identity B covers the bus, including its other meters and aliases. It must be the same resource for every job that could interfere with this exchange.

These identities serve different purposes. Repeating a submission with the same submission identity avoids creating another run. An action's idempotency key lets a cooperating recipient recognize the same external operation. A new attempt gives a worker neither a new business request nor permission to repeat an uncertain command.

Record the request and admit the run

An immediate launch creates durable ready work. A scheduled trigger creates its occurrence and run while advancing its cursor in the same committed transition. Event activation similarly records the accepted event with its run. The controller snapshots the definition, its driver binding and relevant defaults, so later configuration edits do not silently rewrite accepted work. The binding selects the integration used for the job's operations. Submission identities let a client repeat an uncertain request without creating another logical run. Activation, triggers.

Being ready does not mean the bus is available. A claim checks worker compatibility, tenant and group limits, grants and all required resources together. If a resource is busy, the run stays queued without taking partial reservations. The generic ready index does not promise global submission-order FIFO; exclusion comes from the committed admission checks.

Authorize an action before sending it

The handler requests the next operation. Its decision alone grants no permission to perform I/O. The controller must durably authorize the action, and the worker must remain within its verified permission budget. Only then can its driver send the gateway command. The controller must commit the reported effect before execution advances from it. Execution transitions.

For our example, that committed authorization permits attempt A to execute action 2 on bus B. It records permission to send, without establishing whether the driver has sent anything yet. If the worker disappears at this point, a replacement cannot infer that repeating the command is safe.

With MQTT, a broker acknowledgement establishes acceptance by the broker. A resource-backed integration still needs a gateway completion contract that establishes what the response means and whether access is quiescent. Disconnecting from the broker cannot retract a command already queued or forwarded.

Preserve progress without pretending to restore a connection

A checkpoint records the application's progress. The driver may also retain a live connection, selected device or protocol handshake. Restoring the checkpoint cannot prove that the gateway remembers that session. A fresh attempt gets a fresh driver.

The integration author has to make that distinction real. All effects belong in the driver; Execute must stop its I/O before returning. Quiesce must account for recovery and any protocol quiet period, and Close must release local sessions. A closed local connection alone may leave a command in a broker or a device still processing it. The controller relies on trusted worker reports; it cannot observe a legacy bus well enough to establish physical safety independently.

Resource-free waits can yield execution and later resume elsewhere. A wait that retains a shared resource keeps the conversation and its reservation together. The SDK waits for I/O and cleanup to stop before reconnecting; it does not detach an uncooperative driver and declare the previous session safe. Worker reconnection.

Recover the outcome and access separately

Attempt A now disappears after sending action 2, before the controller commits its result. Once ownership expires, djob records the action and run as unknown and blocks bus B. A fresh worker cannot automatically replay the uncertain action. The controller cannot tell this case from a worker that vanished just before sending.

What changes if we learn more? Each row below starts from the same lost-result case, except the last, which moves the failure past the result commit. The rows describe independent possibilities, not an automatic recovery sequence. Scroll horizontally on a narrow screen to keep each case readable.

Job outcome and bus access after failure
Evidence and recovery decision What happens to run R? Can another job use bus B?
No result and no proof that old access stopped Run and action 2 remain unknown No; B stays blocked
An operator establishes stopped access and explicitly resolves the resource, without settling the reading R remains unknown B becomes available; another job can pass normal admission
Independent evidence establishes the completed job's success, and the operator resolves R, without establishing stopped access R becomes succeeded No; resolving the outcome leaves B blocked
Instead, action 2's success was durably recorded before A disappeared, but the conversation had not completed The action result survives; the interrupted run becomes unknown No; a recorded result does not prove that old access stopped

Resource resolution needs evidence that excludes old access, including delayed commands, rather than confidence that the worker is probably dead. Outcome resolution needs evidence for the job's result. djob records those decisions separately, with the operator's reason. Cancellation still has to pass through cleanup; requesting a stop supplies neither kind of evidence.

Keep the code needed by accepted work

Durable jobs may outlive a deployment. Knowing a handler's name and version is useful only while some compatible worker can still execute it.

djob can publish a managed worker deployment as a release, pinning its image digest and configuration references. A run can be pinned to a release as well as requiring specific capabilities. Old and new releases can coexist; an existing run retains its selection through retries and resumptions. Updating a definition affects future direct launches, while triggers retain their own templates until explicitly changed. Beginning retirement refuses new runs for that release. Operators keep its workers available until accepted work has finished. Release state and admission.

Optional Kubernetes management reconciles the desired worker deployments. Externally operated workers remain supported. Starting a replacement Pod restores a process, but does not reconstruct a lost gateway session or settle an uncertain command. Resource recovery still follows the same rules. Managed-worker reconciliation.

Include overload and operations in the comparison

Durability lets work accumulate. It does not ensure that the system can finish it within a useful time. Borg's cluster-management paper treats admission control, resource isolation and monitoring as part of operating a scheduler.fn:6 For these job systems, I would measure lateness from scheduled time to actual execution, recovery delay after a worker failure, queue age and retained-state growth under the intended workload. A fast submission acknowledgement alone says little about any of them.

The storage choices move operational work around. Job Scheduler Platform needs PostgreSQL and Redis; distributed-scheduler needs MySQL and Cassandra. Dapr combines runtime services with its selected state components. Temporal needs its service and persistence infrastructure, unless an offering manages them for us. djob embeds replicated storage in its controllers, leaving quorum, volumes, backup and recovery with the operator. Optional console and Kubernetes worker management add their own responsibilities.

For djob, bounded pending work and tenant quotas protect admission, while retention and storage maintenance address a different budget. Permanent identities and unresolved outcomes can outlive successful payload history. Its console exposes execution records and recovery paths; metrics distinguish queueing, uncertain outcomes and blocked access. These are useful mechanisms, but their presence does not establish a tested capacity or years of operational experience.

distributed-scheduler includes workload and failure reports from a single-machine, emulated deployment, and explicitly qualifies their limits. Its large-scale design targets should not be read as achieved throughput. djob's public preview likewise leaves target-environment capacity, storage behavior and physical gateway fencing to qualification. No common experiment in this article establishes a performance ranking. Scheduler experiments and limits, Preview scope.

Licensing also affects adoption. djob's engine is source-available under its own terms; the complete Go SDK is Apache 2.0. The distributed-scheduler snapshot has no license file, so public access to its source should not be treated as a stated open-source grant. Licensing boundary, distributed-scheduler snapshot.

Choose the responsibility you need the system to carry

For one isolated HTTP operation, I would start with a callback scheduler and a recipient that owns the operation and its deduplication. Persisting a separate conversation would add little if the recipient already handles it. For branching work with durable waits and recovery, Temporal and Dapr Workflow provide broader composition models. If a workflow engine already carries that business process, adding application-owned resource coordination there may be simpler than introducing another engine.

djob addresses the case that led me to build it: jobs whose progress must remain connected to permission to use a shared target. Versioned company code runs independently of the controllers, while reservations and uncertain outcomes remain part of the durable execution state. I accept the possibility that a lost worker stops other work on the same bus until we can establish safe release. The integration still needs physical validation and an operational recovery procedure. If the target instead enforces deduplication and fencing, or the jobs share no target at all, much of that reason for choosing djob falls away.

Before selecting any of these systems, I would walk through the failed command from the opening: find the accepted request, identify the last recorded progress, decide whether repeating the effect is safe, and establish who may use the target next.

Sources and comparison scope

I updated this comparison on 13 November 2026. Project links pin Job Scheduler Platform to 74d5ee5, distributed-scheduler to 9bbd90c and djob's public source to d603e27. The djob contracts discussed here were also checked against my local 81409b1 tree. Temporal and Dapr claims follow their linked documentation; Dapr's pages identify version 1.19. Source inspection supports the architectural descriptions, not a shared runtime qualification of these projects.

[fn:1] Frédéric Suter and colleagues, A Terminology for Scientific Workflow Systems, §2 (five axes), §4 (community process). Future Generation Computer Systems 174, 107974 (2026); DOI. The comparison uses the vocabulary, with questions adapted to background jobs; the paper does not rank these five projects.

[fn:2] Jia Yu and Rajkumar Buyya, A Taxonomy of Workflow Management Systems for Grid Computing, §2.4, "Fault Tolerance." The 2005 report distinguishes task-level and workflow-level techniques, including retry and checkpoint/restart.

[fn:3] Suvam Mukherjee and colleagues, Reliable State Machines: A Framework for Programming Reliable Cloud Services, ECOOP 2019, §2.2. Its runtime persists inbox, outbox and machine state together; external services remain a separate interaction boundary.

[fn:4] Andrew D. Birrell and Bruce Jay Nelson, Implementing Remote Procedure Calls, ACM Transactions on Computer Systems 2(1), 1984, §3.1. A failed call can leave the caller unable to tell whether the procedure ran.

[fn:5] Mike Burrows, The Chubby lock service for loosely-coupled distributed systems, OSDI 2006, §2.4, "Locks and sequencers." A recipient checks the lock's sequencer to reject stale operations; possession of a lock alone does not make an external service enforce it.

[fn:6] Abhishek Verma and colleagues, Large-scale cluster management at Google with Borg, EuroSys 2015, §§2.5, 2.6 and 6.2. Quota, resource isolation and monitoring motivate the operational questions here; Borg's compute scheduling is not evidence of gateway-level fencing.