The design decisions behind djob
I started djob because one stateful service handled all of a platform's Modbus communication over MQTT. If that service struggled, every customer noticed. I wanted to replace that shared bottleneck and support communication with gateways and devices beyond Modbus.
My previous article compared approaches to durable jobs. My preferred design separates device communication from durable coordination. Independent workers handle the communication; replicated controllers record ownership and accepted results.
djob is currently in beta and not yet deployed to any production system. The device example and capacity estimates below are hypothetical; they are not benchmark results.
- Why I started building the engine
- A meter reading is a stateful exchange
- Independent releases for business code
- Job claims and reservations commit together
- When a device acts but the result is lost
- A lost worker can leave the bus unavailable
- Keeping unfinished jobs on their original release
- Recovering an interrupted job
- When I would choose an existing platform
- What would change the design
- Appendix: estimating capacity from bus occupancy
- Sources and scope
Why I started building the engine
I started the project about two and a half years ago, with a pause of roughly a year and a half along the way.
The surrounding platform had accumulated technical decisions made by different teams over several years. Moving the relevant logic to the edge was not an option for organizational and technical reasons. I needed a solution that could work within those constraints and handle other kinds of gateway and device communication too.
I also wanted to learn by building an application with real constraints. My aim was to balance component simplicity with the use cases I needed to support and the performance they required. I wanted to understand the path from a storage transaction to an authorized device operation. CockroachDB's architecture inspired my work on consensus. I could learn from it without taking on a distributed SQL database.
The controller finds job state through known keys and indexes. It does not need SQL, and bbolt provides the embedded key/value transactions those access patterns need. I use etcd Raft to order replicated changes and bbolt to apply local transactions. That keeps storage inside each controller, with no separate database service to operate. I consider it sufficient for the design; whether it has enough capacity for a particular installation depends on the workload and storage hardware.
A meter reading is a stateful exchange
The example uses meters on a shared half-duplex bus. A job selects a meter, requests a reading and waits for the reply before finishing the exchange. Only one such conversation can use the bus at a time. Giving another meter a different address does not give it a separate wire.
Assume one controller cluster manages a fleet of independent buses. Workers can fail and connections can break while devices continue processing. A missing response tells us little about how far a command got. I assume the worker code obeys its execution and cleanup contracts; the engine cannot compensate for a driver that ignores them.
A run is one accepted execution of a job. An attempt is one worker's turn executing that run. An action is an individually authorized external operation within it. A resource identifies the thing competing jobs must not use together. I will call the example's run R and its bus B; these are explanatory labels, not API identifiers.
Now let run R lose its worker after selecting the meter, while waiting for the reading. A replacement process has neither the old connection nor proof of what the device did. The controller must remember the job's progress without treating bus B as available just because the worker disappeared.
Independent releases for business code

A company changing its meter protocol should be able to ship a new worker image without rebuilding the controllers. The image registers exact handler and driver versions, and the controller selects a compatible worker through the supported SDK and worker protocol. It never needs to load the company's Go packages. SDK dependencies and deployment choices.
I put protocol handling in independent workers so a single service would no longer have to own every live conversation. The controllers remain a shared dependency, with their own quorum and storage limits.
Parnas argued for hiding design decisions likely to change behind interfaces.fn:1 Here the company should be able to change its protocol code without having to understand changes to controller storage. The SDK passes recorded state and execution requests as data; company packages can depend on it without importing Raft or bbolt.
A handler receives the recorded state, an event when present, and explicit time. It returns a bounded decision for the controller to validate. Replicas apply that validated transition during log replay; they do not rerun the company's handler. A failure can still cause the worker to call the handler again, so the handler must be pure. External operations belong in a driver, which can keep a temporary session across related actions. These are author obligations: Go does not enforce purity or bounded execution. Handler and driver contracts.
The cost is a remote protocol over authenticated gRPC/TLS, with ordering and cleanup rules for lost connections. Separate processes keep ordinary worker crashes and dependencies out of the controllers. They do not make hostile worker code safe or prove a driver's reports truthful. For a small deployment, embedding company logic with a controller remains an option.
Job claims and reservations commit together
Claiming run R while leaving bus B available would let a second job claim the same bus. The job claim and its reservations therefore have to succeed together. Raft orders the transitions across controllers; the application defines the transaction that applies each one deterministically.fn:2
In djob, that transaction checks compatibility, quotas, grants and every required resource before assigning ownership. If bus B is busy, the claim fails without leaving the run holding partial reservations. Claim and ownership transitions.
I could put resource ownership in a separate lock service. I would then have to handle failures between accepting the job claim and acquiring its resources. Keeping both in one transaction avoids that gap, though it sends their coordination through the same replicated state machine.
Each controller member retains a WAL, snapshots and materialized bbolt state. Ready and wait indexes persist transactionally with that state; only some derived views have explicit rebuild procedures. Restarting a member can recover the recorded coordination history. It cannot recover a device reply that was never recorded. Raft persistence and application, Dispatch indexes.
A three-controller cluster needs two members for a majority. Without them it cannot commit new authority. Already authorized work remains subject to its permission and cleanup limits, but losing quorum does not retract a command from a device. The remote-session checks use quorum and locally applied state; the integration must still enforce exclusion at the device. Remote-worker quorum checks.
This is where action granularity affects performance. The claim, each authorization and each recorded result all add persistence and replication work. Even an idle bus has to wait for those commits. Counting conversations alone misses that cost.
When a device acts but the result is lost
The worker's handler may propose reading the meter, but the proposal gives the driver no permission to send a command. First the controller records an authorized action, identifying its run, attempt and action sequence. The driver may then perform it while that permission remains valid. Once the driver reports an effect, the controller must commit it before using it to advance the job.
Authorization still leaves a gap. A worker can fail before sending, during transmission or after the device acts. If no result commits, all three cases can leave the same controller record.
Birrell and Nelson's RPC analysis describes this ambiguity after a failed call: the caller may be unable to establish whether the procedure executed.fn:3 Recording more controller state does not remove that uncertainty. A receiver that atomically deduplicates a stable operation identity can make repetition safe for that operation; a legacy device may offer no equivalent contract.
Saltzer, Reed and Clark's end-to-end argument helps explain why acknowledgements at different layers are insufficient on their own.fn:4 An MQTT broker accepting a command does not tell us whether the meter completed it. A device reply means whatever that device's protocol promises. Even a worker report arriving at a controller still needs a durable commit before the engine can rely on it.
For the meter reading, the distinction looks like this:
| Failure point | What the controller can establish | Consequence for this resource-backed conversation |
|---|---|---|
| Handler proposes an action, but authorization has not committed | No durable permission for that action | The driver must not send it; lost access may still require cleanup of the existing conversation |
| Authorization commits; worker disappears before a result commits | Permission existed; whether the command ran is unresolved | Ownership expiry preserves an unknown action/run and blocks the bus |
| Broker acknowledges the command; worker disappears before a result commits | Authorization survives; the broker receipt may be lost and would not establish device completion | Broker acknowledgement cannot justify replay or safe bus reuse |
| Action success commits; worker disappears before the conversation finishes | The known action result survives | The interrupted run can still become unknown and the bus can remain blocked |
Suppose meter selection succeeded and that result committed, but the read has only an authorization record. The operator knows which step lacks a recorded outcome. From that record, they still cannot tell whether a reading returned or whether the bus is safe to use. That is a reason to record intermediate outcomes. If none would help the application or operator decide what to do, I would consider making the whole exchange one action instead. Splitting it further adds commits without necessarily making recovery easier.
Reconnecting cannot give a replacement worker permission to replay that uncertain action. Run R retains its history, but that history cannot reconstruct the lost device session. If the execution rules permit another attempt, it starts with a fresh driver. Session supervision, Action completion and expiry.
A lost worker can leave the bus unavailable
Bus B must represent the wire that the meters share. If the inventory gives each meter its own resource, two jobs can each hold a valid claim and still collide. Every relevant alias, address and tenant has to map to that shared resource. Access outside the scheduler needs accounting for too.
The reservation lasts through the reply wait and cleanup, not just the command. A job with no reserved resources can end its attempt while waiting and resume elsewhere later. This conversation cannot: another exchange could interfere with the meter's reply. Its wait keeps both the conversation and the reservation alive.
Leases and ownership epochs let the controller reject stale operations. Stopping stale access to a device requires the recipient's cooperation. Chubby's sequencers illustrate this: the protected recipient checks the generation and rejects an obsolete operation.fn:5 A newer epoch in the controller is useless as a physical fence if the gateway cannot check it.
That leaves the driver with cleanup work. Execute must return only after its I/O stops. Quiesce must cover recovery and protocol quiet periods; Close releases local sessions. Closing a connection alone can leave a command queued or a device still active. The SDK cannot safely detach a driver that refuses to stop.
When access remains uncertain, djob blocks the resource. Every job on that bus has to wait. I accept that loss of availability because a fixed timeout or a new Pod gives me no evidence that the old access stopped. The integration or an operator has to establish that before releasing the bus.
The operator does not have to answer both questions at once. After establishing that the old access stopped, they can release B while R remains unknown and undispatched. Settling R as succeeded, failed or cancelled does not resume its interrupted conversation, unlike an ordinary resource-free wait. Evidence of success could also settle R while B remains blocked. A checkpoint remembers logical progress; it cannot restore the selected device, socket or unfinished handshake.
Keeping unfinished jobs on their original release
Run R may still be waiting when release v2 arrives. If the new handler interprets the checkpoint differently, moving the run would change its meaning even though the stored data survived. Speaking the same worker protocol does not make two releases compatible with the same unfinished job.
djob can select a published worker release when it accepts a run. That selection stays with the run through retries and resumptions. Old and new releases can coexist. Weighted routing chooses a release for new work using a deterministic mapping from run identities. It does not balance live load or guarantee exact percentages in small samples. Existing trigger templates retain their own snapshots until explicitly updated. Release admission and outstanding work, Run activation and release selection.
Pinning a release means keeping its worker implementation available until its accepted work finishes. Waiting jobs and unknown outcomes count too. Retirement stops new admission without migrating those runs. Managed lifecycle guards help with that policy, but cannot prevent an external Kubernetes eviction or infrastructure failure.
The order matters when changing a weighted policy. Suppose a trigger retains a snapshot assigning 20% to v1 and 80% to v2. Closing v1 admission blocks all new runs through that policy, including identities that would select v2: every listed release must be active before selection. There is no automatic redistribution. Update the definition and affected trigger snapshots to remove v1 first, then close its admission while retaining it for accepted work. Updating only the definition leaves existing trigger snapshots unchanged. Weighted-policy validation.
Pinning the image leaves another question: can its configuration change under the same name? By default, a configuration reference does not freeze its contents. The optional immutable-configuration policy checks protected Kubernetes Secrets and pins their UID. It depends on the corresponding admission policy and permissions. Kubernetes configuration checks.
The price is deployment capacity tied up in old releases. State migration could let a system retire them sooner, but it would need rules for in-flight operations and live protocol sessions as well as stored checkpoints.
Recovering an interrupted job
An operator might inspect an unknown run just as someone else resolves it. To avoid applying a stale decision, resolving the run checks the expected checkpoint step. Releasing a resource checks that it is still blocked at the expected ownership epoch, the generation of its reservation. If the inspected state has changed, the request fails. Run and resource resolution.
The controller authenticates the operator, checks their tenant and records who made the decision and why. The operator still needs evidence for the outcome or for stopped access. Recording a resolution cannot withdraw a delayed device command, and tenant-scoped APIs cannot keep an out-of-band client off the physical bus.
A growing queue behind a busy bus needs a different response from a bus blocked by uncertain access. For the first, I would examine arrivals and occupancy. For the second, I need evidence that access is safe before releasing it. If an unknown run prevents an old release from retiring, spare workers will not resolve the missing outcome either.
Cluster recovery can lose more than a worker session. Replacing a member while a quorum survives keeps the existing cluster's authority, and a temporary quorum outage can leave its durable history intact. Losing that history and restoring an older backup is different. The old cluster must be excluded, and work accepted after the backup may be missing.
Suppose the backup shows B available. Later, the original cluster accepts a reading and the device acts, but that history is lost. The restored database may have no request identity and no unknown run to alert the operator. Fencing the old workers cannot bring those facts back. Execution therefore starts held after recovery. Producers must stay paused while operators reconcile the restored work with external effects; a missing record is not enough reason to resubmit. Backup recovery and execution hold.
The worker fault cases and release-admission cases cover parts of the execution contract; a target installation still needs its own storage and physical-exclusion qualification.
When I would choose an existing platform
A SQL-backed engine could keep claims, reservations and results in short transactions while a gateway owner handled the live conversation. PostgreSQL's row locks provide one way to coordinate concurrent changes. The device-specific cleanup and uncertainty would remain. If I already operated a suitable database, I would consider starting there. Avoiding that dependency means taking responsibility for replication, snapshots and recovery myself.
Temporal already supplies durable workflow history, activities, timers, independent workers and Worker Versioning, including pinned workflows. A gateway owner could hold the bus conversation inside one activity, with a resource coordinator managing access across jobs. I would have to reconcile that extra application boundary, but would gain much of the surrounding workflow machinery. For a process already using Temporal, that is worth serious consideration. Worker Versioning, worker deployments.
Building djob lets me develop job execution and resource ownership together. The same model accounts for unknown outcomes and runs that still need an older release. The small storage component helps me understand the system, but leaves me responsible for its distributed behavior and operational qualification. I have no comparative experiment showing it to be faster, cheaper or more reliable than a SQL-backed engine or Temporal.
An adopter also has to consider the license: the engine is source-available under custom terms, while the Go SDK is Apache 2.0. Engine and SDK licensing.
What would change the design
I would revisit these choices if measurements pointed somewhere else. If commits held up work while independent buses had spare capacity, splitting independent resource domains across clusters could help. That split would need a clear ownership boundary: sharing a resource across clusters would bring coordination back. I would want bus service time, queue age, worker-session capacity and lifetime storage growth measured before making that change.
Gateway capabilities could change the recovery policy too. Enforceable fencing and operation deduplication would give the engine better grounds for continuing after a lost worker. A workload dominated by resource-free branching workflows would weaken my reason for specializing in conversations. For the meter jobs, though, the awkward question remains what to do with an unfinished exchange. If it keeps an old release alive for too long, I need a better way to reconcile it; adding workers will not answer it.
Appendix: estimating capacity from bus occupancy
To size a deployment, I would start with the bus. Suppose there are 100 independent buses, each needing 20 readings per minute, and each conversation occupies its bus for two seconds, including waits and cleanup. These are assumptions for sizing djob, not gateway measurements.
| Quantity | Assumption or calculation |
|---|---|
| Offered work per bus | 20 readings per minute |
| Bus occupancy per reading | 2 seconds, including waits and cleanup |
| Ideal capacity per bus | 60 / 2 = 30 readings per minute |
| Average bus utilization | 20 × 2 / 60 = 2/3 |
| Offered work across 100 buses | 2,000 readings per minute, about 33.3 per second |
| Average occupied conversations at that rate | 2,000 / 60 × 2 ≈ 66.7, before additional overhead and headroom |
| Burst of 20 readings on one idle bus | At least 40 seconds until all finish under the stated two-second assumption |
More workers help across independent buses. On one bus, the conversations still have to take turns. Even with spare average capacity, starting every periodic job on the minute can leave the last one waiting a long time.
Across the fleet, each remote worker session serves one conversation at a time, including waits that retain the bus. The assumed rate needs about 67 occupied conversations on average, plus headroom for variation and controller work. The defaults do not provide that: four sessions per tenant per controller give one tenant only 12 slots across three controllers. Adding worker Pods would hit the same ceiling. Worker sessions, controller limits and tenant quotas all need configuring for the workload. Worker sessions and capacity limits.
Little's law relates the average number of jobs in a system to its throughput and average time in that system: L = λW. Assume stable operation with the required finite averages. A per-bus throughput of one reading every three seconds and a mean residence time of six seconds would then imply two jobs on average. Residence includes queueing and execution; the result says nothing about a tail-latency deadline.fn:6
For an installation, I would measure how long each bus stays occupied and how old its queued jobs become, especially after bursts or failures. The controller needs a separate budget. A reading causes several durable transitions, so roughly 33 readings per second means more than 33 storage writes per second. The number of actions and the size of their records and retained results matter, as does commit latency.
Finishing a job does not remove all of its storage cost. Request identities and run tombstones survive cleanup so an old request cannot be mistaken for new work; some receipt classes keep full results permanently. At the example's rate, distinct readings would create 2.88 million runs a day. Compaction can reclaim eligible payloads, but the permanent metadata keeps growing. An installation needs to budget for that growth or eventually retire the deployment. Expiring retry identities instead would change which old requests the engine can recognize. Receipt compaction, Permanent run identities.
If jobs keep arriving faster than a bus can serve them, durable storage will faithfully preserve a growing queue. At that point the application must limit admission or decide which readings have become obsolete. Coalescing replaceable observations can help. It is unsafe when every requested effect matters.
Sources and scope
I checked the implementation on 15 November 2026. Public source links pin revision d603e27; the cited implementation files match the inspected local 2dd5802 tree. The papers supply the reasoning used here, not a certification of the engine.
The supplied system-design interview guide by Arslan Ahmad / DesignGurus informed the sequence: clarify constraints, estimate scale, justify components, examine trade-offs and failures, and stop adding machinery without evidence. It is an editorial checklist, separate from the research references below.
[fn:1] D. L. Parnas, On the Criteria To Be Used in Decomposing Systems into Modules, Communications of the ACM 15(12), 1053–1058 (1972). The modularization argument concerns hiding consequential design decisions; applying it to the controller/worker boundary is my interpretation.
[fn:2] Diego Ongaro and John Ousterhout, In Search of an Understandable Consensus Algorithm, extended version (2014), §§2, 5, 7 and 8. Replicated state machines, quorum-based consensus, log compaction and client interaction concern the controller protocol; they do not make external device effects transactional.
[fn:3] Andrew D. Birrell and Bruce Jay Nelson, Implementing Remote Procedure Calls, ACM Transactions on Computer Systems 2(1), 39–59 (1984), §3.1. Failed calls can leave the caller unable to establish whether execution occurred.
[fn:4] J. H. Saltzer, D. P. Reed and D. D. Clark, End-to-End Arguments in System Design, ACM Transactions on Computer Systems 2(4), 277–288 (1984). The acknowledgement discussion explains why lower-level delivery does not establish the application's completion condition.
[fn:5] Mike Burrows, The Chubby lock service for loosely-coupled distributed systems, OSDI (2006), §2.4. Sequencers require recipient validation; a lease's expiry alone does not force an old client's external access to stop.
[fn:6] John D. C. Little, A Proof for the Queuing Formula: L = λW, Operations Research 9(3), 383–387 (1961). Paper. The theorem relates steady-state averages under its stated assumptions; it supplies no tail-latency or overload guarantee.