Sebastian KaiserA personal engineering notebook

The notebook

Backpressure in a Telemetry Pipeline: What Happens When the Cache Falls Behind?

I wanted to update a cache as device readings passed through an existing telemetry pipeline. The processor would still publish its events; the cache would keep the latest readings ready for queries. A slow cache, however, would leave the processor with updates waiting to be written. I needed a limit on that work and a decision about what to lose when the limit was reached.

The design keeps a bounded set of pending updates, allowing a newer reading to replace an older one for the same device and metric. Revisiting the implementation showed how much depends on what happens after that replacement: which value a writer receives, which write finishes last, and whether a lost reading ever gets another chance. For a device that stops reporting, a dropped final reading may leave the person using the view with a gap that later traffic from other devices cannot repair.

1 When a consumer cannot keep up

In From Time-Series to State, I described keeping the latest known device readings in a cache so that queries would not have to reconstruct them from history. That moves work from readers to the ingestion path. Now a processor that already enriches and publishes events also has to maintain the cache.

I did not want cache writes to hold up event publication. Pranshu Raj's Backpressure in Distributed Systems discusses the choices available when producers outpace consumers, including slowing producers and dropping work. The cache gives me room to discard some updates; the event log has a different delivery requirement.

1.1 A queue buys time

Suppose a consumer completes 80 updates per second while 100 arrive. With room for 200 updates, an empty queue fills in about ten seconds. These illustrative numbers assume constant rates and equal-cost updates: capacity divided by the excess arrival rate gives us the time until the queue fills.

A larger queue absorbs a longer burst. If overload continues, it also makes accepted work wait longer before the queue fills anyway. Launching a goroutine for each blocked send leaves the waiting events in memory, now attached to goroutines. To stay within a memory budget, the system eventually has to accept less work, complete it faster, or discard some of it.

1.2 Where the pressure goes

I use backpressure for feedback that limits what an upstream stage submits. A blocking channel can provide that feedback locally. An asynchronous consumer can advertise demand or stop requesting more work, as in the Reactive Streams model. The submitting thread need not block for the flow of work to slow down.

Rejecting an update without slowing its producer is load shedding. Replacing several pending updates with one is coalescing. These protect the consumer without necessarily changing the arrival rate. My cache design relies on both, so calling the whole mechanism backpressure would leave out an important part of its behavior: ingestion can continue while cache updates are lost.

When required event processing slows down, a durable log can retain the backlog. Lag still grows, downstream views become stale, and retention is finite. The cache is an optional projection, so I can discard work there to preserve capacity for publication.

I can make that trade-off because a newer temperature observation supersedes an older one in the current-state view. It would be wrong for an increment in a consumption total or for an alarm transition or billing event: losing one could change the answer. Even when coalescing is appropriate for a view, historical events may still need to be retained elsewhere.

2 A bounded write path for current readings

My proposal gives the processor a secondary, best-effort path to the current-state cache. It offers each processed measurement to a bounded buffer and continues without waiting for cache I/O. Writers take work from the buffer in batches. If they fall behind, pending updates can be discarded while publication continues.

Both paths still share the processor's CPU and memory. The cache branch needs a resource budget even though it does not wait for I/O on the primary path.

A metric processor publishes required events to a durable stream and separately offers replaceable updates to a bounded pending queue. Fixed workers claim batches and update a current-state cache with a version check. Cache overload ends at admission; it does not regulate the primary path.
Figure 1: The intended separation of responsibilities. Cache admission may coalesce or shed updates; required publication needs its own bounded, acknowledged path. The cache branch is a design, not a guarantee of delivery.

2.1 Coalesce by device and metric

A device reports different metrics independently. Replacing a pending temperature update must not remove its pending voltage reading, so the queue's identity is the pair (device, metric).

The proposed ring buffer has a fixed number of slots and an index mapping each identity to its slot. A newer observation for an existing identity replaces the pending value in place. An older observation is ignored. When a new identity arrives at capacity, the queue evicts its oldest pending entry.

Consider a queue with these pending entries:

Table 1: Illustrative observations, not production data. Only the temperature entry changes when a newer temperature arrives.
Device Metric Observation time Value
A temperature 10:00:00 21.4 C
A voltage 10:00:01 3.5 V
B temperature 10:00:02 19.8 C

If A/temperature arrives with time 10:00:03, it replaces the first value. The queue still has three pending identities, and the writer can skip the superseded temperature reading. The index avoids scanning for it: lookups are expected O(1), while ring insertion and eviction are O(1).

In-place replacement leaves the slot where it was. "Oldest" then means the earliest admitted pending identity, even if its value has just changed. Moving every replacement to the back would change how frequently reporting metrics are evicted. Either policy is possible, but they give different results.

2.2 Writers must consume the state being coalesced

Revisiting the implementation exposed a gap in this design. It keeps recent observations in a ring and sends copies to a separate bounded channel. Writers receive those copies. Replacing a ring entry leaves an older copy already in the channel untouched. If the channel rejects the newer copy, nothing arranges another attempt to deliver the value still held in the ring.

Suppose the device stops reporting at that point. The ring holds its newest reading, but the writer may never see it. The ring is acting as a recent-value registry with best-effort notifications, while the proposal needs it to be the queue of pending work.

I would have writers claim entries directly from that queue. A channel can wake a writer when work arrives without carrying another set of event copies. The following pseudocode sketches the change; these operations would still need to be implemented:

TextAdmission and claiming share one synchronization boundary; network writes happen outside it.Link
offer(event):                         # while holding the queue lock
    reject if admission is closed
    key = (event.device, event.metric)
    if key is pending:
        replace only if event.version is newer
    else:
        evict oldest pending entry if at capacity
        insert key and event
    signal work available

claim_batch(limit):                  # while holding the same lock
    remove up to limit pending entries
    remove their index entries
    return immutable snapshots for the writer

Once a writer claims an entry, a newer observation for that metric can become pending again while the earlier write is in flight. If that write fails, its cleanup must leave the newer pending entry alone. Wake-ups may be combined, provided a worker checks the queue before sleeping and cannot miss the transition from empty to nonempty.

Admission can briefly wait for the lock protecting the ring and index. "Nonblocking" here means that it does not wait for queue capacity or cache I/O. The writer also needs to copy the claimed event under that lock, before another producer can change its contents.

Detailed proposed admission flow: reject closed admission, look up the device-metric identity, replace a newer pending value or ignore an older one, and evict the oldest pending entry before inserting a new identity at capacity. Workers claim immutable batches under the same mutex, then perform deadline-bounded atomic version checks outside it. Slow cache writes reduce claiming and cause pending-entry eviction without throttling the producer.
Figure 2: Proposed admission and writer control flow. The queue and index share a mutex; cache writes happen after claiming and unlocking. Slow writes reduce the drain rate, so admission eventually evicts pending work rather than waiting for cache capacity. This is bounded overload control, not producer throttling. The version policy shown ignores older and equal versions; corrections need the ordering policy discussed below.

2.3 Bound everything that retains work

The ring's fixed capacity limits its entry count. To estimate memory, I also have to account for the values themselves and everything else that retains them: the index, worker mailboxes and batches, encoding buffers, and cache clients. Variable-sized values need a size limit or a byte budget before I can claim a memory bound.

In the proposed single-pending-queue design, a rough payload allowance is (C + W * B) * Emax: pending capacity C, W workers, at most B claimed events per worker, and maximum retained event size Emax. Index and client overhead need their own allowances. The existing implementation also has copies and extra channels, so this expression does not bound its memory usage.

A small worker pool limits simultaneous application-level writes if each worker waits for its write to finish. Counting client objects gives no such assurance, since a client may manage several connections. And without write deadlines, a stalled cache can leave every worker occupied indefinitely.

Batch size and the flush deadline let me trade waiting time for less network overhead. Here too, the implementation needs a closer look: the writer collects a batch, but the storage adapter executes its pipeline separately for each event. Changing the batch size will not give it one network execution per batch.

2.4 Preserve ordering at the final write

A writer can claim a temperature reading just before a newer one arrives. Another writer may claim the newer reading and finish first. If both writes are unconditional, the older reading then overwrites it. Coalescing while observations are pending does not protect them once they are in flight.

The current dispatcher assigns work round-robin, so updates for a key can reach different workers even if they arrived in partition order. Its cache writes are unconditional. The registry's timestamp comparison cannot prevent this later overwrite.

The cache needs to apply the same rule: replace a metric only when the incoming version is newer. With concurrent writers, comparison and replacement must be atomic. Reading the version in a client and then writing leaves room for another writer to act between the two operations. A cache-side conditional update can close that race. Exclusive ownership of a key is another option, provided it covers every writer, including recovery.

Choosing a version also commits us to an ordering policy. Observation timestamps depend on the devices' clocks and reset behavior; equal timestamps, corrections, and sequence resets need explicit handling. Arrival order will not resolve this when replay or delayed delivery can reorder observations.

2.5 A dropped final update does not repair itself

I left retries out of the initial proposal because a later reading could replace a lost one. For a frequently reporting metric, that may be an acceptable delay. But it assumes that another reading arrives and its write succeeds. If the device goes offline instead, its last update may remain missing.

That is where my original claim of eventual consistency needs qualification. To promise eventual recovery of that observation, the system needs replay or reconciliation. Historical storage can supply the data; something still has to find the missing state and repair it.

I also need to know where loss occurs. Ignoring an older observation discards superseded work. Evicting a pending entry or failing a write can lose a value the cache needs. The current path counts outcomes at one boundary, while dispatch can drop silently and the worker discards the returned write error. A single "dropped" counter would hide the distinction even if it counted all of these cases.

Meanwhile, event publication remains a separate responsibility. It is not transactional with cache admission: a cache update can succeed even when publication fails. Launching an unbounded goroutine for each publication would also defeat the memory budget. The required path needs its own publication acknowledgement and input-offset handling.

2.6 What I give up, and where the design stops working

The appeal of this design is that cache trouble does not force the processor to retain an ever-growing backlog. Coalescing removes superseded work, while eviction gives the queue a way to stay within capacity. I pay for that with incomplete state and potentially longer delays before a device's next reading reaches the cache. Avoiding retries keeps the write path simple, but leaves recovery to later observations or a separate repair mechanism.

A process that stays alive through overload has met only part of the requirement. Its cache can still be too stale to use.

2.6.1 When there is little work to coalesce

Repeated temperature updates for one device are cheap to hold: they share one pending slot. The same number of updates spread over different devices and metrics may each need a slot and a write. Event rate alone therefore says little about how much this design saves.

If the writes required to keep those metrics within the chosen freshness target exceed the cache's sustained capacity, no finite buffer can meet that target indefinitely. A larger ring buys time during a burst. Under sustained overload, it can retain more work while the view continues to fall behind. Shedding may keep the service running, but it cannot preserve the promised freshness and coverage at the same time.

The eviction policy also makes no promise to each device. Enough new identities can displace a rarely reporting metric before a writer claims it. If its device then stays quiet, frequent updates elsewhere do nothing to repair that missing value. A single hot key cannot fill this deduplicating queue on its own; competition comes from the number of different keys needing service.

There are two memory limits to consider as well. The pending queue must fit in the processor, and the retained current-state dataset must fit in the cache. Bounding the queue does not bound the number of devices and metrics stored over time. If that dataset exceeds available cache memory, capacity or retention has to change. Otherwise writes can fail or stored entries may be evicted, depending on the cache policy, independently of the pending queue's behavior.

2.6.2 When best effort becomes a broken promise

This design is a poor fit when every observation matters, or when every device must appear in the view within a fixed deadline despite failures. Eviction and unretried write failures explicitly allow those requirements to be violated. A larger buffer cannot change the delivery contract.

Even for a current-state view, accepting occasional loss is a stronger choice for a metric that reports rarely than for one that reports frequently. I would need repair if the last reading before a device goes offline must eventually appear. If the source no longer retains that observation, or no recovery path can read it, the cache cannot reconstruct it.

Version checks depend on trustworthy ordering too. With timestamps as versions, one erroneous future timestamp can cause subsequent normal readings to be rejected until they overtake it. Atomic comparison would faithfully enforce the wrong order. Clock validation or a suitable version and correction policy has to resolve that; coalescing cannot.

2.6.3 When cache trouble spreads to the primary path

The secondary path shares a process with required publication. It still spends CPU on each offered event, takes the queue lock, and allocates or encodes data. A bounded queue limits retained work, not the cost of an unlimited stream of admission attempts. If that work exhausts shared CPU or creates enough memory pressure, publication can slow down even though admission never waits for cache capacity. Fixed workers and byte limits reduce the risk; separate resource budgets, and eventually process isolation, may be needed for a stronger boundary.

Recovery can spread the load in the other direction. If cache misses make readers query history, an outage can move the read workload onto the historical store. If replay then competes with live writes at full speed, recovery can overload the cache again. Google's Addressing Cascading Failures describes how shifting load after a failure can exhaust the remaining capacity. Fallback and repair need admission limits of their own. Repair given only spare capacity will never catch up while live work consumes the entire write budget.

I would use this approach for a replaceable current-state projection whose readers tolerate visible gaps or delay, with enough normal capacity to recover after bursts. If the requirement becomes complete, deadline-bound delivery for every metric, I would need durable pending work or replay, flow control, and capacity sized for that requirement. The bounded ring would still be useful as a local optimization, but it would no longer be the delivery mechanism.

3 How I would evolve the system

Once the cache branch stays within its resource budget, I still need to agree with the people using the view how stale or incomplete it may become. They need to recognize when it is behind, and the people operating it need a recovery path they can manage. Those requirements guide what I would change next.

3.1 Start with the freshness readers need

Queue occupancy alone would tell me too little. I would also track the oldest pending work, the delay until a successful write, and the reasons updates are lost. Observation age needs separate attention: a quiet device can have an old reading while ingestion is healthy. Broker lag helps distinguish that case from a processor that has fallen behind.

A queue full of repeated updates for a few metrics is a good candidate for coalescing. If most entries belong to different metrics, that will buy less; admission or capacity needs to change. Slow writes while little work is pending call for a different investigation again. A larger queue can make the screen staler without speeding up any of those writes.

I would measure retained bytes and write cost alongside events per second. Google's Handling Overload discusses the limits of request-count capacity models and the value of treating work differently according to its criticality. Here, required publication and a replaceable cache projection already have different delivery requirements.

3.2 Give repair a budget

After bounding the write path and enforcing ordering, I would add a repair path that can identify stale or missing state and recover it from the source. Recovery needs a rate limit too. Replaying everything at full speed after an outage can overload the cache just as it becomes available again.

Live updates and repair could share a write budget, with repair using spare capacity. The version rule must apply to both, so a delayed historical read cannot overwrite a newer observation. When the freshness target cannot be met within that budget, the service should report that the view is degraded.

A normal shutdown can leave missing work as well. I would stop admission and join the producers, then drain accepted work within a deadline before closing the clients. Anything left needs to be counted and covered by repair. A fixed sleep cannot tell me whether the writers have finished.

3.3 Scale the constrained part

More workers can help while the cache has spare write capacity. Beyond that, they increase contention or move the wait into connection pools. Partitioning the pending queue would bring ownership and rebalancing into the ordering design. If a noisy source repeatedly crowds out rarely reporting devices, separate allowances may be needed to protect their pending updates.

I would wait for evidence of those problems before adding the machinery. The delivery and ordering gaps need fixing first; tuning an adaptive concurrency controller would be difficult while write errors and missing updates were still indistinguishable.

I would test this with the cache unavailable, checking that publication continues within its own admission limits and pending cache work stays bounded. During recovery, readers must still be able to tell when their view is behind, while repair catches up without displacing live traffic. Until that behavior is established, I would describe the projection as best effort rather than promise eventual consistency.

If something here seems wrong or hard to follow, let me know on LinkedIn.