Fast Writes, Fresh Reads
A client submits a device event. The service replies accepted. An operator refreshes the dashboard and still sees the old value.
The event may already be safely recorded. The table behind the dashboard is still catching up. Without some indication of its progress, the operator has no way to tell a delayed view from a missing update.
Recording an event and making it available to readers can finish at different times. The service needs to say which of those it has promised when it replies.
Updating the table before replying closes this visibility gap for readers of that table. Moving the update into a worker can shorten the acknowledgement path, provided the service tracks the work still waiting. If lookups are rare, it may be cheaper to scan the journal when a query arrives and maintain no table at all.
I want to compare those choices through a small journal design in Go and Rust. The interesting question is where the lookup work belongs, and what a fast acknowledgement leaves the service to finish afterward. The code also gives us a concrete place to compare the languages' ownership and synchronization rules.
In this article: The contract · The cost · Freshness · Backpressure · Ownership · Research · Measurement
Decide what “accepted” means
Consider a service that records unconditional device updates. Ingestion is its main job. Latest-value lookups happen less often and can tolerate some delay, as long as the service reports it. The lookup table is a projection: a representation derived from the journal to serve those reads.
For this design, accepting a write means appending its records to a local journal and successfully synchronizing the file. Only then may the background worker consume them. The reply confirms that the journal has recorded the write; the projection may still be behind.
The writer publishes its progress after synchronization:
append bytes → synchronize file → publish completed position
These excerpts show that ordering for synchronized writes with a background projection. They omit setup and instrumentation. The Rust writer here is a file; a user-space buffering wrapper would have to be flushed before synchronizing the underlying file.fn:1, fn:2
Neither version advances head if synchronization reports an error. The mutex makes the new position visible to the worker; file synchronization handles a separate obligation to storage.fn:3, fn:4, fn:1
Partial writes and failure boundaries
Rust's write_all keeps writing until the buffer is consumed or it encounters an error it cannot retry. The companion's Go helper loops over partial progress and rejects zero progress. Neither operation makes a batch atomic: an error can leave part of it in the file. Both programs stop on a write or synchronization error instead of retrying an uncertain append.fn:5, fn:2
Creating the journal is part of the persistence protocol too. Both programs create a new file exclusively and, in synchronized mode, synchronize the empty file and its containing directory before timing writes. On Linux, file synchronization alone does not necessarily persist the directory entry. These setup steps still do not qualify the programs for every power-loss scenario.fn:6
A crash after synchronization but before the reply can also leave a recorded write that the client believes failed. Retrying safely requires a request identity or an application rule for duplicate updates. Delaying the projection does not resolve this acknowledgement ambiguity.
The lookup projection is optional for accepting these unconditional updates. If acceptance depends on a uniqueness check or a conditional update, the service needs sufficiently current state to make that decision before accepting the write.
Choose who pays for the lookup
The three policies append the same records. They differ in when they do the work needed by readers.
Pay when somebody reads
With no lookup table, a query can scan backward in blocks from a captured journal boundary. It stops at the first record for the device. A recent value may be quick to find; a device with no record requires scanning the entire captured history.
This avoids maintaining a table for events nobody queries. It becomes expensive when an investigation asks for many old values or missing devices. Infrequent reads can still have a response-time requirement.
Pay before replying
The eager writer updates the in-memory table after synchronization and before replying. A subsequent reader of that same table, using the same lock, can see the write or a later update. A different replica or response cache needs its own visibility guarantee.
The table can remain in memory because the journal is enough to rebuild it. Persisting the table would be a separate choice to reduce recovery time. For a small table, updating it directly may cost less than coordinating a worker.
Pay after replying
A background worker replays the published journal prefix into the table. The writer can usually reply without waiting for that replay. The service does, however, pay for another read pass, decoding and checksum verification, as well as worker coordination.
Those reads may hit the page cache and avoid physical I/O, but still use CPU and memory bandwidth. Moving replay out of the acknowledgement path can shorten the response while increasing total work.
For a service with rare lookups, a scan may be enough. Frequent reads can justify a table. A worker is worth adding when the shorter acknowledgement path is useful enough to pay for replay and delayed visibility.
Put freshness in the answer
Suppose the journal is published through record 10,000 and the projection has applied through 9,700. The view is missing 300 events. These are illustrative positions, not benchmark results.
A value alone hides that gap. A value accompanied by as_of_sequence = 9,700 states the view's boundary. The value's own last-update sequence may be older; a particular device need not change in every batch.
The view must represent a contiguous prefix:
A single worker reads and validates the next batch outside the lock. It then applies the table changes and advances its position in the same critical section. Readers must never observe the new position with the old table.
A live reader must copy the entry and the projection position under that same lock. For read-your-writes, the write response can return its journal sequence. A reader waits for indexed > required_sequence= and copies the result while still holding the lock. It may see a later update to the same device: the position is a minimum freshness requirement, not an exact historical snapshot.
A timeout says the view did not reach that position before the deadline. The recorded write may still be waiting to appear.
Here, “latest” means last in journal order. A delayed device observation appended later will replace an earlier entry, even if its measurement timestamp is older. A service that needs the newest measurement instead needs an event-time or version comparison and a rule for ties.
What the lock enforces
Go places a mutex beside the protected fields. The compiler does not require callers to acquire it. Package boundaries, review, and race-enabled tests help enforce the convention; a race detector only observes executed paths.fn:7
Rust's Mutex<State> supplies protected access through a guard, whose scope makes the lock lifetime visible. Arc<Projection> lets threads share ownership; the projection still needs synchronization for its mutable state.fn:8, fn:9
Rust will still compile code that advances indexed too early while holding the correct lock. Preserving the contiguous prefix is our responsibility in either language. Reading and decoding outside the lock also matters because the critical section blocks readers and the writer's progress publication.
Parallel replay workers would need an ordered commit protocol: a later batch finishing first must not advance the position past an unfinished earlier batch. The live freshness API also needs its own tests; queries made only after catch-up cannot exercise it.
Give unfinished work a budget
A worker can finish a short test after traffic stops and still be too slow for the service's normal arrival rate.
| Assumed input rate | Assumed worker rate | Backlog added in 10 seconds |
|---|---|---|
| 300,000 records/s | 270,000 records/s | 300,000 records |
Illustration, not measurement: 30,000 additional records per second × 10 seconds. At 32 bytes per record, the outstanding journal suffix is 9.6 MB.
A larger queue gives the worker longer to absorb a burst. If that rate difference persists, the dashboard falls further behind and shutdown has more work to drain.
The writer can limit that lag by checking capacity before appending another batch:
When the condition fails, the writer waits. Once the budget fills, a sustained write rate is constrained by the worker's progress. Measuring only the earlier fast acknowledgements would hide that limit.
Inspect the Go and Rust admission gates
fn reserve(&self, count: usize, limit: usize) -> io::Result<()> {
let mut state = self.mu.lock()
.expect("projection lock poisoned");
while state.head - state.indexed + count > limit
&& state.failure.is_none()
{
state = self.cv.wait(state)
.expect("projection lock poisoned");
}
match &state.failure {
Some(error) => Err(invalid(error.clone())),
None => Ok(()),
}
}
Both waits release the mutex and reacquire it before continuing. Both loops must recheck the predicate. Go's wait requires notification to return, but notification does not imply the predicate is true. Rust additionally permits spurious wakeups.fn:10, fn:11
The full worker code records I/O or decoding failures under the lock and notifies waiters. Neither wait has a request deadline or cancellation path, though. Storage that stalls without returning an error can leave callers waiting indefinitely.
The full programs reject a lag limit smaller than the configured batch size and cap record counts and batch sizes for their 64-bit target. Those checks matter if this excerpt is reused: an oversized batch can never fit, and unbounded position arithmetic can overflow.fn:12, fn:13
These excerpts assume one writer. Despite its name, the method checks capacity without reserving it. Two writers could both pass the check and then exceed the limit. Multiple writers need reservations or serialized admission, whichever language implements them.
The limit bounds how many accepted records are waiting for replay, not how long they have waited. Even one event can remain stale indefinitely if the worker stops. It also leaves other budgets untouched: the full journal can keep growing and requests may queue upstream. Variable-size or variable-cost events may need a byte or work budget instead of a record count.
Decide who owns the batch
In the replay design, the worker owns a read descriptor and reusable buffers. The writer publishes a position and has no batch memory to share with it. This keeps ownership simple, at the cost of replay and decoding.
An alternative sends the decoded batch directly after persistence. This avoids replay, but the writer must not reuse the batch while the worker is still reading it.
The next two functions show different handoff contracts. Go keeps the caller's original buffer; Rust transfers it. These are teaching examples from examples/handoff, separate from the journal policies being compared.
What changes at the handoff
A Go channel transfers the slice descriptor, not an independent backing array. Sending it does not prohibit later access by the sender, nor establish that processing is finished. This version copies the batch to preserve the caller's ability to reuse the original after the call.fn:14, fn:3
Rust moves the vector into the channel. The caller cannot keep using the moved value, and the backing array is not cloned. If the receiver disconnects before the send succeeds, SendError carries the unsent vector back.fn:15, fn:16, fn:17
Successful submission still does not mean the worker has applied the batch. A buffered send can finish before the receiver takes it; even an unbuffered handoff only confirms receipt. The projection's progress position remains the way to tell whether processing has finished.fn:3, fn:16
| Choice | Go | Rust |
|---|---|---|
| Transfer without copying payload | Possible; sender relinquishes access by convention | Ownership move restricts subsequent use |
| Retain an independently mutable original | Copy the records | Clone or otherwise copy the records |
| Reuse storage after worker completion | Return buffer through an agreed protocol | Return the owned vector through a protocol |
| Waiting in the shown API | Context can cancel the wait | Standard send waits for capacity or disconnection |
Moving a reusable buffer transfers its allocation too:
The writer therefore needs replacement storage for its next batch. A pool with a return path can reuse allocations in either language. Go relies on a convention that the sender stops accessing transferred storage; Rust checks the move at compile time.fn:18
Copying, cancellation, and recovery
These records contain only integers, so the Go slice copy duplicates all record data. With pointers or nested slices it would be shallow. The caller must keep the input stable while copying, and channel closure must be coordinated with sends.
The Go function copies before it checks cancellation. If both the send and cancellation are ready, the select may choose either, so cancellation has no priority. Rust's standard SyncSender::send takes no cancellation token at all; a deadline-aware submission protocol needs additional machinery or another channel API.fn:19, fn:16
A bounded channel limits queued messages, but callers can still allocate batches before blocking on it. In this Go example, each waiting caller retains an extra copy. Bound the number of producers and batch sizes, or acquire capacity before allocating, if the channel is meant to enforce a memory budget. The same issue applies to Rust producers retaining owned vectors while waiting to send.
Direct handoff would still need journal-based recovery after a crash. A batch leaving memory must not become a hole in the projection merely because no worker had consumed it yet.
Look at the allocation pattern
Both table examples use a dense array indexed by bounded integer device IDs. Each entry holds two 64-bit integers. Updating an entry replaces those values without requiring a separate allocation for the update.
Inspect the table layout in both languages
Each logical entry contributes 16 bytes of field data. That excludes collection metadata and allocator overhead; actual memory use needs measuring on the target. Dense IDs make the layout reasonable. Sparse IDs could waste most of the table, while a hash table would introduce its own storage and hashing costs.
In Go, keeping pointers out of this table reduces the collector's scanning work relative to a pointer-rich graph. To judge the remaining cost, inspect allocation rate and retained heap as well as collector CPU, including allocation assists. Pause time tells only part of that story.fn:20
Rust has no tracing garbage collector in these examples. Destruction and deallocation still take time: dropping an owned structure or the last strong Arc can run cleanup on a latency-sensitive thread. The work depends on what is owned; a vector of these integer-only entries has much less to destroy than a graph of objects with custom cleanup.fn:17, fn:9
These examples avoid a long-lived graph of per-event objects. Their allocation pattern says little about a service with string keys or variable payloads.
Ask whether the same work is necessary
Moving work to a different thread is only one way to change the trade-off. Storage research also changes the representation, memory allocation, and amount of maintenance required.
The RUM conjecture describes a trade-off between read cost, update cost, and memory or storage overhead: optimizing two tends to come at the expense of the third. It is a conjecture about access methods, not a rule that fixes the best design for this service. For our projection, it is a reason to include space in the comparison alongside read and write costs. Freshness and backlog are additional service constraints.fn:21
Change which bytes move
WiscKey (FAST 2016) keeps keys and value locations in an LSM-tree while storing values in a separate log. This reduces value movement during compaction, at the cost of managing that log and changing range-query access patterns. It suggests a useful question for our worker: how much of each event actually needs to be copied, decoded, or reorganized?fn:22
Spend memory and maintenance where they help
Monkey (SIGMOD 2017) distributes Bloom-filter memory across LSM levels to reduce the combined false-positive cost of lookups. Giving every level the same bits per key misses that opportunity.fn:23
Dostoevsky (SIGMOD 2018) reduces merging where it contributes little to point lookups, long range queries, or space reclamation. Its design also accounts for the different needs of short range queries. That distinction matters when deciding which maintenance work earns its cost.fn:24
For the journal, the corresponding question is which representation earns its maintenance cost under the queries people actually make.
Schedule maintenance with foreground demand
EcoTune (SIGMOD 2025) treats compaction as an investment of CPU and I/O resources to improve future queries. HATS (FAST 2026), implemented in Cassandra, combines replica selection for reads with adaptive compaction rate control.fn:25, fn:26
Neither paper implements this projection protocol. They address a problem the worker shares with compaction: background tasks compete with foreground requests for the resources intended to make those requests faster.
A writer can slow down even when it never waits directly for the worker. Both still use the same CPU, caches, memory bandwidth, and storage device.
For the journal, this leaves several choices worth measuring: reduce the bytes replayed, maintain less state, or schedule replay around foreground demand. Changing threads is only one option.
Measure acknowledgement and visibility
The producer needs an acknowledgement. The operator needs a sufficiently fresh answer. The service must sustain both without accumulating an ever-growing backlog.
| Whose requirement? | Measure | What a misleading result leaves out |
|---|---|---|
| Producer | Arrival-to-acknowledgement latency | Upstream queueing and batch formation |
| Operator | Visibility delay and lookup latency at a stated freshness | Whether the answer includes the write |
| Service | Lag, catch-up time, CPU, memory, and physical I/O | Work still unfinished when the timer stops |
| Recovery | Time to the required serving state | Which representations must actually be rebuilt |
The companion records batch acknowledgement timing, time in the lag gate, maximum record lag, and catch-up time after writes stop. Its lookups run over the same completed history in all three designs. The gate timer includes acquiring and checking the mutex even when no wait is needed, so it is not a pure blocked-time measure.
Those post-drain lookups still leave the operator's experience untested. To measure live freshness and interference, queries must run while writes continue. A timer around a ready batch also excludes upstream queueing and time spent waiting for the batch to fill.
Keeping the experiment comparable
The companion uses the same 32-byte record format, deterministic input, batching, table layout, worker batch size, and lag limit in both languages. An independently generated fixture checks the encoder, and the comparison runner checks journal hashes and query results. Buffered writes belong in a separate comparison from synchronized writes because their acknowledgement contracts differ.
The harness does not collect process CPU, RSS, allocation profiles, or physical device I/O. Its read-byte counters include data served from the page cache, and its table-size counter covers only logical entry payload. A sustained experiment needs those additional measurements and must run long enough for a slow worker to reach its lag limit. A short run with writes followed by catch-up cannot establish a service-level latency or freshness guarantee.
Measure recovery to the serving state each policy actually requires. A journal-only service need not build a projection before answering queries. A projection-backed service may choose to finish replay before serving or expose partial progress. Recovering from a partial trailing record also needs testing separately from broader power-loss behavior.
What the local runs showed
The measurements below were run locally on this machine:
| Component | Configuration |
|---|---|
| Processor | Apple M3 Max, 16 CPU cores, arm64 (Mac15,9) |
| Memory | 128 GiB |
| Operating system | macOS 26.6.2, build 25G83 |
| Storage | Internal storage, APFS filesystem |
| Toolchains | Go 1.27.3 and Rust 1.99.1, optimized builds |
| Runtime settings | AC power; GOMAXPROCS=16; no CPU affinity or quota |
Runs used normal filesystem caching, without cache eviction or exclusive machine access. Each condition ran seven times in randomized order, with one benchmark process at a time. All 168 measured runs agreed on journal bytes and query results within their workloads; twelve warmup runs were excluded. These results describe this machine and workload, rather than a hardware-independent performance expectation.
For the synchronized baseline, 100,000 records in batches of 64 took roughly seven seconds under each policy. The background worker showed no consistent acknowledgement advantage, and every synchronized background run peaked at just one batch of lag. Synchronization made this a poor workload for testing pressure on a 4,096-record budget.
The longer buffered runs made the maintenance cost easier to see. Each wrote two million records across 4,096 possible keys. The table held only 64 KiB of logical entry data. These are medians of seven runs; the p99 columns summarize the seven per-run batch percentiles.
| Policy | Go write + drain, ms | Rust write + drain, ms | Go batch ack p99, µs | Rust batch ack p99, µs |
|---|---|---|---|---|
| Journal only | 124.95 | 116.19 | 12.46 | 12.21 |
| Eager table | 122.62 | 111.68 | 12.38 | 11.96 |
| Background table | 171.70 | 157.94 | 14.58 | 14.38 |
Here, the background median elapsed time was about 40% higher than eager maintenance in both implementations. Its batch-acknowledgement p99 was higher too. Updating this small table directly was cheap; the worker added replay and coordination, including another logical read of all 64 MB of journal data. Moving that work into a worker did not pay off in this experiment. These buffered numbers do not describe synchronized-write throughput.
Reducing both the lag limit and worker block to 64 records exercised tighter coordination. All fourteen background runs reached that bound without exceeding it. Median write-and-drain time rose to 206.73 ms in Go and 313.19 ms in Rust. Because both settings changed, and the unchanged eager controls also ran faster in this later campaign, I would not attribute an exact slowdown to the lag limit alone.
The journal-only policy paid at query time: post-drain lookup p99 medians were about 28 ms in Go and 20 ms in Rust in the longer buffered workload, which included 36 absent-device queries out of 256. Table lookups were close to the timer's resolution. Those measurements show why the read workload matters; they do not predict HTTP latency or settle a language choice. A larger or more expensive projection could change the maintenance comparison, but this experiment has not demonstrated that benefit.
Choose the language at the constraint that matters
For an existing Go ingestion service with adequate memory headroom and contained ownership boundaries, Go is a reasonable starting point. For a new storage component dominated by buffer recycling, reclamation, and shared mutable state, Rust's static ownership constraints can be a substantial advantage.
Profiling should precede a rewrite. If synchronization accounts for an assumed 90% of operation time and stays unchanged, halving everything else reduces total time by 5%, equivalent to about a 1.053× speedup. That arithmetic does not predict either language's result.
Worker shutdown and failure
The Go worker runs in a goroutine; the Rust worker uses a dedicated standard-library thread. They have different scheduling models. These examples do not compare async runtimes.fn:27
Both programs stop and join their worker on normal completion and returned errors. Go defers the stop-and-wait procedure before file cleanup. Rust performs it explicitly before propagating the write result; dropping a join handle alone would detach the thread.fn:28
The full Rust worker also catches unwinding panics and explicitly publishes failure to wake waiters. That does not make shutdown generally panic-safe: the excerpts still panic on a poisoned mutex, and an abort or permanently blocked I/O operation lies outside that mechanism. Poisoning itself is advisory and does not establish whether the state is intact.fn:8, fn:11
For the operator refreshing the dashboard, the useful response is a value with a stated progress position, or a clear timeout waiting for the requested write. For the service, that promise has a cost: once replay falls far enough behind, new writes must wait or be rejected. I would establish that behavior under a mixed workload before using acknowledgement latency to choose an indexing policy or a language.
Companion code and validation status
The companion contains both journal implementations, an independent byte fixture, tests, and a comparison runner. The separate handoff examples illustrate buffer ownership; they are not another journal policy. Article excerpts omit setup and instrumentation and are not standalone programs.
Checks passed with the project's pinned Go 1.27.3 and Rust 1.99.1 toolchains on macOS/arm64: nine core Go tests and two handoff tests with the race detector, vet for both Go packages, five core Rust tests and three handoff tests in debug and release, and the ownership compile-fail doctest in both profiles. Optimized binaries completed 168 measured runs with matching journal hashes and query checksums within each workload. Every measured background run stayed within its lag limit.
The measurements use one writer, fixed-size records, a small dense table, and queries only after catch-up. They are local comparisons, not a production capacity estimate or a general performance ranking of Go and Rust. Concurrent live freshness, process-resource use, physical device I/O, and power-loss behavior remain unqualified.
References
[fn:1] Rust standard library. File::sync_all.
[fn:2] Rust standard library. Write::write_all.
[fn:3] The Go team. The Go Memory Model, especially channel and lock synchronization.
[fn:4] Go standard library. os.File.Sync.
[fn:5] Go standard library. io.Writer, including partial-write and buffer-retention contracts.
[fn:6] Linux man-pages project. fsync(2), particularly synchronization of directory entries.
[fn:7] The Go team. Data Race Detector.
[fn:8] Rust standard library. Mutex, including lock guards and poisoning.
[fn:9] Rust standard library. Arc.
[fn:10] Go standard library. sync.Cond.Wait.
[fn:11] Rust standard library. Condvar::wait.
[fn:12] The Go language specification. Integer overflow.
[fn:13] The Rust Reference. Integer overflow in operator expressions.
[fn:14] Andrew Gerrand. Go Slices: usage and internals, 2011.
[fn:15] The Rust Programming Language. Using Message Passing to Transfer Data Between Threads.
[fn:16] Rust standard library. SyncSender::send.
[fn:17] Rust standard library. Vec, particularly its allocation and representation guarantees.
[fn:18] Rust standard library. std::mem::take.
[fn:19] The Go language specification. Select statements, including selection among simultaneously ready cases.
[fn:20] The Go team. A Guide to the Go Garbage Collector, especially allocation, latency, and pointer-free data.
[fn:21] Manos Athanassoulis et al. Designing Access Methods: The RUM Conjecture. EDBT 2016.
[fn:22] Lanyue Lu et al. WiscKey: Separating Keys from Values in SSD-conscious Storage. FAST 2016.
[fn:23] Niv Dayan, Manos Athanassoulis, and Stratos Idreos. Monkey: Optimal Navigable Key-Value Store. SIGMOD 2017.
[fn:24] Niv Dayan and Stratos Idreos. Dostoevsky: Better Space-Time Trade-Offs for LSM-Tree Based Key-Value Stores via Adaptive Removal of Superfluous Merging. SIGMOD 2018.
[fn:25] Hengrui Wang et al. Rethinking The Compaction Policies in LSM-trees. Proceedings of the ACM on Management of Data 3(3), Article 207, June 2025. DOI: 10.1145/3725344.
[fn:26] Yuanming Ren et al. Holistic and Automated Task Scheduling for Distributed LSM-tree-based Storage. FAST 2026.
[fn:27] The Go team. Effective Go: Goroutines.
[fn:28] Rust standard library. JoinHandle, including detachment on drop.