Write-Ahead Log Design in Distributed Filesystems

Designing a write-ahead log for a distributed filesystem means deciding both what gets written first and what survives a failure. Replication, fsync, truncation, and recovery must agree on the same durability contract, even under load and concurrent writes. A design that makes each part look correct on its own can still lose the argument when the power goes out.

What a write-ahead log does and why distributed systems complicate it

A write-ahead log, or WAL, records an operation before applying its change to persistent data. The log entry reaches stable storage first, so a crash leaves a record that recovery can replay from the last known-good point. That ordering is the core promise.

Distribution turns that promise into four separate problems. Stable storage becomes a quorum of nodes that may disagree about which entries were flushed, while fsync on one node says nothing about the other replicas. Deleting history that one node no longer needs can leave another node unable to recover. Concurrent writers can also send entries to different replicas in different orders, breaking the assumption of a single append stream. These are not incidental complications to tidy up later, but the design space itself.

How replication strategy sets the durability baseline before any other choice matters

Replication policy defines what the system means when it calls a write durable. A WAL entry is only as safe as the rule that decides how many nodes must acknowledge it before commit proceeds. Synchronous replication to a majority quorum gives a stronger guarantee than asynchronous replication, but every write pays a network round trip. Asynchronous replication lowers that latency by allowing acknowledgment before all the required protection is in place. If the leader fails during that window, acknowledged writes can be lost.

Strong replication everywhere therefore comes with a bill, and cross-region latency collects it on every write. The cost changes which workloads the system can serve, rather than merely making a benchmark less flattering. Placement matters for the same reason. PALF, OceanBase’s consensus-backed logging system, prefers leaders in the same region/IDC as the application system because an extra round trip is needed when the log leader and database leader are apart.

Where consensus sits also changes the price of durability. PALF integrates consensus into the WAL layer, generating and flushing redo logs before applying them to persistent storage. Compared with the replicated state machine, or RSM, model described in its design, this removes the buffering ceiling and lets reads go straight to the storage engine, eliminating the need to merge cache and storage. Its primitives support transaction processing, database restore, and physical standby databases. Logging is the foundation for those guarantees as well as a tool for crash recovery.

Sharing that foundation across partitions reduces per-partition overhead and centralizes durability policy. Multiple partitions can use one stream persisted and synchronized by a dedicated log service. The tradeoff is concentrated responsibility. That service’s availability and throughput ceiling now bound every partition it serves.

What fsync actually guarantees, and where those guarantees break down under distributed load

A replication acknowledgment is only useful if its underlying flush means what the design assumes. Fsync concerns the local storage path. Its durability also depends on how the device handles its write cache and persistence. Treating the call as an unconditional certificate that data reached persistent flash ignores that dependency. Replication cannot repair a mistaken definition of “flushed” that every replica shares.

Batching makes this distinction visible under load. Collecting writes before fsync reduces the frequency of flushes, but a crash before the next flush can discard writes accumulated since the previous one. If the system treats those writes as durable before that point, its acknowledgment promises more than the storage path provides. The optimization is therefore inseparable from the commit rule. Fsync is not a magic word, even when called with great conviction.

Frequent flushes also make SSD write amplification a WAL design input. An aggressive policy can increase wear enough to shorten hardware life below rated endurance. Bohyun Lee, Tobias Ziegler, and Viktor Leis’s 2026 paper, How to Write to SSDs, describes out-of-place writes that reduce amplification at both the DBMS and SSD layers. The paper calls WAL writes “easy to handle” in its amplification accounting without specifying why. That observation does not make amplification irrelevant, especially when a design breaks the log’s append pattern.

PALF can commit once a majority of replicas have flushed. BtrLog likewise replicates records across a quorum of SSD-backed log nodes in a single network round trip. These approaches reduce sensitivity to stragglers, but commit still waits for enough nodes to finish. Under load, the quorum sets the latency.

Log truncation as the operational decision that replication and fsync leave unresolved

Truncation is a correctness policy because a replica may still need an entry that the active writer has finished using. The standard low-water mark tracks the lowest log offset that any replica or consumer still needs. Computing that floor requires accounting for failed replicas as well as responsive ones. Silence is not evidence that a replica has finished its homework.

A network partition exposes the policy choice. A lagging replica can pin the low-water mark at its last acknowledged offset, preventing truncation beyond that point. Retaining its history indefinitely allows unbounded log growth. Truncating past it instead means accepting a full resync or declaring the replica irrecoverable.

Segments make it easier to delete history but add boundaries that recovery must be able to trust. A retained segment needs to be self-consistent, since a partial segment left by a crash can become a corruption hazard. CRC or other integrity checks that do the same job help build that trust. Segmentation controls the units that can be removed, while the low-water mark governs when removal is safe. Neither replaces the other’s correctness question.

Retries add another horizon to the calculation. A failed acknowledgment can cause the same logical operation to appear twice, with the duplicate crossing into a later segment. Removing the earlier segment may then remove the record needed to recognize the duplicate. Idempotent operations or unique identifiers that survive in the retained window address that problem, but truncation must preserve their protection. A durability horizon alone is not enough when replay also depends on a deduplication horizon.

How concurrent writers expose the ordering assumptions WAL makes implicitly

Recovery needs a total order over the entries it replays. In the simple single-writer case, sequential append supplies that order through one log tail. Distributed writers do not inherit it automatically. Entries can reach replicas in different orders, so replay can leave them in different states if the outcome depends on that order. Recording every operation is insufficient if the replicas disagree about which operation came first.

Vector clocks or logical timestamps do not, by themselves, settle the replay contract. Describing relationships between events is not the same as enforcing one sequence that recovery can follow. A partial order leaves decisions unresolved when replay requires a total order. Consensus can resolve those decisions through a leader that serializes entries. Under high write concurrency, however, that leader becomes a throughput bottleneck.

In the database design described for PALF, writes can be applied in memory before log generation, allowing the database’s concurrency control to establish their order. That is distinct from applying changes to persistent storage, which follows the WAL’s generation and flush. The ordering burden moves, but it does not disappear.

Parallel replication pipelines allow progress without waiting for every replica to flush. They can also pre-position consistent log segments for recovery. Both benefits require entries to stay in order as concurrent writers add work, even as more bytes move at once. More writers increase the log volume in flight across clusters, so the network can become the binding constraint before append throughput does.

Recovery path design as the proof that the earlier choices were correct

Recovery is where the durability contract has to produce correct state from incomplete evidence. Starting at a checkpoint or known-good log position, it must distinguish written-but-unapplied entries from records belonging to transactions that never committed. It must also handle cases where the records are missing for changes that reached data. These cases test whether replication, flushing, and application respected the same boundary. A fast steady state cannot compensate for a recovery path that cannot tell what is safe to replay.

Truncation and replication also determine how long that reconstruction takes. Keeping history for longer can leave more to replay, while pre-positioned segments can spare recovery the work of fetching logs. Having the segments there helps only if they arrived in order and without gaps. Otherwise, local availability of the files is not enough. Recovery time sets the floor for how available the system can be in practice, so it belongs in the architecture as well as in measures of how the system runs.

A crash can leave a valid header attached to an incomplete body. Treating that body as complete risks incorrect state, while rejecting too much can stall recovery unnecessarily. End-of-entry markers and entry-level CRC checks allow replay to stop at the last known-good entry. Segment-level integrity alone does not provide that finer boundary.

Agent workloads extend the same consistency problem across filesystem and process state. Dong et al.’s 2026 DeltaBox work shows that when files are rolled back but process memory is kept, an agent is left with stale context for files that are no longer on disk. DeltaFS checkpoints file state by freezing the writable layer and inserting a new one, making a layer switch sufficient for rollback. DeltaBox’s rollback in roughly 1.86 ms reflects a design in which checkpointing and rollback are considered together. Recovery is therefore not a concern addressed after WAL design, but the specification it must satisfy, and an unclear recovery path means the design is incomplete.

How the four tradeoffs interact when all of them are in play simultaneously

Consider what happens when a system relaxes fsync frequency to improve throughput without changing its truncation rule. The deletion point may still assume a durability boundary that the new flush policy no longer provides. Meanwhile, a parallel replication pipeline can have work in flight for a segment that another part of the system considers finished. Safe deletion must account for that unfinished work as well as completed writes. A local optimization has changed the evidence on which another layer relies.

A lagging replica need not block commit, yet its acknowledged offset can still constrain truncation. Removing it from the write-time critical path does not remove it from the recovery contract. Replication, fsync, truncation, and ordering thus act as four interdependent dials. Turning one can move the others, whether the dashboard admits it or not.

Netflix’s WAL system addresses the cost of leaving those dependencies to separate teams. Locally correct solutions had produced a globally inconsistent patchwork, increasing debugging costs and leaving gaps in cross-system guarantees. Centralizing policy behind one WriteToLog interface gives callers one durability guarantee and one replication and retry model. PALF’s co-design of logging and the database expresses the same principle through shared architectural decisions. Both approaches make interactions explicit, although a centralized WAL service also becomes a critical dependency for its consumers.

Persistent agent infrastructure faces the same need for a coherent contract. Agent infrastructure that offers checkpoints inherits the same trade-offs: replication, fsync, truncation, and ordering still have to agree on what a restored checkpoint contains. File semantics fit an agent’s working environment because models are disproportionately trained on bash and file tools. Making that layer durable and consistent aligns what the agent expects with what recovery can actually restore. The tradeoffs that shaped distributed database WAL design are now shaping persistent agent infrastructure, and the lessons transfer directly.