Crash Consistency Models in FUSE Filesystem Implementations

This piece is about a tradeoff that underlies every FUSE-based filesystem: write-back caching makes storage fast, and strong consistency makes storage trustworthy, and FUSE's own architecture won't let you have both without a real engineering fight. The rest of this piece walks through why that fight exists, who's winning it, and where it still loses badly.

FUSE moves filesystem logic into userspace specifically so bugs crash a process instead of the kernel, but the kernel's write-back page cache sits below that userspace layer and operates on its own schedule. Userspace can be as careful as it wants about cross-node coordination and still get blindsided by dirty pages being evicted or served to a reader without the userspace daemon ever finding out, which means strong consistency requires disabling write-back caching entirely and paying a full synchronization round-trip on every write. The library defaults leave write-back caching in its standard state, which is the GlusterFS model: writes return immediately after buffering, performance is strong, and stale reads or conflicting updates on different nodes produce no error and no warning. That disagreement becomes visible later, usually during recovery or an audit, at the worst possible time.

How does FUSE's userspace architecture force a tradeoff between write-back caching and crash consistency, and what happens if you just use the library defaults?

Picture a node that writes data into its local page cache and tells the application "done" before that data ever leaves the building. That's the entire performance case for write-back caching: skip the wait, buffer the write, flush it later. It works beautifully on a single machine. It falls apart the moment a second machine is involved.

DFUSE (SoCC '25) names the failure mode directly. In a distributed setting, if node A writes to its local page cache without coordinating with node B, node B can read stale data or write something that conflicts with what A just did. Nobody crashes. Nothing throws an error. The filesystem quietly disagrees with itself across machines, and that disagreement becomes visible later, usually during recovery or an audit, at the worst possible time.

FUSE's answer to this risk has historically been blunt: disable write-back caching whenever strong consistency is required. Every write then has to complete a full synchronization round-trip before the application gets its "done" signal back. That fixes the race condition. It also means every single I/O operation pays a toll that write-back caching was designed to eliminate.

This isn't a bug waiting for a patch. FUSE moves filesystem logic into userspace specifically so a bug crashes a process instead of the kernel, and so new features ship without touching kernel code at all. That's the whole appeal of the architecture: fault isolation, fast iteration, no kernel rollouts. But the kernel's write-back page cache sits below that userspace layer, operating on its own schedule. Once multiple nodes are in play, there's no way for userspace to guarantee ordering across a cache it doesn't fully control, short of turning that cache off. The tradeoff is structural, built into where FUSE draws the line between kernel and userspace; that is why the field has settled into named camps instead of one obvious best practice.

Why applications' crash-consistency assumptions fail in FUSE

Most application code is written on a comforting assumption: if operations A, B, and C run in that order, a crash will leave the system reflecting A, or A plus B, or A plus B plus C. A clean prefix. Nothing out of sequence, nothing half-finished in a weird way.

That assumption is false even on a local disk running ext4. Modern filesystems reorder writes and metadata updates unless the application forces persistence barriers like fsync(). Optimistic Crash Consistency (SOSP '13) made the underlying point formal: ordering and durability are two separate concerns, and most applications conflate them. That conflation costs you one of two ways. Either you get silent inconsistency because you assumed an ordering guarantee you never actually asked for, or you pay unnecessary performance penalties because you forced durability you didn't need just to get the ordering you did.

A FUSE layer over an object store widens that gap further. Network failures enter the picture. Retry semantics change what "the write succeeded" even means. Authentication affects which reads go through and when. And fsync() can only report back what the underlying system can honestly promise, which over a network connection to a remote store is a lot less than what a local disk promises.

Object storage adds a structural wrinkle on top of all that. It gives you atomicity for a single object, full stop. No batch-level transactions. No in-place mutation, because objects are immutable by design. Operations that POSIX treats as routine, like overwriting a file or renaming it, have no native equivalent in an object store. Something has to be faked, and faking it introduces exactly the kind of timing gap that breaks the sequential consistency application developers assume by default.

That isn't a hidden flaw buried in a bug tracker; it's printed in the README. Developers calling fsync() on top of that layer, expecting the same durability contract they'd get from ext4, are calling a function that does nothing. These clients expose, honestly, what the FUSE-over-object-storage abstraction was never built to promise.

The three consistency models FUSE implementors choose between

Given all that, implementors don't really have a spectrum of options to tune. They have three distinct positions, each internally coherent, each solving one problem by accepting another.

Write-through with strong consistency puts the FUSE process directly in the critical path of every I/O operation. Every write synchronizes to userspace before the application gets its return value. This buys no stale reads and no conflicting updates landing on different nodes at the same time. The cost appears as latency on every single write, because the context-switch overhead, the memory copies, and the lock contention that are already built into the FUSE path get paid in full on every operation, with no buffering available to spread that cost out. DFUSE points to Ceph-FUSE as the reference implementation of this approach.

Write-back with weak consistency flips the priority. The kernel's page cache stays enabled, writes return immediately after buffering, and the FUSE process steps out of the critical path entirely. Performance is strong, since I/O gets absorbed by the page cache and flushed asynchronously in the background. The cost is the one described earlier: no cross-node coordination, so a second node reading the same file can see stale data without any warning that it's stale. DFUSE names GlusterFS as the reference implementation here, and uses it as the baseline weak-consistency system in its own evaluation. The failure mode under this model is quiet. Data races don't announce themselves. They wait until recovery or an audit, and then someone has to explain why two nodes disagree about a file neither of them remembers touching incorrectly.

Optimistic consistency takes a third route, and it's the one that questions the premise rather than just picking a side. Optimistic Crash Consistency (SOSP '13) separates ordering from durability outright, introducing two primitives: osync(), which gives ordering without durability, and dsync(), which gives durability without ordering. Applications get to ask for exactly the guarantee they actually need instead of paying for both properties every time, whether they wanted both or not. The cost here is that this requires the application to know what it's asking for. Code written against the standard POSIX fsync() contract doesn't automatically benefit just because the underlying system supports osync() and dsync(); it has to be rewritten to use them. That model works best for atomic filesystem and database updates, where it preserves the order writes happen in rather than flushing each one quickly to disk.

None of these three is the "right" answer in general. Each is the right answer for a specific workload, which is exactly the judgment call the next section complicates.

How DFUSE resolves the write-back / strong-consistency binary by moving lease management into the kernel driver

DFUSE, built by researchers at Columbia University and Alibaba Cloud and presented at ACM SoCC, held online November 19 to 21, 2025, makes a case that the binary described above isn't fixed law, delivering write-back page caching and strong consistency at the same time as the first FUSE-based distributed filesystem to do so, by relocating where consistency gets enforced.

The core insight is almost embarrassingly simple once stated: write-back and strong consistency conflict in FUSE because the kernel's page cache sits below the userspace consistency logic. Dirty pages can get evicted or served up to a reader without the userspace daemon ever finding out. Userspace can be as careful as it wants and still get blindsided by a cache operating one layer underneath it.

DFUSE's fix is to stop treating that as someone else's problem and move lease management directly into the kernel FUSE driver. When a write lease gets revoked, the kernel itself proactively invalidates the local page cache and flushes the dirty pages tied to it, instead of waiting for userspace to notice and react. That means the userspace process no longer has to sit in the critical path of every I/O operation just to keep consistency intact. The kernel enforces the coordination that previously required a full write-through round-trip on every write.

The result, by DFUSE's own evaluation, is higher throughput and lower latency than the existing write-through designs used by FUSE-based distributed filesystems, outperforming both GlusterFS (the write-back, weak-consistency baseline) and Ceph-FUSE (the write-through, strong-consistency baseline) at once. It also keeps the deployment profile cloud operators actually want: it runs unprivileged, requires only a modified FUSE kernel driver rather than kernel patches, deploys and iterates quickly, and preserves strong fault isolation.

That's a genuine architectural breakthrough for the binary described earlier. It solves the FUSE write-back problem for distributed filesystems operating under a central coordinator model. What it doesn't address is what happens when that coordinator assumption itself becomes the bottleneck, which is precisely the situation the next workload creates.

Why AI training checkpoints expose the limits of FUSE-layer consistency solutions

Distributed AI training doesn't ask for a weaker version of POSIX crash consistency. It asks for a different kind of guarantee altogether, one that no FUSE-layer fix, DFUSE included, was designed to provide: atomic all-rank batch visibility.

Training a modern large foundation model means the data pipeline has to move in lockstep with the training process itself. The batch handed to every data-parallel rank has to be globally consistent across all of them, sequentially ordered from one training step to the next, and recoverable exactly once if a producer or consumer fails partway through. None of that maps onto a POSIX file operation. There's no open(), write(), or fsync() call that expresses "make sure every GPU in this job sees the same batch at the same step, and if one of them dies, don't duplicate or drop any data."

FUSE layers over object storage have tried anyway. The vendor is being upfront about a mismatch between what training jobs need (frequent, fine-grained checkpoints for fault recovery) and what a FUSE layer over an object store can reliably guarantee at that pace and that granularity.

None of this is a criticism of DFUSE specifically. DFUSE's threat model is distributed filesystem clients that need strong consistency under a central coordinator, and it solves that problem well. The training threat model is different: elastic, brokerless data movement at batch granularity, where preprocessing pools scale up and down independently and any coordination component that can't flex with the job turns into either a bottleneck or a single point of failure. Same general topic, consistency under distribution, genuinely different problem.

The architectural counter-move: bypassing FUSE entirely for training-critical consistency

A growing line of work argues the right move for AI training is dropping the file abstraction entirely and building directly on the object store's native unit of atomicity.

BatchWeave, also published as Lakestream (arXiv, May 2026), is one concrete version of that argument. It's an object-store-native training data plane for distributed large foundation model training, and it throws out both the broker/message-queue pattern and the colocated dataloader pattern in favor of versioned-manifest coordination that runs directly over object storage, with no server-side coordination process in the middle at all.

The mechanism at the center of this is what BatchWeave calls the Transactional Global Batch, or TGB. It builds on versioned-manifest ACID storage semantics and then extends them with the specific guarantees training needs: atomic visibility of a batch across every rank at once, a globally ordered sequence of training steps, lifecycle management that stays aligned with checkpoints, and exactly-once recovery from end to end. That's a direct, purpose-built answer to the gap the previous section identified.

Because there's no coordinator process, the brokerless design sidesteps the scaling objection that applies to a central-coordinator approach like DFUSE's. Producers and consumers scale independently of each other, and there's no coordination component introducing a new failure domain into the job.

This approach also leans on a genuine improvement in the underlying infrastructure: Amazon S3 has offered strong read-after-write consistency since December 2020, at no extra cost, so a newly written object is immediately visible to anyone reading it. That single property removes a major hazard for any system, like BatchWeave, that coordinates through versioned manifests rather than a broker.

It's not a universal replacement for FUSE. Multi-object atomicity and the rename/overwrite hazards that come from object immutability are still unsolved by this design. It's a strong, purpose-built answer to one hard problem for training pipelines, distinct from general file access.

Why validating crash consistency in FUSE implementations is harder than building them

Designing a consistency model is one problem. Proving it actually holds under every real failure condition is a separate, harder one, and the difficulty isn't really about tooling.

Research on this, including the paper "Testing Storage-System Correctness: Challenges, Fuzzing Limitations, and AI-Augmented Opportunities" (arXiv:2602.02614), identifies the root issue: many storage-system bugs are latent semantic violations. They don't appear as an immediate crash that a short test run would catch. They appear only under a particular interleaving of operations, or after a system has been running for a long time, or specifically during recovery from a failure. A test that runs for thirty seconds and checks for a crash will miss almost all of them.

Fuzzing, the most widely used automated testing approach in software generally, runs into a structural mismatch with storage systems specifically. Fuzzing assumes failures are local to a single input: feed in a bad value, watch for a bad outcome. Crash consistency properties are properties of long sequences of operations and how persistent state evolves across all of them, not any single operation. A fuzzer optimized to find the next crash from the next malformed input isn't built to notice that the fortieth write in a sequence silently violated an ordering guarantee that only matters if the forty-first operation is a crash.

That's the honest state of the field right now. The consistency models exist. DFUSE shows the write-back and strong-consistency binary can be broken with the right architecture. BatchWeave shows training workloads can route around FUSE's limits entirely. None of that changes the fact that confirming any of these systems behaves correctly, across every interleaving and every recovery path, remains a genuinely unsolved problem, not a testing gap waiting for a faster fuzzer.