Crash Consistency Models in FUSE Filesystem Implementations

FUSE lets a filesystem run as a regular program instead of code jammed into the kernel. That design choice is the whole story here: it's what makes FUSE flexible enough to power object-store gateways and ML checkpointing tools, and it's also what forces a hard choice between fast write-back caching and strong crash consistency. Most engineers don't realize they're making that choice. They just inherit whatever default their FUSE library shipped with, and find out the hard way during a postmortem.

What crash consistency actually means under POSIX's undefined behavior

Crash consistency is narrower than it sounds. It just means that after a crash or power loss, whatever's on disk has to make internal sense. No half-finished renames. The system has no metadata pointing at data that never got written and no half-finished renames. It's a different question than durability (did the write survive?) or availability (can you get back online?). A system can be durable and available and still hand you a corrupted file, if consistency wasn't part of the deal.

POSIX doesn't say what's supposed to happen when a filesystem crashes mid-operation. It defines behavior for the happy path, but the ordering guarantees during a crash are left as an exercise for the implementer. That means application developers are stuck guessing, based on how a filesystem happens to behave, what operations are safe to assume are ordered and which aren't.

Research on this gap, including work formalized in the Ferrite framework out of ASPLOS 2016, has examined how this ambiguity can lead to incorrect assumptions about how a filesystem behaves under a crash. Sometimes that produces mild corruption. Sometimes it wipes out data that everyone assumed was safe.

The comparison to memory consistency models in CPU design applies here. Processors reorder instructions constantly, and programmers rely on documented consistency models to know what orderings are legal. Filesystems need the same thing, but for crashes instead of reordering. A real crash-consistency model needs three ingredients: litmus tests (specific sequences with defined allowed and forbidden outcomes), axiomatic rules any valid implementation has to follow, and an operational description of how state actually evolves. Skip all three, and you're left discovering your filesystem's real behavior the way most teams do: in production, at 2 a.m., via a bug ticket.

The four primary consistency mechanisms available to FUSE implementers

There are four real tools on the table, and they're not mutually exclusive; most production systems mix at least two.

Ordered writes is the soft-updates philosophy: instead of logging changes, you enforce a strict order in which metadata updates hit disk, so the on-disk state is always consistent even mid-crash. SquirrelFS, from OSDI 2024, calls this Synchronous Soft Updates, and the core insight is almost stubbornly simple: crash safety reduces to ordering enforcement, without requiring a separate journal. Ordering bugs stay invisible until a crash happens to hit the wrong window, which makes them miserable to catch in testing.

Write-through caching means every write travels all the way through the user-space daemon, across the network if there is one, and only then does the caller get an acknowledgment. Nothing is ever "sort of" saved. This buys airtight consistency at the cost of speed, since every write is throttled by the slowest hop in the chain.

Journaling means you write the intended change to a journal first, then apply it to the real structure, and replay the journal after a crash to fix anything left incomplete. Reliable, but it costs write amplification (you're writing everything twice), and the journal itself has to be crash-consistent, which is its own puzzle. Even ext4, a filesystem practically everyone treats as a solved problem, exhibited unintuitive journaling behavior under the Ferrite framework's testing.

Copy-on-write never touches the existing copy. Write a new version somewhere else, then atomically flip a pointer once it's fully written. The old data stays valid right up until the swap. This maps almost perfectly onto object storage, which already treats objects as atomic, immutable blobs, and it gets you snapshotting for free as a side effect.

The write-back vs. write-through dilemma in distributed FUSE deployments

Single-node crash consistency is already a hard puzzle. Add a distributed layer, where several nodes share one logical view of the data, and the puzzle multiplies. Write-back caching keeps writes in the local kernel page cache and flushes them out asynchronously, which is fast for the client but risky if that node crashes before the flush happens. Write-through skips the local shortcut entirely and sends every write straight through the user-space daemon and across the network.

Research into distributed FUSE systems lays out why they have leaned so hard on write-through: write-back caching across multiple nodes has historically opened the door to deadlocks, because coordinating cache state between nodes is genuinely hard to get right. So teams pick the safe, slow option and eat the performance cost. That's the consistency-performance gap, a significant inconvenience rather than a minor one. It's the reason write-heavy workloads, like ML training checkpoints or log-structured writes, have often been steered away from FUSE-based distributed filesystems.

CCFS, presented at USENIX FAST, built on earlier findings that applications frequently get crash consistency wrong on their own, and proposed shifting more of that correctness burden onto the filesystem design itself. Whatever model the filesystem picks doesn't just affect the filesystem. It determines how much the application built on top of it is allowed to assume without checking.

Object-backed FUSE filesystems and their tradeoff in concrete operational terms

Object storage backends make this tradeoff visible in a way that's hard to ignore, because objects are atomic and immutable by nature, which lines up neatly with copy-on-write patterns. The friction appears in the moments when POSIX expects things object storage was never built to do well, like in-place overwrites, truncation, or atomic rename. The FUSE layer has to paper over that gap somehow, and how it papers over it tells you everything about its consistency model.

Cloud Storage FUSE (gcsFUSE) uses a close-to-open model. Close a file or call fsync, and a new generation of the object gets created before the call returns, assuming nothing else changed the object first. Concurrent writes to the same object are not safely coordinated, and the general guidance is to avoid letting multiple sources write to the same object at once. If two mount points race to write the same object, the result is undefined, and concurrent writers offer no protection against data loss.

S3FS sends every operation directly to S3, making it straightforward but limited in consistency and performance. It checks for changes using the object's MD5 (via ETag), which helps avoid re-downloading files that haven't changed. It's a fine tool for grabbing something off S3 temporarily, but it's not built for workloads that need strong distributed consistency or serious write throughput, and using it for anything long-term and performance-sensitive means testing it carefully first.

JuiceFS takes a different structural approach entirely, splitting data storage (which can be S3, Azure Blob, GCS, Ceph, MinIO, Swift, take your pick) from metadata storage, which lives in something like Redis, MySQL, TiKV, etcd, or PostgreSQL. That split is the whole trick. Because metadata lives in a system built for fast, consistent updates, a confirmed change becomes consistently visible across servers sharing that filesystem, and the architectural separation gives it a meaningful performance advantage over simpler object-store filesystems. The object store just holds bytes. The metadata engine does the consistency work, which sidesteps the write-through tax entirely on the data path.

AkaveFS doesn't claim full POSIX compliance, because object storage's underlying model is different enough from a local block filesystem that pretending otherwise creates more problems than it solves.

Line all of these up and a pattern falls out: every team building one of these systems is really just deciding where metadata lives. Bury it in the object store, and you get weak consistency but a simple system. Split it into a dedicated metadata engine, and you get strong consistency but more moving parts. Skip the question entirely, and you get weak consistency with none of the complexity, and none of the safety either.

SquirrelFS and the idea of compile-time crash-consistency enforcement

SquirrelFS, from LeBlanc, Taylor, Bornholt, and Chidambaram (OSDI 2024, later extended in ACM Transactions on Storage in 2025), targets persistent memory, not FUSE-over-object-storage. But the idea that generates it travels well beyond its original target.

The mechanism is Synchronous Soft Updates, the same ordering-based approach mentioned earlier, but SquirrelFS enforces it using Rust's typestate pattern. Instead of treating an inode's durability and initialization status as just values that could be right or wrong at runtime, it bakes that status into the inode's type. In typical filesystem implementations, durability status is tracked as a runtime value rather than a type distinction, so the compiler has no way to catch a mistake. SquirrelFS closes that gap, turning a crash-consistency violation into a compiler error instead of a runtime bug. Code that violates the ordering rules simply doesn't build.

That has a practical payoff. Compilation is fast, and a clean compile is itself a strong signal that crash consistency holds, which compares favorably to the cost of runtime crash-consistency testing. SquirrelFS also closes an old, genuinely nasty bug class in classic soft updates, where a crash mid-rename could leave both the source and destination file existing simultaneously. It enforces that writes reach persistent memory in the correct order, which is central to the correctness argument. On benchmarks, it holds its own against NOVA and WineFS.

None of that requires persistent memory to be useful as a principle. The transferable idea for FUSE implementers is that crash safety reduces to ordering enforcement, and if a system's code structure can make that ordering explicit, the compiler (or some equivalent static check) can catch violations before they ever touch disk.

DFUSE, moving write-back caching and strong consistency from opposing choices to coexisting properties

DFUSE, a joint effort from Columbia University and Alibaba Cloud researchers (Haoyu Li, Jingkai Fu, Qing Li, Windsor Hsu, Asaf Cidon), presented at SoCC '25 in November 2025, makes a claim worth taking seriously: it's the first distributed FUSE filesystem to deliver write-back kernel caching and strong consistency at the same time, without trading one for the other.

The deadlock described earlier comes from blind local cache updates: one node updates its cache without knowing what another node just did, and reconciling that after the fact invites exactly the kind of coordination headaches that push teams toward write-through in the first place. DFUSE's fix isn't a clever workaround bolted onto the existing architecture. It relocates lease management logic out of the user-space daemon and into the FUSE kernel driver itself, so coordinated access to the page cache across nodes gets handled at the driver level, before any blind update can happen.

The result: strong consistency cluster-wide, without forcing every write through the user-space daemon and across the network synchronously. In benchmarks, DFUSE posted up to 68.0% higher throughput and 40.4% lower latency compared to the existing write-through approach used by distributed FUSE filesystems.

The takeaway here shapes how the specific numbers should be read. The write-back-versus-write-through split was never a law of physics. It was an artifact of where the consistency logic happened to sit in the stack. Move the logic to a different layer, and the supposed tradeoff stops being a tradeoff at all. That's directly relevant for AI training infrastructure, where checkpoint consistency and write throughput both matter and nobody wants to be told to pick one.

How to verify that a FUSE implementation honors the consistency model it claims

Picking a model on paper is the easy part. Implementations drift from their stated guarantees constantly, sometimes because of a genuine bug, sometimes because the spec was fuzzy to begin with, and the only way to know for sure is to test for it directly.

Ferrite, the same framework mentioned earlier out of ASPLOS 2016, does two things: it gives researchers a formal way to build crash-consistency models, and it provides a toolkit to check real filesystems against those models. It's the same project that caught ext4 behaving in ways its documented model didn't predict, which should tell you something about how much you can trust intuition here, even for a filesystem this widely used and thoroughly studied. The method is litmus tests: specific operation sequences, run mechanically, with clearly defined outcomes marked allowed or forbidden.

More recent work builds on that foundation. SnapCC (Liu, Shen, Xu, Sun, Jiang, published in ACM Transactions on Software Engineering and Methodology in October 2025) tackles the sheer combinatorics of testing crash consistency, since a single write sequence can be interrupted at dozens of different points, and checking every one by hand doesn't scale. Silhouette (Jiao, Goel, Wang, USENIX FAST, February 2025) goes after a scarier failure mode: silent corruption, where data survives a crash looking completely valid but is quietly wrong underneath. And work on crash consistency in block-level caching systems (Duan and Chen, USENIX ATC, July 2025) speaks directly to any FUSE setup using a local block cache as a staging layer before writes go further upstream.

None of this is optional homework. A filesystem's documentation describing its consistency model is a claim, not a guarantee, and the tools above exist specifically because claims and reality have a habit of drifting apart the moment nobody's checking.