Fsync Behavior on NFS and Cloud-Mounted Filesystems

Fsync is supposed to be simple: write the data, don't return control to the app until it's actually on disk. That is the whole contract. On a local NVMe drive, that contract holds up fine. The moment a network shows up between your write call and the physical disk, that contract gets renegotiated, and most developers never read the fine print.

Here's what actually happens under the hood on local storage. The kernel flushes dirty pages and handles the necessary journal work, then signals the device to commit whatever's sitting in its write cache onto the actual persistent media. The call blocks until the device confirms. All that kernel bookkeeping, the page flushing and journal work, is fast. The slow part is waiting on the device to say "yes, it's really there." On a spinning disk, that wait is governed by platter rotation and buffer drain, and it's noticeably slower than on NVMe, where the commit latency is far shorter.

Why bother at all? Because fsync is the "D" in ACID: durability. Skip it, and a crash right after a "successful" write can quietly erase data your application already told a user was saved. Nobody wants to explain that one. On local storage, this is well understood and mostly boring. Once you mount storage over a network, or point your app at object storage instead of a disk, "boring" is the first thing to go.

How NFS versions renegotiate the write-to-disk promise

NFSv2 kept things honest, if slow. The server could only tell the client "write complete" after the data actually hit disk. Client calls fsync, local buffers flush to the server, server writes through, then replies. No shortcuts.

NFSv3 loosened that. It introduced unstable writes, meaning the server can buffer the data and acknowledge it before committing anything to disk. The actual durability now depends on a separate step: the COMMIT request, which the client sends at fsync time or on file close. When COMMIT arrives, the server is obligated to flush any previously unwritten data and metadata to disk before replying. Used correctly, this restores the same durability guarantee as NFSv2, just with better performance in the common case, since writes don't have to round-trip to disk one at a time.

NFSv4 folded COMMIT into its broader stateful design, but the underlying question, whether the server is allowed to buffer before it tells you the data is safe, never went away. It just moved into a different protocol wrapper.

All of this carries real weight. Which NFS version a mount is running determines whether your fsync call triggers a real COMMIT round-trip to disk, or just resolves quietly against a buffer that hasn't gone anywhere yet.

Mount options that silently override protocol-level durability

Mount options can undo all of the above without telling you.

The async mount option is the big one. With async, the server tells the client "it's on stable storage" when it's really just sitting in a buffer. COMMIT, in this mode, does basically nothing. The dangerous part comes next: the client, believing the data is safely held by the server, may reclaim its own cached pages. If the server crashes at that exact moment, the data is gone, or worse, silently corrupted, with no guarantee that an error will reach the application.

The sync mount option is the opposite extreme. Every write syscall flushes to the server before control returns to user space. That gets your durability guarantee back, but it comes at a real cost: no client-side write buffering at all, which tanks throughput for anything write-heavy.

Default mounts (neither sync nor async explicitly set) usually behave like async under the hood. Writes get deferred until memory pressure forces the issue, or until an explicit fsync, msync, sync, close, or lock/unlock happens. Code that was written assuming implicit flushing, the kind of assumption that's harmless on local disk, is exposed here in a way that can bite hard in production.

Then there is the O_DIRECT trap. On Linux, the NFS client treats O_DIRECT as O_DIRECT combined with O_SYNC, and it has no way to track outstanding O_DIRECT RPCs. Practically, that means fsync does nothing useful in that code path. Anyone porting an application built on local-disk O_DIRECT assumptions onto NFS is walking into a guarantee that quietly isn't there anymore.

One improvement worth knowing: since Linux 4.13, writeback errors get reported to every file descriptor that might have written the data that triggered the error. NFS actually tracks this more precisely, per descriptor, than most local filesystems manage.

Why object storage's architecture makes POSIX fsync semantics structurally awkward

Object storage answers a different question than the one fsync asks. POSIX assumes files you can open, seek into, overwrite a few bytes of, and close. Object storage assumes atomic, immutable objects delivered over a RESTful interface, PUT a whole object, GET a whole object, done. There's no concept of "flush just this byte range to disk," because there's no in-place write to begin with.

Writes to an object don't show up in the namespace until the whole thing commits. That means concurrent readers sharing a mount won't see a file mid-write, which sounds like a feature until an application expects to see partial progress and instead sees nothing. Worse: if a crash interrupts a PUT before it finishes, there's no journal to recover from. The data is simply absent. It's not "corrupted," it's absent, like it was never sent in the first place.

The mismatch runs deeper than any one bug. POSIX is IOPS-centric and chatty by design, lots of small operations, lots of round-trips. Object storage is built around throughput over HTTP, big transfers, few requests. Bolting fsync-like semantics onto that requires solving deeper structural problems than writing better code. It's asking a system optimized for bulk shipping to also handle same-day, one-item deliveries efficiently. What should "fsync" even mean here? Replication confirmed across some number of nodes? A transaction log commit somewhere upstream? The VFS layer hands down the fsync() call, but somebody has to invent a translation for it, because object storage never had a native answer.

How FUSE-based S3 mounts handle fsync in practice

FUSE mounts sit in userspace, which gives implementers more freedom than a kernel filesystem driver gets, but also adds overhead and more room to drift from what POSIX actually promises.

s3fs is a good example of the trade-off. Every file operation turns into an HTTP request. It covers a decent slice of POSIX semantics, but it's slow by design, because every operation is a network call dressed up as a filesystem call.

GeeseFS, built by Yandex Cloud, takes a different angle and advertises fast fsync, on the logic that once the upload completes, there is no separate in-place data to flush. Once the upload finishes, there's no separate flush step to wait on. That's a real advantage in the steady state. But new files, metadata changes, and renames still get pushed to S3 as multipart uploads, so fsync is cheaper on average, not free, and the latency floor is still whatever the object store's upload latency happens to be that day. GeeseFS also flatly doesn't support concurrent updates to the same file from multiple hosts. Shared writes aren't a degraded experience here, they're just unsupported.

The general rule practitioners land on: use a FUSE mount when performance needs are moderate and the app doesn't lean hard on POSIX guarantees. Avoid it for anything that both reads and writes actively, and never assume consistency across multiple clients touching the same files. For read-heavy or write-once workloads, FUSE is fine. For anything where fsync durability and shared writes both matter at once, the FUSE layer's guarantees are thinner than the application probably needs.

The fsync overhead problem and what hardware and filesystem research have tried to do about it

Fsync's blocking behavior is a feature for durability and a tax on throughput. Every ordered write has to complete its transfer, then its flush, before the next write can even be dispatched. That serialization gets more expensive, not less, as storage parallelism increases, which is a little counterintuitive: more channels should mean more speed, but ordered writes don't get to use most of them at once.

Research on the Barrier Enabled IO Stack (arXiv:1711.02258) puts numbers on this. On single-channel mobile storage, an ordered write (write plus fdatasync) hits about 20% of plain buffered-write throughput. Scale up to a 32-channel flash array, and that ratio collapses to 1%. More parallelism, more relative penalty, because the flush step can't take advantage of the same concurrency the writes could.

Even hardware protection doesn't fully bail this out. On a device with supercap-backed power-loss protection, meant to guarantee data survives a sudden power cut, ordered write still only reaches about 25% of buffered-write throughput. The transfer-and-flush overhead is a software-level cost, and power-loss protection doesn't touch it.

The proposed fix is a barrier command: a way to preserve write ordering without forcing each write's transfer and flush to fully interleave, so the next command can be dispatched before the prior one has completely finished. In that same research, this approach lifted SQLite performance by 270% on server hardware and 75% on a smartphone. Worth sitting with: if local NVMe fsync already carries this much overhead, tacking a network round-trip onto every COMMIT or object PUT doesn't add a little cost, it multiplies what's already expensive.

Checkpointing in AI training as the highest-stakes real-world instance of the fsync choice

Training a large model without checkpoints is like writing a novel with no save button and a laptop that randomly dies. Every so often, training jobs write out a checkpoint so a crash doesn't mean starting over from scratch. Get the durability wrong here, and the checkpoint itself can come back corrupted, silently, at the exact moment recovery depends on it.

A 2025 study (arXiv:2511.18323) laid out three write modes, ranked by how much durability they buy and what it costs:

• unsafe: no fsync at all, fastest option, zero protection if the write is interrupted
• atomic_nodirsync: file-level durability using fsync(), adding 56.5 to 108.4% overhead versus the unsafe baseline
• atomic_dirsync: durability for both the file and its directory entry, adding 84.2 to 570.6% overhead versus unsafe

That top-end number, 570.6%, isn't a typo. The wide range between the floor and ceiling of atomic_dirsync overhead reflects how differently checkpointing workloads can behave depending on their structure. The same study's integrity checker caught 99.8 to 100% of induced corruptions across 430 unsafe-mode crash tests and 1,600 atomic-mode corruption tests, which says the detection works. It doesn't make the overhead disappear.

The practical answer training teams reach for is asynchronous checkpointing. Once model parameters move off the GPU's VRAM into regular RAM, the GPU goes right back to training while the CPU handles the actual write to storage in the background. This is exactly what AsyncCheckpointIO (in Torch-Lightning) and AsyncCheckpointer (in orbax.checkpoint) are built to do.

NVIDIA's Cosmos 3 project shows this in production. Checkpoint writes get routed through a dedicated Gloo process group, kept separate from the NCCL communicator handling accelerator-to-accelerator traffic, so I/O doesn't compete with training communication. The async design hides the variable latency of writing to object storage instead of exposing it on every step. The async design is described in arXiv:2606.02800 as reducing end-to-end training time compared to synchronous checkpointing. Nobody eliminated the fsync-equivalent flush. They just moved it off the critical path so the GPUs stop waiting on it.

That gap matters more every year. Each new GPU generation brings roughly 2 to 3 times more compute throughput, but storage infrastructure doesn't improve at anywhere near that pace. Checkpoint I/O is one of the clearest places that mismatch shows up, since checkpoint I/O across a training cluster can create synchronization bottlenecks where storage stragglers delay the whole job.

What the NFS async and cloud-mount trade-offs mean for storage architecture decisions

This was a spectrum spanning sync versus async, and local versus cloud. It's a matrix: NFS version, times server write mode, times mount option, times whatever the application actually does when it calls fsync. Get any one variable wrong and the other three don't save you.

For training clusters where checkpoint latency sits directly on the critical path, parallel filesystems like Lustre or GPFS exist for exactly this reason, built for sustained, high-throughput AI workloads. Other distributed filesystem approaches put object storage on the back end with a filesystem interface on the front, pushing the fsync translation problem down into the filesystem layer instead of leaving it for the application to work around.

A cloud-mounted POSIX filesystem sitting in front of object storage, with an NVMe cache layer, can absorb a lot of this pain. Writes land on local NVMe before the fsync call returns, then get flushed to the object store asynchronously, reducing the chattiness that object storage handles poorly. The app sees a fast fsync. The object store sees traffic patterns it's actually good at.

Shared-write workloads are the one place none of this substitutes for the real thing. Atomic rename, flock and fcntl locking, genuine fsync durability, these aren't nice-to-haves, they're the whole contract multi-writer applications are built on. Any storage layer that can't deliver all of them will break those applications quietly, not loudly, which is worse.

For read-heavy or write-once workloads, model inference, dataset serving, most of this discussion is close to irrelevant. Read path performance and cache hit rate decide the outcome, not fsync latency. Fsync earns its keep specifically around checkpoint writes and log-structured workloads, the places where a crash actually threatens to lose something that mattered. The job, in the end, is tracing the full path every fsync call actually takes: syscall, filesystem driver, network protocol, server write mode, backing storage. The guarantee at the end of that chain is only as strong as its weakest link, and that link is rarely where anyone assumed it was.