Unacknowledged Fsync on Cloud Filesystems Leading to Silent Data Loss

fsync is the call an application makes to tell the operating system: stop, wait, don't move on until everything just written is actually sitting on durable storage. It's the hinge that turns "the write probably happened" into "the write happened, guaranteed," which is the whole reason databases, log systems, and anything touching money or model weights depend on it. On a local disk, this guarantee rarely broke quietly. The path was short (application, kernel page cache, filesystem, block layer, disk) and when fsync failed, it was usually because the disk itself was dying, a loud and permanent kind of failure that stopped work and got noticed.

Cloud storage stretches that short path into a long one. Add a FUSE adapter, an NVMe cache, an async replication queue, or an HTTP call out to an object store, and each is a place where something can go wrong quietly, get logged internally, and still get reported back to the application as success. A dying disk screams. A swallowed fsync error whispers, and most monitoring systems aren't built to hear a whisper, so the same function call that used to mean "this is now physically true" can, across a modern cloud stack, mean nothing more than "something, somewhere, said okay."

The Linux kernel's page-cache behavior and silent fsync data loss

The kernel itself has a habit that makes one fsync failure contagious. When fsync() fails, Linux marks the dirty pages involved as clean anyway, even though the data behind them never made it to disk. Call fsync() again after that: the kernel looks around, sees no dirty pages left to flush, and reports success immediately, as if the first failure never happened.

This is exactly the chain that produced what the PostgreSQL community came to call "fsyncgate" in 2018. PostgreSQL retried fsync() after a failure, got back a false success, advanced its write-ahead log checkpoint on the strength of that success, and lost the data that never actually reached disk, with no error ever surfacing to say so. Craig Ringer's original report spelled out the exact problem with the semantics: a successful fsync is supposed to mean "everything since the last fsync is now on disk."

The kernel didn't arrive at this behavior by accident or carelessness. Ted Ts'o, the ext4 maintainer, pointed out that holding failed dirty pages as dirty forever risks taking down the whole system through memory exhaustion. Neil Brown added that doing so would break a core assumption memory management depends on, that dirty pages can always be reclaimed through writeback. It's a defensible tradeoff for keeping a machine alive. It's also a tradeoff that, by design, erases the evidence of the failure that caused it.

There's a second gap in this picture that trips people up even when they've fixed the first one. Durably renaming a file requires fsyncing the directory that contains it, not just the file, because the rename itself is a metadata change sitting in a buffer somewhere. Skipping that directory fsync means a rename can vanish on crash even when the file's own contents were flushed perfectly.

Why replication does not substitute for fsync

The practical implication: "we replicate across three nodes, so who cares about fsync." That argument doesn't hold, and the reason is structural rather than a matter of degree. Replication protocols are built on an assumption called fail-stop: a node is allowed to crash, but when it comes back, it has to come back with the same state it had the instant it crashed. Skipping fsync means a node that crashes can come back with less state than it had, a corrupted memory of its own history. That takes the node out of the fail-stop category the protocol was designed around, so the correctness proof underneath the replication protocol simply stops applying.

What that looks like in practice: a follower node loses power without having fsynced its latest writes, then restarts holding a log that's stuck at an earlier point in time. The leader, meanwhile, believes that follower acknowledged writes it never actually made durable, and keeps advancing the commit point on the strength of an acknowledgment that wasn't real. When the follower comes back, it can't reproduce the state the leader thinks it has.

Spreading replicas across different data centers or availability zones doesn't change this story. If none of the replicas fsynced before acknowledging a write, all of them are carrying the same unflushed state, just in different buildings. Geographic distance protects against a fire in one data center. It does nothing against a crash that happens before the data was ever durable.

There are protocols built to survive worse failures than a crashed node, the Byzantine fault-tolerant family, but they come at a steep structural cost: tolerating n faulty nodes requires 3n+1 nodes total, plus a level of operational maturity and performance overhead that rules them out for most production systems. Redpanda's analysis goes further and proves the point mathematically: at n=1, no protocol built on the smaller 2n+1 model can cover this class of fault at all. Replication is excellent insurance against machines dying. It is not insurance against machines lying about what they've written, and skipping fsync is how a system ends up with liars in its replica set.

New places cloud filesystem architectures introduce for fsync errors to be swallowed

Every layer a cloud storage stack adds between an application and an object store is another spot where an fsync call can go in, bounce around, and come back marked "success" without anything durable happening underneath.

FUSE-mounted object storage, tools like s3fs or rclone mounts, makes this most visible. Object stores are built for HTTP calls and horizontal scale, not for the kind of transactional write-and-confirm pattern a database expects. FUSE adapters sitting on top of them frequently have no real metadata locking and no coherent idea of what fsync should mean once multiple writers are involved. The syscall reaches the adapter, and what comes back reflects the adapter's own cache state, not whether the object store actually has the bytes.

High-performance computing filesystems built around burst buffers run into a related wall, just from a different direction. Burst buffer filesystems like UnifyFS lack crash recovery, so an fsync that returns success may only mean the data reached the burst buffer layer, not that it reached anything stable.

Cloud block storage is the honest exception here, and it deserves to be named as one. AWS EBS, for instance, only acknowledges a write after that write has been durably recorded to non-volatile storage, which is part of why a Lobsters discussion of SurrealDB's durability defaults noted that explicit fsync becomes redundant on that specific stack. That guarantee is narrow, though. It belongs to EBS and stacks like it, and it does not carry over to FUSE mounts, NFS, or any layer that adds its own buffering on top of the block device. Knowing that EBS behaves well doesn't tell you anything about how the FUSE mount three layers up behaves. "Cloud storage" Cloud storage" is a pile of different combinations of backend and access path, and the guarantee, where it exists, belongs to one specific combination, not to the category.

Three concrete cases from production code where fsync errors are discarded

None of this is hypothetical. Four recent cases from production systems show the exact same structural mistake, wearing different clothes.

OmniKV's group-commit code, flagged in Issue #139 filed September 25, 2026, called heap.sync_data() and wal.sync() inside its commit path and then discarded the result with the let _ = pattern. Having thrown away the only signal that would have told it otherwise, the code unconditionally called guard.mark_synced(), which flipped a sync_in_progress flag to false and notified every follower in the commit group that the write was safe. In the specific incident documented, a poisoned lock on the heap file meant no fsync occurred at all, and mark_synced() ran anyway, telling every follower a write was durable that had never touched disk. The issue carries a p0 label: blocking before production claims.

SurrealDB's 2025 benchmark results told a different version of the same story, this time through a default setting rather than a discarded return value. Per-transaction durability was gated behind an environment variable, SURREAL_SYNC_DATA, which defaulted to false with nothing in the documentation flagging it. The published benchmarks ran with fsync disabled across every engine being compared, not singled out for SurrealDB, which meant the numbers described writes that sat comfortably in the OS page cache and never once touched a disk. Corrected benchmarks published in 2026, with fsync properly enabled for every engine, produced materially different throughput figures, which tells you roughly how much of the original performance story was really a story about skipping durability rather than being fast.

A related but separate bug appears in a benchmarking tool connected to the same project, surrealdb/crud-bench, fixed in Pull Request #306. The tool's --optimised flag unconditionally applied MAP_ASYNC to LMDB regardless of whether --sync had also been requested. MAP_ASYNC defers flushing to the OS's own page-cache writeback schedule, which quietly defeats whatever the --sync flag was supposed to guarantee.

The fourth case sits lower in the stack, inside OpenZFS for Windows. Issues #637 and #638, both filed September 4, 2026, found that the Windows file layer has no working conversion from Unix error codes to Windows's native status codes. A switch statement that looks like it should do the conversion exists, but it's dead code, wired to the wrong variable. Unix error codes are small positive integers, and Windows's NT_SUCCESS() check happens to return true for all small positive integers. Write failures, including running out of disk space and genuine I/O errors, come back to the caller as STATUS_SUCCESS.

Every single one of them hides the error in a way that's syntactically invisible, a let _ = here, a dead code path there, a default environment variable nobody read the docs for. Nobody sat down and decided to lie to the caller. The lie is an emergent property of a small omission, which is what makes this class of bug so hard to catch in code review and so easy to catch in production.

Why AI training checkpointing concentrates this failure class's damage

AI training checkpointing is where every pressure behind this failure mode piles up at once: enormous write volumes, time windows too tight for synchronous flushing to be practical, and a failure that can stay invisible for hours of wasted GPU compute before anyone notices.

MLCommons's analysis puts real numbers on the scale. A 1-trillion-parameter model, checkpointing every few minutes on a large accelerator cluster, generates more than 14 petabytes of checkpoint writes per day. Keeping that fully synchronous means sustaining multi-terabyte-per-second throughput across the whole cluster, continuously. That throughput requirement creates a direct incentive to cut corners: checkpoint overlap, writing the next checkpoint while training keeps running, breaks down the moment storage throughput falls below checkpoint size divided by the time window available. When that happens, teams face a choice between letting expensive GPUs sit idle or deferring the flush, and deferring the flush is precisely the move that produces silent loss. Most deep learning frameworks write checkpoints with plain file operations that carry no durability guarantee at all, so a checkpoint can load without a single error and still contain the wrong weights, if the system crashed mid-write and the storage layer reported success for data that was never flushed.

Research into integrity guards for AI checkpointing shows the tradeoff isn't a rounding error. An integrity guard can catch nearly all corruption with no false positives, but it comes with a real performance cost over an unsafe baseline, and a full atomic-dirsync mode costs substantially more on top of that. Cloud object storage like S3 or GCS brings geo-redundancy and affordable capacity for storing checkpoints, but latency can stretch a multi-gigabyte checkpoint write out to minutes, adding yet more pressure to treat the whole operation as fire-and-forget. None of this is a story about sloppy teams cutting corners. It's a story about a genuine structural conflict between correctness and throughput at a scale where both matter enormously and neither one gives way easily.

Correct fsync handling across the stack

Durability has to be earned at every layer of the stack, not asserted at the top and assumed to hold everywhere below. A storage system that swallows an error anywhere in its chain, application, kernel, filesystem, network, object store, cannot honestly claim to guarantee durability, no matter how clean its API looks from the outside.

At the application layer, that starts with treating the return value of fsync as real information rather than a formality to step past on the way to the next line of code. The OmniKV bug, that let _ = heap.sync_data() pattern, is this failure mode reduced to its simplest possible form: an error exists, gets generated correctly, and then gets thrown away by one keystroke's worth of code. Checking the return value costs nothing and catches exactly this class of bug before it reaches production.

Further down the stack, the same discipline has to hold at the kernel and filesystem level, where retries need to be checked against actual on-disk state rather than trusted blindly, and at the network and object-store level, where an added layer needs to mean what a caller assumes it means. None of these layers get a pass just because they're further from the application and harder to inspect. The chain is only as strong as its weakest swallowed error, and right now, that weak link tends to be wherever an engineer assumed somebody else was already checking.