Unacknowledged Fsync on Cloud Filesystems Leading to Silent Data Loss

The PostgreSQL incident: how Linux itself can return fsync() success after silently discarding data

In 2018, a documented Linux bug made database engineers everywhere spill their coffee. A dirty page, meaning data sitting in memory waiting to get written to disk, gets evicted from the OS page cache. A write to that page then fails. The failure never makes it back to the application. Postgres calls fsync(), expecting to hear about any trouble, and the kernel says everything's fine, because from its point of view there's nothing left to sync. The page got marked clean on eviction, the error vanished with it, and so did the data.

Once a page gets marked clean, a failed write against it doesn't reliably get reported to whatever called fsync() on it. The application ends up holding a receipt for something that got thrown out without telling anyone.

PostgreSQL spent months chasing this down, and the conclusion should worry anyone who assumed fsync() means "durable." It doesn't, not reliably, not on Linux, not always. The project had built years of durability logic on a belief about fsync() that turned out flat wrong. PostgreSQL 11 shipped a fix in response, backpatched to earlier supported versions, with a further refinement landing in PostgreSQL 12.

That wasn't a one-off scare either. A peer-reviewed study in ACM Transactions on Storage looked at how three widely used Linux filesystems (ext4, XFS, Btrfs) and five widely used applications (PostgreSQL, LMDB, LevelDB, SQLite, Redis) handle fsync failures. Every single application tested picked a different strategy for handling those failures, and every single one of them lost or corrupted data anyway. Pick your poison; none of them worked.

This problem predates cloud computing by nearly a decade, too. Back in 2009, Linux.com documented an ext4 quirk where a crash at the wrong moment during a rename could leave a zero-byte file sitting at the new path. Not the old contents, not the new contents. Nothing. An empty placeholder where the data used to live.

If a local kernel, running on one machine with no network in the loop, can eat an error and still report success, a cloud filesystem doesn't stand a better chance. Adding buffering layers, network hops, and an object store dressed up in a filesystem costume does not shrink the problem. It multiplies rather than shrinking.

Why replication does not substitute for fsync()

Replication gets treated like a backup plan for fsync() constantly. It isn't one, and pretending otherwise is where a lot of "highly available" systems quietly lose data. Redpanda made this argument in 2023, and it holds no matter which replication protocol runs underneath. A single node loses power, loses data that never made it to disk, and that local failure can turn into silent data loss across the entire replicated cluster, not just on the one machine that dropped.

The mechanism is almost embarrassingly simple once someone points it out. In non-Byzantine replication protocols, the kind most distributed systems actually run, a node can acknowledge a write, lose power before that write hits disk, restart, and hand the rest of the cluster stale state as though nothing happened. The cluster has no reason to distrust its own member, so it accepts that stale state as fact. The acknowledged write disappears from every replica, because the "authoritative" version of events just got rewritten by a node that skipped its homework.

Picture a group project where one teammate promises to save the file before logging off, but never actually clicks save. It doesn't matter how many other teammates have their own copies open, because if the group later treats that unsaved version as the source of truth, everyone's copy gets overwritten with the broken one. Replication spreads risk around. It doesn't remove it. Multi-AZ setups, multi-region setups, even strong consistency bolted on at the application layer: none of it closes the local-durability gap if one node in the quorum skips fsync before saying "got it."

New fsync failure surfaces in cloud and object-storage filesystems

Object storage dressed up as a filesystem is where this problem changes costumes and comes back scarier. S3-backed FUSE mounts are object stores wearing a filesystem's clothes, and that costume creates failure modes ordinary local filesystems never produce.

Take rename(). On a real filesystem, renaming a file is atomic: an instant, all-or-nothing swap of a name. S3 has no native rename operation, so FUSE layers over S3 typically fake it with a copy, followed by a delete. Crash in the middle of that sequence, and the result is a failure mode no local filesystem would ever generate on its own.

fsync() fares no better. In plenty of these implementations it's either a complete no-op, or it flushes only to a local FUSE-layer cache and never reaches the object store. The s3fs-fuse project has a documented, open issue on exactly this: users assumed their data got uploaded to S3 before fsync returned. In release 1.86, the implementation actually called an asynchronous flush (Flush(false)) instead of the synchronous version, so nothing guaranteed the data had reached S3. Users thought they'd locked the front door. They'd left it propped open.

A 2026 review of filesystem-on-S3 projects found this quiet-drop pattern repeating everywhere. Most of these systems cheat somewhere: a separate database has to track metadata, directories full of small files choke the system, or POSIX guarantees like hardlinks and fsync get dropped without much fanfare. Goofys, a project still in common use, documents fsync as explicitly ignored. Not a bug. A design decision, written right into the docs.

Even the more capable offerings carry the scar. Amazon S3 Files, launched in April 2026 and positioned as a major step forward for POSIX-on-S3, delivers a genuinely impressive list: POSIX permissions, atomic rename (via EFS), symbolic links, file-level locking, directory operations. But it explicitly sacrifices real-time fsync durability (a 60-second window applies) and drops hard links. The best-funded, most capable product in the category still ships with a documented fsync gap, and that says less about sloppy engineering than about how genuinely hard this problem is to solve.

The pattern also appears in systems outside single vendors. A 2026 GitHub issue on an edge gateway project documented a degraded-volume scenario: the storage volume kept accepting file writes and renames, but stopped syncing directory metadata. Every operation after that point returned errors indefinitely, while data kept queuing up behind the scenes, invisible until someone finally went looking for it.

The thread running through every one of these examples is the same. The filesystem hands back a normal-looking success code while the durability guarantee behind that code hasn't actually been met. That silence is a structural feature, baked into how these systems get built from the ground up. It's structural, baked into how these systems get built from the ground up.

The WAL-sealing bug as a clinical specimen of silent fsync omission

For a real-world specimen of this exact failure, look at GitHub issue #335 in the cntryl/midge repository, opened September 19, 2026, found during a full-system audit at commit 4273d661 and flagged priority:p1, area:wal. The issue illustrates the mechanics of silent fsync omission in concrete detail.

The setup: a write-ahead log (WAL) sealing routine called CloudAsync seal only calls writer.flush(), inside a function named, almost too on the nose, flush_for_cloud_upload_within. The doc comment right above it says the function should "do a flush+fsync only when sealing." The fsync call never actually happens. The rotate step afterward renames wal.log to something like N.wal and fsyncs the directory, not the file itself. Meanwhile, a function called complete_cloud_upload_seal goes ahead and sets state.wal.local_durable_seq = max_sequence, regardless of whether any of that data actually reached disk.

Walk through what that means. The CloudAsync engine accumulates writes, then seals them by renaming wal.log to, say, 7.wal, a durable-looking directory entry, and queues that segment for upload. Power drops before the upload finishes. Segment 7.wal never got its data pages fsynced, so the file on disk is either empty or cut off mid-frame, like a phone call that dropped mid-sentence.

On restart, the system realizes the segment never made it into the catalog and needs the local copy back. It inspects the sealed WAL file and fails, either with "sealed WAL segment is empty" or an incomplete-frame error. A strict recovery mode returns RecoveryFailed. Credit where it's due: that part is the system working as designed. It noticed something was wrong.

Before any of that recovery check ran, local_durable_seq had already moved forward. Before any of that recovery check ran, local_durable_seq had already moved forward. Idempotency checks and local-durability waiters throughout the system had already treated those writes as durable. Anything downstream that trusted that sequence number was trusting a number that lied to it.

The project's own durability contract (docs/user-guides/transaction-durability-contract.md, lines 205 to 222) states that cloud_async doesn't treat the local append barrier as equivalent to fsync, and that the write may be lost if the local bytes don't survive. Losing the data is, technically, within the stated contract. The real defects sit somewhere narrower and nastier: strict recovery fails open instead of hard-failing after a power loss, where the contract implies data loss rather than a loud crash, and local_durable_seq feeds visibility and idempotency logic for bytes that never got fsynced.

The trigger condition is the cruelest part of the whole thing. Only a power loss or kernel crash sets it off. Only a power loss or kernel crash sets it off. A regular process crash never reproduces it, so the failure stays invisible to basically every routine test suite, every ordinary chaos-engineering drill, every unit test anyone's likely to write. The power actually going out is what triggers it, and by then it's too late to do anything but count the damage.

This one bug carries all three properties that make cloud fsync failures genuinely dangerous, together, at once. The omission sits documented right there in the contract. The success codes returned along the way are completely real; nothing's broken in the flush logic itself. And the data loss stays silent until a power failure turns it permanent.

AI training checkpoints as the highest-stakes context for this failure mode

Nowhere does this gap cost more than in AI training checkpoints, and the math is genuinely brutal. Training a large model relies on periodic checkpoints, snapshots of model weights and optimizer state, specifically so a GPU preemption or a network timeout doesn't erase a multi-day run along with the compute budget spent getting there.

Futureagi.com identifies two canonical ways checkpoints fail. First, non-atomic checkpoint writes, where a crash mid-write can leave a corrupt file. Second, weight-only saves that drop optimizer state entirely, so a "successful" resume restarts training from a file that looks structurally fine but is missing half of what it actually needs.

A study from arXiv:2511.18323 put numbers to the cost of doing this right. Researchers tested three write modes on macOS/APFS: unsafe (no fsync), atomic_nodirsync (file-level fsync only), and atomic_dirsync (fsync on both file and directory). Across 430 crash-injection trials and 1,600 corruption-injection trials, the integrity guard caught 99.8 to 100% of corruptions with zero false positives, a near-flawless result. That's the good news. The bad news: performance overhead ran 56.5 to 108.4% for file-level fsync, and 84.2 to 570.6% for full directory-sync, against the unsafe baseline.

Doing it right, in plain terms, costs somewhere between roughly two and six times the throughput of doing it recklessly. That gap tempts engineering teams to quietly skip fsync on checkpoint writes. The unsafe path runs so much faster it's hard to say no to, especially when a training run is already burning tens of thousands of dollars an hour in compute.

Hardware isn't helping either. Meta's fleet analysis found silent data corruption affecting roughly one in a thousand machines, and at the scale of large AI training runs, these silent corruption events occur every one to two weeks. Soft error rates have gotten measurably worse as chips have shrunk too: soft error rates have worsened measurably as transistors have shrunk. Smaller transistors leave more room for a stray cosmic ray to flip a bit.

The stakes per failure keep climbing because the files keep growing. Checkpoints for 70-billion-parameter models now run 150 to 200 gigabytes each, and a corrupt checkpoint means regenerating a multi-hundred-gigabyte object from the last known-good state, a rewind that might cost hours of compute.

One widely cited example, sourced from a vendor blog at introl.com (worth treating with some caution, since it isn't a primary disclosure from OpenAI itself), claims a GPT-4 training interruption cost 72 hours of progress to checkpoint corruption, with a cited price tag of $8.6 million in wasted compute and a two-week product delay. Take the number with a grain of salt given the source, but the shape of the story lines up with everything else here.

When the specifics are stripped away, the same gap appears again. Bytes get written, success gets reported, and later somebody discovers the bytes never survived. The only thing that changes is the currency it's counted in: GPU-hours and dollars, instead of database rows.

Object-storage performance constraints and the fsync trade-off

Global AI infrastructure spending crossed $250 billion in 2025, with storage and networking growing at a pace nearly matching compute itself. And yet MinIO's research found that more than half of organizations say data and storage bottlenecks are actively limiting how well their AI systems perform and scale. Money keeps pouring in at the top, and the plumbing keeps clogging anyway.

Part of the problem is just latency. According to JuiceFS, routine metadata operations against object stores (listing files, pulling attributes, and the like) typically take 10 to 30 milliseconds. That sounds tiny until it gets multiplied across every query happening during training or inference at real scale; at that point it is no longer a rounding error but a hard ceiling on throughput.

Meta put a dollar figure on what idle GPUs cost, in a presentation quantifying that roughly 20% GPU idle time translates into losses of tens of millions of dollars per hour, at the scale of a data center the size of Manhattan. At that scale, slow storage is not an abstraction but a line item on a budget spreadsheet.

The fsync trade-off isn't theoretical here, and it's genuinely hard to resolve. Synchronous fsync against an object-storage backend adds latency to every single write on the path. The overhead numbers from that checkpoint study, 56.5 to 570.6%, were measured on local APFS, on a machine's own disk. Pushing that same durability guarantee out to a cloud object store makes the numbers structurally worse, because an unavoidable network round-trip to the backend sits in the middle of every write.

The downstream consequence is visible in abandonment rates. Ideas2it reports that 42% of businesses abandoned AI projects in 2025 due to data quality issues, including undetected anomalies, up sharply from 17% the year before. That's the storage-reliability gap appearing on a different balance sheet: teams hit the failure long before anyone figures out what actually caused it.

Skip the false choice between "always fsync" and "never fsync." Neither rule survives contact with a real system. Pick a side instead of splitting the difference: know exactly which layer in the stack actually owns durability, and verify that it closes the gap rather than assuming it does. That takes understanding the whole stack, not trusting whatever number the API happens to hand back.

What a cloud filesystem must do to close the fsync gap

The minimum bar for a cloud filesystem that means what it says isn't complicated to describe, even if it's hard to build.

fsync() has to flush all the way to the real backend, such as an object store or something equivalent, and not merely to a local cache or a FUSE-layer buffer sitting in between. If the write hasn't reached durable storage, the call has no business returning success, no matter how inconvenient that is for latency numbers on a benchmark slide.

rename() has to be genuinely atomic at the storage layer. Copy-then-delete is two separate operations wearing a rename's name tag, and a crash between them produces exactly the kind of half-finished, neither-old-nor-new file that's shown up throughout this piece, from ext4 in 2009 to S3-backed FUSE mounts today.

Everything else, the replication strategies, the checkpoint formats, the recovery logic, gets built on top of those two guarantees actually holding. Skipping either one means no amount of redundancy elsewhere in the stack saves the data that quietly vanished at the bottom.