POSIX Compliance Gaps in S3-Compatible Object Storage

S3-compatible object storage and POSIX file systems solve different problems on purpose: S3's design choices are deliberate trade-offs that make it excellent at scale but structurally incompatible with what POSIX-dependent software assumes at the process level. Engineers have spent over a decade trying to paper over that gap, and the pattern of their failures tells you exactly where the fix actually has to live.

Why S3 and POSIX represent two genuinely different contracts

POSIX is a contract between a process and a file system. It promises atomic operations, writes that become visible the instant they happen, the ability to read or write any byte range on demand, and a shared, coherent view of state across every process touching that file. A database engine, a build tool, a training script, all of them lean on that contract without ever having to think about it. They assume one coherent storage entity sitting underneath them, behaving the same way every time.

S3 is a flat-namespace, HTTP-based object store, designed for horizontal scale, stateless access, and consistency that started eventual and became strong, never built to make that promise. Those are the right trade-offs for storing exabytes of data across a fleet of machines that fail constantly and get replaced without anyone noticing. They are the wrong trade-offs for a process that wants to lock a file, seek to a specific offset, and overwrite a few bytes in place.

The resulting gap isn't a missing checkbox on a feature roadmap. S3 was built for massive distributed durability; POSIX was built for process-level coordination guarantees, those two goals pull in opposite directions at the architecture level, and you can't bolt one onto the other without giving something up.

Atomic rename: the load-bearing primitive

Of all the POSIX primitives S3 lacks, atomic rename carries the most weight, because it's the mechanism nearly every piece of crash-safe software quietly depends on. The pattern is almost universal: write your new data to a temporary file, then call rename to swap it into place. On a real file system, rename is a single metadata update. It either happens completely or it doesn't happen at all, so even if the process dies mid-write, the original file is never corrupted. It's either the old version or the new one, never something in between.

S3 can't do that. A rename on standard S3 is a copy followed by a delete, two separate operations with a gap between them. If anything goes wrong in that gap (a crash, a dropped connection, a retry that fires twice), the store can land in a state that matches neither the old file nor the new one. That's not a theoretical edge case. A common framework's standard atomic-write pattern, writing to a temporary file and then renaming it into place, is the idiomatic way Python code commits data safely, and it silently broke the moment it ran against S3. Every framework that relied on rename-as-swap had to either get rebuilt for S3's semantics or run on top of a POSIX layer that, for years, simply didn't exist.

Checkpoint recovery in model training leans on this same primitive. Without atomic rename, a partially written checkpoint looks identical to a complete one from the outside, which makes rollback a gamble instead of a guarantee. That's the quiet failure mode: a training job doesn't crash loudly when this happens, it just hands you a checkpoint that may or may not be real.

There's real movement here. Amazon S3 Express One Zone shipped a RenameObject API in June 2025, a single API call that renames an object inside the same directory bucket with no data movement at all. Renaming a 1-terabyte log file now takes milliseconds instead of the hours a copy-and-delete approach would need. That's a genuine capability shift rather than a workaround bolted on top. But it's scoped specifically to S3 Express One Zone directory buckets. It doesn't extend to general-purpose S3 buckets, so most of the ecosystem is still writing software against the gap this section describes.

File locking: how the absence of flock and fcntl turns concurrent writes into race conditions

Rename solves one coordination problem: making sure a single writer's update lands cleanly. File locking solves a different one: making sure two writers never step on each other in the first place. POSIX gives processes flock() and fcntl() for exactly this. One process grabs an exclusive lock, and every other process either waits or gets told immediately that the file is busy. Databases, logging daemons, and distributed systems all lean on this to keep concurrent writers from clobbering each other's work.

S3 has nothing like it. There's no lock primitive anywhere in the API. Two concurrent PUT requests to the same key are a straightforward last-writer-wins race, and neither writer gets so much as a notification that the other one existed. Whoever hits save last simply erases the other person's work, silently.

Database engines that coordinate through lock files, build systems that track lock state, and ML frameworks checkpointing from multiple workers at once all assume this kind of exclusivity exists. Pointing any of them at S3 without redesigning their coordination logic means they lose that guarantee without any error message telling them so. That's what makes this failure mode genuinely dangerous: it doesn't look like a failure. The write succeeds. The corruption just sits there, invisible, until something tries to read the data back during recovery or validation and the numbers don't add up.

It's tempting to think S3's consistency improvements fixed this, but they didn't. AWS S3 reached strong read-after-write consistency in December 2020, meaning that the instant you write an object, the next read reflects it. But consistency describes visibility: whether readers see the latest write. Locking describes exclusivity: whether a writer can collide with another in the first place. Those are different problems, and solving the first one does nothing for the second. No amount of S3-compatible storage tuning manufactures a coordination service out of thin air. That requires something sitting in front of the object store, acting as a referee.

In-place writes and write amplification: why every file modification becomes a full-object rewrite

A PUT doesn't patch an object. It replaces the whole thing. There's no seek, no append-in-place, no way to touch just the bytes that changed.

A POSIX file system doesn't work that way at all. A process can open a file, seek to a specific offset, write a handful of bytes, and close it, touching only the bytes it meant to touch. On S3, that same small edit means reading the entire object into memory, changing the relevant bytes, and writing the whole thing back out. A 4 KB random write to a 100 MB object means moving all 100 MB, not 4 KB, and the bigger the object gets, the worse that math gets.

This is why the 2010s FUSE clients, including Goofys, broke under production load: the write amplification and lack of atomic in-place modification made them unsuitable for any workload that modifies existing data, as when it kept no on-disk data cache and had no support for random writes, so any edit to an existing file meant recreating the entire object from scratch. A 4 KB random write turned into a full-object PUT many times larger.

Write amplification and lack of atomic in-place modification made 2010s FUSE clients unsuitable for any workload that modifies existing data. It wasn't sloppy code. Write amplification combined with the total absence of atomic in-place modification made these tools structurally unfit for any workload that modifies data that already exists. Log files that append new lines, index structures that update in place, model weights adjusted mid-training, none of these can be pointed at raw S3 without a redesign somewhere in the stack.

Latency: why the HTTP round-trip makes S3 structurally slower for small, frequent operations

The fourth gap is the one no API update can patch, because it's baked into the transport itself. Every single S3 operation is an HTTP request. That means a TCP connection (or a reused keepalive), a TLS handshake, authentication headers getting processed, and replication across multiple availability zones, all before the operation is even acknowledged. A single operation costs somewhere between tens and hundreds of milliseconds.

Compare that to a POSIX file system running on local disk or NFS, which handles small reads and writes in the microsecond-to-low-millisecond range. That gap isn't closed by tuning timeouts or picking a faster region; it's built into the fact that one model talks over HTTP and the other doesn't have to.

This matters in a specific, narrow way: it hits per-operation overhead on small, frequent requests, not aggregate throughput on large sequential reads. S3 is genuinely good at the second thing, streaming a large file sequentially plays to its strengths. It's the first thing, many small requests fired rapidly, where the HTTP floor turns into a wall.

Training pipelines feel this directly, because streaming training data is bound by how fast the storage layer can hand over bytes on the read side. When storage can't keep up, GPUs sit idle, burning electricity and doing nothing. Introl Blog's analysis found that Meta discovered 56% of GPU cycles were stalled, simply waiting for training data to arrive. Loading data straight from object storage into GPU memory runs into the IOPS ceiling that typical cloud object storage imposes. More than half the compute sitting unused because the data can't arrive fast enough.

Retrieval-augmented generation systems hit the same wall from a different angle. A retrieval step that pauses for a few hundred milliseconds per query breaks the conversational feel a user expects, and that tail latency appears in high-concurrency small-object reads, where S3's HTTP overhead piles up.

Compounding gaps in a real AI training pipeline

None of these four gaps occurs alone in a production training job. They stack on top of each other, hitting a pipeline from both directions at once. Slow reads starve the GPU during the forward pass. Broken atomic writes threaten to corrupt or lose checkpoints during the backward pass. One storage gap, two separate ways to lose a training run.

Checkpoint writes make this painfully concrete. A large checkpoint written synchronously to S3-compatible storage means the GPU sits fully idle for the entire duration of that write. There's no atomic, point-in-time capture available, so many Kubernetes-based ML pipelines fake versioning through naming conventions inside a bucket, with no real rollback mechanism behind it. A checkpoint named "v47" is only a label.

IBM's approach to this problem is instructive because it treats the compounding, not the individual symptoms. It inserts a high-performance file system between the object store and the GPUs, acting as a caching layer. Data loads into GPUs far faster when starting or restarting a training job, and model weights get checkpointed to that file system at a much faster rate, with the checkpoint data pushed asynchronously back to object storage in a way that never blocks the training job's progress.

This pattern resolves all four gaps simultaneously, not one at a time. Rename and locking happen at the cache tier, where POSIX semantics actually exist. Write amplification gets absorbed locally instead of traveling all the way to the object store. Latency gets served off NVMe hardware instead of crawling through the HTTP path. One architectural move, four problems addressed, because all four problems came from the same root mismatch to begin with.

Why FUSE clients failed

They were a reasonable attempt at an impossible job: synthesizing POSIX semantics on top of a storage service that structurally cannot provide them at the client layer. You can't negotiate a lock with a server that has no concept of a lock. You can't make a rename atomic by wrapping it in cleverer client code when the underlying operation is still a copy and a delete underneath.

S3FS-FUSE, Goofys, AWS Mountpoint, and rclone mount each documented their own version of this gap, often restricting writes to sequential-only or offering limited to no support for editing existing files. The old blanket claim that none of these tools support atomic renames doesn't hold anymore: AWS Mountpoint now supports atomic renames for S3 Express One Zone objects, as of June 2025. But that's a server-side capability flowing through the client, not something the client invented on its own. These limitations were never documentation footnotes. They were hard architectural limits of trying to solve a server-side problem from the client side.

Three flaws occurred across every one of these 2010s clients: write amplification from object immutability, no real locking, and no atomic rename. Three symptoms, one root cause. The lesson that falls out of a decade of this is straightforward: you cannot fix POSIX-on-S3 purely at the client layer. Either the storage service itself changes, or a caching tier with genuine POSIX semantics has to sit between the application and the object store.

HPC workloads feel this lesson hardest. Applications built around POSIX assumptions can't be pointed at S3-compatible storage without a rewrite to use S3's own APIs directly, and that rewrite adds cost and risk that many teams simply won't take on for existing software. That's a large part of why object stores in HPC contexts mostly appear in new applications built for them from the start, rather than old ones retrofitted onto them.

Where the gap is closing

Progress here is landing unevenly: the POSIX gap is closing on the server side through a combination of storage-layer changes and caching intermediaries, with some gaps resolved and others structurally persistent for general S3-compatible storage.

Rename is the clearest case of direct, server-side progress. S3 Express One Zone's RenameObject API, shipped June 2025, applies inside that specific product's directory buckets. Amazon S3 Files, slated for April 7, 2026, is set to extend similar rename functionality beyond directory buckets. Locking has no equivalent server-side fix anywhere in the picture: S3's consistency improvements describe visibility, not exclusivity, and nothing on the roadmap turns an object store into a coordination service by itself. That gap gets closed, if it closes at all, by a POSIX-compliant caching layer sitting in front of the object store, the same pattern IBM's architecture demonstrates.

In-place writes and latency follow the same pattern as locking. Object immutability is foundational to why S3-compatible storage is cheap and durable at scale, so expecting it to disappear misreads why the trade-off was made in the first place. The fix lives in caching: absorb the small writes locally, serve hot reads off NVMe, and push the final state back to the object store asynchronously once it's ready.

The practical takeaway for anyone choosing between these two models is to stop asking whether S3-compatible storage will eventually become POSIX-compliant. It won't, by design. A POSIX-compliant layer needs to sit somewhere in the stack so that software that assumes atomic rename, locking, in-place writes, and low latency gets to keep believing those assumptions are true, while the data underneath still gets the scale and durability that made object storage worth choosing to begin with. Archil, a cloud file system for AI agents that mounts existing S3, GCS, R2, and Azure Blob buckets as a POSIX volume without migration, is one concrete implementation of that pattern.