POSIX Compliance Gaps in S3-Compatible Object Storage

S3 was never built to act like a file system, and most enterprise software still assumes it will anyway. That mismatch quietly burns engineering hours and cloud budget across the industry, and it shows up hardest in the workloads that write constantly: AI training, checkpointing, anything that expects a rename to just work. This piece walks through where the gap actually sits: which POSIX guarantees S3 skips, why those skips are baked into the architecture on purpose, and what they cost teams running training pipelines at scale.

POSIX (the interface standard published as IEEE 1003.1, last updated in 2024) is the contract most enterprise software was written against. Atomic rename, file locking, in-place writes, symbolic links: none of that is a nice-to-have. It's assumptions baked into decades of code. Teams pick S3-compatible storage anyway, not because they love how it behaves, but because the economics leave no real choice. The old assumptions sit there quietly until the day they don't, and then someone's on call at 2 AM wondering why a checkpoint write silently ate itself.

What S3 was actually designed to do, and where its model diverges from POSIX by construction

S3 is an object store. Amazon built it for massive scale and eleven nines of durability, reached over stateless HTTP calls, not the kind of syscalls a Linux kernel hands to a mounted file system. That distinction explains almost everything downstream. The gap between S3 and POSIX isn't a bug someone forgot to patch. It's the design doing exactly what it was told to do.

Take the "folder" in the console: logs/2025/march/app.log. Looks like a directory path. It isn't one. S3 has a flat namespace, and that string is just a prefix glued to one object key. No hierarchy sits underneath it, no directory inode, nothing to actually cd into.

Objects are also immutable. No partial writes, no updating a byte range in place. Change one byte and the whole object gets rewritten from scratch. Strong consistency for PUTs and DELETEs only showed up in December 2020. Before that, a read right after a write could hand back stale data, which is exactly the kind of landmine that made early FUSE workarounds so shaky.

None of this is a knock on S3. For write-once, read-many work, like data lake partitions, long-term backups, huge immutable training sets, the design holds up fine. The trouble starts the moment an application built on POSIX assumptions gets pointed at S3 and nobody updates the ground rules underneath it.

The four POSIX primitives S3 does not implement and what each one breaks in practice

Atomic rename. POSIX rename() is one atomic step. S3 has no equivalent. The standard workaround is CopyObject to a new key, then DeleteObject on the old one. That's two calls, not one, and the gap between them is a real correctness hazard: something can read the old key mid-flight, or hit the new one before it's fully written.

This isn't hypothetical. PyTorch's standard checkpoint pattern, write to a temp file, then os.rename(tmp, final), quietly broke on S3 for years, because the whole pattern leans on rename being instant. Checkpoint frameworks had to get rebuilt around the limitation. Table formats like Apache Iceberg, Delta Lake, and Apache Hudi exist partly just to paper over this one hole.

There's a partial fix now. As of June 2025, S3 Express One Zone shipped a RenameObject API that does true atomic rename without moving data. But it only works inside Express One Zone directory buckets, not general-purpose S3. Mountpoint for S3 mirrors the same split: no mv support on regular buckets, only on Express One Zone. Rsync-style workflows that treat rename as the commit step just don't run on standard buckets. Full stop.

File locking. S3 has none. No flock, no fcntl record locks, no advisory or mandatory locking of any kind. Any app that needs to coordinate concurrent writers has to build its own lock service on top, solving a problem the storage layer was supposed to solve in the first place.

Random and in-place writes. Because objects are immutable, a 4KB change to a large file means rewriting the entire object. That's brutal write amplification, and it gets worse as the file grows. Mountpoint for S3 doesn't support random-access writes at all, and FFmpeg needs them for common formats like MP4. Storj's benchmarks show Mountpoint failing standard FFmpeg jobs outright. Even a basic append with >> fails under Mountpoint's --allow-overwrite mode. Append only works through a separate --incremental-upload flag, and only on Express One Zone.

Inodes, symlinks, directories. S3 has no inode concept, so no hard links, no symbolic links, no per-file ownership the way POSIX expects. Directories aren't objects, they're inferred. Mountpoint will make an empty directory vanish the moment nothing inside it exists in S3, because there was never anything backing that directory in the first place. Any script that does cd, chmod, or follows a symlink hits a wall, usually a silent one.

Why the rename problem is harder than it looks at the distributed systems level

Object storage hits its scale numbers by sharding metadata across independently managed pieces of infrastructure. Atomic rename needs one guarantee across two locations at once: consistency and atomicity, together, at the same instant, everywhere. When the source key's metadata and the destination key's metadata sit on different shards, there's no shared storage to lean on and no global clock to line the two operations up.

That's the actual reason Express One Zone's RenameObject stays boxed into directory buckets. A single zone can manage a consistent namespace internally, which sidesteps the cross-shard problem entirely. General-purpose S3, spread across a much bigger and more scattered footprint, doesn't get that luxury.

So the next time a vendor claims atomic rename across a fully general, flat namespace, take that claim with a grain of salt and ask which bucket type they actually mean. This isn't a policy call some team could reverse in a sprint. It's closer to a physics constraint, wired into how distributed metadata has to work once you're operating at that scale.

How S3's performance constraints compound the POSIX gaps for high-frequency workloads

S3 also caps how fast anything can happen against a single prefix, and this is where the POSIX gaps stop being theoretical and start showing up on invoices. The ceiling sits at 3,500 PUT/POST/DELETE and 5,500 GET/HEAD operations per second, per prefix. Pile too much activity under one prefix, and it doesn't matter how big the bucket is: that ceiling is the ceiling.

Throughput per connection tops out around 80 MiB/s, too. A single GET or PUT can't push past that number even sitting on a 100 Gbps instance. And metadata calls, LIST especially, aren't cheap: each one is a full API request against a distributed index, with its own latency and its own line on the bill.

Uber ran into this directly. S3FS-FUSE had to enumerate prefixes to fake an ls command, and that alone generated tens of thousands of dollars a month in LIST charges, before counting a single byte of actual data moved. FUSE adapters like s3fs, Goofys, and Mountpoint tend to buffer operations locally and replay them later, which opens the door to stale reads and write-ordering bugs under any real concurrency. Goofys, for one, only offers close-to-open consistency, so multiple clients writing at once can and do race each other.

None of this is an edge case, either. Any pipeline running thousands of metadata calls per epoch, or any training job writing checkpoints while multiple workers stay active, runs straight into these limits.

Where the gaps hurt most: AI training pipelines and checkpoint workflows

AI training loops do two things constantly: read data, and periodically write checkpoints. Both land right on top of S3's gaps at once, which makes this the workload where the POSIX mismatch bites hardest.

Meta found 56% of GPU cycles sat idle waiting on training data. That's not a rounding error. That's the single most expensive resource in the entire stack sitting there doing nothing because storage couldn't keep up. Separate profiling at Uber, Shopee, and AliPay pinned roughly 80% of total training wall-clock time on data loading, just moving bytes from S3 into GPU memory during synchronous steps.

Checkpoints fail in a specific, repeatable way here. A job writes 95% of a checkpoint file, then needs to seek back and patch the header, and the seek just doesn't work, because S3 through FUSE was never a real file system to begin with. So teams route around it: copy the dataset to EBS, run training locally, copy checkpoints back out. That workaround burns 30 to 45 minutes of dead time per run, and it means paying to store the same data twice. Throw in a spot instance interruption, and the whole copy cycle runs again from scratch. Datasets built from lots of small files make it worse still, since every file open turns into its own round trip against S3's metadata index.

What the FUSE adapter generation got wrong, and what it revealed about the real requirement

S3FS-FUSE was the first serious attempt to paper over all of this: translate POSIX calls into S3 API calls underneath. It worked fine for one person poking around a bucket. It did not survive contact with production load, and it was never going to.

Three problems kept showing up across that whole generation of tools. Before the December 2020 consistency fix, read-after-write could return stale data, about the worst possible failure mode for checkpoint recovery. Metadata calls like stat, rename, and chmod have no cheap native match in S3, so each one turned into its own API round trip. And every small write got amplified into a full-object rewrite: Goofys turned any random write into a complete object overwrite, every single time.

Uber's fix wasn't a better FUSE client. It was dropping POSIX emulation for that workload entirely and moving the pipeline to a direct S3 API pattern, speaking object-native instead of pretending to be a file system. The lesson from that decade still holds: client-side emulation can't manufacture guarantees the storage layer never agreed to provide. What actually has to change is the storage service itself: real server-side atomic operations, a metadata layer that isn't reinvented per API call, and write semantics that don't turn every small edit into a full-object PUT.

How current solutions approach the gap: a survey of what is actually available

Mountpoint for Amazon S3 (AWS, open source) connects S3 buckets to Linux file systems and to Kubernetes through a CSI driver. It doesn't support overwriting a file without deleting it first, appending with >>, random-access writes, mv, or standalone empty directories. Storj's benchmarks show it failing standard FFmpeg jobs outright over these limits. It's a fine fit for cloud-native, sequential-read work. For HPC, ML checkpoint writes, or anything multi-cloud, it's the wrong tool, and no amount of tuning changes that.

s3fs-fuse (open source) handles single-user file browsing fine and falls apart under concurrent production traffic. CunoFS's benchmark numbers give a sense of scale: writing the Linux kernel source tree to S3 through s3fs takes just over two hours, reading it back takes around 15 minutes. On GCP, PyTorch running over the s3fs gateway writes at 260 Mbps. The LIST cost problem is the same one Uber hit, just waiting to happen again at whoever's scale next.

Goofys (open source) is lighter than s3fs and skips a lot of FUSE's metadata emulation, which makes sequential reads faster. But its write-back cache can't guarantee consistency across more than one client, so race conditions on concurrent writes show up regularly, and the write amplification problem never goes away. A 4KB random write still triggers a full-object PUT.

JuiceFS (open source, Apache 2.0) takes the most different approach of the four: a full POSIX file system where data lands in object storage like S3, but metadata lives separately, in Redis, MySQL, or TiKV. That split buys real POSIX locking, both flock-style and POSIX fcntl record locks, plus built-in LZ4 and Zstandard compression. Redis-backed metadata lookups are fast, and the project has picked up a substantial and growing open-source community. The catch is operational: running a separate metadata store is real infrastructure to babysit, which is exactly the overhead the simpler FUSE tools were built to avoid in the first place.

None of these four is the free lunch it sometimes gets pitched as. Faster reads cost weaker consistency. Full POSIX behavior costs a second infrastructure component someone has to keep alive. Broad compatibility costs raw performance. Nobody gets atomic rename or real file locking for free on top of S3's flat namespace, and anyone telling you otherwise is selling something. If the workload is sequential and cloud-native, Mountpoint is the right call and JuiceFS is overkill. If it's checkpoint-heavy and needs real locking, JuiceFS earns its operational cost and the FUSE tools are the wrong bet from the start. Picking a storage layer for a POSIX-native workload means picking the tradeoff the team can actually live with, not hunting for the option that supposedly skipped the line.