Sparse File Support in Object Storage Filesystems

A file can claim to be an enormous size and take up zero bytes on disk. That's not a trick; it's a sparse file, and it's one of the oldest tricks in the POSIX playbook. Object storage, the thing a growing share of the industry is now trying to build filesystems on top of, has no idea what a hole is. This piece is about that gap, and about who's actually closing it versus who's just talking about it.

Why object storage has no native concept of a hole

Start with what object storage actually is: a flat namespace of whole objects, accessed with PUT and GET. You write the whole thing, or you read the whole thing. There's no verb in the S3 API for "write these bytes at offset 4,000,000,000 and leave everything before it alone." That operation doesn't exist because objects are blobs with a name tag, not files. They're blobs with a name tag.

POSIX, meanwhile, assumes a pile of things object storage was never built to offer. Byte-addressable writes to arbitrary offsets. A real distinction between "allocated" and "unallocated" space inside one logical file. Calls like fallocate, and the lseek variants SEEK_HOLE and SEEK_DATA (only standardized across POSIX.1-2024, though they've existed as OS-specific extensions for years) that assume something, somewhere, is keeping a map of where the holes are.

Here's the specific failure point. On a normal Linux filesystem, when a process calls ftruncate to stretch a file out, or fallocate with the punch-hole flag to blow a hole in the middle of one, the filesystem's inode and block map just note it. Cheap, fast, done. An object store has no inode. It has no block map. It has an object and a key. So when a filesystem is built on top of one, whatever layer sits between the application and the bucket has to invent hole-tracking out of thin air, because the storage underneath will never do it for free.

This is one entry in a longer list of things object stores lack (atomic rename, flock, a real hierarchical namespace among them), but sparse files are a nastier case than most. Rename or locking failures are loud. They throw errors. A botched sparse file implementation is quiet: it just silently writes real zero-filled bytes where a hole should be, and now the "empty" region is costing storage and bandwidth for no reason. It also breaks any tool, like cp --sparse or tar -S, that goes looking for hole boundaries and finds none.

The architectural patterns implementations use to bridge the gap

Every workable design faces the same job: store only the real data in the object backend, but keep enough of a map on the side to rebuild the holes convincingly when someone reads the file back.

The most common approach is a sidecar index. One version of this pattern, described in patent literature around parallel log-structured filesystems, works like this: the filesystem accepts a sparse file from whatever application wrote it, stores only the data-bearing chunks as a compact object, and keeps a separate metadata record of which byte ranges actually have data. Anything not listed in that index is, by definition, a hole. Reads into it return zeros, manufactured on the fly, without the object store ever being asked for anything. This sidecar pattern appears, in some form, in nearly every object-backed filesystem worth using.

A second pattern, described in US patent 7,194,579, stripes pieces of a file sparsely across multiple volumes or file servers. Each volume only stores the slice assigned to it and treats the rest as a hole, tracked by pointer rather than by stored zero. It's a reasonable fit for distributed multi-volume systems, less so for a design built around a single bucket.

A third pattern, from Symantec's US patent 8,965,855, uses the term "sparse object" for something else entirely: a lightweight placeholder object used to manage object placement and redirect access away from overloaded storage. It's not file-hole logic at all. But it's a useful reminder that "sparse" as a general idea, metadata standing in for missing bytes, appears across object-storage design in different costumes.

Whichever pattern gets used, three things have to actually work, or the implementation is cosmetic. SEEK_HOLE and SEEK_DATA need to report real boundaries, since tools like cp --sparse=always trust them blindly. Punching a hole with fallocate needs to free real backend storage, not just paint over it with zeros. And reads into a hole need to return zeros without ever touching the object store, no fetch, no phantom object created for a range that was never written. Every one of these adds a metadata operation, and metadata operations against an object store cost latency and, often, real money per API call. There's no version of this that's free. There's only a version that's worth it.

Why AI training workloads make sparse file semantics matter more now

Object storage's role as the backbone of AI infrastructure is expanding fast, with the cloud object storage market growing at a 16% compound annual rate, from $8.14 billion in 2024 to a projected $9.49 billion in 2025, and AI pipelines are a big chunk of why. Checkpointing is where that growth collides directly with the hole left in checkpointed data.

During training, models get checkpointed at intervals, weights written out to storage as a hedge against the inevitable crash, the inevitable spot-instance eviction, the inevitable something. That checkpoint write is one of the two heavy I/O phases of a training run, and it's brutal on hardware if it's not handled well. A fast filesystem sitting between the GPUs and the object store lets the checkpoint land quickly and get flushed to the bucket asynchronously afterward, so the GPUs aren't sitting there burning electricity waiting on a write to finish. MLCommons clearly thinks this matters: MLPerf Storage v2.0, released August 4, 2025, added a dedicated checkpointing benchmark for LLM training on scale-out clusters. That's a benchmark body putting a stopwatch on checkpoint I/O specifically, which tends to mean the industry has stopped treating it as an afterthought.

Checkpoint files happen to be prime sparse-file territory. Checkpoint files can contain large regions that were never written, leaving much of the allocated space empty. Without real hole tracking, the filesystem has two choices, and both are bad: fill the untouched region with stored zeros (inflating the checkpoint's actual size and burning object-store API calls for data that says nothing), or slow the async flush down trying to figure out what to do with it. At real training scale, none of this is academic. A cluster running 64 GPUs at 2 GB/s each needs 128 GB/s of aggregate throughput just to keep up, and at that throughput, writing fake zeros instead of tracking a hole is not a rounding error.

Microsoft Principal Engineer Glenn Lockwood has made the case that checkpointing, with its big sequential block writes, is "ideally suited for object stores." True enough, but only if the layer between the training job and the bucket is honest about what was actually written and what wasn't. That honesty is exactly what sparse file semantics provide. Skip it, and the suitability Lockwood describes turns into wasted throughput on the very system that was supposed to be avoiding waste.

How Amazon S3 Files approaches POSIX semantics

AWS announced Amazon S3 Files on April 7, 2026, pitched as "fully-featured, high-performance file system access" to S3 data, with nothing required to leave S3 to get it. It was available in all commercial AWS regions at launch.

The claimed feature set: full filesystem semantics, POSIX permissions with UID and GID stored as object metadata, and no more duplicating data between an object copy and a file copy just to satisfy two different access patterns.

What the launch materials didn't say is the more interesting part. There's no mention, one way or the other, of hole punching, fallocate behavior, or SEEK_HOLE/SEEK_DATA support, the specific mechanics that make sparse files work rather than just exist in name. There were no throughput benchmarks published alongside the announcement. No pricing specific to S3 Files was disclosed. Nothing was said about how consistency behaves if the same object gets touched simultaneously through the S3 API and through the file-system mount, or about how the feature plays with existing S3 machinery like versioning, lifecycle policies, Intelligent-Tiering, or cross-region replication.

That silence on sparse files means something and deserves a second look. "Full file system semantics" is an easy phrase to write in a press release and a hard one to actually deliver on every corner case. Sparse file support is precisely the kind of corner case that's simple to leave out of a launch and painful to bolt on afterward, because it touches how data gets stored at the lowest level, not just how permissions get checked. The granular answers, the ones that would say whether a VM disk image or a checkpoint workflow would actually behave correctly, weren't in the announcement itself.

How JuiceFS and flexFS handle the POSIX gap in practice

Two systems reward direct examination, because both were built by people who ran into this problem and had to solve it rather than announce it.

JuiceFS layers POSIX semantics on top of object backends like S3, MinIO, and another cloud object storage service, and it does the one thing that makes any of this tractable: it keeps metadata somewhere else entirely, in Redis, MySQL, or TiKV, separate from the object data itself. Among the usual crop of S3 mount tools (s3fs-fuse, goofys, geesefs, rclone mount), JuiceFS stands out as the one that implements POSIX properly, hard links, atomic rename, fcntl locking included. The others tend to get lumped into "fine for backups, log collection, and media streaming," which is a polite way of saying don't run a database on them. Splitting metadata out from the object layer is exactly what makes fine-grained hole tracking possible in the first place, since checking or updating a hole boundary doesn't require a round-trip PUT to a bucket every time. One large language model training company migrated onto JuiceFS and built a cache pool pulling 70 GB/s out of otherwise idle resources. Tuhu, working between 2025 and 2026, built a unified AI storage platform combining JuiceFS, TiKV, and Ceph RADOS specifically to stop copying datasets back and forth between a training pipeline demanding POSIX access and object-API-hungry tooling around it.

Paradigm4's flexFS came out of a more specific frustration. Existing open-source options, s3backer among them, couldn't handle concurrent reads and writes from multiple servers hitting the same backend at once, which is a fairly hard requirement to fail if the goal is a real shared filesystem. Gary Planthaber has described the goal in direct terms: a proper POSIX network filesystem with the pricing, durability, and aggregate throughput of S3, not a workaround pretending to be one. flexFS stores file data in physical blocks specifically to cut down on S3 API charges and latency, and that same block-granular design happens to make hole tracking cheaper too, since holes can be recorded at block boundaries instead of needing byte-level bookkeeping. It also supports Linux advisory file locking.

The thread connecting both: neither system asks the object store to remember anything about file structure. The metadata layer carries that weight entirely. That's not a stylistic choice; it's the precondition for any sparse file implementation that isn't faking it.

What a filesystem needs to do for existing workloads to run unmodified

Here's the actual bar: an application that already creates sparse files using standard POSIX calls should not need a single line changed to run correctly. If it does, the filesystem hasn't closed the gap, it's just moved the burden onto the application developer.

Four behaviors decide whether that bar gets met. Extending a file with ftruncate past its written data has to create a real hole, not silently allocate and zero-fill a backend object. Punching a hole with fallocate has to actually free the backend storage tied to that range, not just record a zero and call it done. lseek(SEEK_HOLE) and lseek(SEEK_DATA) have to return byte offsets that are actually correct, because cp --sparse=alwaysand tar -S rely on those offsets to identify hole boundaries. And reads into a hole region need to return zeros without the backend ever seeing a fetch request or creating a phantom object for bytes that don't exist.

There's a simple test that exposes a broken implementation fast: copy a sparse file within the filesystem and check whether the copy stays sparse. If a 512 MiB apparent-size file with zero bytes actually on disk turns into a 512 MiB real file after a plain cp, the implementation is cosmetic. VM disk images and database snapshot workflows push this hardest, since they create, punch, extend, and query holes constantly and in quick succession. A filesystem that only handles the "create a hole and forget about it" case will fall over the first time a real workload touches it.

There's also a cost angle that doesn't get talked about enough. An implementation that quietly materializes holes as real stored objects isn't just wrong about semantics, it inflates both storage footprint and API call volume, which translates directly into cost. Sparse file support belongs on the same list as atomic rename, flock, mmap, fsync, hard links, and symlinks: not nice-to-haves, but requirements, if the goal is running existing code with zero SDK changes. Sparse files just happen to be one of the harder items on that list, because it demands active, ongoing metadata tracking rather than a permission check that can be enforced once and left alone.

Evaluating object storage filesystems for sparse file support

Anyone testing one of these systems should skip the marketing copy and go straight to the calls that actually matter: fallocate with the punch-hole flag and both SEEK_HOLE and SEEK_DATA. Run them, then check whether the object store shows the storage reduction it should. If punching a hole in a test file doesn't shrink the actual bytes billed, the filesystem is faking sparsity, dressing zeros up as an absence when they're really just zeros.

Run the cp test by hand instead of just reading about it. Create a sparse file, copy it within the same filesystem, and compare apparent size to actual size on both ends. Then try tar -S on it and see if the archive stays small or balloons to the full logical size. These are five-minute tests that reveal more than any spec sheet will.

For anyone evaluating this in the context of AI infrastructure specifically, checkpoint-style write patterns are the real proving ground: partial writes into a pre-allocated file, followed by an async flush to the object backend, is close to exactly what large training jobs do at checkpoint time. A system that handles that pattern cleanly, without inflating storage or burning excess API calls, has demonstrated the thing that matters. A system that only demos well on a single small file in a slide deck hasn't demonstrated much at all.