Sparse File Support in Object Storage Filesystems

Sparse files are a deliberate design feature built into POSIX, not some accidental gap or edge case a filesystem forgot to close. The whole idea rests on a simple split: a file's logical length and its actual allocated storage don't have to match. A file can claim to be enormous on paper while taking up almost nothing on disk.

To reason about this correctly, keep three separate numbers in your head at once: how long the file claims to be, what bytes actually come back when you read it, and how much physical storage the filesystem has set aside for it. These three numbers are allowed to disagree, and that disagreement is the entire point.

Two kernel-level moves create this gap. The first: an application jumps past the current end of a file using lseek and writes further out, and the kernel never bothers to allocate blocks for the stretch it skipped. The second: fallocate(2), called with the flags FALLOC_FL_PUNCH_HOLE and FALLOC_FL_KEEP_SIZE, which takes an existing file, an offset, and a length, and deallocates the blocks in that range, handing them back to the filesystem's free pool, all while leaving the file's reported size exactly where it was. Opening a file, seeking to 4 GiB, writing a single byte, and closing it produces a 4 GiB logical file that occupies essentially no disk space.

Reads from any part of that file you never wrote come back as zeros, since those blocks were simply never allocated. That gap between what a file says it is and what it actually costs to store is the thread the rest of this piece pulls on.

Object storage's flat, immutable-object model and its lack of a native equivalent

Object storage follows a different design contract and was never built to recognize this gap. Object storage organizes data as a flat key namespace with HTTP REST access (PUT/GET/DELETE), and an object of length N is exactly N bytes, full stop, with no metadata anywhere in the API that marks interior byte ranges as holes. The object format simply has no vocabulary for "this part doesn't exist yet."

Immutability makes the problem worse. On a local filesystem, you can punch a hole in the middle of a file and leave the rest untouched. Object stores don't allow partial edits like that: writing into the middle of an object, or appending to one, generally means rewriting the entire object from scratch. The partial deallocation that hole punching performs on a local filesystem has no S3-API equivalent.

Object storage also skips the hierarchical directory tree and the allocation bookkeeping that a real filesystem uses to track what's actually stored versus what's merely claimed. There's no inode, no block map, no concept of "this directory's quota is half-used." It's keys and blobs, nothing more.

Put those two facts together: anything that wants to offer POSIX sparse-file behavior on top of an object store has to build the entire illusion itself. The zero-filled reads from holes, the divergence between logical size and real storage cost, all of it has to be synthesized in a layer sitting above the object store, because the store itself offers no help whatsoever.

Translating hole creation across the object boundary

Building that illusion starts with chopping a file's logical address space into fixed-size chunks, where each chunk corresponds to one backing object (or a slice of one) in the store underneath. When an application seeks past the end of a file and writes further out, the filesystem layer updates the file's recorded size in its own metadata, but it skips the PUT entirely for any chunk that falls inside the hole. Those chunks simply don't exist as objects; they are absent, unallocated and missing.

Later, when something tries to read from one of those absent chunks, the layer hands back a buffer full of zeros without ever issuing a GET request to the backend. This mechanism makes the POSIX promise ("reads from holes return zero bytes") hold up even though the object store underneath has no idea a hole exists. The layer's own bookkeeping maintains the illusion; the object store does nothing to support it.

Nutanix's granted patent, titled "Concurrent multiprotocol access to an object storage system," is one of the clearer documented descriptions of how this kind of allocation model gets built, but it does not describe a sparse file approach where offsets get skipped before writing to higher offsets. The pattern it covers is multiprotocol consistency rather than the seek-and-write hole mechanism specifically, and the two should not be conflated.

Whatever the exact design, the hard requirement is the same: the layer has to persistently track, for every single file, a complete record of which chunk indices have a real backing object and which ones are holes. That map has to survive a restart. It has to stay consistent even when multiple writers are touching the same file at once. Lose track of that map, even briefly, and the file's contents become unreliable, because the system no longer knows which chunks are real and which are fiction.

Chunk size is a trade-off baked into the design. There's no universally correct chunk size, only a balance tuned to the kind of files a given system expects to store.

Implementing hole punching after the fact

Punching a hole into a file that already exists is a harder problem than creating one during a write, because now you're modifying the allocation state of objects that are already sitting in an immutable store. When an application calls fallocate with FALLOC_FL_PUNCH_HOLE and FALLOC_FL_KEEP_SIZE, it's handing the filesystem an offset and a length and asking it to deallocate exactly that byte range while keeping the file's reported size untouched. The file looks the same length from the outside. The punched region, on the inside, becomes a hole.

Misaligned punches are where things get messy. If the requested range starts or ends partway through a chunk, the layer has to read that chunk's object, zero out the requested sub-range inside it, and write a new object back in its place. Because the store won't let you edit in place, this becomes a read-modify-write cycle, and that cycle adds latency and opens a window for consistency problems if anything interrupts it midway.

The FALLOC_FL_KEEP_SIZE flag isn't a minor detail, it's the whole semantic contract of the operation. The file's logical length is not supposed to move after a punch. A layer that lets the size shift, even if the zero-fill behavior is technically correct, has broken the POSIX contract.

Shallow implementations fail quietly, in a way that's easy to miss during testing. Picture a layer that handles a punch request by just zero-filling the requested range inside the existing object, without ever deleting that object or reclaiming any storage. From the application's point of view, everything looks fine: the fallocate call returns success, and a subsequent read across that range comes back as zeros, exactly as expected. But no storage was ever reclaimed. The entire economic argument for punching holes, freeing up storage you no longer need, silently fails to happen, even though every visible signal says the operation worked.

Getting this right requires two things to happen together: the storage actually gets reclaimed, through object deletion or replacement, and the metadata map gets updated to reflect it. Both of these have to happen with enough atomicity that a crash in between doesn't leave the file in a broken state, where a hole looks allocated or an allocated region looks like a hole. Anything less than that leaves a gap between what the system claims and what it actually did.

Maintaining the logical-versus-allocated size split across the object layer

Both of the mechanisms covered so far, creation-time holes and after-the-fact punching, come down to the same underlying requirement: the filesystem layer has to track logical size and allocated size as two genuinely separate pieces of metadata, because the object store beneath it has no concept of either.

POSIX draws this line explicitly. st_size is the logical length, the number stat reports. st_blocks is the count of allocated 512-byte units, the number du actually uses. A correct sparse-file implementation has to be able to answer both questions with different numbers, on demand, accurately.

There's a tempting shortcut here that doesn't hold up: skip tracking allocated size separately, and instead derive it on demand by listing the objects that belong to a given file. This approach is, at best, only eventually consistent with reality, and at worst slow enough to be unusable, since every size check now costs a network round trip to the object store. The correct approach persists the hole map, the full set of absent chunk indices, as part of the inode metadata itself, so that st_blocks can be computed by counting present chunks locally, without touching the object store at all.

The Nutanix patent's scope is relevant again here, specifically around concurrent multiprotocol access: the hole map has to stay consistent even when the same file is being touched through a file-based protocol and an object-based protocol at the same moment. That adds a concurrency dimension on top of an already delicate bookkeeping problem, since two different access paths now have to agree on which chunks are real.

None of this is purely an accounting exercise, either. Applications that genuinely care about sparse-file structure, backup software, checkpoint loaders, VM hypervisors, use the SEEK_DATA and SEEK_HOLE modes of lseek to find hole boundaries directly, without reading every byte of a file to figure out where the holes sit. Sparse-file compliance is about whether the interface tells the truth fast enough to matter, not only about saving storage.

FUSE-based adapters and the additional gap they introduce between the contract and the implementation

FUSE-based object mounts start from the same structural limitations as the object model itself, and then add a cost of their own on top. The combination makes full POSIX sparse-file compliance both harder to build and more expensive to run.

The translation path FUSE sits on runs from POSIX calls to HTTP requests, and every metadata operation along that path becomes its own network round trip. The s3fs-fuse project documents this directly: operations like listing a directory suffer from poor performance specifically because of that network latency. A hole map lookup, needed to answer a stat call, resolve a SEEK_HOLE request, or serve a read from an absent chunk, has to cross the FUSE boundary between user space and the kernel on every single occurrence, adding CPU overhead before the request has even reached the network.

It's a reasonable trade-off for workloads that fit its model, but the design choices made upstream of it rule out full POSIX sparse-file support from the outset.

There's a concrete counterpoint that shows FUSE isn't the only road available. At the OpenZFS Developer Summit 2025 in Portland, MayaNAS introduced objbacker.io, a native ZFS VDEV implementation for object storage that skips FUSE entirely. It reported 3.7 GB/s of read throughput straight from S3, GCS, and Azure Blob Storage, by keeping metadata and small blocks under 128 KB on local NVMe SSD while streaming the larger blocks directly from the object store. It's a working demonstration that avoiding FUSE is a real architectural option.

Sparse-file compliance requirements at the AI training workload scale

Model checkpoints are where every gap covered so far turns into real money and real GPU hours. A checkpoint is a snapshot: optimizer state, weight tensors, training metadata like step count and the RNG seed. Tensors inside that checkpoint are frequently pre-allocated to their full eventual size and then filled in gradually as training proceeds, which is exactly the seek-and-write pattern that creates sparse files in the first place.

Without native awareness of where the holes actually are, an object store ends up transferring zero-filled bytes that carry zero information every time a checkpoint gets read back. That wastes network bandwidth, sure, but worse, it wastes the GPU-feed time that read was supposed to serve in the first place. Every second spent pulling meaningless zeros across the network is a second the GPU sits there waiting instead of computing.

Meta's experience building Tectonic, its distributed filesystem for storing exabytes of training data, shows what that waiting looks like at scale: GPUs sat idle waiting on training data, with utilization dropping in ways that pointed straight at a data-loading bottleneck. Sparse-file overhead on checkpoint reads is one contributor feeding into that kind of stall.

There's a second cost layered on top of the first: request amplification. Without hole-aware retrieval, zero ranges travel across the network right alongside the actual data, inflating the total volume moved for every single batch. NVIDIA's AIStore takes a direct swing at that problem with GetBatch, a storage primitive that replaces thousands of separate object reads with one deterministic, fault-tolerant streaming operation. In a production training run on 128 NVIDIA A100 GPUs, that collapsed thousands of individual requests into a single stream, cutting out exactly the kind of per-object overhead that sparse-file-blind storage piles on by default.

The real stakes of this whole translation problem are GPUs that stay fed versus GPUs that sit there burning money while they wait on zeros that never needed to travel the network in the first place.