Hard Links and Symlinks on Object Storage Filesystems

Hard links and symlinks are two of the quietest, most load-bearing features in any POSIX filesystem, and object storage was never built to support either one. That's a design decision made decades apart from the workloads now running on top of it. It's a design decision made decades apart from the workloads now running on top of it, and understanding the mismatch explains exactly what you gain, and what you give up, when a cloud filesystem lets you mount a bucket and call it a drive.

Start with the plumbing. A hard link is just an extra directory entry pointing at the same inode. No new data gets written anywhere; it's the same file wearing a second name tag. A symlink is different: it's its own file, with its own inode, and its content is nothing more than a path string that the kernel follows when something tries to open it. The inode is the thing doing all the real work here. It holds permissions, ownership, size, and the link count. It does not hold the filename. Filenames live in the directory, which is really just a lookup table mapping names to inode numbers. That separation, name over here, data over there, is the entire reason hard links and symlinks can exist.

Hard links come with some fine print. They can't cross filesystems, because inode numbers are only unique within one filesystem (an inode number on your laptop means nothing on a server across the room). They can't point at directories either, since that would let you build a loop a traversal tool could wander into forever. And a file's data doesn't actually vanish until every hard link to it is gone. Symlinks trade those constraints for a different set of tradeoffs: they can cross filesystems, they can target directories, and they can even point at something that doesn't exist yet (a "dangling" symlink, which sounds like a yoga injury but is really just a broken promise).

None of this is academic. rsync --link-dest builds incremental backups by hard-linking unchanged files into each new snapshot directory, so every snapshot looks complete while actual disk usage only grows by the delta. Debian's update-alternatives swaps program versions by redirecting a chain of symlinks, so upgrading Python is really just changing where an arrow points. Enabling a systemd service is, underneath the hood, creating a symlink. These are small, boring mechanisms. They also run half the internet.

Why object storage was built without these primitives

Object storage doesn't have directories. It doesn't have inodes either. What looks like a folder, bucket/photos/2024/vacation.jpg, is really just one long string serving as the object's key. There's no hierarchy underneath it, just a flat address space with billions of keys sitting side by side. Each object is standalone, immutable, and identified by that one key. There's nothing to point a second name at, because there's no inode layer separating "the name" from "the thing."

S3 specifically lacks several things POSIX systems take for granted: atomic renames, file locking, symbolic links, and directory inodes. Renaming shows the whole issue in miniature. Renaming a file on S3 isn't a rename at all, it's a copy to the new key followed by a delete of the old one, done per object. During that window, a concurrent reader can see both the old name and the new name existing at once. That's the same consistency gap that makes hard links impossible: there's no atomic operation to bump a link count, because there's no link count to begin with. Azure Blob Storage has the same shape of problem. On Azure Blob Storage, symlinks aren't supported and get skipped during transfers, and hard link relationships between files are not preserved.

This isn't a theoretical inconvenience. ML training jobs chewing through millions of small files run into brutal per-request latency and prefix throttling, burning compute meant for model training while waiting on storage instead of using it. ETL teams end up building their own staging layers and lock services just to fake the coordination POSIX would've given them for free. Research pipelines written assuming POSIX semantics hit race conditions that fail silently, which is the worst kind of failure, because nobody notices until the numbers are already wrong.

And the timing is brutal. Industry analysts estimate that roughly 90% of all data in existence was created in just the prior two years, with about 80% of it unstructured. That's exactly the data type that lands on object storage by default, and exactly the data type that decades of POSIX tooling was built to handle. The gap is a structural feature that nobody patched by oversight. It's structural. Closing it means re-engineering the storage layer, bolting on a metadata tier, or doing both at once.

Why the POSIX gap became an AI infrastructure problem at scale

The gap stops being a filesystem curiosity because it now shows up as a line item. Most deep learning frameworks, PyTorch and TensorFlow included, are built assuming POSIX underneath them, a JuiceFS user story blog post notes, because their I/O and file-handling logic relies on that layer being present. Skip that assumption, and a platform needs its own custom storage protocol just to get data to the GPUs. AI training also needs a lot of processes reading the same files, at once, from different machines, with the throughput to keep expensive accelerators fed. Without that, teams either copy the dataset to every node (burning disk and time) or make everyone queue for the same copy (burning patience).

The dollar figures make the stakes concrete. MinIO reports that global AI infrastructure spending topped $250 billion in 2025, with storage and networking representing a significant and fast-growing share of that total. More than half of organizations report storage bottlenecks actively limiting AI performance, while the same source finds 57% say their data isn't even AI-ready. Experimentation is racing ahead of the plumbing meant to support it.

IDC's Worldwide Digital Infrastructure Sentiment Survey puts numbers on the frustration: 78% of enterprises now call AI infrastructure the single largest cost element in their AI return-on-investment math, and 73% say infrastructure complexity is what delays moving a model from pilot into production. Meta's own experience with PyTorch training is the sharpest illustration available. Introl reports that training jobs previously spent 35% of compute time simply waiting for data to arrive. After adopting GPUDirect Storage, streaming data straight to the GPU at 192 GB/s instead of routing it through a CPU-bottlenecked path capped around 50 GB/s, training speed improved 3.8x. That's not a rounding error; it's the difference between a training run finishing this week or next.

Storage architecture built for this kind of throughput can hit up to 5x the performance of a conventional protocol layered over standard web requests, MinIO's benchmarks show, exceeding 100 GB/s aggregate read against roughly 20 GB/s on the standard path. And this is exactly where link semantics stop being a footnote. Tools that lean on hard links for zero-copy dataset deduplication, or symlinks for dataset versioning and train/test splits, don't throw a helpful error on raw object storage. They just break, quietly, and teams end up either rewriting the tooling from scratch or copying the data outright, both of which pile more work onto the exact bottleneck everyone's trying to escape. IDC projects a 21% compound annual growth rate for scale-out file and object storage through 2030, which means this problem is growing. It's compounding right alongside the data.

Three approaches to bridging the gap, and the trade-offs each one accepts

There's no clean fix here, just three different places to hide the complexity, and three different bills that eventually come due.

The first approach adds a metadata sidecar, a separate fast store (Redis, MySQL, TiKV, take your pick) that holds filenames, permissions, directory structure, and, critically, link counts. Object storage becomes a dumb bucket of data chunks, while the metadata engine does the actual bookkeeping. This can support full hard link and symlink semantics, since something now actually owns the link count. The cost is a new dependency: the metadata store's uptime, consistency, and scalability all become the operator's problem now, not the cloud provider's.

The second approach puts a POSIX-capable caching tier in front of the bucket, something like local NVMe or an edge cache, and lets that tier absorb every filesystem operation while the bucket sits underneath as durable backing storage. Writes land on the cache first and flush out to the object store asynchronously. Rename, chmod, and symlink creation all work immediately, because they're happening on a real filesystem. Hard links are usually the casualty here, since the cache layer's semantics rarely stretch that far.

The third approach is a gateway layer that translates POSIX calls into object storage API calls on the fly. Applications don't need to change a single line of code, which is the appeal. But performance leans entirely on how well the gateway handles small files and metadata-heavy workloads, and link semantics support varies by implementation.

Every one of these approaches has to answer the same question: where does the link count actually live, and who's responsible for keeping it correct when multiple writers touch it at once? That question is also why symlinks are so much easier to support than hard links. A symlink target is just a stored string, cheap to write and cheap to keep. A hard link count needs a consistent, atomic decrement across every holder of that link, every time. One is a sticky note. The other is a ledger.

How Amazon S3 Files handles POSIX semantics after its April 2026 launch

S3 Files launched in April 2026, rolling out broadly across AWS regions on day one. Its architecture pairs an EFS caching tier at the edge with S3 Standard sitting underneath as the durable store. POSIX operations happen immediately against the EFS tier, and changes flush out to S3 asynchronously, within 60 seconds.

A lot works well here. Rename, chmod, and symlink creation all get full POSIX semantics on the EFS tier, exposed through a standard NFS v4.1/v4.2 interface that supports up to 25,000 concurrent NFS connections. That's a real filesystem experience sitting on top of S3, not a simulation of one.

Hard links, though, are explicitly excluded. Anything that leans on them, rsync --hard-links, cp -l, inode-aware backup tools, either throws an error or quietly falls back to making full copies instead. Advisory locks (flock, fcntl) operate against the EFS tier, but they don't extend across the boundary between NFS and the underlying S3 API.

The 60-second write-back window is a deliberate tradeoff, not an oversight. Writes are durable and visible to every client on the EFS tier the instant they land, well before that window closes. For AI training checkpoints, which tend to arrive in bursts followed by long stretches of pure computation, 60 seconds is nothing, and batching those writes into fewer, larger S3 objects actually saves money on PUT requests. AWS, building on its own infrastructure, with every incentive and resource to solve this problem, still chose to leave hard links out. That's a pretty good signal that hard links are the genuinely hard half of this problem, and symlinks are the tractable one.

How JuiceFS implements both symlinks and hard links via a metadata engine

JuiceFS takes the metadata-sidecar approach and builds the whole system around it. A JuiceFS client sits between an object storage backend and a separate metadata engine, and that metadata engine is where filenames, sizes, permissions, timestamps, directory structure, and link counts all actually live.

The metadata engine itself is pluggable: Redis, MySQL, PostgreSQL, SQLite, TiKV, among others, and it's built to scale horizontally, with a single namespace designed to scale to very large numbers of files. The data backend, meanwhile, doesn't care which cloud it's talking to, S3, GCS, R2, HDFS, even plain local disk all work, so whatever bucket already exists gets mapped into a POSIX filesystem and mounted like any other local path.

Because there's a real metadata layer doing real bookkeeping, JuiceFS supports both one style of advisory locks (flock) and POSIX record locks (fcntl), the exact lock types that don't survive the boundary between a network filesystem and object storage in S3 Files. It also runs LZ4 or Zstandard compression on data blocks before they ever leave for object storage, and it exposes itself through POSIX, Hadoop, Kubernetes, and an S3 gateway interface, so frameworks like PyTorch and TensorFlow just work, no rewrites needed.

None of this is free, and it shouldn't be sold as if it were. The metadata engine is a genuine operational dependency: if it goes down, the filesystem goes down with it. The pitch of "just point at a bucket" quietly becomes "point at a bucket, and also run and maintain a metadata store." Hard link support on object storage is achievable, this proves it, but it comes with a second system to keep alive, not a shortcut around one.

How AWS DataSync preserves link semantics across migrations as a partial workaround

DataSync is a migration-time trick worth understanding anyway, and it makes no pretense of being a filesystem. What it offers is a migration-time trick worth understanding anyway, because it shows how link semantics can be encoded as plain object metadata and reconstructed later, on purpose, by tooling built to expect it.

During a transfer to S3, each file referenced by a hard link gets moved exactly once, so storage isn't wasted duplicating the same bytes under multiple names. On later incremental transfers, new objects only get created if new references actually show up in the source. As long as that hard link stays unchanged in S3, it gets correctly rebuilt when transferred onward to an NFS file server, FSx for Lustre, FSx for OpenZFS, FSx for ONTAP over NFS, or Amazon EFS.

Symlinks get similar treatment: the target path is encoded so that symlink relationships can be reconstructed later by tooling built to expect it, and it gets correctly restored on transfer to any of those same destinations. Azure Blob Storage doesn't play along the same way. Symlinks aren't supported at all and get logged as skipped, and hard links transferred from Azure Blob Storage arrive as separate, unrelated files, with the link between them gone for good.

The takeaway is narrow but real: link semantics can survive a round trip through object storage, but only when the tooling on both ends explicitly knows to encode and decode them. That works inside a curated migration pipeline. It is not a general property of object storage, and it's not something to expect by default. If a POSIX filesystem is heading to the cloud and link semantics matter, the destination service matters just as much as the source, since not every object store or managed file service treats this the same way.

What a cloud filesystem that mounts your existing bucket needs to do to support both link types

Put the previous sections together and the requirements list writes itself. Symlinks need their target paths stored in a persistent metadata layer, not smuggled into the object key, and that metadata has to survive across different mounts and different clients touching the same filesystem. Hard links need something harder: a link count that gets atomically decremented across every holder, enforced by the metadata layer itself, because the object store underneath has no concept of "how many names point at this."

Atomic rename isn't optional here either, it's a prerequisite. Plenty of tools use a rename-to-commit pattern as their safety net, and some of those same patterns are how hard links get created. Skip atomic rename, and those tools aren't just missing a feature, they're operating unsafely without realizing it. Full lock support, both flock and fcntl, matters just as much for anything running concurrent, multi-process workloads, which describes most AI training pipelines running today.

None of this is exotic engineering. It's the same four ingredients threaded through every section above: a real metadata layer, an atomic rename, an honest link count, and locks that actually hold across every client touching the file. Object storage was never going to grow these on its own, they have to be built on top of it, deliberately, by something willing to own the bookkeeping the bucket was never designed to do.