Flock and Fcntl Locking Semantics on Distributed Filesystems

Two lock APIs handle file locking on Linux: flock() and fcntl(). Both were built assuming one kernel, one machine, one set of processes that all die together when something goes wrong. Spread that file across NFS, SMB, Lustre, CephFS, or object storage, and every one of those assumptions has to answer a question it was never designed for: what happens when the thing on the other end of the lock isn't a process anymore, it's a network connection?

That question doesn't have a clean answer. It has several messy ones, depending on which filesystem you're on. Here's the full breakdown, layer by layer.

Why the process-association rule makes fcntl() locks quietly dangerous before any network is involved

Before any network gets involved, fcntl() already has a landmine built in.

Open the same file twice in one process. That's two calls to open(), which means two separate file descriptions, even though it's the same file and the same process. Lock one of them with fcntl(). Now close the other one, the one you never locked, for some completely unrelated reason (cleanup code, a loop that closes every fd above 2, whatever). The lock you were counting on just vanished. No error. No warning. The kernel just quietly drops it, because fcntl() locks belong to the process, not to the specific file descriptor that requested them.

This is not some obscure corner case, it's a known failure mode that clustered filesystem documentation flags precisely because people keep getting bitten by it.

There's a thread-safety version of the same bug. Two threads share one process ID. fcntl() locks are scoped to the process, so those two threads can never use fcntl() to lock against each other, since the kernel sees them as the same lock owner. If a program needs threads to coordinate over a file, flock() or OFD locks are the right tool. fcntl() just isn't built for it.

Some programs, trying to be extra safe, historically grabbed both an exclusive flock() and an exclusive fcntl() lock on the same file, figuring belt-and-suspenders coverage against any peer process regardless of which API it used. Locally, the two lock types never conflict with each other. Over a network filesystem, as the next section shows, that habit turns into a bug.

One more asymmetry worth flagging: the kernel checks for deadlocks with traditional POSIX record locks. It does not do this for OFD locks. Keep that trade-off in mind, because distributed filesystems lose even the local detection.

How NFS translates flock() locks and where that translation breaks

Before Linux 2.6.11, flock() on an NFS mount stayed local. Two different clients could each grab an "exclusive" flock() lock on the same file at the same time and neither one would ever hear about the other. Locking that does nothing across machines isn't really locking, it's a courtesy call to nobody.

Linux 2.6.12 fixed the visibility problem by converting flock() calls into POSIX byte-range locks that span the whole file, sent over the wire using the NLM protocol (that's NFS v2/v3 locking). Check /proc/locks on the client and it still says FLOCK. Check the server, and it's a POSIX lock. Same lock, different label depending on which machine you ask.

That mismatch causes a specific, documented bug: POSIX locks and flock() locks never conflict with each other, even after translation. So a server-side process holding an exclusive flock() and an NFS client holding what it believes is an exclusive flock() on the same file can both succeed. Both sides think they have exclusive access. Neither one is wrong, exactly, they're just both wrong at the same time.

This is where the dual-lock habit from the last section turns from harmless redundancy into a self-inflicted wound. A program that grabs both flock() and fcntl() locks "for safety" ends up sending two POSIX locks over NFS. Those two lock types do conflict with each other. The defensive pattern becomes the thing that deadlocks you.

Linux 2.6.37 added a local_lock mount option, letting clients treat flock() as purely local again. Rather than a fix, it functions as an opt-out. You get back the pre-2.6.12 behavior, meaning no cross-machine coordination at all, which was the exact problem 2.6.12 was trying to solve in the first place.

NFS's cache model piles on more risk. NFS uses close-to-open consistency: caches flush and invalidate around open and close operations, not on every write. Any gap where neither side actually holds the lock (which the flock()/POSIX mismatch creates constantly) is a window where a reader can pull stale cached data straight off a machine that hasn't heard about the latest write.

Two more details worth knowing. Process identity embedded in lock metadata loses meaning across machines, since a PID only identifies a process on the box it came from. And ENOLCK, an error rarely seen on a local filesystem, is worth handling explicitly over NFS, since the remote locking protocol can fail in ways that have zero local equivalent.

How SMB/CIFS changes the rules further by making locks mandatory

SMB took a different road entirely, and it's arguably the more dangerous one for code that wasn't written with it in mind.

Before Linux 5.4, flock() over SMB/CIFS didn't reach the server at all. Same story as pre-2.6.12 NFS: local-only, invisible to anyone else on the network. Linux 5.5 changed that by emulating flock() with SMB byte-range locks across the whole file, producing the same fcntl()/flock() interaction problem that NFS has.

Here's the part that actually matters more: SMB locks are mandatory. On a typical operating system filesystem, a lock is a handshake between cooperating processes, an "I'll check if you're using this" contract. Nobody's forced to respect it. On SMB, any I/O attempt from a separate file descriptor on a locked file just fails outright with EACCES, whether or not that code ever checked for a lock in the first place.

That's a real problem for anything written to the Unix locking model, because it assumes locks are hints. Code built on that assumption has never had to handle a hard I/O failure caused by a lock it didn't know existed. Drop that code onto an SMB mount and it can start throwing errors it has no code path for.

And the behavior isn't even consistent across SMB itself. Protocol version, mount options, and server implementation all shift how locking actually plays out, so "SMB" isn't one behavior, it's a family of them.

Lustre and CephFS: cluster filesystems that implement locking but still carry caveats

Lustre and CephFS were both built for clusters from the ground up, which means locking gets more attention than it does as an afterthought bolted onto NFS. That doesn't mean it's free of caveats.

Lustre supports flock() cluster-wide by default, as BSD-style whole-file advisory locks. It also supports fcntl() byte-range locks, describing them in its own documentation as "nearly POSIX-compliant," an honest admission that no distributed filesystem can fully satisfy every POSIX fcntl() requirement while also functioning across machines.

On Lustre, the scope is a mount-time choice: -o localflock keeps locking node-local, -o flock makes it cluster-wide, and cluster-wide is the current default, though the documentation still lists known problems with that mode. The risk here isn't really about which option is better, it's that a misconfigured node can silently drop back to local-only locking, and the application has no way to know. The application has no way to detect the difference.

CephFS handles both flock() and fcntl(), but every lock request gets routed through the MDS (Metadata Server). There's no capability-based shortcut around it. CephFS aims for something stronger than NFS offers too: same-host behavior even when clients are on different hosts, a tighter consistency goal than NFS's close-to-open model.

That strength costs something in performance, and the cost is not small. A binary search across a roughly 75 GB file on CephFS, using mmap without any explicit locking, ran for close to two hours. Adding one flock() call to mark the file as shared cut that down to five minutes. The MDS's handling of page-locking overhead was the likely culprit. Correct locking rescued the workload here, but the reverse point holds just as well: skip the lock, and you can pay for it in ways that have nothing to do with correctness.

One more caveat that matters for anything doing heavy writes: if a client crashes mid-write on CephFS, the write isn't guaranteed atomic. Crash recovery doesn't restore whatever invariant the lock was supposed to protect. The lock can be perfectly correct and the data still ends up in a half-written state.

What object storage's absence of locking primitives means for applications that assume them

Object storage, S3, GCS, Azure Blob, R2, doesn't have a flock() or fcntl() equivalent built into the protocol. There's no lock to request and no lock to fail. (Azure Blob has a native Lease API, and S3 added conditional writes in 2024, but neither one is a POSIX lock, and neither behaves like one.) Dropping the lock table and dropping a huge source of coordination overhead is a deliberate trade for horizontal scale, letting the system scale in a way a POSIX filesystem never could.

Eventual consistency compounds the gap. A write can complete and still not be visible to every client immediately, which is close to the opposite of what POSIX guarantees after a successful write() call returns.

Code that assumes POSIX locking correctness runs straight into a wall here. There's no lock to acquire, no error to catch, no fallback path, the assumption just fails silently, because the API it depends on was outside the contract to begin with.

Object storage was built for write-once, read-many access patterns. Random writes and byte-range locking sit entirely outside what it was designed to do, and that's by design, not a missing feature waiting to be added.

That gap matters more now than it used to, since object storage has become the default place AI training data actually lives. The distance between where the data sits and where POSIX locking behavior is expected keeps growing rather than shrinking. Teams stuck with file-locking assumptions have two real options: keep the data on a filesystem that actually provides lock semantics, or build the coordination themselves at the application layer, leases, conditional writes, version checks, essentially rebuilding by hand what a kernel would otherwise give for free.

The design assumptions that break at each layer of distribution

Line them up side by side and the pattern is obvious: every layer of distribution strips away one more guarantee that application code was quietly leaning on.

Advisory locking holds fine everywhere flock() and fcntl() exist, right up until a remote client can perform I/O without ever seeing the lock state, which is exactly what pre-2.6.12 NFS and any object storage layer allow. Open-file-description association for flock() survives fork() and dup() just fine on one machine, but breaks the moment the "file description" is a kernel object on one box while the lock itself is tracked somewhere else, since an NFS server tracks POSIX locks by lockd, not by the fd that originally requested the lock.

Process-death cleanup is maybe the biggest gap. Locally, the kernel guarantees a lock releases on exit or on the last fd closing. Distributed, a client node crashing can leave a lock held until NLM recovery runs or the server otherwise resolves the stale state. There's no dependable equivalent of "the kernel cleaned it up for you."

Byte-range granularity means something concrete on a local disk. Over NFS v3 and NLM it's transmitted across the wire, but how precisely the range gets enforced depends on the server's implementation. Over object storage, it doesn't exist, full stop.

The clean local rule that flock() and fcntl() locks never interact gets broken by both NFS and SMB, since NFS translates flock() into a POSIX lock and creates cross-type interactions that local code was written without expecting, while SMB emulates flock() with byte-range locks spanning the whole file. And deadlock detection, available for POSIX record locks locally, is generally missing across nodes entirely, since distributed deadlock detection needs coordination the kernel just doesn't provide.

Every step away from a single kernel, to NFS, to a cluster filesystem, to object storage, removes at least one guarantee that some piece of application code out there is silently counting on.

What engineers should verify before relying on either lock family in a distributed deployment

Start by figuring out which lock type the code actually uses and whether the target filesystem translates it into something else. flock() over NFS is not flock() semantics anymore, it's a POSIX lock wearing a flock() costume. Check /proc/locks on the client to see what lock types are actually being held.

Check mount options directly, and don't assume the default matches what you expect. Lustre's -o flock versus -o localflock changes the scope of coordination. NFS's local_lock option does the same thing, quietly reverting to local-only behavior.

Never mix flock() and fcntl() locks on the same file across a network boundary. That NFS translation turns two lock types that never conflict locally into two POSIX locks that do conflict on the server, exactly the self-deadlock trap described earlier.

On SMB/CIFS mounts running Linux 5.5 or later, test what happens when a non-locking file descriptor tries I/O on a locked file. Mandatory locking means EACCES failures the advisory model never prepared the code to handle, so that error path needs to exist before it gets hit in production.

Treat process-death cleanup as something that isn't going to happen for you. Design around a crashed client leaving its lock behind: timeouts, heartbeats, lease-based protocols, something that doesn't depend on the kernel noticing a machine disappeared.

For thread safety inside a single process, use OFD locks (F_OFD_SETLK) instead of traditional POSIX record locks, which were added to Linux in 2015.That API exists specifically because process-scoped fcntl() locks are unsafe once more than one thread is involved.

For object storage backends, there's no lock to lean on at all. Coordination has to happen at the application layer, conditional writes, ETags, version checks, or the coordination file itself needs to live somewhere with real lock semantics.

For AI and data-heavy workloads where the data sits in object storage but the code still expects POSIX locking, the cleanest fix is a filesystem layer that mounts the bucket with genuine POSIX behavior: real flock, real fcntl, OFD locks, atomic rename. That closes the gap without moving the data and without rewriting the application, because the lock behavior the code expects gets enforced by the filesystem layer instead of the object store underneath it.