Data Corruption From Concurrent Uncoordinated Writes on Shared Filesystems

Shared filesystems let more than one writer touch the same files and the same metadata at the same time. That access pattern, multiple writers, same target, no referee, is the root of every failure this piece covers: torn writes, read-modify-write races, and lost updates. All three corrupt data quietly, and all three follow predictable rules once you know what to look for.

Traditional filesystems were built assuming one host, one writer. ext4-dax uses per-host transaction IDs. NOVA uses CPU-local structures to reduce contention. Neither design expects two different hosts to reach into the same metadata at the same time. And even when two hosts map the exact same physical memory, each one keeps its own copy of the bookkeeping: its own superblock, its own inode cache, its own dentry cache, its own page cache. Two hosts can be looking at "the same" file and quietly disagree about what's in it.

Research on concurrent filesystem access makes the point: concurrent filesystem writes can corrupt shared data with no error signal visible to the writers. Every writer thinks it succeeded. The disk ends up wrong anyway. None of this is random noise, either. It follows directly from the lack of coordination, which is actually good news: if the failure is systematic, so is the fix. This piece sticks to write-side coordination problems on shared POSIX filesystems, not disk failures, not network corruption. Just what happens when writers don't talk to each other.

Torn writes: when a logical write is not delivered as an atomic unit

A torn write happens when one logical write gets split into multiple physical writes, and somebody else reads the file in between them. Picture a relay race where the baton gets dropped mid-handoff, except the filesystem doesn't know it dropped anything.

Most filesystems only guarantee atomicity at the block or sector level. Anything bigger than that gets broken into a sequence of smaller writes, and no lock covers the whole sequence. POSIX doesn't promise atomicity for writes larger than PIPE_BUF when other writers are active; that's a documented feature of the spec. That's a documented feature of the spec. It's a documented feature of the spec.

For structured data, this gets ugly fast. Say a file has a header and a few fields that straddle a block boundary. A reader showing up at the wrong moment sees field A already updated and field B still holding last week's value. The record still parses. It's just lying to you.

Network filesystems make the window bigger. SMB, NFS, anything remote breaks the write into round-trips over the wire, so there's more time for a half-finished write to sit there in plain view.

A Linux kernel SMB client patch from August 2026, submitted by Paulo Alcantara for the stable tree, is a real-world case worth walking through. The function cifs_do_truncate() flushed dirty pages and then truncated the file on the server, but it did both steps without holding i_rwsem or invalidate_lock. That gap let a concurrent buffered write through netfs_perform_write() sneak in and dirty new pages between the flush and the truncate. Those new pages got silently thrown away by cifs_setsize() calling truncate_pagecache(). Nobody got an error. The data just vanished.

The fix grabs three locks: i_rwsem exclusively (which blocks any buffered write, since those only take the lock in shared mode), filemap_invalidate_lock (which blocks page faults from reads), and it wraps the server truncate plus the local resize inside netfs_wb_begin/netfs_wb_end to block writeback collection too. Three locks, three different paths that needed shutting down. That's the tell here: this wasn't some rare edge case. Buffered I/O, page-cache management, and file-size changes appear together constantly in multi-writer setups. The ingredients for a torn write are sitting on the counter in basically every shared-filesystem deployment.

Read-modify-write races: how concurrent agents corrupt shared state without either write being wrong individually

Writer A reads a value. Writer B reads the same value. A computes something new and writes it. Then B, still working off its stale read, writes over A's update. Last write wins, and A's work just disappears, no error, no trace.

The gap between the read and the write is the danger zone. Anything that finishes its own write inside that gap will get quietly erased. And here's the frustrating part: neither write did anything wrong. Each writer wrote exactly what it computed from what it read. The bug isn't in either operation, it's in the timing between them. No amount of code review on either writer alone will catch it.

That's also why logging doesn't help. Both writers log success. Both see a valid file afterward. Nobody sees an anomaly, because there isn't one, locally.

Research on race conditions in production multi-agent systems (tianpan.co, 2026) found something telling: read-modify-write corruption in these deployments looks exactly like a model hallucinating. Same source found failure rates across production multi-agent deployments ranging from 41% to 86%, with state corruption and coordination bugs as major drivers. Teams see weird output and go debug the model. The model's fine. The infrastructure underneath it produces its own race conditions, as shown by state corruption and coordination bugs identified as major drivers in the cited failure rates.

Picture two agents both reading a shared progress file, each bumping a counter by one, each writing the file back. One increment vanishes. Running that loop a thousand times causes the drift to compound invisibly, until the numbers stop making any sense and nobody can say when it started.

Filesystems don't stop this by default because nothing forces them to. O_APPEND gives atomic appends. flock and fcntl offer locking. But both are optional. Writers that skip the locking protocol just race, and the filesystem shrugs.

Lost updates: the special case where a write succeeds and leaves the file worse than before

A lost update is its own animal, different from a torn write (which is about partial delivery) and different from a read-modify-write race (which is about a stale read clobbering something newer). A lost update is when a write finishes, completely, durably, and then a later operation wipes it out anyway.

The classic setup: open with O_TRUNC, then write. If a second writer opens that same file for truncation after the first writer wrote its data but before it closed the file, the first writer's bytes are simply gone. Both operations were individually correct. The outcome still isn't.

That SMB client patch is relevant again here. As the patch documents, pages dirtied after the flush but before the local truncate were silently thrown away when truncate_pagecache() ran. Completed and lost, in the same breath.

AI training checkpoints are especially exposed to this. The usual pattern is: write a new checkpoint, then rename it over the old one. If two workers write to the same checkpoint path without coordinating, one worker's checkpoint gets clobbered, and when recovery kicks in after a fault, it restores from a checkpoint that might not correspond to any real, consistent training state.

Research on safe checkpoint installation (arxiv.org/html/2511.18323v1) put a number on how fragile the naive path already is: unsafe writes survived crash injection 0 times out of 430. Zero. That's before you even add a second uncoordinated writer into the mix. Adding one makes an already-broken path worse.

POSIX says rename is atomic. Testing across production Linux filesystems (Pillai et al., OSDI 2014) found that rename isn't always atomic with respect to crashes in practice, and follow-up work using the CrashMonkey tool turned up additional bugs in production Linux filesystems. So the guarantee everyone leans on for the "safe" checkpoint pattern has documented holes.

The unsettling part of a lost update is that the filesystem can land in a state worse than either writer wanted. Not a blend of the two writes. Not a partial write. A regression to something older, or in the worst case, a zero-length file where a checkpoint used to be.

The absence of detection at the filesystem layer

All three failure modes share one trait: every individual operation reports success. No error code, no exception, nothing flagged. The filesystem did its job, technically.

Filesystems were never built to check meaning. They track byte ranges, block addresses, and metadata consistency. They have no concept of what value should be in a field, so they can't flag an overwritten update as wrong, because from where they're standing, nothing is wrong.

Advisory locks are, well, advisory. flock and fcntl only work if every writer chooses to use them. Skipping the lock lets the filesystem allow the write through anyway. Nothing forces participation.

So the failure signal, when it finally appears, is visible way downstream, dressed as bad output. A model that behaves like it's undertrained. An agent producing results that contradict what it just did. A counter that never converges. All of it gets blamed on logic bugs or model limits, when the actual problem is sitting in storage. Research on multi-agent deployments (tianpan.co, 2026) makes this exact point: parallel sub-agents corrupting shared state looks just like hallucination. That confusion follows directly from how the problem is baked into how quiet the failure is. It's baked into how quiet the failure is.

Hardware adds another layer of the same problem. Research on LLM training reliability (arXiv:2604.00726, 2026) found that NaN or infinity results are only a small slice of hardware-caused corruption. Most corrupted values are still numerically plausible, so they propagate through training undetected. The same logic applies at the filesystem layer: the bytes written are valid bytes. They're just the wrong bytes, and nothing downstream can tell the difference without help.

Catching any of this needs something outside the filesystem: checksums built into the file format, version counters that only ever go up, or application-level checks that verify an invariant actually held. None of that comes free.

AI training pipelines and multi-agent systems

Two kinds of workloads pack all three failure modes into one place: distributed AI training, where a lot of GPU workers write checkpoints, gradients, and data-loader state to a shared filesystem, and multi-agent systems, where a lot of agent instances read and write the same shared context files.

Checkpointing eats real time. AWS's guidance on fault-tolerant distributed training on Amazon EKS notes that synchronous checkpointing consumed up to 40% of total wall time on tested cluster sizes. Under that kind of time pressure, teams have every incentive to cut corners on coordination, which is exactly the condition that produces lost updates. That same AWS guidance (September 2026) points to Amazon FSx for Lustre as the shared filesystem underneath both async and synchronous checkpointing, and it flags checkpoint loading, not the restart mechanism, as the thing that actually dominates recovery time at scale. Lose or corrupt a checkpoint, and a fault that should've been recoverable just isn't.

Data loading has its own version of the problem. Training ResNet-50 on ImageNet at scale can spend up to 85% of per-epoch runtime on I/O (per research at arxiv.org/pdf/2508.11035). Multiple data-loader workers reading and writing shared shuffle state or sample indexes under that kind of I/O load is a textbook setup for read-modify-write races on the loader's own coordination files.

Multi-agent systems hit the same wall from a different angle. Agents sharing a scratchpad or a shared knowledge file will race by default: each one reads, appends or edits, writes back, with zero coordination against the others running in parallel.

And scale doesn't help. More writers, more writes per second, bigger files, all three push the odds of a collision up. A setup that works fine with two writers can fall apart at twenty, not because anything changed in the code, just because the dice get rolled more often.

The coordination mechanisms that prevent each failure mode

Torn writes get fixed by staying inside whatever atomicity guarantee the filesystem actually offers. For anything bigger than a single block, write to a temp file first, then rename() it into place. POSIX specifies that rename should be atomic with respect to readers, so readers should see either the old file or the new one, never something in between.

Read-modify-write races need an exclusive lock held across the entire read-compute-write sequence. fcntl with F_SETLKW, or flock with LOCK_EX, held from before the read all the way through the write and its flush. Skipping any part of that window brings the race back.

Lost updates need the atomic-rename pattern paired with coordinated truncation. Never open a shared file with O_TRUNC while other writers might still be active. That SMB kernel fix, grabbing i_rwsem and invalidate_lock before touching the flush-truncate-resize sequence, is this exact principle implemented at the kernel level.

POSIX locks are advisory, so the scheme only holds if every single writer plays along. One writer that skips the lock breaks the guarantee for everyone else, even the ones doing it right.

Version counters help too. Stick a monotonic version number in the file header. After grabbing the lock, check that the version still matches what was read before writing. If it doesn't, someone else got there first, and the current writer needs to re-read and recompute instead of blindly overwriting.

None of this works without real POSIX behavior: atomic rename, working flock/fcntl, real fsync all have to function correctly. A filesystem that only half-implements these (recall the documented gaps in rename atomicity across real filesystems) undercuts the whole strategy even when the application code above it is written correctly.

And if locking on every single operation becomes a throughput problem, the fix is to change the write pattern. It's to change the write pattern. Partition the file space so each writer owns its own slice. Use append-only logs and merge them later, offline. Or move the shared state somewhere built for concurrent writes.

Which points at a boundary worth keeping in mind: per the Oracle/developers research, when multiple agents or users need to touch the same shared memory at once, that's a job for database-level guarantees, not filesystem tricks. Filesystems and databases aren't competing for the same job, they're built for different ones. Structured state that needs concurrent updates belongs in something designed for exactly that. Unstructured data, big blobs, agent scratch files, that's still filesystem territory, and it always will be.