Consistency Models in Distributed Filesystems for Agent Workloads

Distributed filesystems for agent workloads force a trade-off between consistency guarantees and the coordination overhead agents can tolerate. Get that trade-off wrong and you don't get a slow system. You get silent corruption, the kind nobody notices until a training run produces garbage or two agents flatten each other's work. The spectrum from strict linearizability to eventual consistency matters here in ways it never did for stateless services.

Why distributed filesystems create a distinct consistency problem for agents

Two agents sit on a live Kubernetes cluster. No lock gets violated. The filesystem delivers consistent reads to each agent, yet the compound result is wrong.

This is not a theoretical edge case. CoAgent, from Lyu et al. at Shanghai Jiao Tong University, identifies the core mismatch: classical transactions are bounded and fast, but a single agent transaction spans minutes of inference. Read sets are broad and opaque. You can't statically infer what an agent will touch. And writes take effect the moment they execute. You can't fork or buffer live state, so speculative isolation is off the table by default.

The scale is documented. A recent audit of over 200 multi-agent traces, from Cemri et al. in 2025 and cited by CoAgent, attributes more than a third of all failures to inter-agent misalignment. You get a wrong answer that looks plausible.

The consistency spectrum from linearizability to eventual consistency, and what each costs agents

Consistency models form a spectrum, and every point on it trades a different combination of correctness guarantees, coordination overhead, and availability. Strict linearizability sits at one end. Every read sees the most recent write. All operations appear to execute instantaneously at a single point in time. That's the strongest guarantee you can get, but it requires synchronous coordination across replicas. You pay for it in latency and availability risk under partition.

Causal consistency relaxes the requirement. Operations that are causally related appear in the same order everywhere. Unrelated operations may diverge. This is weaker than linearizability, but it preserves the semantics agents care about most. If you wrote a file and then told another agent to read it, that agent should see your write. That's the guarantee.

Eventual consistency is the far end of the spectrum. All replicas converge given no new updates. There's no ordering guarantee and no visibility guarantee at any specific moment. Eventual consistency persists in caches, replicas and some self-hosted stores; the major cloud object stores are now strongly consistent per key. Where it remains, you pay its costs in agent logic, not infrastructure.

The CAP theorem remains the governing constraint. No distributed system simultaneously provides consistency, availability, and partition tolerance. You choose which two to favor, and that choice determines which failure modes you inherit. Then the failure modes start compounding.

Where eventual consistency breaks for long-running, parallel agent transactions

Eventual consistency is not a safe default for parallel agent workloads. The convergence it promises arrives too late and too quietly. A single agent transaction spans minutes of inference. The write that invalidates it has already propagated before the agent acts. That timing asymmetry is the core issue.

The gradient corruption case makes this vivid. In distributed training, all ranks must derive their projections from the same batch. The batch must become atomically visible across the cluster before any rank begins its optimization step. BatchWeave, documented in arXiv 2605.09994 from May 2026, spells this out. You can't run a training job where rank three sees the batch and rank seven doesn't.

Last-writer-wins is the default conflict resolution under eventual consistency. MELD, from Lovén et al. at the University of Oulu and documented in arXiv 2608.16357 from August 2026, shows what happens: one of two contradictory claims is silently discarded.

Non-determinism compounds the problem. Crew Scaler's security analysis, arXiv 2603.09002, identifies non-determinism as the most under-addressed domain across all reviewed frameworks for multi-agent systems. The objection that eventual consistency works fine for informational layers like recommendation engines is true. That's exactly the partition the case studies in the next section will argue for.

How strong-consistency systems handle the agent workload: DeepSeek 3FS and BatchWeave

Strong consistency is achievable at AI training scale. The architectural choices required to get there reveal why it's not a universal answer. DeepSeek's Fire-Flyer File System, open-sourced in February 2025, is the infrastructure-level exhibit. 3FS uses CRAQ, or Chain Replication with Apportioned Queries, for strong consistency without a leader bottleneck. That allows highly concurrent data access and removes the partial-visibility problem that eventual consistency creates for training ranks.

The numbers are serious. 3FS achieves total read throughput of 6.6 TiB/s in a 180-node cluster. Single-node KV cache query peak throughput exceeds 40 GiB/s. FoundationDB handles metadata. NVMe SSDs and RDMA handle the data plane. The kernel/userspace hybrid design matches storage throughput to GPU compute throughput. RDMA is central to those numbers, and RDMA remains largely absent from contemporary cloud object storage, which relies on protocols that are not RDMA. 3FS's consistency model and its performance are jointly contingent on infrastructure most teams don't have.

BatchWeave is the complementary example. It's a library-level answer, not an infrastructure-level one. Implemented entirely as a client-side library with no server-side processes, it uses the object store as the sole shared substrate. The core construct is the Transactional Global Batch, or TGB. It provides atomic all-rank batch visibility, a globally ordered step sequence, checkpoint-aligned lifecycle management, and end-to-end exactly-once recovery. These are strong-consistency semantics built on versioned manifests and conditional object writes.

BatchWeave governs data visibility through versioned metadata manifests rather than record offsets. On 64-GPU evaluations, it outperforms colocated dataloader throughput while providing full failure isolation. No partition assignment, no cold-start overhead.

The common thread matters more than either system's specifics. Both 3FS and BatchWeave achieve strong consistency by pushing coordination into the data structure itself. Versioned manifests in one case, chain replication in the other. Neither reintroduces the leader bottleneck that makes linearizability expensive. That's the design move that lets them scale.

How agent-specific concurrency control fills the gap classical locking cannot

Classical concurrency control mechanisms fail for agent workloads in ways that are structural rather than tunable. Two-phase locking blocks long inference intervals. You can't hold a lock for minutes while an agent thinks. CoAgent states this problem precisely, and its alternative flips the model.

MTPO, or Monotonic Trajectory Pre-Order, fixes a serialization order at launch. When a conflict occurs, a one-way notification asks an affected reader to re-judge and patch its plan. The framework mechanically undoes and reorders misplaced writes through the saga-style inverse each tool registers in advance. The key insight is that the LLM inside each agent can judge whether a conflicting write actually invalidates its plan. It can repair exactly the operations that depended on it. Coordination becomes advisory rather than blocking.

The results hold up. Two-phase locking and OCC surrender nearly all concurrency gains on the same workloads. A filesystem interface where agents issue bash commands is the workload CoAgent is most naturally suited to, and it aligns with how frontier models are trained and how agents overwhelmingly prefer to manipulate data.

The objection is obvious: this puts the correctness burden on LLM reasoning, which is non-deterministic. The answer is that MTPO's mechanical undo-and-reorder layer handles structural conflict resolution. The LLM judgment is invoked only for semantic repair of the plan, not for enforcing serializability. The deterministic layer does the bookkeeping. The non-deterministic layer does the judgment. Each gets the job it's actually good at.

The case for eventual consistency with merge protocols in distributed agent memory

Eventual consistency is the right model for distributed agent memory. The gap MELD identifies is precise: no protocol lets two agents' memories link related facts held apart, or reconcile contradictory knowledge without silently discarding either claim. Strong consistency applied to knowledge stores would require a coordinator to adjudicate truth. That's semantically wrong. You don't want a lock manager deciding which agent's belief is correct.

MELD's approach treats each agent's memory as a versioned, typed-link knowledge graph. Every incoming claim goes through a five-outcome procedure: insert, merge, relate, conflict, or reject. The decision comes from three signals. Scoped claim-key identity, embedding similarity, and a natural-language-inference verdict. A per-claim status CRDT keeps sovereign brains coherent without a coordinator. And a detected contradiction is preserved for later adjudication, never silently resolved. That's the property last-writer-wins destroys.

The partition is the practical architecture decision. The evaluation ran on a real computing continuum spanning an operator-grade 5G edge, national HPC, and a local tier. CRDTs are the enabling primitive here. MELD's status CRDT is a concrete instance of that pattern applied to knowledge-graph claims.

What the infrastructure layer must provide: POSIX semantics, shared mounts, and elastic storage

Agent protocols still depend on the storage beneath them for a few things: core primitives, shared mounts, and elastic capacity. File locking is one: flock and fcntl make exclusive writes enforceable, but only where the filesystem honors them across every client that mounts it.

Atomic rename gives file-based workflows a single-step publish. BatchWeave gets the same effect on object storage by publishing each batch as a conditionally written new manifest version, not a multi-step update. Shared mounts across parallel agent runs are a prerequisite for persistent agent context. Agents need to work on the same source data across multiple conversations, with state surviving session boundaries. Local filesystems that disappear when serverless functions end fail this requirement outright.

Storage elasticity matters because agent workloads are unpredictable in volume. An agent may need a megabyte of working state or a gigabyte. Pre-provisioned block storage forces a choice that's wrong in one direction or the other. A CNCF blog post by Atlassian engineers treats shared training storage as a platform design problem, exposing a Lustre filesystem through a CSI driver so ML teams can run distributed training without managing the underlying infrastructure.

The cloud-agnostic data layer is the last piece. One where S3, GCS, R2, and Azure Blob all mount the same way, with no migration, no ETL pipeline, and no copies. A filesystem that mounts over an existing bucket and keeps the bucket as the source of truth gives agents a stable workspace without locking data to a storage vendor.

Choosing a consistency model for your agent workload: the decision the architecture actually forces

The consistency model choice for an agent filesystem is not a single decision. It's three separate decisions made at different layers of the stack, and conflating them is the source of most silent corruption in production multi-agent systems. Layer one is shared mutable state under concurrent writes. The filesystem layer. This requires strong consistency or agent-specific concurrency control like CoAgent-style MTPO. Eventual consistency here produces the gradient corruption and silent overwrite failures documented earlier. POSIX atomic rename and file locking are the primitives that make strong guarantees implementable.

Layer two is training data visibility across distributed ranks. This requires atomic all-rank batch visibility. BatchWeave's versioned-manifest approach achieves it without a coordination service in the critical path. Stale views from caches or lagging replicas are most dangerous at this layer, which is where 3FS's CRAQ-based strong consistency earns its complexity cost. Layer three is distributed agent knowledge and memory reconciliation. Eventual consistency with semantic merge, MELD-style CRDTs and five-outcome merge procedures, is correct here. Strong consistency would require a coordinator to adjudicate semantic truth, which is the wrong abstraction. Contradictions should be preserved, not resolved by the infrastructure.

The practical rule follows. Strong consistency belongs at the layer where a stale read causes incorrect computation. Eventual consistency with explicit merge belongs at the layer where divergent views represent genuinely different knowledge that must be reconciled semantically. Operational complexity is a real cost at layer one. 3FS is the right answer only for teams running AI at serious scale who can absorb its operational demands. Teams that cannot should verify which guarantees their storage layer actually provides across clients before building on them.

Google DeepMind's Decoupled DiLoCo, from April 2026, represents a deliberate relaxation of synchronous consistency across training islands in favor of resilience and multi-datacenter scale. The unresolved question remains. The choice is still yours, and it's not one you get to make once.