Torn Writes in Cloud Storage and Agent Checkpoint Corruption

A checkpoint sounds like a save button. Hit it, the system freezes a moment in time, and training picks back up like nothing happened. That mental model is wrong, and it matters: a checkpoint is not one file, but a sprawling, multi-part snapshot of a distributed system, and that sprawl is what makes partial writes so dangerous.

Research presented at SCA/HPCAsia 2026 calls checkpointing a "big data problem" in its own right, with volume, variety, and velocity all working against you at once. Start with volume. A model checkpoint at 16-bit precision runs about 12 bytes per parameter, compared to the 2 bytes you'd see in a serving copy of the same model. The extra 10 bytes carry the optimizer state, the math needed to pick training back up exactly where it left off. Skipping that state means restarting rather than resuming.

Then there's variety. Modern training spreads a model across thousands of GPUs using 3D parallelism (tensor, pipeline, and data parallelism stacked together), and each checkpoint event has to gather weights, optimizer states, and metadata from all of them at once. That produces hundreds to thousands of separate files per checkpoint, hitting whatever storage system sits underneath with a pile of simultaneous metadata requests.

Velocity closes the loop. Most of a training run is read-heavy. GPUs chew through tokenized input in small batches, and Glenn Lockwood's analysis at blocksandfiles.com points out that per-GPU data consumption over a long run is actually modest. Checkpointing flips that pattern on its head: every GPU in the cluster stops computing at once and tries to flush a huge burst of data to storage in a short window. That burst, large objects, many files, all written at the same time, is what turns a structural quirk into a live hazard. The checkpoint window is the moment a crash does the most damage, because it's the one moment everything is in motion at once.

Why object storage does not prevent torn writes

A torn write happens when a crash lands in the middle of a write, leaving an object that looks finished but only holds part of what it was supposed to. The file loads. It just loads wrong.

Most training frameworks write checkpoints with ordinary file operations that don't guarantee durability. A crash mid-write can leave behind a file that opens cleanly and runs fine for a while, then quietly feeds the model bad numbers. Across a checkpoint made of hundreds of files, some objects finish writing and some don't, and the resulting mix is internally inconsistent: not broken enough to throw an error, not whole enough to trust.

Object storage does guarantee atomicity, but only at the level of a single object. Each file either writes or it doesn't. The trouble is that a checkpoint isn't one object, it's hundreds or thousands of them, written by different processes at the same time, and nothing lower in the stack is watching to confirm that all of them landed together. Per-object atomicity is a real guarantee and a useful one. It simply stops at the object boundary, and a checkpoint's boundary is much wider than that.

The scariest version of this failure is the quiet one. A torn checkpoint often loads without complaint, and training proceeds on top of it for a long stretch before anyone notices the trajectory has gone strange. By the time someone traces the problem back, several more checkpoints may have been built on the bad one.

Silent data corruption gets filed under the same symptom through a different cause. The SCA/HPCAsia 2026 paper notes that checkpoints are the primary recovery mechanism for unstable training runs, including the sudden loss spikes that occur in large training jobs. Checkpoints are the actual defense against fault tolerance, not a side benefit bolted onto it. A 2026 paper out of Technische Universität Berlin (Altenbernd, Wiesner, and Kao, arXiv:2604.00726) confirms that as models scale up, hardware faults can slip past system-level detection and corrupt gradients and weights without tripping any alarm. A torn write and silent data corruption produce the same outward sign, a checkpoint that looks fine and isn't, coming from two different directions. That overlap makes it harder to tell, after the fact, which failure actually happened.

Why the checkpoint window has the highest GPU costs

A torn write doesn't just ruin a file. It can erase the only record of days or weeks of GPU compute, because the checkpoint is the sole proof that any of that work happened.

Multi-GPU, multi-node training jobs run at a steep daily cost, and a single unplanned failure without a clean checkpoint to fall back on burns through the entire GPU budget spent since the last good save. Whether the checkpoint write happens synchronously (the training loop just stops and waits) or through a background thread racing to flush state, storage latency during that window turns directly into idle accelerator time. Thousands of GPUs sitting there, paid for, doing nothing.

Cloud object storage can stretch a multi-gigabyte checkpoint operation out to several minutes, and fault tolerance demands frequent checkpoints, so this isn't a one-time cost. It repeats every cycle, compounding across a long run.

This isn't a hypothetical risk living only in academic papers. The broader industry has reached the same conclusion from a different angle: Omdia's September 2025 analysis of the AI performance storage tier names checkpointing overhead as a load the right storage tier has to absorb, with GPU starvation as the bill that comes due when it can't. A torn write, in financial terms, converts paid compute time into waste.

How the underlying I/O architecture determines torn-write exposure

The gap between what object storage promises and what checkpoint safety actually needs is a mismatch between two different consistency models, and closing it means adding durability guarantees somewhere above the storage layer, in the application or the framework.

The SCA/HPCAsia 2026 research frames the core challenge as a journey through several tiers of hardware: GPU memory, host memory, local storage, and finally some external, remote repository, with orders of magnitude of performance difference between each stop along the way. Every handoff between tiers is another place a crash can land badly.

One specific finding from that research: small, uncoalesced write operations can cut throughput in half compared to clean synthetic benchmarks. Aggregating those writes into larger, batched operations restores most of that lost bandwidth and cuts down on metadata overhead, and the same research reports substantially higher throughput than existing checkpointing engines like DataStates-LLM, with a sizable edge over TorchSnapshot as well. This matters for torn-write risk specifically, because a slower write holds a wider window open. A crash during a drawn-out, poorly batched write has more time to land mid-operation. Faster, coalesced writes shrink that window and with it the odds of catching a crash in the act.

Making a checkpoint write fully crash-consistent is not free, and the costs have actually been measured. A 2025 study on crash-consistent checkpointing (arXiv:2511.18323) built three write modes with increasing durability guarantees on the APFS file system, then ran controlled crash-injection and corruption-injection tests against all three. An integrity guard layered on top caught nearly every injected corruption with zero false alarms. The atomic write modes ran anywhere from a moderate markup to several times the cost of the unsafe baseline, depending on the mode. That's a real number on a real trade-off between safety and speed, not a rounding error.

That trade-off is why asynchronous checkpointing has become the default move. Flush state from GPU memory to CPU memory fast, let training pick back up right away, and push the data out to remote storage in the background. Teams accept a window of exposure in exchange for keeping accelerators busy instead of idle.

Whether object storage belongs in this picture at all is a live argument, not a settled one. Lockwood's analysis argues that large, sequential writes, the kind checkpointing actually produces, suit object stores well. The counter-argument says bucket-level throughput on object storage can't keep up with active checkpoint writes at real scale, and that parallel file systems still have a role to play. The honest answer depends on cluster size, how often checkpoints run, and how much asynchronous risk a team is willing to carry. One detail narrows the problem, though: unlike a typical HPC job, a large LLM checkpoint is the same size whether it's trained on a handful of nodes or many thousands, because global synchronization keeps every data-parallel copy identical. Torn-write risk per checkpoint event stays bounded even as fleet size and checkpoint frequency climb.

Systems the research community has built to address these gaps

The field has taken torn writes seriously enough to produce several purpose-built systems, each chipping away at a different piece: shrinking the write window, verifying integrity after the fact, or moving the durability burden off the main training loop.

Asynchronous distributed checkpoints are the current standard for large runs. Each node writes its own slice of state, flushes quickly from GPU memory to CPU memory, and lets training resume almost immediately while the actual move to remote storage happens in the background.

ByteCheckpoint, built by ByteDance and published at NSDI 2025, is a unified checkpointing system built for large foundation model training. It takes direct aim at the heterogeneity and scale problems the SCA/HPCAsia research laid out: many file types, many parallelism strategies, one coherent system trying to manage all of it.

Checkmate, described in a 2026 NSDI paper, makes the boldest claim in the field: zero performance overhead, achieved through network gradient replication instead of writing a full checkpoint to storage.

The integrity-guard approach from the arXiv:2511.18323 study takes a different angle. It doesn't try to prevent corruption, it catches it, and it does that job well (near-perfect detection, no false positives). The overhead involved, though, points to a role as a monitoring layer rather than a full durability strategy on its own.

None of these systems actually eliminates the torn-write hazard sitting down in the storage layer. Each one relocates it or puts a boundary around it. Asynchronous checkpointing still leaves a gap between the CPU-memory flush and the eventual remote commit, and a crash landing in that gap still costs the state that hadn't made it out yet. Integrity guards catch damage after it's done without stopping it from happening, which makes them useful for spotting silent data corruption but doesn't change what the storage layer actually commits during the write. Every one of these approaches is compensating for something storage itself doesn't guarantee: atomicity across an entire checkpoint, not just within one object.

What storage infrastructure can do that application-layer workarounds cannot

Application-layer systems narrow the torn-write window. They can't close it, because the size of that window is set by how fast and how reliably the storage underneath can move data, and that makes storage architecture a central design decision, not a detail to sort out later.

Google Cloud's Next announcements state that storage is "no longer just a place to keep data," it's "the engine that feeds data-hungry accelerators," and how well it performs decides whether accelerators stay busy or sit idle. Google Cloud Storage Rapid Bucket, now generally available, delivers very high bandwidth, millions of requests per second, and sub-millisecond latency inside a single zonal bucket. Google reports that Cloud Storage Rapid, announced in May 2026, makes checkpoint writes substantially faster than traditional object storage, with restores substantially faster as well. Google Cloud Managed Lustre has also picked up a large throughput jump over the prior year, with both checkpoint writes and restores meaningfully quicker, and Google describes it as the most performant managed Lustre offering available in any cloud at the time of the announcement. A feature called Rapid Cache writes data into cache at the same moment it's written to a bucket, which removes the usual cache-miss penalty on first read, a detail that matters directly for how fast a team can restore from a good checkpoint after a torn write forces a rollback. CoreWeave's AI Object Storage takes a related approach, built around data mobility for GPU-driven workloads, with direct integration into Weights & Biases (which CoreWeave recently acquired) for storing and pulling back training artifacts.

Speed solves half the problem. Giving the training process a way to express atomicity in the first place matters just as much as moving bytes faster. Full POSIX semantics, atomic rename, flock and fcntl, mmap, fsync, hard links, symlinks, are the operations existing checkpointing code already assumes it has. A storage layer that supports them lets current frameworks run as written, with no rewrite needed. Atomic rename carries particular weight here: write the new checkpoint to a temporary file, then rename it into place. That rename is a single operation. It either completes or it doesn't, and there's no in-between state for a reader to stumble into.

Speed and semantics together shrink the torn-write window from both directions at once: less time spent writing, and a cleaner mechanism for marking a write as done. Application-layer tricks can manage around a slow, semantics-poor storage layer for a while. They can't substitute for a storage layer built to close the gap.