Torn Writes in Cloud Storage and Agent Checkpoint Corruption

Torn writes happen when a machine crashes mid-write and leaves behind a file that's part old data, part new data, stitched together into something that looks fine and isn't. Cloud object storage makes this worse, not better, because S3 and GCS don't give you the same atomicity guarantees a real filesystem does. That gap is where agent checkpoints go to die quietly, and nobody notices until someone hits resume.

Why object storage cannot substitute for filesystem atomicity during a checkpoint write

Object storage was built for a different job: huge sequential reads, batch jobs, pulling one big file and moving on. It was never built for the fast, bursty, small writes that checkpointing throws at it every few minutes during a training run.

There's no atomic rename on S3 or GCS. On a real filesystem, renaming a file is instant and safe: the new name exists or it doesn't. On object storage, a "rename" is a copy followed by a delete, which doubles the I/O cost every time a checkpoint finalizes. Append operations and write-ahead log patterns don't fare any better. Some providers support them halfway, some not at all, so frameworks end up bolting on extra round trips just to fake behavior they should get for free.

Then there's latency. A standard S3 PUT incurs meaningful per-operation latency. That sounds trivial until hundreds of training ranks try to checkpoint at the same moment, and that per-operation delay turns into one coordinated stall across the whole cluster. Pile enough requests onto a single bucket or prefix, and S3's own rate limiting throttles the very burst you needed to finish fast.

Object storage skips every tool checkpointing frameworks were built around: no atomic rename, no fsync, no flock. Write a checkpoint straight to raw object storage, and there's a window where a partial write survives as a perfectly normal-looking object, with nothing anywhere flagging that something went wrong.

How a corrupt checkpoint goes undetected until an agent tries to resume

A crash mid-write doesn't throw an error. It produces a file. The loader opens it, sees a valid header, and calls it good, because nothing about the file format screams "broken" from the outside. The corruption sits quietly in the interior state, invisible until an agent actually tries to use that state to resume work.

That detection gap can stretch for days. By the time anyone notices, the corrupted checkpoint might be the newest one on record, which is the worst possible moment to find out it's garbage.

A training interruption reported by Introl is a useful example. Silent corruption sat undetected for days. Basic validation could have caught it earlier, and the existence of older checkpoint versions offered a potential path to recovery. That incident illustrates why checkpoint verification matters. The same Introl piece attributes an $8.6 million figure to a GPT-4 training incident, tied to wasted compute and a two-week product delay rather than hardware replacement; that number should be checked against a primary source before being treated as settled fact.

Hardware adds its own bad luck. Meta's fleet analysis found roughly 1 in 1,000 machines affected by silent data corruption, and at large training scale, expect an SDC event every one to two weeks. Soft error rates climbed hard as chips shrank, reaching one failure per 1.5 hours at 16 nanometers. Torn writes and hardware SDC come from completely different places, but they land on the same symptom: a checkpoint that opens fine and lies to you.

The scale at which checkpoint failures compound: what production data shows

Meta's Llama 3 training run puts real numbers on this. Across 16,000 accelerators over 54 days, the run logged 419 failures, roughly one hardware failure every three hours (per the Alluxio guide citing the Llama 3 paper). At that rate, frequent checkpointing is the only thing standing between the team and losing weeks of compute.

The math gets worse, not better, as clusters grow. Mean time between failures shrinks inversely with cluster size, so bigger runs break more often, not less. Anyone assuming a large training job runs more stable than a small one has that relationship backwards.

Checkpoint size sharpens the exposure further. Introl reports that 70-billion-parameter model checkpoints run 150 to 200 gigabytes, and optimizer state tacks on roughly a 4x multiplier over the raw weights alone. Scaling up to trillion-parameter models means a single checkpoint event can hit 15 terabytes or more, amounting to a small data center's worth of state trying to land safely in one shot. That's a small data center's worth of state trying to land safely in one shot, not a file anymore. That's a small data center's worth of state trying to land safely in one shot.

A 2025 to 2026 study on a B200 cluster (63 nodes, 504 GPUs, running Solar Open for 55 days) analyzed 523 checkpoint events and found something odd: measured throughput averaged only 21.5% of maximum read bandwidth and 16.0% of write bandwidth. Storage sat underused, and not for lack of capacity. The bottleneck was latency and contention on the write path itself, which happens to be exactly where torn writes live. MLPerf Storage v2.0 took notice, adding a checkpointing workload to measure backup and recovery speed at LLM scale, something earlier versions (v0.5, v1.1) never bothered testing since they only benchmarked training data ingestion.

The storage properties that prevent torn-write corruption in checkpoints

Preventing a torn write comes down to a short list of properties. Skipping any one of them reopens the door.

Writes need to be atomic: fully complete or fully absent, with no partial object ever visible to a reader, under any failure condition. Finalization needs to be atomic too. The rename or commit step that makes a checkpoint official has to land as one clean move, so a reader sees the full new checkpoint or the previous one, never a mix of both. Durability has to come before acknowledgment, since a storage layer that says "done" before the write hits stable media just reopens the torn-write window it was supposed to close. And ordering matters: if checkpoint data has to land before the pointer that references it, that sequence has to hold even when things fail midway.

POSIX filesystems already give you all of this, through atomic rename, fsync, and flock/fcntl. Checkpointing frameworks like PyTorch Distributed Checkpoint, DeepSpeed, and Megatron-LM were built assuming a POSIX-compliant shared filesystem underneath them, because POSIX filesystems supply atomic rename, fsync, and flock/fcntl directly, and that assumption is visible in how those frameworks fail when the filesystem beneath them isn't POSIX-compliant. Modern tooling, including the Amazon S3 Connector for PyTorch and Megatron-LM's Multi-Storage Client, now lets these frameworks talk to object storage directly. Talking to object storage directly doesn't teach it POSIX manners, though. Raw object storage still can't offer atomic rename or an fsync equivalent on its own, full stop.

The fix that actually works: put a POSIX filesystem layer on top of the object store, something that mounts S3, GCS, R2, or Azure Blob and behaves like a real filesystem, translating POSIX calls into safe operation sequences against the bucket. Pair that with a local NVMe cache that absorbs the write burst and returns acknowledgment before the slow S3 PUT even happens, flushing asynchronously afterward. That pulls the latency clean out of the critical path without giving up durability. The object store still holds the job of source of truth. It stops standing between a training run and a torn write once the S3 PUT happens, flushing asynchronously afterward, because the object store still holds the job of source of truth.

What infrastructure teams have shipped to close the gap

Google Cloud announced two relevant pieces at Google Cloud Next '26 in April 2026, and they solve different problems.

Rapid Bucket runs on Google's Colossus distributed storage system and delivers more than 15 terabytes per second of bandwidth, 20 million requests per second, and sub-millisecond latency inside a single zonal bucket. Checkpoint restores run 5x faster and writes run 3.2x faster than traditional object storage, with GPU blocked time cut in half. It plugs into PyTorch and JAX through both a high-performance gRPC path and an S3-compatible API. Rapid Cache (a rename of the old Anywhere Cache) adds 2.5 terabytes per second of aggregate read throughput to existing buckets with zero code changes, plus an ingest-on-write feature that caches data the moment it's written, enabling up to 2.2x faster restores. Both are genuinely fast. Neither one, on its own, hands training frameworks the POSIX atomic rename they're missing when writing through the S3-compatible path. Speed and atomicity are not the same fix, and it's easy to buy the former thinking it solves the latter.

Google Cloud Managed Lustre, announced at the same event, is built on DDN's Lustre expertise and EXAScaler technology, pushing up to 10 terabytes per second of throughput and writing or restoring checkpoints 2.6x faster than other Google Cloud storage options. Its Dynamic Tier prices at $0.06 per gigabyte-month and serves data straight from persistent disk instead of an object-based cache, removing the performance cliff that caching architectures sometimes hit. Because it's a real parallel filesystem, POSIX semantics come baked in, atomic operations included. Google Cloud reports it cuts mean time to first token by more than 40% versus keeping a KV cache in host memory alone, and a 75% improvement in total inference throughput. Checkpointing frameworks run on it unmodified, no workaround required. Of the two announcements, this is the one that actually closes the atomicity gap rather than just widening the pipe.

Write-back caching architectures follow a related idea: write checkpoint data to a local cache, acknowledge the write immediately, flush to the object store later. That kills the 30 to 40 millisecond PUT latency sitting in the critical path and skips the copy-plus-delete rename, since data lands at its final path the first time. Whether a given cache layer actually provides atomic rename and fsync itself falls straight through to the object store underneath, unresolved, unless checked case by case.

The broader pattern ties these ideas together: a POSIX cloud filesystem mounted over S3, GCS, R2, or Azure Blob, with no data migration and no changes to training code. NVMe cache absorbs the burst at sub-millisecond speed, writes get replicated before acknowledgment, and an asynchronous flush keeps the bucket as the durable source of truth. That setup provides atomic rename, fsync, flock/fcntl, hard links, symlinks, the full toolkit checkpointing frameworks expect. Capacity gets billed on active cache rather than a fixed provisioned volume, which suits checkpoint workloads well since demand spikes at each interval and sits idle the rest of the time. And because the filesystem interface stays the same no matter which cloud's object store sits behind it, the checkpointing framework never has to know or care which cloud it's talking to.

IBM Storage Scale hit 656.7 GiB/s in reads training a 1-trillion-parameter model in MLPerf Storage v2.0 results, a useful marker for what parallel filesystem infrastructure can deliver at that size.

Run every option through the same test: does it provide atomic write finalization, POSIX rename, fsync, and burst absorption? A setup that only improves throughput cuts down stall time. On its own, that does nothing to make the torn-write risk disappear, and mistaking one for the other is the most common buying mistake in this space.

Verifying checkpoint integrity when the storage layer cannot be fully trusted

Storage-layer fixes handle one half of the problem. Validation matters just as much, because a corrupt checkpoint that loads without error sails straight through any filesystem-level check. Only something that actually looks at the content catches corruption sitting inside the file.

The incident covered by Introl drove that lesson home: basic validation could have caught the corruption far earlier. The practice that spread industry-wide afterward was simple. Validate on write, and validate again on load.

A few concrete habits make that real. Checksum the checkpoint object the moment it's written, then verify that checksum before ever loading it, which catches both torn writes and hardware-level SDC in one move. Beyond that, check a sentinel value or a known-good slice of parameters after loading, since a checksum computed over a partial write can pass clean even when the underlying data is garbage. Keep multiple checkpoint generations around, too, so if corruption appears when a run resumes, there's a fallback to the previous version instead of restarting the whole run. Log write duration and object size for every checkpoint: an unusually short write time or a smaller-than-expected object is often the earliest sign a write got torn.

Storage-layer atomicity and validation aren't substitutes for each other. If the storage layer guarantees atomic finalization, a checkpoint is either fully written or simply absent, and validation is now catching hardware SDC rather than a storage tear. The two defenses cover different failure sources that happen to produce the identical symptom, and skipping either one leaves the other doing a job it wasn't built for.

Agents make the stakes sharper. An agent resuming from a broken checkpoint doesn't necessarily crash. It might just keep running, quietly producing wrong outputs for an entire session before anyone catches on, which is a far worse outcome than a clean failure would be. Validating before resume beats waiting for a crash to announce the problem. Reliable checkpointing needs a storage layer built for atomicity and an application layer that checks its own work. Skipping either one, at production scale, opens exactly the gap where things go missing.