Mmap on Cloud-Backed Filesystems for Agent Inference Workloads

Inference is quietly eating the entire AI compute budget. It's projected to grow from a $106 billion market in 2025 to $255 billion by 2030, and by next year it's expected to make up roughly two-thirds of all AI compute, up from a third just a few years earlier. That shift changes what infrastructure gets built for, and one detail that determines whether it works is whether the runtime loading your model weights can use mmap the way it was designed to.

Training jobs care about throughput. Run the batch, get the result, move on. Inference doesn't get that luxury: it's a live service that has to answer fast, stay up, and do it again a few milliseconds later for the next request. Some estimates put inference at 80 to 90 percent of a production AI system's total lifetime cost, mostly because it never really stops running. And now that reasoning models are in the mix, chewing through far more compute and tokens per answer than a standard chat response, the line between "training-scale workload" and "inference-scale workload" is getting blurry fast.

Agents make this worse in a specific way. An agent isn't one request, it's a chain of them, often running in parallel across sessions, each one needing the same model weights available immediately. Where those weights physically live, on local disk, over a network mount, or behind a cloud API pretending to be a filesystem, decides whether every step in that chain pays a tax just to get started. That's the thread this piece pulls on.

What mmap does during model loading and why runtimes depend on it

Mmap is short for memory map, and the basic idea is almost stubbornly simple: instead of copying a file into a buffer, the operating system just points to it. It reserves a chunk of virtual address space, and only pulls in the actual pages of data when something goes to read them. Nothing gets front-loaded. Nothing gets eagerly copied between kernel and user space. If your inference process touches every single weight in the model (which, to be fair, it usually does), all those bytes still have to come from storage eventually. Mmap doesn't reduce how much gets read. It changes when and how.

That distinction is exactly why llama.cpp uses mmap by default. Large models don't have to fit entirely in RAM up front, because the OS pages weights in and out as needed, and when memory gets tight, it evicts pages instead of killing your process. That's the difference between a graceful slowdown and a crash. Research on warm-up behavior backs this up too: once a model's pages are already sitting in memory from a prior run, subsequent inference reuses them, and latency per request drops.

There's a multiplayer bonus here as well. When the OS maps a file rather than copying it, memory pressure is managed at the page level, which can help when multiple processes need access to the same weights.

The apparent magic of mmap has a cost: it isn't a performance free lunch. It trades a big memory spike at load time for a bunch of small page-fault delays at runtime. During decode, especially with mixture-of-experts style models, weight access is anything but sequential, so the page fault handler ends up firing over and over for pages that aren't currently resident. Llama.cpp actually gives you an escape hatch here, a --no-mmap flag that loads the whole model into RAM before inference starts. No faults, no jitter, but you pay for it with a bigger memory footprint and a slower first token.

Why cloud object storage cannot support mmap natively and what gets inserted in between

Here's the wrinkle: everything above assumes mmap is talking to a real filesystem, of a conventional flavor, sitting on a local block device. Object storage like S3, GCS, and Azure Blob doesn't work that way at all. It's an HTTP API. You ask for a range of bytes with a GET request, and it hands them back. There's no concept of a byte-addressable file you can randomly seek into at the block level, because there's no block level to speak of.

So mmap, strictly speaking, has nothing to talk to. Something has to sit in between and pretend otherwise, and that something is usually a FUSE filesystem. Tools like gcsfuse or s3fs-fuse mount a storage bucket at a local path, so from the point of view of llama.cpp or a Safetensors loader, it looks like an ordinary folder on disk. Underneath, every filesystem call gets intercepted and translated into an object storage request.

Translation isn't free. mmap tends to generate a flurry of small, scattered reads as it maps different tensor regions, and each of those reads, once translated through FUSE, becomes its own network round trip. Model files streamed this way over HTTP, without prefetching or parallel reads, can take an order of magnitude longer to load than they would off local disk. And network conditions aren't always polite about it: models have reportedly failed to load entirely on services like Cloud Run because gcsfuse hit latency spikes north of 30 seconds.

s3fs-fuse has its own, separate problem, and it's a structural one rather than a tuning issue. It leans on C++ STL maps to cache the results of filesystem operations, and those maps are memory-hungry. Handling around 100 million objects can require roughly 32 gigabytes of DRAM per map, and with several maps in play, that adds up to something like 128 gigabytes just to track metadata. AI storage setups routinely deal in the billions of small objects rather than the tens of millions, making this a mainstream case. It's a wall.

How mmap access patterns interact with cloud latency during decode

Knowing that FUSE adds overhead is step one. What matters more is what that overhead actually feels like once tokens start generating.

During decode, weight access, again, especially for expert layers, is scattered rather than sequential. A page fault on local NVMe resolves in microseconds, practically instant. The same fault, routed through FUSE to an object store, can take milliseconds, sometimes tens of milliseconds, because it's now waiting on a network round trip instead of a disk seek. Every one of those unresolved faults stalls the decode step that triggered it, and across a long generation, that jitter piles up token by token.

The worst-case numbers here are genuinely rough. Experimental results have shown mmap dropping to somewhere between 0.08 and 0.67 tokens per second when parameters can't stay warm in memory, meaning they get swapped out before the next time they're needed and have to be re-fetched from storage synchronously for every decode step. That's about as bad as this pattern gets. An alternative system called FlexInfer, tested against that same scenario, showed a 5 to 12-fold improvement, which gives some sense of how wide the gap is between mmap under cold-storage conditions and something built to handle it.

Cold starts compound this for agents specifically. With a model snapshot sitting on local NVMe, cold start can land in the 2 to 5 second range for a 7 to 13 billion parameter model. A 70 billion parameter model at full precision is roughly 140 gigabytes, and even on fast local NVMe running at 3.5 gigabytes per second, restoring that takes around 40 seconds on its own. Cloud-backed paths that can't match local NVMe bandwidth stretch that further, and agents that spin up new sessions in parallel just multiply the bill.

The real distinction isn't warm versus cold in the abstract, it's which one an agent actually experiences. Once the OS page cache holds the working set, mmap on a cloud-backed mount can behave reasonably well. But every new replica, every restart, every fresh agent session that hits a weight not already cached pays full price. And agents, by their nature, running repeatedly, in parallel, across sessions, are structurally set up to hit that cold path constantly rather than occasionally. One study of high-severity inference incidents found that roughly 60 percent traced back to failures in the inference engine itself, and about 40 percent of those were timeouts. Slow model loading over a cloud-backed path is a plausible contributor sitting right there in that timeout bucket.

How model format choice changes the mmap I/O pattern hitting a cloud-backed filesystem

Two file formats dominate here, and they behave very differently once FUSE gets involved. Safetensors is the Hugging Face standard: no pickle serialization, supports memory-mapped loading, stores weights efficiently. GGUF, a single-file binary format built specifically with mmap compatibility in mind, is the format of choice for local and edge inference. A typical pattern is to store Safetensors on the model hub and serve using a format optimized for local inference.

Safetensors has a sharding habit that doesn't play nicely with object storage. Large models get split into multiple files, and each shard is its own separate object in the bucket. Mmap has to be issued per shard, which means every shard is its own open-and-map operation through FUSE, each one incurring its own network round trip. Documentation on this pattern is blunt about it: Safetensors' reliance on mmap produces frequent, random reads across shards, and in a cloud storage setting, that turns into repeated round-trip penalties.

GGUF sidesteps a chunk of this simply by being one file. One object open, one mmap call, and metadata overhead is contained to a single object rather than multiplying across shards. Quantization inside GGUF adds another layer of savings: dropping a Llama 3 70B model to 4-bit precision shrinks it substantially from its roughly 140-gigabyte FP16 footprint. Fewer bytes on disk means fewer bytes that need to cross the FUSE boundary on a cold start.

Google Cloud's own guidance leans the same direction, calling 4-bit quantization "the ultimate cold start hack," since smaller weights directly speed up the download portion of loading. Moving off pickle files toward Safetensors offers zero-copy loading benefits, though on a cloud-backed path the shard-multiplication cost doesn't go away just because the serialization format improved.

Format sits upstream of basically everything else discussed here. Pick the wrong one for a cloud-backed deployment, and no amount of caching or infrastructure tuning downstream fully makes up for the overhead that a better format choice would have avoided in the first place.

Infrastructure approaches that change what mmap actually reads from

Once format is sorted, the remaining question is what to do about the network latency that's still baked into the path. Broadly, there are two moves: skip the FUSE layer entirely, or put something faster in front of it.

Skipping FUSE means using a runtime that talks to object storage directly. SGLang, for example, supports loading weights straight from s3://, gs://, and az:// URIs through a native streaming loader, pulling bytes from the object store without ever pretending there's a POSIX filesystem in the middle. That removes the FUSE translation tax completely for the model-loading step. This only works if the runtime has actually built that native path. Llama.cpp, and most tools that assume a standard filesystem mount, can't take advantage of it without real modification.

The other approach is a caching layer that sits between the FUSE mount and the object store, keeping frequently used model files closer to the compute that needs them. The first mmap call still pays the network cost, but once that cache is warm, subsequent calls, from new replicas or repeated agent sessions, hit the cache instead of round-tripping to the bucket every time. That reframes the cold-path cost from something paid on every single load to something paid once, which matters enormously for agent workloads specifically, since the same weights get hit again and again across parallel runs rather than just once per job.

Format choice narrows the I/O pattern. Infrastructure choice decides what's actually sitting behind that pattern when the page fault fires. Get both wrong, and mmap quietly turns from the fast path it was designed to be into the exact bottleneck it was built to avoid.