Sandboxed Persistent Volumes as the Right Primitive for Per-Tenant Agent Workspaces

For multi-tenant AI agent applications, the right way to keep each tenant’s storage isolated is a dedicated microVM backed by a persistent, forkable volume. The microVM separates each tenant’s execution, the volume preserves work across restarts, and cheap forks make parallel attempts and rollback practical. These must be one architectural decision.

Why per-tenant agent workspaces create a three-way architectural tension

In December 2025, a coding agent issued rm -rf tests/ patches/ plan/ ~/. The stray ~/ wiped all of the developer’s home directory. The command needed shell access without a boundary between the model’s decision and the developer’s files to cause damage. In a multi-tenant application, that boundary must also separate one customer’s mistakes from another customer’s data.

Isolation is only the first requirement. Coding and research agents build their environments incrementally through installed packages, intermediate files, and session context. If that work disappears at restart, the next invocation must repeat setup before continuing the task. Persistence therefore protects correctness as well as convenience, especially when later steps depend on earlier results.

The third requirement is cheap branching. Parallel attempts, speculative execution, and recovery from a bad action cost more when each branch needs a copy of the whole workspace. Ephemeral containers discard state, shared volumes weaken tenant separation, and full-copy snapshots make experimentation expensive as workspaces grow. The architecture has to preserve the work without preserving an agent’s ability to damage everything around it.

Why containers alone are not the right isolation boundary for agent-generated code

Standard Linux containers use namespaces and cgroups while sharing the host kernel. That shared kernel is the weak point. A container escape can reach the host and other tenants’ workloads, with the resulting exposure bounded by privileges. For reviewed application code, that can be an accepted tradeoff. With arbitrary model-generated code, the threat model is different.

An agent can be confidently wrong or quietly manipulated by text it reads. Destructive commands are the accidental case, while prompt injection, data exfiltration, credential theft, and resource abuse add adversarial paths. The familiar container configuration is therefore not automatically a sufficient boundary just because it works for ordinary services. The tenant next door should not inherit the consequences of a persuasive paragraph.

Meta’s Agents Rule of Two, published in October 2025, makes the conflict explicit. An agent session should have no more than two properties among processing untrustworthy input, accessing sensitive systems or private data, and changing state or communicating externally. A task requiring all three needs a fresh session or a human checkpoint. Stronger execution isolation limits what compromised code can reach beneath those controls. This is why a user-space kernel or microVM belongs in the agent architecture discussion.

The isolation spectrum: gVisor, microVMs, and where each is appropriate

Beyond standard containers, gVisor inserts a user-space application kernel that handles the container’s system calls. Most of those calls never reach the real host kernel. Google Cloud Run uses it in production, and Anthropic uses it for code execution in claude.ai on isolated infrastructure. It also integrates with Kubernetes through RuntimeClass without requiring a full VM boundary.

That middle ground has practical tradeoffs. Not every syscall or /proc file is implemented, and syscall-heavy or file-heavy workloads can run more slowly. Teams can use gVisor for routine execution while reserving microVMs for builds and arbitrary third-party code. The choice need not be all-or-nothing.

Firecracker gives each workload its own Linux kernel, so a guest-kernel compromise is separate from a host-kernel compromise. Archil operates this pattern in production for AI agents, with each tenant’s workload running in a distinct microVM. Firecracker’s low overhead makes this practical, while emerging work such as ZeroBoot and copy-on-write KVM fork demonstrates sub-millisecond VM startup. Those emerging results are not a promise about every deployed sandbox.

Nor is a microVM invulnerable. CVE-2026-5747 describes an out-of-bounds write into Firecracker’s VMM heap, with CVSS v4 8.7. Host code execution requires additional preconditions, such as a custom guest kernel or specific snapshot configurations. CVE-2026-1386, with CVSS v4 6.0, concerns arbitrary host file overwrite through a jailer symlink and is a host-local privilege escalation. Practical exploitation remains hard, but absolute claims about impossible escapes are not defensible.

Network and credential controls complete the boundary. The recommended pattern keeps credentials out of sandbox images and gives the sandbox only a session-scoped token, while a proxy enforces what it can reach. In deployments such as Archil’s, this proxy-based pattern complements the microVM. An isolated shell with unrestricted egress is still a very capable delivery service for stolen data.

Why durable, volume-backed persistence is the right complement to microVM isolation

A volume that outlives the microVM lets compute stop without erasing the agent’s work. Isolation and persistence serve different jobs. The execution boundary contains code, while the storage boundary preserves packages, files, and intermediate results for later sessions or pipeline stages. Keeping durable state outside ephemeral compute avoids rebuilding the environment or adding a sidecar solely to rehydrate it.

“Persistent” does not always mean the same thing. Some systems keep files, others keep process memory and running processes, and still others use snapshots to make new environments. Fly.io Sprites, launched January 2026, offers up to 100 GB of persistent POSIX storage and approximately 300ms checkpoint creation. Its entire filesystem persists across runs, which is a more precise claim than simply saying the environment remembers.

Windmill’s AI sandboxes, launched April 2026, take a synchronization approach. They download what is in the volume from workspace object storage before code runs, bind-mount the volume while it runs, and sync changed files back afterward. Names can interpolate workspace and argument values to give sessions separate storage. This fits workflows where persistence between executions is the key requirement.

For a continuously available file interface, Archil’s persistent sandboxes pair Firecracker microVMs with durable disks. Installed packages, intermediate work, and bash history can remain across invocations in a mounted POSIX filesystem with atomic operations. However, durability alone does not settle concurrent access. Separate durable local views in userspace-filesystem-based multi-agent synchronization can drift apart, so agents collaborating within a tenant need well-defined write semantics.

Giving agents file access to large object-store datasets without copying them into the sandbox

Object-store data creates a practical extension of that problem. When a dataset lives in S3, GCS, or R2, downloading it into every sandbox ties access back to the sandbox’s lifetime. Session resets can then discard the local copy. With a mounted file interface, the dataset stays where it is and agents can use the file tools they already know.

AWS launched S3 Files in April 2026 to address this mismatch. Its engineering teams had seen agents use tools such as Kiro and Claude Code that defaulted to local file access even when the data lived in S3. S3 Files connects EFS technology to S3, exposing a bucket as a shared NFS filesystem without migration. Both file and object APIs can still access the same data.

That access has a defined deployment boundary. S3 Files serves AWS compute resources within the same VPC, including EC2, Lambda, ECS, and EKS, and requires VPC, IAM, and mount-target setup. It is not a local or non-AWS mount service. The brief identifies no equivalent native-mount product from Google Cloud or Cloudflare R2.

For existing buckets across those stores, Archil disks expose S3, GCS, R2, Azure Blob, and S3-compatible storage through POSIX without migration. Writes are durable once fsync returns and reach the bucket in the background. Each disk also has an S3-compatible API, so other parts of the application can use object access while the agent uses files. Because models get so much training on bash and file tools, this saves the context that would go toward teaching them a custom data-access vocabulary.

Branching and checkpointing agent environments without full copies

Cheap branching changes how an agent can work. If a fork takes milliseconds regardless of workspace size, parallel attempts and rollback stop being exceptional operations. Full copies make each experiment pay rent on the whole environment, giving agents less reason to try. Production tools already address parts of this problem, but at different layers.

Git worktrees are a lightweight, production-ready option for coding agents. Each agent gets a branch and working directory, and they share the same underlying .git object store. Conflicts move to merge time, where standard Git tooling detects them, and storage cost scales with working file count independently of repository history. Worktrees do not, however, claim to checkpoint an entire running environment.

At the disk layer, Archil’s checkpoints and branches run in constant time regardless of disk size. Research systems explore broader state handling. BranchFS provides isolated copy-on-write workspaces with O(1) creation, atomic commit to the parent, and sibling invalidation. DeltaBox treats filesystem and process memory as a transactional, change-based pair, duplicating only changes between checkpoints. Its evaluations on software-engineering benchmarks and RL micro-benchmarks achieved millisecond-level checkpoint and rollback, but both systems belong in the research discussion.

Checkpoint completeness remains a separate engineering trap. One production bug forked the conversation journal during rewind but omitted the file checkpoint data. Switching back to the resulting branch always reached the “No file checkpoint” path. The conversation had travelled back in time. The files had missed the train. Atomic recovery across conversation and filesystem state remains unresolved in many agent frameworks.

How do you give each customer isolated storage at thousands of tenants?

Giving each tenant a dedicated sandbox and volume makes security easiest to reason about at scale. Each customer has a distinct execution environment and storage workspace, rather than separation resting entirely on software rules around a shared mount. That is the silo side of the silo-versus-pool decision. This is affordable only if storage can grow as tenants need it, even when demand is hard to predict, without large reserved allocations.

Billing therefore belongs in the architecture review. Archil’s 2026 pricing charges $0.20 per binary gigabyte-month for active data. Windmill’s interpolated volume names address a related operational burden by scoping storage through workspace and argument values, removing the need for manual provisioning for every tenant.

The same fork primitive supports more than parallel attempts. Tenant onboarding from a template, environment cloning for debugging, and rollback after a bad action can use branches. These operations retain the per-tenant model rather than making a shared workspace the shortcut for easier administration.

Bring-your-own-bucket also gives enterprise customers control over residency. Their storage can remain in their own S3, GCS, R2, or Azure Blob account, with access revocation leaving the source data in place without migration. Cache nodes run alongside compute, and moving data across regions needs to be explicitly configured. A security review still needs to name the mechanisms, spell out the egress policy, and state how secrets are handled. SOC 2 Type II and HIPAA compliance can narrow the field for regulated deployments, but “secure sandbox” alone is not an auditable architecture.

Why the microVM-plus-persistent-forkable-volume combination satisfies all three requirements where alternatives do not

An ephemeral container can offer useful separation while still discarding the environment at restart, and a standard shared-kernel container also falls short of the strongest execution boundary. A shared volume preserves work but exposes tenant separation to software-layer mistakes if multiple tenants can read the same mount. Full-copy snapshots can preserve isolated environments, yet their copying cost makes branching increasingly burdensome.

The combined primitive assigns each requirement to the right layer. The microVM separates guest kernels, the persistent volume survives compute replacement, and disk-layer copy-on-write makes forks cheap without duplicating the whole workspace. The file interface then lets existing agent tools operate on that durable state without a separate access language.

This is already a viable production pattern. Archil’s Firecracker infrastructure has handled over a year of production load, its persistent disks have been generally available since September 2026, and it serves millions of agents weekly.

The unfinished work sits at the seams. Checkpoints still need to capture conversation and filesystem state atomically, research systems need production hardening, and proxy-based credential handling and controlled egress are not universal. The architectural conclusion is clear, but the operational test remains specific. A workspace needs a verifiable execution boundary, durable write semantics, and a complete recovery contract (not merely three reassuring labels on a diagram).