Working Directory Conventions for Reproducible Agent Runs

Isolated per-tenant storage for multi-tenant AI agent applications needs a durable workspace for each tenant, a real sandbox boundary, and clear controls for the network and credentials. Reproducible runs then depend on how that workspace is laid out, its persistence guarantee, and how it behaves on a fork, as well as pinned models or tool settings.

Why the working directory is the actual reproducibility boundary for agent runs

An agent accumulates more than its final answer. Crawled datasets, installed packages, conversation history, and warm caches become the context that makes its next action possible. When a sandbox expires or is garbage-collected, that context can disappear with it. Building it again uses up the compute budget and engineers’ time to redo work. A pinned model cannot bring back a missing dependency tree.

That is why the working directory defines the practical reproducibility boundary. A run that succeeds once is not reliably resumable if its files vanish, its paths change, or a fork begins with different state. The gap between a sandbox that runs code and one that remembers its work becomes a production constraint. Three questions expose that gap early. Does the directory survive a restart, who else can read it, and how fast can it be forked for parallel attempts?

What "persistence" actually means for a working directory, and why the distinction matters

Filesystem persistence means files and installed packages survive stops and restarts. With process-state persistence, memory and running processes survive too. A coding agent halfway through a dataset needs to keep its dependency tree and files. An agent waiting for human approval also needs its process state preserved, or an explicit checkpoint from which it can resume.

The storage lifecycle introduces a second distinction. Sandbox-level snapshots retain state while the sandbox has not been deleted, which can support standby without guaranteeing long-term retention. Durable volumes survive sandbox deletion, but only files written to the volume receive that guarantee. In a volume-only design, files outside the mount remain ephemeral and may be lost when the sandbox pauses. “Persistent” is therefore the start of a question about the storage specification.

How current sandbox platforms differ in what they actually preserve across sessions

The platforms teams evaluate in 2026 make different architectural promises. Fly.io Sprites uses Firecracker microVMs with persistent, solid-state-drive-backed ext4 storage per sandbox. Its checkpoints take about 300ms to create, and restore reconnects in roughly a second. Compute charges stop while the sandbox is idle, but storage is still billed. This works more like a persistent development machine than a slot that is thrown away after code runs.

Archil preserves both disk and kernel state across restarts, so installed packages, files, and running processes can persist without explicit snapshots. Its durable, forkable disks attach to Firecracker microVMs. A developer must still understand the lifecycle promised for the disk, the process, and the sandbox itself.

Other platforms put different capabilities at the center. CodeSandbox uses a snapshot-and-fork model suited to reusable environments and has no fixed session time limit. Vercel Sandbox provides SDK-managed microVMs with configurable sessions and snapshots, through a standalone SDK usable from non-Vercel environments too. Cloudflare Sandbox exposes command execution, file operations, background processes, and application previews from Workers. Those interfaces describe how work runs, but do not by themselves answer every retention question.

The developer’s job is to turn those promises into a directory contract. For volume-only persistence, the volume mount must be the durable root. Anything written elsewhere sits outside that guarantee. Choosing storage is only half the design.

Directory layout conventions that make persistence guarantees usable in practice

To make that contract usable, place a structured handoff file such as PROGRESS.md at the project root and pair it with an append-only event log. A resumed session should begin with pwd, read the handoff file, and review the tests. This gives the agent a known starting point. Context can then be discovered from the directory.

Put repositories, installed dependencies, generated outputs, and handoff files under the volume mount. Keep a clearly labeled temporary directory outside it for work that need not survive a pause. The distinction should be visible in the tree. Otherwise, an agent can lose intermediate outputs and leave behind a saved state that looks complete but cannot reproduce the run.

Keep paths relative to a project root that stays the same. Hard-coded absolute paths can break when a directory is forked or mounted on another machine, even when every required file survives. Commit to git before each major step for a lightweight recovery point that works without platform snapshots. Then use file-permission middleware to set the path read/write scope for each session. That scope limits workspace drift, where unrelated edits undermine the state the handoff file and checkpoint are supposed to describe.

Isolation model and egress policy as prerequisites for a trustworthy working directory

For each customer’s workspace to be trustworthy, its isolation boundary must hold. Rules for directories cannot keep files safe if agent-generated code can reach the host kernel, neighboring tenants, or node credentials through an instance metadata service. A separate folder is not a separate security boundary. The design needs per-tenant storage along with workload isolation, limits on the network, and controls on how secrets are handled.

Standard Linux containers share the host kernel and are acceptable only for trusted code under this threat model. CVE-2024-21626, the “Leaky Vessels” escape, illustrates how a runtime vulnerability can threaten co-tenants. gVisor instead intercepts syscalls through Sentry, its Go-implemented Linux-compatible kernel, without exposing the host kernel to agent code. That adds moderate overhead for heavy I/O and syscall-intensive work. Northflank Cloud uses it for GPU sandboxes.

MicroVM approaches, including Firecracker and Kata Containers, give workloads dedicated kernels through hardware virtualization. They provide the strongest boundary among these approaches, with fast startup and minimal memory overhead per microVM. However, isolation is not a permanent certificate of good health. A 2026 arXiv security comparison, 2606.08433, found escape-class CVEs on public record for microVM engines including Firecracker. Network controls remain necessary.

For a team evaluating those controls, the vendor should name its hypervisor, state its default egress policy, and explain where secrets live. The appropriate outbound default is deny-by-default with an explicit allowlist. Use network policy to block IMDS access, and account for how providers differ in the risks that EC2, GKE, and AKS expose. AWS documentation also describes requiring IMDSv2 with a hop limit of 1 as a distinct mitigation for EC2/EKS.

Credential placement completes this boundary. Environment variables are readable by sandbox processes and can leak into logs, while runtime vault retrieval shortens the theft window but still places secrets in memory. Proxy-based injection applies credentials at the network layer without admitting them to the sandbox. That avoids turning a saved workspace into a saved credential leak. Locked-down egress and credentials kept outside the working directory support artifacts whose provenance can be trusted.

Giving agents file access to large object-store datasets without copying them into the working directory

Large datasets need not become duplicate files inside every workspace. When the source bucket is mounted, agents can access its files without a separate migration or ETL step. S3, GCS, and R2 can all work this way, though they use different drivers. The design that agents use can thus stay storage-agnostic. Copying everything into each sandbox is not a persistence strategy just because it takes a long time.

Cloud Storage FUSE, or gcsfuse, translates file operations into GCS object API calls. For example, ls /mnt/my-bucket/ becomes a list-objects request, while cat /mnt/my-bucket/data.csv triggers a get-object download. For R2 and other S3-compatible endpoints, an S3-compatible FUSE driver provides the mount path. S3 Files is an option for S3.

Conventional FUSE drivers translate operations, and their limitations include refusing unsupported file operations or using metadata approaches that can disrupt the object API view. Drivers also differ a lot in how well they handle small-file workloads. Vercel added FUSE-based filesystem support on July 3, 2026, but care is still needed with credentials passed to mount commands.

Choose the driver for the backend and file-size profile. Mounting the canonical bucket avoids a stale private copy when the source changes. It does not, by itself, freeze that source in time. Reproducibility still depends on understanding which data the run actually sees.

Checkpointing and branching the working directory for parallel attempts and rollbacks

LangGraph checkpoints state at every super-step, using MemorySaver for development and PostgresSaver for production. In a sequential 10-step workflow where step 7 fails, execution resumes from the checkpoint after step 6. Its branch-and-fork debugging also supports exploring alternatives from saved checkpoints. Recovery becomes part of normal execution. The same saved-state pattern also helps with human approval gates and debugging later on.

At the infrastructure level, snapshot-then-fork gives each task a copy-on-write image whose parent already has dependencies installed and caches warmed. With persistent microVM approaches such as Sprites and Archil, durable, forkable storage makes this workflow practical. Checkpoints in these approaches complete in 90–300 ms regardless of size. Forking can then be used to try out ideas in everyday work as well as in emergencies.

Fast capture is not the whole problem. A 2026 arXiv paper on agent checkpoint semantics, 2608.22928, presents an algorithm for deciding whether checkpoint, fork, restore, and merge edits are safe. DeltaBox, described in an arXiv paper, targets millisecond-level sandbox checkpoint and rollback for stateful agents.

Workspace drift can make a branch diverge from the state its checkpoint represents. The developer’s discipline is to combine git checkpoints with declared path permissions and per-session workspace mounts. Parallel attempts need room to differ without quietly rewriting one another’s premises.

A checklist for designing a working directory that survives production agent workflows

Verify the persistence contract. Check whether the platform provides volume-backed persistence, sandbox snapshots, or both, and distinguish filesystem retention from process-state retention. Check what survives restart. Check what survives sandbox deletion. Put durable artifacts under the volume mount. Label scratch space outside it so the retention boundary is visible.

Verify tenant isolation and outbound access. Check for a named hypervisor or isolation mechanism and a stated default egress policy to substantiate the phrase “secure sandbox. Use deny-by-default outbound rules with an explicit allowlist. Block IMDS access for workloads that run agent-generated code, taking into account how the cloud behaves. Favor credential injection at the network boundary over the use of environment variables.

Choose the data-access path. Mount source buckets rather than copying datasets into each working directory. Use S3 Files for S3, gcsfuse for GCS, or an S3-compatible FUSE driver for R2 and compatible endpoints as appropriate. Test the driver against the workload’s file-size profile. Check small-file behavior on its own, since bulk transfers are no basis for assuming how it will work.

Define the session handoff. Maintain PROGRESS.md or an equivalent file at the directory root and require resumed sessions to read it before changing files. Record completed actions in an append-only event log. Keep paths relative to the project root. Commit before major steps so there is a way to recover without relying only on the sandbox’s snapshot mechanism.

Set the branching contract. Choose infrastructure where forks take milliseconds regardless of directory size, and verify which state the fork preserves. Declare path read/write scope per session or branch to limit unrelated edits. Use per-session workspace mounts to keep drift in check. A branch should provide an alternative attempt isolated from the shared workspace.