Erasure Coding vs. Replication in Distributed Filesystem Durability

Erasure coding and replication solve the same distributed filesystem durability problem, but they spend different resources to do it. Replication generally suits hot, latency-sensitive data, while erasure coding suits large, cold data. For persistent agent storage, including Archil’s durable, forkable disks, the choice also has to account for recovery and workspace lifecycle needs. The right design balances capacity, latency, and repair behavior.

What distributed filesystem durability actually requires

Commodity disks fail, nodes crash, and network partitions happen. A filesystem with only one copy has a single point of failure. Durability means keeping data when hardware fails, but the promise has to set limits on which failures it covers. A drive and a whole machine are different failure domains. A useful durability contract names the components that can fail and the combinations of simultaneous failures the system can withstand.

That contract is usually expressed as failure tolerance. The system can lose a specified number of components before data becomes unrecoverable, and both replication and erasure coding put parameters around that promise. The same requirement applies to AI agents whose sandbox disks must survive restarts and node failures. Archil pairs isolated Firecracker microVMs with durable, forkable disks delivered through a userspace filesystem with a POSIX-compatible interface. Isolation addresses where execution happens, while durable storage addresses whether its state survives.

How replication works and what it costs

Replication stores N full copies of each data block on N distinct nodes. Each replica contains the whole block, so a read does not need to assemble data from other copies. With 3-way replication, any two simultaneous node losses can be survived, provided the copies occupy distinct nodes. No decoding is needed to retrieve the surviving block.

The capacity bill is equally straightforward. Storage overhead is exactly N×, so 3-way replication consumes three times the raw data volume. In return, any replica can answer a read, allowing requests for hot data to be spread across copies. That is useful when many readers want the same data at once. The extra copies provide both protection and places to send traffic.

Writes still have to satisfy the system’s replication policy. A synchronous write waits for the slowest replica needed to complete its quorum, so a network hiccup on that path can delay completion. Asynchronous replication can reduce the wait, but changes the durability guarantee while copies remain outstanding. A replacement node copies each missing block from a surviving replica, transferring the size of the lost data without reconstructing it.

How erasure coding works and what it costs

Erasure coding splits a data block into k data shards and computes m parity shards. It distributes those k+m shards across distinct nodes, and any k shards can reconstruct the original block. This article uses k+m notation throughout. A 6+3 scheme therefore tolerates three simultaneous node losses, with storage overhead of (k+m)/k, or 1.5×. The earlier 3-way replication example uses 3× storage but tolerates two node losses, so those examples do not offer identical failure tolerance.

The savings come from storing recoverable fragments. That requires computation as well as extra disk space. Writes compute parity, while degraded reads reconstruct data from surviving shards through decoding. Reed-Solomon codes are the most common mathematical basis for this approach. Locally Repairable Codes, or LRC, add shards to reduce how much data must be gathered during repair.

Read granularity introduces another cost. In implementations that perform full-stripe reads, fetching a small object can require reading k full shards. Healthy reads can therefore move more bytes than the request itself suggests, even without a failure. That distinction is important for small or random reads.

Storage overhead is not the whole cost picture

Erasure coding uses less space but adds CPU work and network traffic, especially when missing shards have to be rebuilt. In a standard Reed-Solomon repair, reconstructing one lost shard requires reading k surviving shards before writing its replacement. Replication instead reads one surviving copy of each lost block. A tidy capacity spreadsheet can hide a rather untidy network bill.

The comparison depends on what is being repaired. With replication, recovery traffic grows with the amount of data lost, while standard erasure-code repair reads grow with k times the size of the lost shard. A higher k can thus mean more repair amplification even when the scheme uses little storage in steady state. Concurrent failures compound that demand and can saturate internal links. LRC changes this balance by reconstructing a lost shard from a smaller local group, accepting additional storage overhead in exchange for less repair traffic.

A model of total cost must therefore account for how often failures happen, the average shard size, k, and the price of network bandwidth. A useful parallel appears in agent storage, where Archil bills on active data. That billing basis reflects actual use patterns rather than a provisioned worst-case allowance, but it does not identify which underlying durability scheme is cheaper.

Read and write latency under normal and degraded conditions

Under healthy conditions, the performance gap can be small. A replica serves a whole block directly, while a full-stripe erasure-coded read asks k nodes for data and waits for the shards it needs. Its completion time is exposed to the slowest response in that fan-out. As k grows, more participants can contribute a late response. The request is only as punctual as the group it needs.

A failure makes the difference more pronounced. Replication can send a read to another surviving copy without extra reconstruction work. Erasure coding may instead need to collect k surviving shards and decode them in memory before returning the data. That adds CPU work and waiting at the same time the cluster is dealing with a missing or slow node.

Writes resist such a simple ranking. Replication sends the full block to N nodes, whereas erasure coding sends smaller per-node shards but must first compute parity. For large sequential writes, the bandwidth savings can make the two approaches comparable. Small random writes tend to favor replication because encoding work can outweigh the saved transmission. A write-heavy workload therefore needs more description than its label provides, including the size and pattern of those writes.

Data temperature as the practical decision axis

Data temperature describes how often data is read relative to how long it remains stored. It is the most useful starting point for this decision because it connects access costs to capacity costs. Hot data is read often, and those reads usually need low latency. Replication offers lower per-read work and spreads concurrent requests across complete copies. With erasure coding, reads pay for fan-out, and decoding adds to the cost when data has to be reconstructed.

Cold data reverses the economic balance. Data that is written once and rarely read spends most of its life taking up storage, with few requests to serve. Erasure coding reduces that long-lived footprint, making occasional reconstruction work easier to accept. Backups, archives, and compliance data fit this pattern. They need a durable home.

Object size sharpens the temperature test. Small objects face disproportionate encoding and shard fan-out costs, which makes hot, small objects a strong fit for replication. Large, cold objects are the clearest erasure-coding target. Active databases, caches, and collaborative filesystems with frequent small writes tend toward replication for the same reason.

Many production systems avoid choosing one scheme for everything. They replicate newly written or frequently accessed data, then migrate it to erasure-coded storage as access falls. This tiered approach follows the data’s changing behavior rather than treating its first storage placement as permanent. It also separates the cost of being ready now from the cost of being retained for a long time.

Repair cost and its effect on cluster stability during failure events

A failure starts two clocks at once. Users still need service, and the filesystem has to rebuild the lost protection before more losses go beyond what it can withstand. During that repair window, the system has less redundancy than its configured scheme promises at full health. Another failure can cause data loss if the system can no longer withstand it. Repair duration is therefore part of durability.

Standard Reed-Solomon repair adds to the work during this window. Rebuilding even one missing shard requires k surviving shards, amounting to the original block’s worth of data in the basic reconstruction scheme. Across the shards lost with a disk or node, those reads draw traffic from many survivors at once. The internal network must handle that work while serving production requests. Repair traffic does not politely wait for the users to go home.

Replication usually repairs faster because it takes less work to rebuild each block. A surviving copy supplies the replacement, without a coordinated reconstruction across k sources. Depending on the configuration and amount of redundancy being restored, replication may still transfer more total bytes, so simplicity does not mean every byte comparison favors it. Its advantage is the less coordinated work needed for each missing block. That can make recovery less disruptive to foreground traffic.

LRC narrows the repair fan-out to a local shard group. Systems that need erasure coding’s capacity savings but also need fast recovery often choose it over plain Reed-Solomon. The design must also account for the chance of another failure before repair finishes. Both k and m influence that risk, because reconstruction demand and the number of tolerable losses jointly determine exposure.

How to match the scheme to the workload

The practical decision begins with four questions, starting with the storage budget. Is raw capacity the binding constraint, and is the data large and cold? If so, erasure coding’s lower overhead can justify its compute and network costs. If capacity is ample or data is hot, replication’s simpler read and repair paths are usually the better fit.

The second question concerns read/write ratio and latency tolerance. Does the workload need high read concurrency with tight response times, or can reads tolerate reconstruction work? Replication favors the former, while archival workloads and suitable large-write workloads can absorb erasure coding’s costs. Small random writes favor replication even when most requests are writes. The latency requirement must cover both degraded operation and the healthy cluster.

Third, are objects large enough to use erasure coding efficiently, or will small requests repeatedly trigger disproportionate work? Small objects should be aggregated before encoding or kept replicated. Fourth, where internal repair bandwidth is constrained, replication or LRC-based erasure coding is preferable to plain Reed-Solomon with a high k value.

Replicating hot data and erasure-coding cold data puts each scheme to work where its costs are easiest to justify. A hybrid design is a way to avoid charging every workload for requirements that only some of them have.

Agent workspaces add a final architectural requirement. An environment may need to survive restarts, checkpoint quickly, and fork for parallel attempts, making durable, forkable storage such as Archil’s relevant beyond the replication-versus-coding choice. These workloads also need fast, size-independent branching. That favors a filesystem layer that handles checkpointing independently of how underlying blocks are replicated or coded, without assuming that either durability scheme alone supplies it. Surviving a failed node and starting the next parallel attempt are different jobs, and the storage design has to account for both.