The Petabyte Math: What Internet Archive's Self-Hosted Servers Taught Me About Leaving S3

Our S3 bill for the media archive tier at my last job jumped 40% in one quarter, and it wasn't because we'd stored more data. It was egress. A research partner had started pulling full monthly exports of our video catalog for a training pipeline, and every one of those pulls came straight out of our margin. I spent an afternoon staring at Cost Explorer trying to figure out which bucket was bleeding us, and that same week I happened to read the Internet Archive's September fundraiser post about keeping their servers running.

What stuck with me wasn't the donation ask. It was one line: they hold 210 petabytes of data, and rather than contracting that out to a cloud vendor, they build and run their own infrastructure. That's not a hobbyist number. That's a nonprofit with a skeleton engineering team quietly operating storage at a scale most Series C startups never touch, and doing it without a single line item to AWS or Google. I got curious enough to actually go dig through how they'd built that, and it turned into a week of spreadsheets figuring out whether the same logic applied to our own, much smaller pile of video files.

This post is that spreadsheet, cleaned up. It's not a pitch to rip out S3 and go full self-hosted — for a lot of teams that's a bad trade. But if you're sitting on a growing archival storage bill and you've never actually run the break-even math, it's worth doing before you renew that reserved capacity.

The AWS bill that started this

Our setup was unremarkable: about 300TB of video and thumbnail assets in S3 Standard, most of it write-once and read rarely, except for that one research partner doing bulk monthly pulls. Storage alone was already north of $7,000 a month at standard rates, and the egress from those exports added another few hundred to a couple thousand dollars depending on how big that month's pull was.

None of this was a crisis. It was just the kind of bill that creeps up quietly until someone in finance asks why the AWS line grew faster than revenue. The instinctive fix is to reach for a cheaper S3-compatible provider — Backblaze B2 or Wasabi both undercut AWS meaningfully on raw storage price. But cheaper managed storage still leaves you paying per gigabyte forever, and our growth curve wasn't flattening. That's the part that made me actually consider owning the hardware instead of renting it.

What Internet Archive's fundraiser post actually revealed

The post itself is a donation appeal, not an engineering writeup — it's asking people to start a recurring $25-or-more monthly gift in September 2026, which gets matched 2:1 during the campaign. But buried in the pitch are a few concrete facts worth taking seriously as a backend engineer. The Internet Archive currently preserves and serves 210 petabytes of knowledge, it charges nothing for access, sells no user data, and runs no ads — and it explicitly says it builds and maintains its own systems rather than contracting its core technology out to corporations.

Brewster Kahle, the organization's Digital Librarian, is quoted in the post making the case for why any of this matters at all:

"Universal access to all human knowledge is within our grasp." — Brewster Kahle, Internet Archive

That's a mission statement, not a systems design doc. But it explains the constraint everything else is built around: if you refuse to charge users and refuse to sell their data, your infrastructure budget has to come entirely from donations averaging around $25, which means every petabyte you store has to be as cheap as physically possible. Cloud storage at retail rates was never going to fund a library that size.

How PetaBox nodes and mirroring actually work

The fundraiser post doesn't get into hardware, so I went looking at what the Internet Archive has published elsewhere about its own architecture over the years. The core unit is what they call a "datanode" — a commodity Linux box packed with dozens of data drives plus a couple of OS drives, organized into racks of ten machines that talk to each other over high-speed Ethernet to form a storage cluster. Drive capacity per node has scaled with the industry, from 2TB drives years ago up to today's much larger drives, without needing to add more physical slots.

The part I found genuinely clever is the mirroring scheme. Instead of RAID within a box, they replicate a given disk slot to the identical slot on the identical relative datanode in a different rack, usually in a different physical data center. A file on drive 7 of datanode 5 in rack 12 of one facility has its twin on drive 7 of datanode 5 in rack 12 of another. That naming convention alone is what lets a small ops team track tens of thousands of drives without losing their minds, because failures get resolved by comparing two predictably-named locations instead of consulting a spreadsheet.

That 2x multiplier is easy to forget when you're comparing a cloud storage quote against a pile of hard drives, because cloud providers already bake their own replication into the sticker price. When you self-host, you have to remember to double it yourself, or your break-even math ends up way too optimistic.

Running the real numbers for a 300-terabyte archive

My first pass at modeling this was embarrassingly naive. I wrote a quick script that just divided hardware cost by drive lifetime and compared it to the S3 sticker price, and it made self-hosting look like a six-times cost win. Here's roughly what that first version looked like:

def naive_monthly_cost(tb_stored, hardware_cost_per_tb=180, drive_lifetime_months=48):
    return (tb_stored * hardware_cost_per_tb) / drive_lifetime_months

s3_monthly = 300 * 1024 * 0.023  # S3 Standard, $/GB
selfhosted_monthly = naive_monthly_cost(300)

print(f"S3: ${s3_monthly:,.0f}/mo")
print(f"Self-hosted (naive): ${selfhosted_monthly:,.0f}/mo")
# S3: $7,065/mo
# Self-hosted (naive): $1,125/mo

That result should have made me suspicious immediately, and it did, because it ignores the mirroring overhead from the previous section, colocation power and cooling, and the fact that somebody still has to physically drive to a data center when a drive dies. I rewrote it to account for those:

def realistic_monthly_cost(tb_usable, hardware_cost_per_tb=180, drive_lifetime_months=48,
                            replication_factor=2, colo_power_per_tb=1.10,
                            engineer_hours_per_month=6, engineer_hourly_rate=85):
    raw_tb = tb_usable * replication_factor
    hardware = (raw_tb * hardware_cost_per_tb) / drive_lifetime_months
    power_and_colo = raw_tb * colo_power_per_tb
    labor = engineer_hours_per_month * engineer_hourly_rate
    return hardware + power_and_colo + labor

print(f"Self-hosted (realistic): ${realistic_monthly_cost(300):,.0f}/mo")
# Self-hosted (realistic): $3,420/mo

Still cheaper than S3's $7,065, but by roughly half, not six times. That's a very different pitch to make to a CFO. The naive version would have gotten a project greenlit on numbers that couldn't survive contact with an actual invoice from the colo provider, and I'd have been the one explaining the gap three months later.

The TCO gotchas nobody lines up in a spreadsheet

Even the "realistic" script above is still generous, because it assumes a steady six hours a month of engineering time, and that number is a lie the first time a drive fails at 2am the week before a launch. Drive failure at scale is a Poisson process, not a monthly average — you'll go quiet for weeks and then lose three drives in the same rack in one afternoon, usually right when you're deep in something unrelated.

The bigger trap is egress that doesn't show up until after you've already sunk the capital into hardware you can't return. Our research partner's bulk export pattern was exactly the kind of workload that looks trivial on paper — one big pull a month — but if that partner base grows, or someone spins up a second consumer of the same data, you're suddenly serving traffic out of a network link you sized for quiet archival storage, not for repeated large downloads.

There's also a people problem that spreadsheets don't capture well: self-hosting means someone on your team now owns firmware updates, RMA paperwork, and the pager for a physical facility. That's a real, ongoing cost even when nothing is on fire, and it's the reason plenty of engineers I respect stay on managed storage well past the point where the raw dollar math favors owning hardware.

What we actually shipped

We didn't go all-in on self-hosting, and I don't think most teams reading this should either. What we landed on was a split: the hot tier that our own application serves directly stayed on S3, because the convenience of not owning hardware was worth the premium for anything customer-facing. The cold archival replica — the copy that exists purely so we never lose the catalog, and that research partners pull from — moved to a two-node, mirrored MinIO cluster we colocate, borrowing the same idea the Internet Archive uses of keeping a full second copy in a physically separate location rather than relying on RAID alone.

Here's roughly how the numbers broke down for that 300TB slice once we actually built it:

That's a real savings, roughly 56% on this slice, but it only pencils out because the workload is archival, predictable, and already had a clear egress pattern we could plan a network link around. If our traffic were bursty or customer-facing, I'd expect that gap to shrink or flip entirely.

The real takeaway

Reading about an organization running 210 petabytes on its own hardware didn't convince me to rip out our cloud infrastructure — it convinced me to stop assuming the cloud bill was just a fact of life and actually price the alternative honestly, including the parts that make self-hosting look worse. The Internet Archive's model works because their storage growth is enormous, predictable, and the mission specifically forbids the kind of revenue that would make cloud vendor lock-in tolerable. Most of us aren't operating under that constraint, and that matters.

If you're deciding whether this applies to you, the honest threshold isn't a terabyte count — it's whether you already have, or can hire, someone who's genuinely comfortable owning physical infrastructure and being on call for it. Below roughly 50 to 100TB of steady-state storage, or if your access pattern is unpredictable, staying on managed cloud storage is probably still the right call even after you run the numbers. Above that, and once egress starts showing up as its own line item on your invoice, it's worth an afternoon with a spreadsheet before you sign another annual commitment.

Sources: Internet Archive — "Keep Our Servers Running: Your Recurring Donation Goes 3X This September" and Internet Archive — "20,000 Hard Drives on a Mission"