We moved our database off MongoDB Atlas and onto a $6 droplet

Since November 4, Remoet's production database no longer lives on MongoDB Atlas. It runs on a single DigitalOcean droplet in the same private network as the app, for $6 a month. The visible part of the move was a maintenance page that stayed up for about 13 minutes, and this post is the rest of it: why a free database had become our most expensive problem, what we run now, how the cutover went, and what we gave up.

Why we wanted off a free database

The bill was never the issue. We were on Atlas's free tier (M0), and in September we measured what that tier means from inside the production app: the primary answers about 100 operations per second no matter how many you send it. Send it 300 concurrent reads and you get exactly 100 a second back. Bulk reads ran at around 93 KB/s. For months we had been chasing slow pages one query at a time (a cache here, a projection there), and every one of those investigations ended at the same wall. The "5.7 ms per document" constant we kept rediscovering was the governor all along, not our code.

There is also a 512 MB storage cap. On October 28 a headless-browser crawler sent us tens of thousands of pageviews in a few hours, the events collection pushed the cluster over its quota, and Atlas blocked every write. Every request that wrote anything, from authenticated API calls to the scraper recording new roles, returned errors until we deleted the junk and shipped a filter.

Why not just pay Atlas?

That was the obvious first option, and we priced it twice. Here is what was on the table, using MongoDB's current published prices:

Flex is the honest middle ground, and for a lot of apps it is the right answer. For us it moves the ceiling from 100 to 500 operations per second without removing it, and our scraper writes continuously in the background while pages and crawlers read, so a hard limit is exactly the kind of wall we had just spent months running into. M10 removes the limit, and at roughly ten times the price of the droplet it is what you pay for not having to think about the database at all. We decided we would rather think about it.

There is also the network. The app runs on DigitalOcean in New York and our Atlas cluster lived in GCP's Iowa region, so every query crossed the public internet and a cloud boundary. A paid cluster could have lived closer (Atlas runs in AWS's Virginia region, for example), which would shrink the round trip, but as far as we know DigitalOcean is not among the networks Atlas can peer with, so the traffic would still go over the public internet. A droplet in the app's own VPC gives us a private connection a couple of milliseconds long.

What you give up by not paying is real, and the last section of this post is about it. Atlas handles failover, backups, upgrades and monitoring, and the moment you leave, all four are yours.

This was our second attempt. The first, in September, stalled on index builds that never finished on a single-node replica set. The cause turned out to be our TLS setup: before committing an index build, the primary sends a vote to itself over the network (SERVER-48235), and our certificate did not match the address it was calling. We reproduced it on a throwaway droplet on the morning of the move, and dropping TLS inside the private network removed it.

What we run now

The app needs a replica set, because MongoDB only runs transactions on one (we use them for Stripe webhooks, subscription reconciliation and stars). One droplet runs a single-member replica set, which is the same setup our local Docker environment has used for months:

net:
  port: 27017
  bindIp: 127.0.0.1,<private IP>
security:
  authorization: enabled
  keyFile: /etc/mongo-keyfile
replication:
  replSetName: rs0

mongod listens only on localhost and the private interface, a cloud firewall allows port 27017 only from the VPC range, and the app connects through App Platform's VPC integration, so the database has no public endpoint at all. The app's database user can read and write one database and nothing else. The MongoDB packages are pinned with apt-mark hold, because a surprise server upgrade under production is a data-layer change, not a patch.

Backups used to be Atlas's job and are ours now. Every night the box runs mongodump with the oplog, keeps seven copies locally and thirty in a private Spaces bucket, and DigitalOcean takes a weekly image of the whole droplet on top of that. On day one we downloaded a dump back from Spaces and restored a collection from it, because a backup nobody has restored is a guess.

The box is a 1 GB droplet. (I asked for something cheaper. The 512 MB size cannot hold WiredTiger's minimum cache next to our working set, so 1 GB is the floor.)

The cutover

Before touching production we ran the whole migration as a dry run: dump from Atlas, restore onto the new box, compare. The comparison is the part I would insist on if you do this yourself. Matching collection and document counts are not enough, because data can restore perfectly while indexes go missing, and an index-starved database on a migration that exists for performance is the worst possible outcome. So we compared every index on every collection, by name, keys, uniqueness, TTL, partial filter and sparse flag. Our first attempt in September is where we learned that one.

The real cutover went like this:

1. Maintenance mode on, for both the frontend and the backend. App Platform supports a maintenance block in the app spec that serves an offline page.

2. A final mongodump from an Atlas secondary, a restore, and the full comparison. Every index and every document count matched.

3. A deploy with the new connection string. Our pre-deploy migration job connects to the database first, so an unreachable database would have failed the deploy and left the old version running on Atlas.

4. A catch-up pass for anything written to Atlas after the dump, a smoke test from inside the app, and maintenance mode off.

Two things went sideways, both small. Step 2 says "from a secondary" because we forgot that on the first try, and the dump crawled through the throttled primary to about half done in ten minutes before we killed it. Against a secondary the same dump takes 17 seconds.

The second one taught us more. Maintenance mode stops visitors, but it does not stop background work, so while the old instances drained, the old scraper kept writing to Atlas for about a minute and a half after the new deploy went live. Our catch-up script copies anything whose _id, updatedAt or createdAt is newer than the dump, keeping whichever version is newer, and it handled most of the gap before hitting a duplicate key. The old and new scrapers had both discovered the same new job, under different _ids, and a unique index on active jobs allows only one. The copy on the new database was the live one, so we skipped the Atlas duplicates and ran it again. No user data changed during the window, which is exactly what the maintenance page was for.

So, was it worth it?

We ran the same benchmark script from the new droplet against both databases. Atlas is idle now, so its numbers show the free tier's limits with no app load on top.

The middle row undersells the new box, because the benchmark shared its single vCPU with the database and opened up to 100 authenticated connections at once. The bottom row is the one we actually feel: any read that returns a lot of documents is roughly 60 to 85 times faster.

For pages, the number we care about is how fast they load in the US, which is also where Googlebot crawls from. Timed from a machine in New York, with a Googlebot user agent, 12 random pages per type from our sitemaps and the cache bypassed, every page type now comes back in 0.16 to 0.22 seconds at the median. We don't have a clean before number for that path, so I won't put a multiplier on it. The first table is the honest comparison.

What we gave up, and what we don't know yet

A single droplet has no failover. If it dies, we restore from last night's dump or the weekly image and the site is down while we do it. Atlas M0 is a three-node set under the hood, and we gave that up knowingly. Memory is tight (the box sits around 625 MB of its 1 GB), and it moves to 2 GB the day it starts swapping. Upgrades are ours too: the packages are pinned, so nothing changes until we decide it should, which also means nothing gets patched until we do it.

What I don't know yet is whether Google notices. We have a lot of job pages that Google has discovered and never crawled, and Google sets its crawl rate partly by how fast and reliably a site responds. I'll look at Search Console's crawl stats in a couple of weeks and report back. If you have moved a site to faster infrastructure and watched the crawl rate respond (or not respond), I would love to know how long it took.

Would you run your production database on a single node to get out from under a free tier, or is that a trade you would never make? Come tell me in Discord, or reply here.