Meta's AI Crawler Kept Crashing Our Job Board for Two Days. We Have the Receipts.

On August 20 our backend went down at 22:10. It went down again six minutes later, and again five hours after that. That evening it stopped being discrete events at all: the backend fell into a crash loop, out of memory, dying and restarting every few minutes, the pressure holding it under as fast as it could get back up. It crashed once more the following night. A single container, sized comfortably for our real traffic, the same cause every time.

We spent the day fixing real bugs that the pressure exposed: a connection pool that queued forever instead of shedding load, three caches that rebuilt once per concurrent request instead of once, a missing health check. All of it needed fixing and all of it is shipped. But the pressure itself came from somewhere, and our analytics could not say where, because like most small sites we logged pageviews without user agents or IP addresses.

So we added them. It took one evening. The next morning we had a name.

The fingerprint

Between 21:00 and 03:00 UTC that night, our events collection recorded about 5,800 pageviews. 5,660 of them, 97 percent, came from one crawler: meta-externalagent/1.1, Meta's AI training crawler.

What the capture showed:

• 76+ rotating IP addresses, all inside 2a03:2880:f806::/48. That block belongs to AS32934, which is Facebook. This is not somebody spoofing a Meta header from a basement; the traffic comes from Meta's own network.

• Rotating fabricated browser identities. The requests present as Chrome on Windows, Chrome on Mac, Edge, Firefox on Linux, Chrome on Android, with the real crawler token appended at the end of each string. One agent wearing a different face per request.

• It executes JavaScript. We know because our pageview analytics only fire client side. A plain HTTP scraper would be invisible in that collection. This crawler runs a full browser.

• A rate of roughly 1,000 pages per hour, all night. Not a burst. A sustained pull, every hour, from 9pm to 3am and beyond.

• It crawled URLs that robots.txt forbids. Two days earlier we had added explicit disallow rules for our job board's filter URLs. Meta's own documentation says this crawler honors robots.txt. It fetched those exact URLs for at least 19 hours after the rules went live, roughly 5,400 times in one night.

Stop on that third item, because it is a strategy, not an implementation detail. Meta does not fetch your HTML; it renders your pages in a full browser, the way a human visitor would. That choice buys them everything a plain scraper misses: content that only appears after JavaScript runs, the data your frontend loads from your own APIs, the page as a user actually experiences it. And it means the target pays full fare on every take: the server render, the API calls behind it, the client-side execution, all of it. A plain HTTP fetch costs a site milliseconds; a full render costs everything the page costs, multiplied here by thousands of URLs a night. It is the collection method that takes the most complete copy at the maximum possible cost to the host, and the disguise comes free, because a real browser wearing a Chrome identity looks like a person everywhere except the token at the end of one string.

What it was crawling makes it worse. Not our job posts, which are public and linked and sitemapped, and which we would happily hand to any crawler. It was enumerating filter combinations: /jobs?company=X&techStack=A,B,C, thousands of machine-generated permutations of the same underlying data, 3,800 distinct URLs in a single day, 93 percent of them visited exactly once. Every one of those pages triggers a database aggregation over our whole catalogue. It is the most expensive URL space we have, it contains nothing a training corpus needs that the canonical pages do not already offer, and it is the exact space our robots.txt tells crawlers to skip.

What it cost us

On our current host the damage was downtime and a bigger server. Our backend ran comfortably for months on a box sized for our real traffic, humans and polite crawlers alike. Under this pressure its memory need did not grow, it multiplied: at the peaks the process wanted three to four times what real traffic had ever required. We doubled the box and the doubling barely holds it, still running near its ceiling whenever the crawler leans in. That line item is now twice the price, indefinitely, to serve a training pipeline.

But here is the part that keeps me up. Until days before the heaviest wave, this site's frontend ran on usage-based billing.

Before we moved, we measured why our hosting credit kept evaporating: 759,489 server renders of the jobs board in 24 hours. Sustained 9 to 10 requests per second, around the clock, from external crawlers enumerating those same filter URLs. Each render took 4 to 10 seconds. That is roughly 24 million function invocations a month against the 1 million included in the plan, and on the order of 30,000 hours of function compute a month. Multiply 30,000 compute-hours by any serverless provider's list price and you do not get a scary number, you get a four-figure monthly bill, for a page whose human traffic was modest in comparison. The crawlers were not a rounding error on top of our audience; the audience was a rounding error on top of the crawlers.

And a four-figure month is only where our meter happened to stop. Usage billing has no ceiling of its own: the invoice scales with the crawler's appetite, not with anything the site owner controls, and nothing in the arrangement says next month's crawl cannot be ten times this one. The spend is unbounded by design, and the party generating it is not the party paying it.

I had a spend cap, and it worked exactly as designed, which is the problem. When crawler traffic burned through the credit, the cap tripped and took my sites down. Not the noisy route. Not the one site being crawled. Everything under the account, dark, because a crawler I never invited spent my budget. That is the choice usage billing hands you: leave the cap off and a machine you do not control writes your invoice, or leave it on and that same machine decides when all of your sites go offline.

The move off usage billing was not foresight. It was a reaction, done under fire, moving everything onto fixed-price boxes where the worst case is bounded. Days later the heaviest crawl wave yet arrived. On the old setup that would have been either an unbounded line item or another account-wide blackout, whichever the cap decided. On a fixed-price box it was a crash loop on one backend, which we could diagnose, fix, and write this post about.

If your side project is on usage billing, understand that a spend cap does not protect your uptime. It protects your wallet by sacrificing your uptime, all of it, across every project that shares the account. That is still the right trade. But nobody should have to make it because of traffic they explicitly told the sender not to send.

Our own part, plainly

Two things were our fault, and fixing them was step one.

We shipped the unbounded URL space ourselves. Filter chips that toggle technologies in and out of the query string generate a combinatorial explosion of crawlable links, a classic crawl trap, and we built it without thinking about who would walk into it. We have since stopped emitting those links and disallowed the space in robots.txt.

And our backend handled overload badly: an untimed queue that grew without bound instead of shedding load. A better-behaved server would have degraded instead of dying. It does now.

But here is the thing: we fixed our half. The crawler kept coming. Robots.txt is our half of a handshake, and the other half did not show up.

The block

So yesterday we shipped the last resort, and it is narrower than you might expect:

• Requests whose user agent contains meta-externalagent get a 403 on filter URLs, the same URL space robots.txt already disallows. The 403 costs us a string comparison instead of a render and a database aggregation.

• Everything else about Meta stays welcome. meta-externalfetcher, the agent that fetches a page live when a real person asks Meta AI about it, is untouched. facebookexternalhit, which builds link previews when someone shares a Remoet page, is untouched. Our canonical pages remain open to every crawler, Meta included.

• Googlebot, which crawled politely alongside all of this, a few dozen pages a day, is not affected in any way.

We are not anti-crawler. Our whole product is built on the belief that structured job data should be readable by machines; we run an MCP server so that AI agents can query us directly. We are against one specific behavior: autonomous bulk collection that ignores robots.txt, disguises itself as human browsers, and externalizes its compute bill onto the sites it takes from.

The word for this

Describe the traffic without naming the sender: requests from 76+ rotating IP addresses, under rotating fabricated identities that defeat per-IP rate limiting, ignoring the target's published opt-out, sustained until the target was not going down and coming back but cycling, a crash loop held under by the pressure. If a person did that to us on purpose, everyone would agree on the word for it.

The only element missing is motive. A denial-of-service attack intends the denial; Meta intends the extraction, and the denial is collateral damage it never has to notice. I am not claiming Meta attacked us. I am observing that from the receiving end, the two are indistinguishable, and we are not the only ones saying so: SourceHut has publicly described LLM crawlers as DDoSing their infrastructure, and industry reporting this year names Meta and OpenAI as the heaviest crawler load on the open web.

And here is what makes the indifference impossible to excuse: politeness was free. A crawl at a tenth of the speed collects the same corpus in ten nights instead of one; training data does not spoil. Crawl-delay conventions are older than Facebook. Rate limiting a fleet per target domain is not a research problem for a company running some of the most sophisticated infrastructure on earth; it is a setting. The speed was a choice, and the choice encodes exactly how much the target's survival was worth in the plan: nothing. I know what that choice looked like from the receiving end, because I was up for all of it, my inbox filling with memory alerts and downtime notifications, watching our container die and come back in minute-long cycles at 3am while a crawler with an effectively infinite budget kept the pressure on. Nobody at Meta was awake for this. Nothing at Meta even noticed.

Which raises the question I actually want answered: who is responsible? Here is Meta's entire accountability surface, as far as I can find: a documented promise that this crawler honors robots.txt. That is it. There is no crawl-rate control a site owner can set, the way Google has offered for years. There is no channel to report that their crawler is overloading you. There is no mechanism by which any of the cost lands on them. And the one promise on offer is the one we watched not being kept, for at least 19 hours, roughly 5,400 times in a night.

The market's answer has been for third parties to patch around them: CDNs now sell per-crawler allow, charge, or block controls, an entire product category that exists because the crawler operators accept none of the cost they create. The responsibility got privatized to every site owner individually, one 403 rule at a time.

And follow the value through to the end, because every arrow points one way. The crawl consumed our compute, our bandwidth, and our uptime, at our expense. What it took feeds models that generate revenue for one of the most profitable companies in history. Training data earns no citation, no link, no referral traffic, and no payment; there is no line on any ledger where anything flows back. A search engine at least ran on a bargain, crawl me and send me readers. This is the crawl without the bargain: we paid to be taken from, the site was crash looping in the taking, and the only compensation on offer was the opportunity to pay for a bigger server so it could happen faster.

Let me say the stakes without any varnish. This is a bootstrapped business. This traffic sends us zero users, zero revenue, zero anything, and in return for nothing it kept the product crashing for two days in front of the early users we can least afford to lose, and it doubled a hosting bill that comes out of my pocket. On my previous billing setup, one unchecked box, no spend cap, and the same 48 hours would have produced an invoice with the power to end the company. Not damage it. End it. A business like mine can be killed by a crawler that will never know it was here, run by a company that would never notice the difference. That is not a pricing quirk or an engineering war story. That is the actual arrangement currently on offer to everyone who publishes on the open web, and nobody agreed to it.

If you run a small site

Four things, in the order I wish someone had told me:

1. Put a spend cap on anything usage-billed, and know what it will do. Today, before reading further. The cap converts the worst case from a bill into an outage, and an outage is the better failure. But know the blast radius: mine took down every site sharing the account, not just the one being crawled. If some of your projects matter more than others, do not let them share a budget.

2. Log user agent and IP on your traffic, server side. Ours was one evening of work. Without it you are debugging a ghost; with it, attribution takes one database query.

3. Treat robots.txt as documentation, not enforcement. It states your terms. Enforcement is a 403 at the edge, and it is cheaper than the render it replaces.

4. Watch for the crawl trap you built yourself. If your filters generate URLs, you have handed every crawler an infinite space. Bound it before someone walks in.

The compute this crawl cost Meta would not show up in any report they run. The compute it cost us kept the site crashing for two days and doubled a hosting bill. That asymmetry is the entire story: for them a rounding error, for a small site an existential number, and the party that pays is not the party that decides.

The jobs are still public. The data is still open. The front door was never locked, which is what makes it strange to keep finding someone climbing through the window.

If this happened to your site too, or you think the block was the wrong answer, I read everything in Discord. Reply here too, I answer those.