Collecting Training Data For LLMs Without Getting Blocked

The bottleneck is never the model

Teams building on their own corpora almost always underestimate collection. A few million pages sounds trivial until you meet per-IP rate limits, regional content differences and sites that serve a stripped-down version of the page to anything that looks automated. What you end up training on is not the web — it is the web's opinion of your crawler.

Collect the page a person would see

The most damaging failure is silent: the site returns HTTP 200 with a thin, degraded page. Your pipeline stores it, your dataset fills with near-empty documents, and nobody notices until evaluation.

Guard against it:

• Compare content length against a per-domain baseline and flag outliers
• Keep a small set of known-good pages and re-fetch them periodically as canaries
• Log the exit country and IP with every stored document
• Reject pages containing challenge markers instead of storing them

Geography changes the corpus

News, commerce, forums and search results differ by country. If everything is collected from one region, your model inherits that region's view. Spread collection across the markets you care about and record the exit country as metadata — it lets you weight, filter or audit later instead of re-crawling.

Politeness is a technical requirement, not a courtesy

Respect robots.txt, keep to public pages, avoid anything behind a login, and crawl at a rate the site would not notice. Beyond the legal and ethical case, aggressive crawling is what gets a domain hard-blocked for everyone, which costs you the source permanently.

Practical architecture

Queue by domain, not by URL, so one slow host cannot stall the run. Give each domain its own concurrency ceiling and its own sticky session. Use residential exits for consumer-facing sites and datacenter for open APIs and archives — the price difference at corpus scale is real. Deduplicate before storage using a content hash; the web repeats itself far more than you expect.

Bandwidth planning

Text-only collection with media blocked averages roughly 100-300 KB per page. A ten-million-page crawl is therefore in the 1-3 TB range. Price that before you start, and pick a plan whose per-GB rate holds at that volume rather than one that looks cheap at 5 GB.