Scraping Search Results: Location, Pacing And Clean Data

There is no single set of results

Two people searching the same phrase see different pages. Location changes the local pack and the ordering. Device changes the layout and which features appear. Language settings change the sources. If you collect from one place with one configuration, your rank data describes that configuration and nothing else.

Control the variables that matter

Location, precisely. Country is not enough for anything local. City-level exits are what make a local pack meaningful.

Device. Track mobile and desktop separately. They differ enough that averaging them destroys the signal.

Language and locale. Match Accept-Language and timezone to the exit country. A mismatch produces a hybrid page that reflects nobody's experience.

A clean session. No accumulated history. One session per query set, then release it.

Pacing beats volume

Search engines rate-limit per IP aggressively and challenge anything that reads like a script. Keep concurrency low per exit, randomise the gap between queries, and spread a keyword set across many sessions rather than pushing one hard. Collecting 1,000 keywords slowly across an hour works; collecting them in a minute does not.

Store the context, not just the rank

Every record should carry the query, exit country and city, device type, language, timestamp and the IP. Without that, a change in position is uninterpretable — you cannot tell a real ranking move from a different exit city.

Validating what you collected

Keep a handful of stable queries as canaries and re-run them regularly. If a canary shifts dramatically, suspect your collection before you suspect the index. Check page length against a baseline to catch degraded or challenge pages that returned HTTP 200.

Setup that holds up

Residential exits in the target cities, sticky for the duration of a keyword batch, mobile and desktop tracked as separate runs, randomised pacing, and full context stored with every row. That is unglamorous and it produces data you can actually make decisions from.