How Search Engines Work: Crawl, Index, Rank

Every SEO tactic makes sense — or stops making sense — the moment you understand what search engines actually do. The machine has three jobs: crawl the web to find pages, index the ones worth keeping, and rank the index against each search in real time. Three jobs, three failure points, three categories of SEO work. Here's each stage in plain language, and what it asks of your site.

Stage 1: Crawling — the discovery machine

Search engines run fleets of crawlers (Googlebot being the one that matters) that fetch pages and follow their links to find more pages, forever. Your site enters the web's map through links pointing at it and through the sitemap you declare; how often the crawler returns depends on how important and how frequently-changing your site appears to be. What crawling asks of you: be reachable (no robots.txt accidents), be fast and stable (slow servers get visited less), and connect everything with internal links — a page nothing links to barely exists. The deeper mechanics live in our technical walkthrough.

Stage 2: Indexing — the library's selection desk

Crawled pages get rendered (JavaScript executed, the page seen roughly as a browser sees it), analysed (what is this page about? which entities, topics, questions?), de-duplicated (one version chosen among duplicates), and then — the step beginners don't expect — judged: the index is curated, and thin or redundant pages are declined. Only indexed pages can ever rank; that's why "why isn't my page on Google at all?" is a different diagnosis from "why is it ranked low?" — the first is a technical/quality conversation, the second a competition.

Stage 3: Ranking — the instant auditions

When someone searches, the engine pulls every indexed candidate and orders them in milliseconds using hundreds of signals that reduce to three questions: Is it relevant? (does the page answer this query, in the format the intent demands); Is it trusted? (does the web vouch for this site — backlinks being the loudest vouching mechanism, per the link building discipline); Is it usable? (speed, mobile experience, security — the tiebreakers). Crucially, ranking is per-query and comparative: nothing about your page is "rank 3" in itself; it's rank 3 against these competitors for this search from this place — which is why local and personalised results differ, and why competitor moves change your positions without you touching anything.

The consequences that pay your salary

1. Every SEO problem maps to a stage. Not found → crawling (links, sitemap, robots). Found but not indexed → quality/duplication. Indexed but low → relevance and authority. The diagnostic ladder just walks the stages in order.
2. The pipeline explains SEO's speed limit. Changes count when pages are recrawled and re-evaluated — days for hot pages, weeks for cold ones — one big reason SEO takes months.
3. Machines read structure. Clean HTML, descriptive titles, structured data, sensible architecture — everything that helps the analysis stage understand you is ranking groundwork.
4. The index is a moving target. Re-evaluation is continuous: improvements get credited, decay gets debited, and core updates re-weigh everyone. The machine never finishes.

Frequently asked questions

How do I get Google to find my new site in the first place?

Submit the sitemap in Search Console, request indexing on key URLs, and — the accelerant — get a few real links from already-crawled sites; crawlers arrive along links. The full sequence is in ranking from zero.

Does Google index every page it crawls?

No — selection is real, and "Crawled — currently not indexed" in Search Console is the machine declining politely. Persistent declines on pages you care about are feedback about distinctiveness and depth, not a bug to report.

Do other search engines work differently?

Same three stages everywhere — Bing, DuckDuckGo (largely Bing-fed), and the rest differ in signal weights, not architecture. Optimise for the pipeline honestly and you've optimised for all of them; the reason everyone says "Google" is market share, and the reason everything in the fundamentals series holds is that the pipeline is the physics — with authority as its strongest current (our specialty).