Before a page can rank for anything, Google has to find it, fetch it and decide it is worth keeping. Most SEO conversations skip past that and start arguing about keywords, which is fine until a page you wrote, published and linked to simply is not in Google at all. Then keywords are irrelevant. The page is not losing a competition, it never entered one. This guide covers the plumbing that decides whether a page can appear: sitemaps, robots.txt, noindex, canonicals, and the free report that tells you which of them has gone wrong.
None of it is difficult. It is just invisible, which is why it goes unchecked for years and does most of its damage on the day a new site goes live.
Key takeaways
- Not indexed and not ranking are different problems with different fixes, and confusing the two wastes months.
- A sitemap is a hint that a page exists, not an instruction to index it, so submitting one guarantees nothing.
- Blocking a URL in robots.txt stops Google fetching it, which is not the same as removing it from the index.
- A staging noindex left on after launch is the costliest mistake in this subject, and nothing looks wrong while it happens.
- A canonical tag is a vote about which version of a page is the real one, and Google is free to disagree.
- The Search Console Pages report names the exact reason each URL was excluded, and most of those reasons are ordinary.
- Crawl budget is real, but almost no small business site will ever hit it, so worrying about it is a distraction.
Who this guide is for
This is for the owner of a small business website who published a page, waited, searched for it and found nothing. Maybe the whole site vanished after a rebuild. Maybe Search Console said a URL was crawled but currently not indexed, and you had no way of knowing whether that was a shrug or an emergency.
It assumes nothing technical. You will not have to edit a file, though you may end up asking whoever built the site to change one setting. The vocabulary is the real barrier here, not the ideas underneath it. Once you know what each file and tag is for, diagnosis becomes a process of elimination.
It matters less if you run a large ecommerce catalogue, where indexing is an ongoing discipline with real crawl budget constraints. For a site of ten to two hundred pages it is a one-afternoon subject you check after any change. We build websites, so we have an obvious interest in you worrying about this, which is why the honest answer sits at the top.
Crawling, indexing and ranking are three different jobs
Almost everything confusing here comes from collapsing three stages into one word. Getting a page in front of a customer needs three things to happen in order, and each can fail on its own. Name the stage that is failing and you have very nearly named the fix.
Crawling: can Google reach the page
Crawling is discovery and fetching. Google finds URLs by following links from pages it already knows, by reading sitemaps and by picking up mentions elsewhere. If nothing links to a page and no sitemap lists it, Google may never learn it exists, which is an absence rather than a punishment.
Indexing: is Google willing to keep it
Being fetched is not the same as being kept. Google renders the page, decides what it is about, checks whether it duplicates something it already holds, then decides whether to store it. A noindex instruction is an order and it obeys. Sometimes it simply judges the page too thin to bother with, the commonest reason of all on small sites.
Ranking: where the page sits against everyone else
Ranking only happens to pages that made it into the index. It is your page against every other stored page that could answer the same query, weighed on relevance, usefulness, credibility and where the searcher happens to be standing.
Not indexed and not ranking are not two degrees of the same problem. One means your page never entered the race. The other means it ran and came thirty-fourth.
Why naming the stage decides what you do next
Nothing indexed means one blunt fault, almost always a single setting, and it is urgent. Most pages indexed with a few gaps means the technical side works and the gaps are about page quality. Everything indexed but no enquiries means the plumbing is fine and your pages are not competitive.
What an XML sitemap is, and what it is not
A sitemap is a file, usually at /sitemap.xml, listing the URLs you would like search engines to know about. It is machine readable and thoroughly boring. Most platforms generate one automatically and update it when you publish, so you probably have one already.
What it actually does
It helps with discovery, and that is the whole of it. If you publish a page and nothing links to it yet, the sitemap is how Google learns it exists. On a well-linked ten-page site Google would have found everything anyway. Priority and change-frequency fields also exist, and Google ignores both.
What it is not
A sitemap is a hint, not a command. Listing a URL does not force Google to index it, and there is no appeal if it declines. The reverse catches people out too: leaving a page out does not hide it, because anything linking to it leads Google straight there.
How to submit one, and what to check afterwards
In Search Console there is a Sitemaps section in the left menu. You type the path, usually sitemap.xml or sitemap_index.xml, and press submit. That is the whole job, and you do it once per property unless the path changes.
What matters is the report afterwards. You want a status of Success and a discovered-URL count roughly matching the pages you believe you have. Three on a forty-page site means it is broken. Nine hundred means it is stuffed with tag archives you never meant to publish.
robots.txt, and the mistake almost everybody makes with it
The robots.txt file sits at the root of your domain and tells well-behaved crawlers which paths they may fetch. It is the oldest and bluntest instrument here, it takes four seconds to read, and it causes more accidental disappearances from Google than anything else on this page.
What it is legitimately for
The honest uses are narrow: keeping crawlers out of an admin area, an internal search page, a checkout, or a directory of files that would burn requests without ever being useful. Most platforms ship a sensible default and most small sites never need to change it.
Blocking a URL does not remove it from the index
Disallow does not mean remove, it means do not fetch. If Google already holds the page and you then block it, Google can no longer fetch it to see anything has changed, so it can sit in the results for a long time afterwards, sometimes with no description at all.
Combining the two instructions is worse. Block a page and add a noindex, and the noindex never gets read, because reading it would mean fetching the page you just forbade. Allow crawling, apply noindex, wait for the recrawl, then block the path only if you still want to.
Never block your CSS and JavaScript
Google renders pages roughly the way a browser does, loading the stylesheets and scripts and looking at the result. Block the folders those files live in and Google gets a skeleton: unstyled text, missing navigation, content that never loaded. The site still looks perfect to you, which is what makes this one nasty.
The noindex tag, and the disaster it causes on launch day
If robots.txt is the blunt instrument, noindex is the precise one. It is a small instruction in the head of the page saying: you may read this, but do not keep it. Because it is an order rather than a hint, it is the only reliable way to keep a page out of search, and the most dangerous setting on your website.
When noindex is genuinely the right tool
There are pages you want reachable but not listed. Thank-you pages that only make sense after a form submission. Internal search results. Author archives on a one-author site. The test: would a stranger landing here from a search be confused or disappointed?
noindex versus robots.txt, side by side
| What you want | The right tool | What goes wrong with the other one |
|---|---|---|
| Keep a page out of Google | noindex, crawling still allowed | A robots.txt block leaves it in the results and stops the noindex being read |
| Stop crawlers wasting requests on a huge useless directory | robots.txt Disallow | noindex still needs every page fetched first, so nothing is saved |
| Remove a page that no longer exists | Return a proper not-found, or redirect it | Blocking it leaves it in the index with no way to learn it has gone |
| Hide a site while it is being built | Password protection at the server | A staging noindex gets copied to live, which is the disaster below |
The staging noindex left on after launch
Here is how it happens, and it happens constantly. A site is built on a staging URL, and to keep the half-finished version out of search somebody ticks the box that adds a noindex to every page. WordPress has exactly such a checkbox. The build gets approved, the site goes live, and the checkbox goes with it.
Nothing looks wrong afterwards. Every page loads, the form works, the design is better than the old one. Then over the next two to six weeks the pages drop out one at a time as Google recrawls each and is told to discard it. Enquiries fade rather than stop dead, which is why it goes unnoticed for a quarter.
Canonical tags and duplicate content
Duplicate content causes more unnecessary anxiety than any other topic in search. There is no penalty for it. When Google finds several URLs holding substantially the same content, it picks one to keep and ignores the rest. That only becomes your problem when it picks the one you did not want.
Where duplicates come from without anyone creating them
Most duplicates are byproducts of how the web works rather than anything a person did. The same page with and without a trailing slash. With and without www. With a tracking parameter added by an email campaign. On a competently built site the platform handles nearly all of it before you notice.
What a canonical tag actually does
A canonical tag is a line in the head of a page naming the address you consider the proper one for this content. It is a vote, not a command. Google usually accepts it and sometimes overrules it. URL Inspection shows the canonical you declared and the one Google selected on separate lines, and when those two disagree you are looking straight at your answer.
The self-referencing canonical and the location page trap
The normal state for a page is a canonical pointing at itself, which protects you when somebody shares it with a tracking parameter bolted on. The riskier duplicates are the ones you create on purpose: ten location pages identical apart from a swapped town name. Google keeps one and ignores the rest, which is why the ninth town page never does anything.
Reading the Pages report in Search Console
The Pages report, which older guides call index coverage, is where the answer usually lives. It splits every URL Google knows about into two piles, indexed and not indexed, then names the reason for everything in the second pile. Those reasons are written in clipped shorthand that reads as far more alarming than it is.
Two rules of thumb before the table. A scattering of exclusions across several reasons is normal. A large group excluded for the same reason is a real signal, and the reason usually names the fix.
The exclusion reasons in plain English
| What Search Console says | What it actually means | What to do about it |
|---|---|---|
| Crawled, currently not indexed | Google fetched it, read it and chose not to keep it. A quality judgement, not a fault | Make the page more useful, and link to it from pages that already work. Resubmitting does nothing |
| Discovered, currently not indexed | Google knows the URL exists but has not fetched it yet | Wait, and link to it from an indexed page. Across many URLs, check server speed |
| Alternate page with proper canonical tag | A duplicate, and you told Google to keep a different version | Check the canonical points where you intended, then leave it alone |
| Duplicate, Google chose different canonical than user | You voted for one version and Google kept another | Look at the page it chose. Make yours distinct, or accept the choice |
| Excluded by noindex tag | The page carries a noindex instruction | Fine if you meant it. If your service pages are here, go and fix it now |
| Blocked by robots.txt | Google was not permitted to fetch the page at all | Find the rule. Removing it is what lets a noindex be read |
| Soft 404 | Returned a success code but looked empty, or looked like an error message | Common on empty categories and thin thank-you pages. Add content or return a proper not-found |
| Page with redirect | The URL redirects elsewhere, so the destination is indexed instead | Normal. Only investigate if it should not be redirecting |
| Not found (404) | The URL does not exist | Fine if deliberate. If you still want the page, restore it or redirect it |
| Server error (5xx) | Your site failed to respond properly when Google asked | Talk to your host. Repeated errors suppress crawling site-wide |
The two that get misread most often
Crawled, currently not indexed causes the most panic and is almost never a technical fault. Google came, looked, and decided the page was not worth storing. There is no button that overrides that. The honest response is to make the page substantially better or accept that it will not appear.
Discovered, currently not indexed is different. Google has the URL on a list and has not fetched it yet. On a new site that is ordinary and resolves itself. Across hundreds of URLs for months it means Google has decided your site is not worth crawling hard.
What a healthy report looks like
Healthy means your homepage, service pages, contact page and articles in the indexed pile, with tag archives, paginated listings, redirected old URLs and a few thin pages in the other. It does not mean zero exclusions.
The only question worth asking is whether anything you would print on a business card has ended up in the wrong pile. If the report is unfamiliar, our walkthrough of Google Search Console covers setting it up.
Redirects, orphans and the routes into your pages
Two structural problems account for a large share of pages that quietly fail to index, and neither shows up in an audit as a satisfying red number. The first is redirects piled into chains. The second is pages nothing on the site links to.
Redirect chains and loops
A permanent redirect tells Google an address has moved and that the old URL's standing should transfer. The problem is accumulation: a 2019 URL points at the 2022 equivalent, which points at the 2024 one. Each hop is harmless alone, but strung together they slow crawling, and past a few hops Google may stop following. Point the oldest URL straight at the current destination.
Redirecting everything to the homepage
When a site is rebuilt and all the URLs change, the lazy option is to send every old address to the homepage. It makes the errors disappear, so the report looks clean, and it is worse than doing nothing. Google treats a redirect to an irrelevant destination as a soft 404 and transfers nothing.
Orphan pages, and why linking is an indexing issue
An orphan page is one nothing on your site links to. It might sit in the sitemap, it might even be indexed, but it has no route in and little standing, because internal links are how you tell Google which of your pages matter. If a page matters, link to it from pages that already get traffic. If not, delete it and redirect it.
Crawl budget, and why most small sites can ignore it
Crawl budget is how much crawling Google is willing to do on your site: how many requests it makes and how often it comes back. It gets mentioned in audits constantly, in a tone suggesting you are losing something. For most small business websites it is a non-issue.
What the term actually means
Two things set it. Crawl capacity is how much your server can take without slowing down, and crawl demand is how badly Google wants your pages. Google has said plainly that sites under a few thousand URLs need not think about this.
When it starts to matter
It becomes real on large ecommerce and listings sites, and anything generating URLs combinatorially. Faceted navigation is the classic case: filters for colour, size, brand and price multiplying into tens of thousands of near-identical addresses, almost none worth indexing. That is where robots.txt genuinely earns its keep.
What to do instead if you are small
Keep the sitemap accurate so Google is not fetching URLs that no longer exist. Flatten redirect chains so each request lands first time. Keep the server responding quickly, because a slow site gets crawled less. And do not publish hundreds of near-empty pages. That is ordinary housekeeping under an expensive-sounding name.
Common mistakes
Almost every genuine indexing fault we come across was introduced by a change rather than by neglect. A new site, a new plugin, a new host, a developer tidying something on a Friday. Sites do not drift out of the index on their own, they get pushed.
The moment to check any of this is immediately after something changes. Fifteen minutes on launch day beats a year of monthly reports about a site nobody has touched. If nothing has changed and the site was fine last month, it is almost certainly still fine.
The list below is roughly ordered by how much damage each one does. The first three have taken entire businesses out of search, and each is a single setting rather than a project.
- Leaving the staging noindex on after launch. The most expensive ten seconds of work in this subject.
- Blocking a page in robots.txt to get it out of Google. It does the opposite, and prevents the instruction that would have worked.
- Blocking the folders holding CSS and JavaScript. Google renders a broken skeleton and judges the page on that.
- Assuming a sitemap forces indexing. A page Google does not rate stays out however prominently it is listed.
- Requesting indexing repeatedly for the same URL. Once after a real fix is sensible, weekly is a nervous habit.
- Redirecting every old URL to the homepage. It hides the errors, transfers nothing and loses the visitor.
- Canonical tags pointing at the homepage from every page. A plugin fault that tells Google nothing else is worth keeping.
- Publishing near-identical location pages. Swapping a town name across ten pages produces nine that Google ignores.
- Never opening the Pages report. The answer to most of these is already sitting in it, for free.
How to diagnose a page that will not index
Work through the following in order and stop the moment you find a cause, because the first thing you find is nearly always the whole answer. It takes about twenty minutes for a single page and needs nothing beyond Search Console and a browser.
Before you start, be clear about what you are diagnosing. If the page appears when you search a distinctive sentence from it in quotation marks, it is indexed, and your problem is ranking. Do that check first.
One caution about timing. Indexing is not instant even when everything is correct, and a new page on a healthy, well-linked site is typically picked up within a few days to a fortnight. Judging it after forty-eight hours sends you hunting for a fault that does not exist.
- Search for it properly first. Put a distinctive sentence into Google inside quotation marks. If it appears, the page is indexed and this list is not your problem.
- Run URL Inspection. Paste the full URL into the bar at the top. It reports whether the URL is on Google and, if not, why. This resolves most cases.
- Read what it says about crawling. If the fetch was blocked, open robots.txt and find the rule. Remove it if it was not deliberate.
- Check whether indexing is allowed. If a noindex tag was detected, that is your answer, and on a recently launched site it is almost certainly the staging setting.
- Compare the two canonicals. The panel shows the one you declared and the one Google selected. If they differ, go and look at the page Google chose.
- Check what links to the page. If nothing does, add links from two or three relevant pages that are already indexed.
- Look at the rendered page. Use the live test and view the output. Unstyled text or missing content means something the page needs is blocked.
- Ask whether the page deserves indexing. If it is two hundred words that could belong to any business in the country, that exclusion is a review rather than a bug.
- Request indexing once, then leave it. Use the button after you have actually fixed something. Once. Then wait a fortnight.
If you reach the end without an answer, what remains is usually a setting applied above your website at the server or CDN level, or a manual action, which Search Console reports separately. Both are rare, and both are jobs for somebody technical.
Frequently asked questions
How long should a new page take to appear in Google?
Does submitting a sitemap guarantee my pages get indexed?
My page says crawled, currently not indexed. What do I actually do?
If I block a page in robots.txt, will it disappear from Google?
How do I know whether a staging noindex was left on my new site?
Should I request indexing for every page I publish?
Do I need to worry about crawl budget?
Will duplicate content get my site penalised?
Where this fits
Indexing is a gate rather than a growth lever. Getting it right does not make anybody want to hire you, and it will not turn a weak page into a strong one. It only means your pages are eligible to be found, which is the precondition for everything else.
That is why it is worth checking before you buy anything. An SEO retainer aimed at a site with a noindex left on is money set on fire, and so is a content programme feeding pages into a site that cannot crawl them.
For the wider picture, our guide to what technical SEO actually is puts this in context alongside speed, mobile and site structure, while structured data and rich results covers what happens once your pages are indexed and you want them to look better in the listings.
If you would rather not think about any of it, that is reasonable. We build sites where this is handled at the start rather than diagnosed later: our web design work ships with the sitemap, robots and canonical side already correct, and our SEO work begins by checking foundations. If you are not sure whether your site has a problem, get in touch and we will give you an honest answer.
