Search Console says 41,000 pages are indexed. Your CMS says you have 380. That gap is not a reporting glitch. It is index bloat, & it is one of the quietest ways a site undermines itself.
Index bloat happens when search engines store large numbers of URLs that were never meant to be entry points. Filter combinations, session parameters, internal search results, tag archives, print versions, paginated series & staging leftovers all multiply quickly. Individually harmless. Collectively, they blur what your site is about & spend crawl attention on pages nobody should land on.
Why it costs you
Three concrete problems, in descending order of severity.
Diluted relevance. Search engines build an understanding of what a domain is authoritative about. When most indexed URLs are thin filter permutations, that signal gets muddy. Your genuinely strong pages end up representing a smaller share of the site's identity.
Wasted crawl attention. Crawl budget is only a pressing constraint on large sites, but it is real. If a crawler spends its allocation on colour & size filter combinations, your new service page waits longer for discovery.
Self-competition. Near duplicate pages compete for the same query. Search engines pick one, & it is often not the one you would have chosen. Users land on a filtered subset instead of the category page you designed.
The goal is not the smallest possible index. It is an index where every URL could plausibly be the best answer to a real query.
Audit it before you delete anything
Resist the urge to start noindexing. First understand the shape of the problem.
- Pull the indexed URL list. Use Search Console's page indexing report & the site: operator as a rough cross-check. Export everything you can.
- Group by pattern, not by URL. Sort by path segment & query string. You will find that 41,000 URLs collapse into perhaps twelve patterns. Decisions get made per pattern.
- Attach data to each pattern. For each group, find clicks, impressions & any internal links pointing at it. A pattern with zero impressions over twelve months is a straightforward candidate for removal. A pattern with steady long-tail impressions is not.
- Check for external links. Occasionally a filtered URL has picked up genuine external links. Deleting it outright throws away that equity. Redirect instead.
This grouping step is the part teams skip, & it is the part that prevents damage. Without it, you are making thousands of individual guesses rather than twelve informed decisions.
Choose the right tool per pattern
There are five viable treatments & they are not interchangeable. Picking the wrong one is how sites lose traffic during a cleanup.
- Canonical tag. Use when the page should exist for users & is a variant of a master page. Sorted or lightly filtered category views are the classic case. The page stays crawlable & consolidates its signals.
- Noindex, follow. Use when the page has user value but should never be an entry point, such as internal search results or deep filter combinations. Crawlers still follow links out of it.
- 301 redirect. Use when the page is genuinely obsolete & a clear successor exists. Never redirect en masse to the homepage. An irrelevant redirect target is treated as a soft 404 anyway.
- 410 or 404. Use when the content is gone & nothing replaces it. A clean 410 is honest & resolves faster than leaving thin pages alive.
- robots.txt disallow. Use only for URL patterns that must never be crawled & are not already indexed. Critical caveat: disallowing an already indexed URL prevents the crawler from seeing your noindex tag, so the page can stay in the index indefinitely. Noindex first, allow it to be recrawled & dropped, & only then consider blocking.
Faceted navigation deserves its own plan
Ecommerce & directory sites generate most of their bloat here. The mathematics are unforgiving: five facets with six options each produce tens of thousands of combinations, & they link to each other, so crawlers find every one.
A workable policy looks like this. Decide which single-facet combinations have genuine search demand, for example a category plus a brand, & let those index as real landing pages with unique copy. Everything else gets noindexed or rendered through a mechanism crawlers do not follow. Keep facet parameters in a consistent order so the same selection never produces two different URLs. Then make sure your XML sitemap contains only the URLs you actively want indexed, since a sitemap is a statement of intent.
Tag pages, author archives & date archives
Content sites accumulate bloat through taxonomy. Every tag ever used creates an archive page, most of which list two or three posts & contain no original writing.
Be honest about which taxonomies serve readers. A small set of well-populated topic hubs with a genuine introduction can rank & can help internal linking. A tag applied once in 2019 cannot. Consolidate the useful ones into real hub pages, & noindex the rest. The same applies to author & date archives on sites with a single author.
Prevent it from coming back
Cleanups decay. Within a year, a new plugin, a new filter or a new campaign structure will have reintroduced the problem unless you build a rule into the process.
- Document an indexation policy per template & keep it with the development documentation, not in a marketing folder.
- Add an indexation question to the acceptance criteria for any new template.
- Monitor total indexed pages monthly. A sudden rise is a symptom worth investigating the same week.
- Keep the XML sitemap generated from the same rules, so it acts as a check on itself.
What to expect afterwards
Indexed page counts fall gradually, not overnight, because search engines need to recrawl each URL to see the new instruction. Expect weeks, not days. Impressions may dip briefly as low-value pages leave the index, which is the intended outcome & not a regression. Watch clicks & conversions rather than impressions, because impressions from pages nobody clicked were never worth anything.
This kind of cleanup pairs naturally with an internal linking review & a content consolidation pass. If you want a structured approach, see our technical SEO & content marketing work, or get in touch with your indexed page count & we will tell you whether it warrants attention.