Technical SEO audit for e-commerce: crawl budget, faceted navigation and indexation traps

A crawler shows what can be found on the site. Logs show what Googlebot actually does, and that is usually a different picture. We take 30 days of logs, filter to verified Google IP ranges, and group hits by path pattern: product pages, categories, categories with parameters, internal search, static assets.

Three numbers matter. The share of hits on parameterised URLs, where anything above 30 per cent means faceted navigation is eating the budget. The share of product pages crawled at least once in 30 days, because if 40 per cent of the live catalogue was never fetched, new products will take weeks to appear. And the response code distribution, where any noticeable share of 5xx errors or redirect chains is a direct loss.

Alongside that we read the Search Console crawl stats: average requests per day, average response time and the breakdown by crawl purpose. A jump in average response time above 800 ms usually precedes a drop of tens of per cent in pages crawled, which makes it an early warning indicator rather than a curiosity.

Six filters with eight values each, combinable, produce several hundred thousand permutations per category. The shop never notices, because no human looks at those URLs, but Googlebot follows links and stops visiting product pages as a result. The fix is a deliberate decision about which combinations deserve to be URLs and which should never exist as a link at all.

We usually keep only single facets with genuine search volume indexable, typically category plus colour or category plus brand, and treat those as full pages with their own title and copy. Everything else stays reachable for users but unreachable for the crawler as an indexable link. In practice that means handling the filter change without emitting a clickable anchor in the HTML, or moving the state behind a hash fragment.

A frequent mistake is blocking parameters in robots.txt while relying on canonical tags at the same time. A blocked URL is never fetched, so Google never sees the canonical and may leave the URL in the index with no description. Order matters: stop linking first, wait for the URLs to drop out, and only then consider a robots.txt rule.

Internal search results in the index are a classic and the easiest win in any audit. Query-parameter URLs create an infinite space, frequently with zero results, and they look to Google like low-quality pages. Blocking that path in robots.txt and removing links to popular searches from navigation modules usually removes a low double-digit percentage of bot hits within the first week.

The second common trap is out-of-stock products returning 200 with an empty body and an availability notice. Google classifies those as soft 404s and over time starts treating similar URLs with suspicion by default. The third is pagination with canonical tags pointing at page one, which leaves products on deeper pages with no path into the index at all.

It is also worth reading the indexing report for the discovered, currently not indexed state. If it covers more than 30 per cent of your sitemap URLs, this is not a patience problem, it is Google saying it sees no reason to fetch those pages. The cause is usually manufacturer-supplied duplicate descriptions or missing internal links into the deeper catalogue.

What happens to a product leaving the range has a direct traffic impact, because those pages often hold the most links and history. If the product returns within a few weeks, keep the URL at 200 with a clear availability message and a list of alternatives from the same category. Only the availability property in the structured data changes, so the search result does not lie about stock.

If there is a direct successor, a 301 to its page is correct and carries the link value across. Redirecting every discontinued product to its category en masse, on the other hand, is a fast route to soft 404s, because Google recognises redirects to unrelated pages. For a permanent withdrawal with no successor, 410 plus removal from the sitemap is the cleanest option.

All of this should follow rules encoded in the system rather than decisions made by whoever adds products. On a store rotating several hundred SKUs a month, managing the process by hand has no chance of working, and the errors accumulate for years before landing on our desk as an unexplained visibility decline.

Google renders JavaScript, but it does so in a second pass, and on large sites the delay can stretch to days. If a category listing fetches products client-side, part of the catalogue will be indexed late or not at all. The test is straightforward: fetch the page with JavaScript disabled, or compare raw and rendered HTML in Screaming Frog's rendering mode.

On Next.js builds the problem usually appears where critical content sits in a client component loaded dynamically with server rendering disabled. Category copy, product data and internal links must be in the initial HTML; the rest can arrive later. Check this on the mobile version specifically, since mobile-first indexing means that is the version being evaluated.

An audit that ends in 120 observations is useless, because nobody knows where to start. We describe every task with three numbers: estimated traffic impact, confidence in that estimate, and implementation cost in days. Sorting on those three tends to push dull items to the top, such as internal search in the index or redirect chains, and push ambitious rebuilds down.

The second filter is revenue linkage. Fixing indexation in a category worth 2 per cent of turnover matters less than the same fix in a category worth 30 per cent, even when the technical work is identical. So we ask for the revenue split by category before the audit starts, and the report is written in that order rather than in the order of the tool's tabs.