Google Can't Find Your Pages? Fix Crawlability and Indexing

You published a page two weeks ago. It's good. You checked the on-page SEO. You built a couple internal links. And yet when you run a site: search, it's not there. Or worse, Google Search Console shows it under "Crawled – currently not indexed."

Nobody searches for "crawlability and indexing." They search for why their pages aren't showing up.

The answer almost always lives in the gap between Googlebot visiting your page and Google deciding it's worth showing. That gap is where crawlability and indexing live, and if you don't understand how they work together, you'll keep hitting the same wall. For the broader picture of everything technical SEO covers, see our technical SEO overview.

What crawlability actually means (and what it doesn't)

Crawlability is the simplest concept in technical SEO, and also the most misunderstood. It's one question: can Googlebot reach your URL and fetch its content?

If the answer is yes, the page is crawlable. That's it. Crawlability is not about whether the page is good or whether it ranks. It's a door. Open or closed.

Google discovers pages by following links, reading XML sitemaps, and checking previously crawled URLs. When Googlebot finds a URL, it sends a request to your server. If your server responds with a 200 OK and delivers the page content, the crawl succeeds. If it hits a block, a redirect chain, or a server error, it doesn't.

Start your free technical SEO scan — see which of your pages Google can actually reach.

The three things that control crawlability:

• robots.txt — tells crawlers which URLs they can or can't request. A single Disallow: / blocks everything. A misplaced wildcard can nuke entire sections of your site.
• Internal links — crawlers follow links. Orphan pages (pages with zero incoming internal links) are effectively invisible, even if they're in your sitemap.
• Server response — 5xx errors, DNS failures, and redirect loops all stop crawlers cold.

What indexing actually means

Indexing is the step after crawling. Google takes the content it fetched, analyzes it, and decides whether to store it in the Google index. A page that's indexed can appear in search results. A page that isn't indexed cannot, period.

But here's the part most guides skip: crawling does not guarantee indexing. Google can crawl a page perfectly, render all the JavaScript, and still decide not to index it. You'll see this in Search Console as "Crawled – currently not indexed."

The indexing decision comes down to a few signals:

• Content quality and uniqueness — thin, duplicate, or auto-generated pages get crawled and dropped.
• Canonical tags — if Google picks a different canonical than you intended, your page gets folded into another URL's index entry.
• noindex directives — the <meta name="robots" content="noindex"> tag or X-Robots-Tag: noindex HTTP header. Often left on staging pages by accident.
• Rendering completeness — if JavaScript loads your main content and Google's renderer doesn't run it fully, the page looks empty.

Here's the simplest way to think about the relationship:

	Crawlability	Indexing
Question it answers	Can Googlebot reach this URL?	Will Google store and show this page?
Happens when	First	Second
Blocked by	robots.txt, server errors, no internal links, redirect loops	noindex tags, canonical conflicts, thin content, rendering failures
You check it in	GSC URL Inspection Tool, server logs	GSC Pages report (Indexing tab)
A failed crawl means	Indexing can't even start	N/A
A failed index means	N/A	Page is crawled but won't appear in search

Diagnose your crawlability issues

The fastest way to check if Google can crawl a page is the URL Inspection Tool in Google Search Console. Paste the URL, hit enter, and look at the "Page fetch" section. If it shows "Failed," you have a crawlability problem.

But one page at a time won't scale. Here's what to do instead:

Check the GSC Pages report

Go to Indexing → Pages in Google Search Console. Look at the "Not indexed" section. Filter for reasons that point to crawl failures:

• Blocked by robots.txt — Googlebot was told not to crawl this URL.
• Server error (5xx) — Google tried to fetch the page and your server returned an error.
• Not found (404) — the URL doesn't exist. Google calls this a "soft 404" if the page exists but looks empty.
• Redirect error — too many redirects or a redirect loop.

Crawl your own site first

Use a crawler like Screaming Frog SEO Spider or Sitebulb to simulate what Googlebot sees. Set it to crawl your site and filter by:

• Blocked by robots.txt
• 4xx and 5xx responses
• Orphan pages (in the sitemap but not linked internally)
• Redirect chains longer than 2 hops

Most crawlability problems are configuration errors, not content problems. A staging site's robots.txt that accidentally blocks the production site. A plugin that noindexes category pages. A redirect chain left over from a migration three years ago.

For a full audit checklist you can run right now, see why your pages aren't ranking because Google can't find them.

Fix the most common crawlability problems

Your robots.txt is blocking too much

Open /robots.txt on your site. Look for Disallow: rules. The most dangerous pattern is:

Disallow: /

That blocks everything. But subtler mistakes do real damage too:

Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /*.js$

That last line blocks all JavaScript files. Google needs those to render your pages. If you block JS, Google sees a blank page and won't index it. Google's own documentation says you should allow crawlers access to CSS, JS, and image files.

Fix: remove the Disallow: /*.js$ and Disallow: /*.css$ lines. These were common advice a decade ago and they're still in a lot of WordPress setups.

Your internal linking is weak

Google discovers pages through links. If your most important page sits five clicks from the homepage and has one internal link pointing to it, Google will crawl it slowly or not at all.

A flat site architecture helps. Every important page should be reachable within three clicks from the homepage. Use descriptive anchor text that tells Google what the linked page is about.

Run a crawlability audit on Lookelo to find orphan pages and weak internal links before they cost you rankings.

You're serving the wrong status codes

A 200 OK means the page is fine. A 301 redirect tells Google to go somewhere else. A 404 means the page doesn't exist. A 500 means the server is broken.

But soft 404s are the sneaky ones. Your page returns a 200 OK but the content is empty, a thin error message, or a "no results found" page. Google sees this and treats it as a 404. Fix by either adding real content or returning a proper 404 with a helpful message.

Diagnose your indexing issues

Back to the GSC Pages report. The "Indexed" and "Not indexed" tabs tell the story. The most common indexing problems:

"Crawled – currently not indexed"

Google fetched the page, read it, and decided not to index it. This is the most frustrating status because it tells you the crawl worked but something else failed.

The usual causes:

• Thin content. The page has fewer than 300 words of unique text. Google can crawl it, but it won't index something that offers no value. This is common on tag pages, filtered category pages, and auto-generated location pages.
• Duplicate content. The page is nearly identical to another page on your site or on another domain. Google sees no reason to index both.
• Low authority. New sites, sites with few backlinks, and sites in competitive niches often see this status. Google crawled the page but doesn't trust the domain enough to index every URL.

Fix: consolidate thin pages with canonical tags or 301 redirects. Add unique, useful content. Improve internal linking to distribute PageRank to orphaned pages.

"Discovered – currently not indexed"

Google knows the URL exists but hasn't crawled it yet. This is a crawl budget issue at heart. Google has a queue of URLs to crawl, and yours didn't make the cut.

Common on:

• Large sites with thousands of URLs
• Sites with slow server response times
• New pages on low-authority domains

Fix: improve server speed (time to first byte under 200ms). Reduce the number of low-value URLs Google is crawling by blocking them in robots.txt. Submit the URL directly in GSC's URL Inspection Tool and click "Request Indexing."

"Page with redirect"

Google found a redirect and followed it. The destination page might be indexed instead. This is normal for 301 redirects, but check that the redirect chain isn't too long. Two or more consecutive redirects waste crawl budget and can confuse Google.

"Excluded by noindex tag"

The page has a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP header. This is often intentional for admin pages, thank-you pages, and internal search results. But it's also common to find noindex tags on pages that should be indexed — left over from development, or applied by a checkbox in a SEO plugin.

Fix: check the page source. If you're on WordPress, check the "Allow search engines to index this page" setting in Yoast or Rank Math. If you're not on a CMS, check the raw HTML for noindex in the robots meta tag.

Crawl budget: when it matters and when it doesn't

Crawl budget is the number of URLs Googlebot will crawl on your site in a given day. For most sites, it's not a constraint. Google has said that sites under a few thousand URLs rarely need to worry about crawl budget.

But if you run a large ecommerce site, a job board, a news site, or any site with thousands of pages, crawl budget is real. Google allocates crawl capacity based on two things:

• Crawl demand — how popular and fresh your pages are. Pages with more backlinks and more frequent updates get crawled more often.
• Crawl rate limit — how fast Googlebot can crawl without overwhelming your server. If your server is slow, Google slows down.

Wasting crawl budget is easy. Every faceted navigation URL, every session ID parameter, every paginated page beyond page 5, every thin tag page — these all eat crawl budget. Googlebot spends time on them instead of your product pages.

Fix: block faceted navigation URLs in robots.txt. Use canonical tags to point parameterized URLs to the clean version. Remove or noindex thin pages. Keep your sitemap clean and only include canonical, indexable URLs.

JavaScript and indexing: what you're probably getting wrong

Google renders JavaScript. It has since 2019. But rendering takes time and resources. Google crawls the HTML first, then queues the page for rendering in a second wave. That second wave can take days or weeks.

If your content lives entirely in JavaScript — React apps, Vue apps, client-side rendered pages — you're gambling that Google's renderer will process your page before it makes an indexing decision.

The safer approach:

• Server-side rendering (SSR) or static generation for content-heavy pages. Google gets the full HTML in the first crawl.
• Dynamic rendering for pages that must be client-side. Serve a fully rendered HTML snapshot to Googlebot while users get the JS version.
• Test with the URL Inspection Tool. Click "View Crawled Page" and look at the screenshot. If your content is missing, Google didn't render it.

Mobile-first indexing adds another layer. Google now crawls and indexes the mobile version of your page. If your mobile site loads a stripped-down version with less content, that's what Google indexes. Make sure your mobile and desktop content match.

Canonical tags: the silent index killer

Canonical tags tell Google which URL is the "real" version of a page. When you have the same content on multiple URLs, the canonical tag consolidates them into one index entry.

But canonical tags go wrong in predictable ways:

• Self-referencing canonicals that point to the wrong URL. If your page is at example.com/page but the canonical tag says example.com/page?utm_source=newsletter, Google might ignore your preferred URL.
• Canonical chains. Page A canonicalizes to page B, which canonicalizes to page C. Google will eventually figure it out, but it wastes crawl budget and can cause indexing delays.
• Cross-domain canonicals that aren't honored. Canonicalizing to a different domain is a suggestion, not a directive. Google may ignore it if the content is significantly different.

Fix: every page should have a single, self-referencing canonical tag. Audit your canonicals with a crawler. Look for pages where the canonical doesn't match the actual URL.

How to check your work

After you fix crawlability and indexing issues, verify that the fixes worked:

1. URL Inspection Tool — test the live URL. If Google can fetch it, crawlability is fixed.
2. Request Indexing — click the button in the same tool. This tells Google to re-crawl and re-evaluate the page.
3. Check the GSC Pages report — come back in a week. The "Not indexed" counts should drop.
4. Log file analysis — if you have server access, check your logs for Googlebot activity. Look for 200 responses on previously blocked URLs.

Once your pages are crawlable and indexed, the next question is what actually moves them up the results — see how Google ranks websites for the ranking factors that take over from here.

Start monitoring your crawlability and indexing on Lookelo — weekly scans that catch these issues before they become traffic killers.

FAQ

What's the difference between crawlability and indexing?

Crawlability is whether Googlebot can reach and fetch your page. Indexing is whether Google stores that page in its database and makes it eligible for search results. Crawling happens first. A page can be crawlable but not indexed — for example, if it has thin content or a noindex tag.

Why does Google Search Console show "Crawled – currently not indexed"?

Google fetched the page successfully but decided not to add it to the index. The most common reasons are thin content, duplicate content, or low domain authority. Add more unique, useful content and improve internal linking to the page.

What does "Discovered – currently not indexed" mean?

Google knows the URL exists but hasn't crawled it yet. This is usually a crawl budget issue — your site has more URLs than Googlebot can crawl in a given timeframe. Speed up your server, reduce low-value URLs, and submit the URL manually in Search Console.

How do I check if my robots.txt is blocking Google?

Open yourdomain.com/robots.txt in a browser. Look for Disallow: rules. You can also use the robots.txt Tester in Google Search Console to test specific URLs against your rules.

Does JavaScript affect indexing?

Yes. Google renders JavaScript, but not immediately. Pages that rely entirely on client-side JavaScript for content may experience delays in indexing. Server-side rendering or dynamic rendering are safer options for content-heavy pages.

What is crawl budget?

Crawl budget is the number of URLs Googlebot will crawl on your site per day. It's determined by crawl demand (how popular and fresh your pages are) and crawl rate limit (how fast your server can handle requests). Most small sites won't hit their crawl budget limit. Large ecommerce sites, news sites, and sites with thousands of URLs need to manage it.

How long does it take for Google to index a new page?

Anywhere from a few hours to several weeks. New sites, pages with few internal links, and pages on slow servers take longer. Use the URL Inspection Tool to request indexing and speed up the process.

Can a page rank without being indexed?

No. A page must be in Google's index to appear in search results. If a page isn't indexed, it can't rank for any query.

What's the fastest way to fix indexing problems?

Consolidate thin and duplicate pages using canonical tags or 301 redirects. Remove accidental noindex tags. Improve your internal linking so important pages are linked from high-authority pages on your site. Submit a clean XML sitemap in Google Search Console.

Should I block CSS and JavaScript files in robots.txt?

No. Google specifically recommends allowing crawlers access to CSS, JS, and image files. Blocking them prevents Google from rendering your pages correctly, which can hurt indexing.

---

You don't need to understand every edge case in crawlability and indexing to fix your site. You need to know the three things that block crawling (robots.txt, internal links, server errors) and the four things that block indexing (noindex tags, canonical conflicts, thin content, rendering failures).

Start with the GSC Pages report. Sort by the biggest bucket of "Not indexed" pages. Fix the root cause, not the symptom. And check back in a week.

Most sites that struggle with crawlability and indexing don't have a technical problem. They have a configuration problem that someone set and forgot. Find it, fix it, and your pages will finally show up.