Keep key pages indexable

Keep key pages crawlable, indexable, one hop away and in the sitemap.

From seo-audit, Technical SEO › Crawlability and Indexation.

Test robots.txt

Edit either file, load a broken one, or test a path.

Written in a hurryBad
    AuditedGood
      User-agent: *
Disallow: /admin/
Disallow: /cart
Disallow: /search
Allow: /

Sitemap: https://acme.com/sitemap.xml

      Will this page be indexed?

      Flip a signal or load a preset.

      Signals on /products/blue-kettle
      What Googlebot seesIndexed
      1. Crawl
      2. Index
      3. Sitemap
      
        
      
        
      
      
      
    
      
    
      // app/products/[slug]/page.tsx
export default async function Page({ params }) {
  const product = await getProduct((await params).slug);
  if (!product) notFound(); // a real 404, not a 200 that says "not found"
  return <Product product={product} />;
}
      
  
      

  
      
    
      One redirect hop
      
    
      Press Crawl, then loop the chain and crawl again.
      
    
      
      
      
      
      
    
      
    
      
      
      
        
      Piled up over timeBad
      
        
        
        
        
      
        
      
        
        
        Old URL → final URLGood
        
        
          
        
          
      
          
    
          
    
          // next.config.ts
async redirects() {
  return [
    { source: "/old-pricing", destination: "/pricing", permanent: true },
    { source: "/pricing-2023", destination: "/pricing", permanent: true },
  ];
},
trailingSlash: false,
          
  
          

  
          
    
          Why it matters, pros and cons
          
    
          
      
          Why it matters
          A page Google can't fetch or won't index earns nothing.
          
      
          Pros
          • Cheap to check: indexed or not.
          • One hop saves requests and link equity.
          • A clean sitemap is a trusted signal.
          
      
          Cons
          • robots.txt is not access control; blocked URLs still index.
          • Some blocks are right: admin, cart, internal search.
          • Small, well-linked sites index fine without a sitemap.