How to use the site crawler
- 1
Sign in with your Google account. The crawler is free; the sign-in is the only requirement.
- 2
Enter the address to start from, usually your homepage, for example https://example.com.
- 3
Start the crawl. It reads the start page, then follows links to other pages on the same hostname, one page at a time, until it has read 100 pages or runs out of links.
- 4
Check the summary and the table, then export the report as a CSV file to sort and filter it in a spreadsheet.
What the crawler checks
For every page it reads, the report shows: • the HTTP status code (200 is fine; 4xx and 5xx are errors); • the title and its length, marked too short under 30 characters and too long over 60; • the meta description and its length, marked too short under 70 characters and too long over 160; • the number of H1 headings (one per page is the usual goal); • how many images have no alt text; • the number of internal and external links; • the size of the HTML and how long the page took to load. The summary adds up missing titles, descriptions and H1s, images without alt text and pages that returned an error.
What it doesn't check
The crawler reads the HTML your server sends. It doesn't run JavaScript, so content that only appears after scripts run isn't seen. It doesn't read robots.txt or your XML sitemap, it follows only links on the exact hostname you entered (www.example.com and example.com count as different sites), and it stops at 100 pages. Treat it as a quick check of the on-page basics, not a full technical audit: indexing, canonical tags, hreflang and speed metrics such as Core Web Vitals need other tools, like Google Search Console.
Questions
Is the site crawler really free?
Yes. There's no charge and no card. You only need to sign in with a Google account to start a crawl.
Why did it read fewer than 100 pages?
It only finds pages that are linked from pages it has already read, on the same hostname. Pages linked only from JavaScript menus, listed only in the sitemap, or not linked at all won't be found. Pages that don't load within 10 seconds are skipped.
How long should titles and meta descriptions be?
The report marks titles of 30–60 characters and descriptions of 70–160 characters as good. These are common guidelines, not rules: Google shortens snippets by pixel width and sometimes rewrites them, so write for the searcher and put the important words first.
Can I crawl a site I don't own?
The crawler reads public pages the way a browser does. Only crawl sites you own or have permission to check, and remember that each crawl sends up to 100 requests to that site.
Does it show my pages the way Googlebot sees them?
Not exactly. It reads the HTML your server sends, which is what Googlebot fetches first, but it doesn’t run JavaScript the way Google’s renderer does. To see how Google renders a page, use the URL Inspection tool in Google Search Console and open “View crawled page”.