How to Manage Google Search Console Crawl Issues

TL;DR

To manage crawl issues in Google Search Console, focus on removing unnatural links, improving page speed, and ensuring server stability. Regularly monitor for crawl errors and address them promptly to maintain optimal site performance. Use the disavow tool if necessary, and ensure your site's technical setup supports efficient crawling.

Transcript

JOHN MUELLER: All right, welcome, everyone, to today's Webmaster Central office-hours Hangout. My name is John Mueller. I am a webmaster trends analyst here at Google in Switzerland, and part of what we do are these Webmaster office-hours, where people can join in and ask any web search, website-related question that might be on their mind. A bunch... Read More

Key Insights

• Unnatural links can harm your site; consider disavowing them if they form a clear pattern.
• Page speed impacts crawl rate; faster servers allow more efficient crawling.
• Server errors like 500 or 503 can slow down crawling; ensure server stability.
• Googlebot's crawl rate is adjusted based on the site's response speed and error rates.
• Crawl budget issues are rare for small to mid-sized sites; focus on larger sites with frequent content updates.
• Search Console's coverage report includes all indexed URLs, while other reports use samples.
• Absolute URLs can prevent issues with canonicalization, but use what's easier for your setup.
• A site move from a country domain to a generic domain should include geotargeting settings.

Questions & Answers

Q: How to handle unnatural links affecting my site?

If you identify a pattern of unnatural links pointing to your site, consider using the disavow tool to prevent these links from affecting your ranking. Focus on removing or improving problematic links, especially if they are from widgets or other manipulative tactics. Cleaning up link issues proactively can prevent manual actions from Google's web spam team.

Q: Why is my site's crawl rate slower than expected?

Crawl rate can be affected by server speed and error rates. If your server response time is high or if there are frequent errors like 500 or 503, Google might reduce the crawl rate to avoid overloading your server. Aim for response times between 100-500 milliseconds for optimal crawling.

Q: What is the impact of server errors on crawling?

Frequent server errors, such as 500 or 503, can lead Google to slow down crawling to prevent server overload. It's important to ensure your server is stable and can handle requests efficiently. Regularly monitor server logs for errors and address underlying issues promptly.

Q: How does Google determine the crawl budget for a site?

Crawl budget is determined by a site's popularity, the number of pages, and server performance. Most small to mid-sized sites don't need to worry about crawl budget. However, for large sites with frequent updates, optimizing server speed and reducing errors can help Google crawl more efficiently.

Q: How can I compare URL counts from crawl stats to access logs?

Crawl stats in Search Console include requests from Googlebot, rendering, and other bots, which might not match your access logs. Logs provide a detailed view of requests, including response times and file sizes. Use both to understand overall crawling patterns, but expect discrepancies due to different data sources.

Q: What should I consider when moving from a country domain to a generic domain?

When moving from a country-specific domain to a generic one, ensure you set geotargeting settings in Search Console to maintain local search visibility. Use the site move tool to guide Google in recognizing the new domain. Monitor the transition to ensure search rankings remain stable.

Q: Does using absolute URLs help with SEO?

Absolute URLs can help prevent issues with canonicalization by consistently pointing to the preferred version of a page. However, whether you use absolute or relative URLs depends on what's easier for your site's setup. Both can be effective if your site is correctly configured.

Q: How does Google handle blocked URLs in robots.txt?

URLs blocked by robots.txt won't be crawled, but they can still be indexed if linked from other sites. Over time, Google may only index the URL without content. To remove these URLs from search, use the URL removal tool, especially if they shouldn't be indexed.

Summary & Key Takeaways

• Managing crawl issues involves addressing unnatural links, improving server response times, and monitoring server errors. Google Search Console provides tools and reports that help identify and resolve these issues. Regularly check for technical issues and consider using the disavow tool for harmful links.

• Page speed and server stability are crucial for efficient crawling. Faster response times enable higher crawl rates, which is important for sites with frequent content updates. For most small to mid-sized sites, crawl budget is not a significant concern.

• When moving from a country-specific domain to a generic one, ensure geotargeting settings are configured. This helps maintain search visibility. Use the site move tool in Search Console to guide Google in adjusting to the new domain setup.