Googlebot: SEO Mythbusting

TL;DR

Googlebot crawls web content and organizes it in an index, while a separate ranking system uses information from Googlebot to choose results for particular queries. Crawling can begin through a discovered link, a submitted sitemap, or a re-indexing hint in Search Console, and its frequency varies with content changes and server capacity. Read on to understand its browser-like behavior, crawl budget, JavaScript processing, and mobile-first indexing.

Transcript

SUZ HINTON: A lot of confusion revolves around SEO because no one understands how the Googlebot actually works. [MUSIC PLAYING] MARTIN SPLITT: Hello and welcome to another episode of "SEO Mythbusting." With me today is Suz Hinton from Microsoft. Suz, what do you do at work, and what is your experience with front end SEO? SUZ HINTON: Yeah, so right ... Read More

Key Insights

• Googlebot performs three main functions: crawling, indexing, and influencing the ranking process, which is separate from Googlebot itself.
• Suz Hinton, a former front-end developer, discusses her experience and confusion about how Googlebot works, leading to a detailed explanation by Martin Splitt.
• Googlebot behaves like a web browser during the crawling process, fetching content from the internet to understand and index it.
• The crawling frequency of Googlebot depends on the website's nature, such as whether it's a news site or a static museum page.
• Googlebot uses a crawl budget to avoid overwhelming servers, balancing the need to index new content with server load constraints.
• JavaScript-driven websites pose challenges for crawlers, but Googlebot can execute JavaScript, albeit at a later stage, for indexing.
• Mobile-first indexing involves Googlebot using a mobile user agent to index content, while mobile friendliness is a separate ranking signal.
• There are over 200 signals influencing how content is ranked in search results, emphasizing the importance of quality content over SEO tricks.

Questions & Answers

Q: What does Googlebot do?

Googlebot crawls content from the internet and helps organize that content in Google's index. Its information supports ranking, but Martin Splitt explains that the ranking process itself is separate from Googlebot.

Q: How does Googlebot crawl a website?

During the initial crawling step, Googlebot behaves much like a browser visiting a page. It may reach the page through a discovered link, a submitted sitemap, or another input to Google's systems.

Q: How often does Googlebot crawl a website?

Crawl frequency depends partly on how often the site's content appears to change. A newspaper may be checked relatively frequently, while rarely changing museum content may be revisited less often.

Q: Can Search Console trigger a Googlebot crawl?

Search Console can provide Google with a hint by requesting re-indexing. According to Martin Splitt, that request triggers a crawl, although the broader crawling schedule still considers factors such as expected changes and server capacity.

Q: How does Googlebot avoid overwhelming a server?

Googlebot uses a crawl budget to balance gathering content with the amount of load a server can bear. Server demand may vary, for example, load can be higher on Black Friday, so crawling must account for available capacity.

Q: How does Googlebot handle JavaScript-driven websites?

Googlebot can execute JavaScript for indexing, but this may occur at a later stage rather than during the initial crawl. As a result, dynamically rendered content may not be processed immediately.

Q: What is the difference between mobile-first indexing and mobile friendliness?

Mobile-first indexing means Googlebot uses a mobile user agent when indexing content. Mobile friendliness is separate: it is a ranking signal concerning how well a page works on mobile devices.

Q: Why should SEO focus on quality content instead of tricks?

The discussion says that more than 200 signals influence ranking. Because ranking considers many factors, Martin Splitt recommends serving user needs with valuable content instead of trying to manipulate the system with SEO tricks.

Summary & Key Takeaways

• In this episode of 'SEO Mythbusting,' Martin Splitt and Suz Hinton discuss the complexities of Googlebot, focusing on its role in crawling, indexing, and ranking. They explore how Googlebot behaves like a browser and the importance of understanding its functions for effective SEO.

• Suz shares her experiences and questions about Googlebot, leading to a detailed discussion on how it manages server load through crawl budgets and how it handles JavaScript-driven websites. They emphasize the importance of creating quality content for users.

• The conversation covers the distinction between mobile-first indexing and mobile friendliness, highlighting the many signals Googlebot uses for ranking. Martin advises focusing on user needs and quality content rather than trying to game the SEO system with tricks.