Web Deduplication (WMConf MTV '19)

TL;DR

Allan Scott discusses Google's web deduplication process and its benefits.

Transcript

ALLAN SCOTT: So my name is Allan. I'm here to talk about web deduplication. So probably I should explain what web deduplication is first. So what we do is we identify and cluster duplicate web pages, anything that looks the same, basically. And then we take these clusters. We pick representative URLs that actually get put into the index and served ... Read More

Key Insights

• Web deduplication involves identifying and clustering duplicate web pages to improve search results and index efficiency.
• Deduplication helps search users by preventing repetitive search results, enhancing the overall search experience.
• By removing duplicate pages, more space is created in the index, allowing for better handling of unique and long-tail queries.
• Webmasters benefit from deduplication as it helps retain signals when redesigning or moving pages, maintaining site relevance.
• Alternate names in deduplication aid in localization and site rebranding, ensuring continuity in search results.
• Key signals for deduplication include redirects, page content, and rel=canonical tags, with redirects being the most reliable.
• Localization challenges arise when similar content is presented for different regions, necessitating the use of hreflang tags.
• Representative URLs are selected through a machine-learned system, prioritizing user experience and security.

Questions & Answers

Q: What is web deduplication?

Web deduplication is the process of identifying and clustering duplicate web pages to improve search results and index efficiency. It ensures that users do not see repetitive results and helps make space for unique and long-tail queries in the search index.

Q: Why is web deduplication important for search users?

Web deduplication is crucial for search users because it prevents them from encountering the same search result multiple times, enhancing the overall search experience. By removing duplicates, users are more likely to find diverse and relevant content quickly.

Q: How does deduplication benefit webmasters?

Webmasters benefit from deduplication as it helps retain signals when they redesign or move pages, maintaining site relevance. Deduplication ensures that the signals from old pages are forwarded to new locations, aiding in site continuity and improving search visibility.

Q: What are the key signals used in web deduplication?

The key signals used in web deduplication include redirects, the actual content of the page, and rel=canonical tags. Redirects are the most trustworthy signal, as they clearly indicate the movement of pages and help forward signals appropriately.

Q: What challenges does localization present in deduplication?

Localization challenges arise when similar content is presented for different regions, making it look like duplicate content. This requires the use of hreflang tags to indicate the intended audience for each page, ensuring proper clustering and deduplication.

Q: How are representative URLs selected in deduplication?

Representative URLs are selected through a machine-learned system that evaluates pairs of pages based on chosen signals. The system prioritizes preventing hijacking and ensuring a good user experience, such as avoiding slow meta refreshes or expired certificates.

Q: What suggestions does Allan Scott offer to webmasters?

Allan Scott suggests webmasters use redirects for site redesigns, serve meaningful HTTP results, check rel=canonical links, and use hreflang for localization. It's also important to secure dependencies on secure pages and keep canonical signals unambiguous to aid in deduplication.

Q: Why is it important to keep canonical signals unambiguous?

Keeping canonical signals unambiguous is important because conflicting signals can confuse the deduplication system, leading it to make incorrect assumptions about which URL should be canonical. Clear signals help ensure that the system selects the intended representative URL.

Summary & Key Takeaways

• Allan Scott explains web deduplication, a process used by Google to identify and cluster duplicate web pages. This process helps improve search results by removing repetitive pages, making space for unique queries, and retaining signals when web pages are moved or redesigned.

• Web deduplication benefits both search users and webmasters by enhancing search experiences and maintaining site relevance. Key signals used include redirects, page content, and rel=canonical tags, with redirects being the most reliable. Localization can pose challenges, requiring hreflang tags for clarity.

• Representative URLs are chosen using a machine-learned system that prioritizes user experience and security. Suggestions for webmasters include using redirects, meaningful HTTP results, checking rel=canonical links, and ensuring secure dependencies to aid in the deduplication process.