Figuring out finicky 404s

TL;DR

Geo-specific content led to a spike in soft 404 errors.

Transcript

[MUSIC PLAYING] MARTIN SPLITT: Hello, and welcome everybody to another episode of SEO Fairy Tales. With me today is Mike King, founder, and CEO of iPullRank, where you help bigger brands to be more successful in Google Search results and other search engines probably as well. And you are objectively the best rapper, at least that's what it says on ... Read More

Key Insights

• Soft 404 errors can be caused by geo-specific content issues, where inventory is not available for certain locations.
• Log file analysis is crucial in diagnosing crawl issues and understanding Googlebot's behavior on large websites.
• Using tools like Botify can help combine crawl data with log file data to identify rendering and crawling issues.
• Algorithm updates are often mistakenly blamed for SEO problems; it's essential to rule out other factors first.
• Cloaking concerns arise when different content is shown to users and Googlebot, but it's not always an issue if user experience is consistent.
• Understanding and utilizing log files can provide valuable insights into site performance and Googlebot interactions.
• Geolocation can affect content visibility, leading to soft 404 errors when no relevant content is available for a location.
• Collaboration between SEO and development teams is key to resolving technical issues and leveraging log data effectively.

Questions & Answers

Q: What caused the spike in soft 404 errors?

The spike in soft 404 errors was caused by a geo-specific content issue. Googlebot was crawling from a location where the client's site had no inventory, leading to pages being perceived as soft 404s. By defaulting to national inventory, the issue was resolved.

Q: How was the issue with soft 404 errors resolved?

The issue was resolved by displaying national inventory to all users, rather than location-specific content. This approach ensured that even when Googlebot crawled from locations with no inventory, pages would not return soft 404 errors, thus improving the site's SEO performance.

Q: Why is log file analysis important in SEO?

Log file analysis is crucial in SEO because it provides insights into how Googlebot and other crawlers interact with a site. It helps identify crawl issues, understand which pages are being indexed, and can reveal problems not visible through standard crawl tools. This information is vital for optimizing large sites.

Q: What tools were mentioned for analyzing crawl and log data?

Botify was specifically mentioned as a tool that combines crawl data with log file data. Other tools like Screaming Frog's Log File Analyzer and resources like Kibana Tutorials were also referenced as useful for understanding and leveraging log file data in SEO.

Q: What misconceptions exist about algorithm updates?

A common misconception is that algorithm updates are the primary cause of SEO issues. However, changes in site content, technical aspects, or user behavior are often the actual causes. It's crucial to investigate these factors before attributing problems to algorithm updates.

Q: What is cloaking, and when is it a concern?

Cloaking involves showing different content to users and search engines, which is a concern when it misleads users. However, showing different content based on user context, like location, is not considered cloaking if it aligns with user expectations and improves their experience.

Q: How can geolocation affect SEO performance?

Geolocation can affect SEO performance by leading to soft 404 errors if content is not available for certain locations. This happens when Googlebot crawls from a location with no relevant content, causing pages to be indexed incorrectly. Providing default content can mitigate this issue.

Q: What are the challenges in accessing and using log files?

Challenges in accessing and using log files include difficulty in obtaining data from development teams and a lack of understanding of how to interpret the data. Tools and resources, like those from Builtvisible and Onely, can help SEOs learn to analyze log files effectively.

Summary & Key Takeaways

• In this episode of SEO Fairy Tales, Michael King discusses a spike in soft 404 errors at a car comparison site, caused by geo-specific content issues. By analyzing log files and using Botify, they identified the problem and resolved it by displaying national inventory to all users.

• The episode highlights the importance of log file analysis in diagnosing SEO issues and understanding Googlebot's behavior. Michael emphasizes that algorithm updates are not always the cause of SEO problems and suggests using tools like Botify to combine crawl and log data.

• Martin Splitt and Michael King explore the topic of cloaking and user experience, clarifying that showing different content to users and Googlebot is acceptable if it's consistent with user expectations. They also discuss the potential of log files for gaining insights into site performance.