3D-printed Female Googlebot
Sometimes the small coincidences in daily life amaze me. Just earlier today I was explaining to one of my clients about the terrible search presence they unintentionally got because of the locale-adaptive pages (definition below) and their incorrect indexing by Googlebot. One of those terrible consequences was about their snippets in various Google searches – their titles, descriptions, and sitelinks were provided in different languages. And now we have big news from Google on exactly the same issue.
We already knew that Gooblebot mostly crawls from the United States (Mountain View, California) and sometimes from Russia (!) and South Korea. Now Googlebot got geo-distributed crawling and should crawl from more (non-US) locations (country-level). In other words, Googlebot will use additional IP addresses that are identified as being from the non-US range.
Googlebot uses well-established IP addresses that appear to come from the United States. With geo-distributed crawling, Googlebot can now use IP addresses that appear to come from other countries, such as Australia.
Also, the second enhancement to Googlebot is the support of language-dependent crawling. Now Googlebot may set different Accept-Language headers in the HTTP request. It’s hard to say which exactly or even how many languages will be covered by this change, probably not many in this starting point.
There were no announcements whether the new Googlebot locale-aware crawling configurations already is or ever will be supported (reflected) in Google Webmaster Tools “Fetch and render” feature, which is your best friend in finding the website crawling problems of this kind.
Finally, it’s worth noting that these new configurations will be enabled algorithmically for pages Googlebot detects to be locale-adaptive.
Google notes that these are the baby steps only towards the truly efficient locale-aware crawling. So despite the promising news, the use of separate locale URL configurations and rel=alternate hreflang annotations are still recommended.