Search index bug: Google explains why your website lost traffic in April

In April, you might have noticed a drop in your search results. This is due to bug that temporarily lost part of the Search index. What does this mean? This had got nothing to do with your website breaking down or SEO done poorly.

Basically Google was deploying a new version or the latest version of its Search index to all of its data centers across the world. AS this very routine process was happening, somewhere in the interwebs,  a “small number” of documents were “dropped” (or basically weren’t included in the updated index), that means when the update was done, some content was not listed therefore ‘missing’ leading to Google Search delisting hundreds of websites. The result meant that for any affected websites there was lost traffic and subsequently revenue.

Read How to use Jetpack for uptime or downtime monitoring for WordPress websites

As a consequence, the Google’s Search Console, a tool used by all of us to gauge how our websites are performing by the minute in Search, started showing “inconsistencies.” Therefore for most users, what appeared on the console is Search metrics statistics flatlining through the variety of reports. This happened on April 7th but took until April 11th to roll back every data center back to an earlier Search index

Google explains the saga in detail:

Basically, when serving search results to users, to accelerate the speed of the service, the query of the user only “travels” as far as the closest of our data centers supporting the Google Search product, from which the Search Engine Results Page (SERP) is generated. So when there are modifications to the composition of the index (some pages added and removed, documents are merged, or other types of data modification), those modifications need to be reflected in all of those data centers. The consequence is that users all over the world are consistently served pages from the most recent version of the index.

Keeping the index unified across all those data centers is a non trivial task. For large user-facing services, we may deploy updates by starting in one data center and expand until all relevant data centers are updated. For sensitive pieces of infrastructure, we may extend a rollout over several days, interleaving them across instances in different geographic regions. source

So, as we pushed some planned changes to the Search index, on April 5th parts of the deployment system broke, on a Friday no-less! More specifically: as we were updating the index over some of our data centers, a small number of documents ended up being dropped from the index accidentally. Hence: “we lost part of the index.”


Communications going forward

Google has decided a proactive method going forward in the wake of the April indexing issues, to improve how to better communicate with webmasters in case of large system failures. Some of key decisions are to:

  1. Explore ways to more quickly share information within Search Console itself about widespread bugs, and have that information serve as the main point of reference for webmasters to check, in case they are suspecting outages.
  2. More promptly post to the Search Console data anomalies page, when relevant (if the disturbance is going to be seen over the long term in Search Console data).
  3. Continue tweeting as quickly as we can about such issues to quickly reassure webmasters we’re aware and that the issue is on our end.

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 152 other subscribers

%d bloggers like this: