If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!

Several weeks ago, I came across a post by Nathan of Not So Boring Life that explains how to get your blog out of Google’s supplemental index — also known as the Google sandbox — to improve your position within SERP’s. If you are unfamiliar with the supplemental index, it’s an additional listing of results related to a search engine query that is not included with the primary results by default. The reason for not including it is due to search engines like Google assuming that it is repeat content.

Unfortunately, blogs are especially prone to having pages enter the supplemental index due to site structuring. In WordPress for example, after setting up permalinks, a page with a post slug (URL path) of “seo-beginners-guide” could potentially be accessible via the following:

• http://yoursite.com/seo-beginners-guide/
• http://yoursite.com/seo-beginners-guide/feed/ (RSS)
• http://yoursite.com/2007/06/ (monthly archive)
• http://yoursite.com/2007/ (yearly archive)
• http://yoursite.com/seo/ (category tag)
• …and more!

Because this creates a plethora of duplicate content, Google throws the extras into the supplemental index. The problem with having content in the supplemental index is that it hurts your result rankings. According to Tropical SEO, what having your pages within the supplemental index means to Google is that they don’t trust your web pages, and they won’t rank for anything of importance.

Below is a merged screen shot of a tool I’ll introduce you to soon that includes my SEO statistics on May 24th (left) and June 11th (right). The date of May 24th is significant, as that is the date I made major changes similar to what is described by Nathan.

The two figures in question relate to the “Supplemental” index number and the “Cached” number. As you can see, prior to my changes to my robots.txt file, Google’s supplemental index contained 175 pages from my site and 198 cached (non-supplemental) pages. In other words, for every 1 page I had cached, I had 0.88 pages in the supplemental index.

Compare that to the numbers after the simple update I made. I now have 339 cached pages with 220 in the supplemental index. Since Google only removes pages previously listed in the supplemental index every 36 days according to Nathan, none of the original 175 listed on May 24th have been removed. As such, I’ve had 132 pages cached with only 55 supplemental pages. In other words, for every 1 page I had cached, I had 0.42 pages in the supplement index for a 212% improvement!

Hopefully after reading that, you’re interested in getting similar (or perhaps even better) results for your own website! Nathan’s advice for getting out of Google’s supplemental index is pretty simple in theory, though slightly difficult if you’re non-technical or new to the robots.txt file.

The first step is to first find out how many pages from your website is in the supplemental index with SEOBook’s SEO for Firefox plug-in. Look for the “Supplemental” number as circled in the screen shot above.

For the enthusiastic that are looking to learn a little bit more about SEO and Google’s sandbox, double-click on the “Supplemental” row in the window, as seen below. Your browser will navigate to Google, listing all pages from your site that are listed within the sandbox. If you scroll down the list while looking at each URL that falls within the sandbox, you may start to notice a pattern among the pages stuck here. Again, for the enthusiastic, you can use with your soon-to-come robots.txt knowledge to help you cut even more pages from the sandbox than what this article explains!

Once you see how many pages you have in the supplemental index, the next step is to update the robots.txt file on your server so that Google won’t spider pages matching certain criteria listed here. The goal is to add “disallow” certain URL paths via the robots.txt file such that Google won’t attempt to index archive pages, category pages, RSS feeds, and trackbacks. For an example, take a look at the robots.txt file used on my blog. Although I haven’t made all the changes I intend to yet, you’ll see that I’ve added the following four significant lines (in addition to several other as well).

• Disallow: /category/
• Disallow: /page/
• Disallow: */feed/
• Disallow: */trackback/

That’s all there is to it! I highly suggest updating your own robots.txt file to include similar changes depending on your blogging software or website structure. Feel free to leave any questions here for guidance.