A Dangerous Search Engine Optimization Mistake.
By: Julia Udovenko | Posted: 12th June 2006
Whether you are an experienced SEO or a novice, reading this article may save you a lot of grief later on. While the situation I'm going to describe here is not common, if it does happen it's capable of completely erasing the results of your efforts to optimize your website or even get it indexed by major search engines.
Has it been months since your website went live and although you've been actively promoting it all this time, there is no sign of it in Google, MSN and some other search engines?
You are now sure that at least a hundred large websites link back to yours, yet your website is still not indexed by major search engines except Yahoo. If this applies to you, read on and you may find the ultimate solution to your troubles.
How are the robots.txt and the "200-OK" response code related to any of this? Let me tell you my story. My first encounter with side effects of code "200-OK" was when I tried to upload one of my websites' sitemap to Google Sitemaps. It wouldn't verify my website because apparently my web hosting provider set its servers to return code "200-OK" no matter what. So if someone would enter a bad URL instead of displaying the error page it would display some ads and return a "200-OK" HTTP response instead of "404-not found". I decided that it's not a big deal and I will do just fine without Google Sitemaps and continued actively promoting my new website in the web community. I didn't bother creating robots.txt file because I wanted all my pages indexed at that time.
Months went by, but my efforts showed no result: my website seemed virtually invisible to Google and MSN. Months of active promotion didn't seem to make the slightest difference; I had no idea what to do. Numerous emails to Google support and other fellow SEO's didn't seem to help. Everyone referred to the so-called "Google dance", where pages get dropped from index occasionally, but I knew it wasn't the case for me. Finally I gave up and moved on.
A couple of moths later I was checking my Google Sitemaps account and I noticed the strangest thing: the stats for the website that caused me so much grief showed that Google crawled many of its pages, but each and every one of them showed some "robots.txt unavailable" error. Reading on, I discovered that Google did crawl my website, in fact it crawled it many times, but before indexing any of my pages, it looked for the robots.txt file, but since I didn't have one, it couldn't access it. Also having received a "200-OK" code instead of "404-not found", it assumed that robots.txt exists but is presently unavailable and therefore, rescheduled the crawl for a later time. In my case it's been rescheduling it for a few months while I was desperately trying to promote my website.
Imagine my reaction when I found out this. The evil combination of always returning code "200-OK" and absence of the robots.txt is deadly for search engine optimization. So if you are experiencing a similar situation, ask your web hosting provider whether they return any HTTP error codes, 404 in particular, and make sure you have the robots.txt file in your root directory.
This article is free for republishing
Printed From: http://www.articlealley.com/article_62428_62.html
Back to the original article
Tags: google, servers, yahoo, other search engines, novice, major search engines, msn, seo, grief, web community, web hosting provider, fellow, first encounter, ultimate solution, moths