The other day, I ran into an old friend and colleague of mine, who told me that an unfinished site on which he was working showed up in Google search results. Fortunately, he was able to remove the entry from Google's search index after submitting a request to Google, but I imagine his client was disappointed by the situation and unhappy while waiting for Google to process the removal request. My friend was surprised that the site appeared in Google's index, because he had set up his robots.txt file to block search engines from the site.

Learning the hard way

I can't claim any moral high ground on this issue; I made a similar mistake years ago on one of my first professional web projects. I wrongly assumed that, if there were no sites linking to the development copy of the site on which I was working, search engines would have no way to find and crawl the development site. Wrong! I got a nasty surprise when my client sent me an angry e-mail asking why his half-complete new site was showing up in Google. I had to rush to send a removal request to Google, check several other major search engines to confirm that they had not indexed the site, and apologize profusely to the client. Fortunately he forgave me, and the development site dropped out of Google's search index.