So search engines are tricky to understand, especially if you want the search engines to find your site and people to find your site via the search engines. One of the things you may want to do is restrict pages that the search engines find, that might seem silly to exclude yourself but I don’t want people inadvertently finding my site.
Here’s an example, I don’t let* the search engines search my monthly archives. Because on the page of 20-50 unrelated posts the information on that page is pretty much unrelated by the end of the month. In December 2006 the mention of “iPods” on the 3rd, has nothing to do with any of the mentions of “family” or “Christmas” but if some searched for “iPods” & “family” or “iPods” & “Christmas” they may come upon that page and while I want traffic I’m not trying to lure people here under false pretenses. This is why I have categories like “Apple” and “Friends + Family” that will have related posts in one location (and even then it’s still kinda a wide range of articles).
So far I’ve just told you why you want a ROBOTS.TXT file but I haven’t told you how and I’m not planning on it, because Google just put together on “Controlling how search engines access and index your website with ROBOTS.TXT”. It’s got a lot of links that take you all over the place but there’s lots of good info there. Continue reading
