Posted by Jonathan on September 5, 2002, at 20:56:07
In reply to Re: Search engines, posted by oracle on September 5, 2002, at 0:50:30
Jane D > I don't suppose there is a way to prevent Social (and 2000 and admin) from being picked up by the search engines, is there?
Oracle > That would be impossible ...
Actually, it *is* possible, at least for search engines that look for and respect a "robots.txt" file in the site's root directory - e.g. for this site, http://www.dr-bob.org/robots.txt , which doesn't (yet?) exist; I think most popular, reputable search-engines do this, including Google - http://www.google.com/webmasters/3.html#B3 .
Further details at http://www.robotstxt.org/wc/exclusion-admin.html .
Like you, Jane (and perhaps Phil?), I have a personal interest in this: one of my now regretted Social posts, from which I'm instantly recognisable by anyone who knows me in real life, is among the few hits that would be returned in the plausible event that any of my mathematician colleagues should search Google for "secaps.cirtem.laimonylop" (spelt backwards this time to confound those robot spiders!)
> An ever paranoid Jane
An even more paranoid Jonathan :)
poster:Jonathan
thread:7296
URL: http://www.dr-bob.org/babble/admin/20020725/msgs/7305.html