Posted by pseudoname on January 8, 2006, at 10:46:28
In reply to Re: searchability, posted by Dr. Bob on January 7, 2006, at 3:25:17
> But I still think a non-Googleable board might be interesting. Would a board like that need to be restricted to registered members, too?
Not at all. You can hide any page from Google & other search engines while keeping it open to everybody else with the Robots Exclusion Protocol -- a "robots.txt" file.
A couple pages explaining it begin here:
http://www.robotstxt.org/wc/exclusion-user.htmlYou need root-level access to the dr-bob.org domain, which I assume you have.
I'm pretty sure, however, that *any* page's URL can still appear in a Google search-results list if a link to that page is on any other web page. If you hide msg #123, for example, with your robots.txt file, I can still put a link to that msg on my home page, which is indexed by Google. If I put the word "opioids" as the link's hypertext, then Google may return the URL of msg #123 in the results-list for an "opioids" search. It still won't report what's actually on page #123, but Google will report that such a URL exists and that it probably has something to do with opioids.
This is on reason why you sometimes see just a link, with no snippets, in a Google results list.
I don't think there's a way to block that type of listing of a hidden page, although I don't know. For Babble posts, it probably wouldn't be a big problem.
Another way any author can hide an individual page is to put meta tags in its html header:
<meta name="robots" content="noindex">
Google obeys this instruction, but apparently many search engines don't.
poster:pseudoname
thread:593798
URL: http://www.dr-bob.org/babble/admin/20051205/msgs/596547.html