From: Cameron Kaiser <spectre@stockholm.ptloma.edu>
Message-Id: <200101150544.VAA10768@stockholm.ptloma.edu>
Subject: [gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)
In-Reply-To: <87y9wdpnaf.fsf@complete.org> from John Goerzen at "Jan 14,
 1 09:22:16 pm"
To: gopher@complete.org
Date: Sun, 14 Jan 2001 21:44:29 -0800 (PST)
MIME-Version: 1.0
Content-type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org


> This proposal would be extremely difficult to implement for gopher
> sites that carry mirrors, especially mirrors of other gopher sites.

Well, it's just a framework for a pragma to contain such information. 
Consider possibly a menu that had, as one of its offerings, such a mirror,
under 1/stayout, and maybe another resource under 1/dontindex also.

You could instead put in something like

iF1/stayout;1/dontindexFerror.hostF909

in the menu that references these selectors.

Mind you, I'd be happy with any approach that works on a per-menu level,
just as long as the bot doesn't have to cache every server's particular
robot policy and can determine the policy for a selector from the menu(s)
that reference that selector. This is just one way I can think of.

-- 
----------------------------- personal page: http://www.armory.com/~spectre/ --
 Cameron Kaiser, Point Loma Nazarene University * ckaiser@stockholm.ptloma.edu
-- if (you.canRead(this)) you.canGet(new job(!problem)); -- Seen at JavaOne ---