From: Cameron Kaiser <spectre@floodgap.com>
Message-Id: <200306181744.KAA12996@floodgap.com>
Subject: [gopher] Re: Veronica-2 and robot exclusion
In-Reply-To: <009401c335b6$24e89c50$43da5982@killspy> from Ruliz Galaxor at
 "Jun 18, 3 06:24:50 pm"
To: gopher@complete.org
Date: Wed, 18 Jun 2003 10:44:55 -0700 (PDT)
MIME-Version: 1.0
Content-type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org

> Should the robots.txt be visible in the index if you request it normally, or
> is it also allowed to be invisible?
> 
> Invisible as in:
> Requesting the index doesnt show any robots.txt -> gopher://foo.bar.com/
> but requesting robots.txt works -> gopher://foo.bar.com/0/robots.txt
> 
> Is this 'invisible' use of robots.txt allowed?

This would be the way I'd do it, and this is easy to do with a link list
(like a Bucktooth gophermap file). There's probably no harm if the robots.txt
file does get presented to the user, though.

In any case, itemtype=0 sel=robots.txt is what the crawler will ask for.
(Note no leading slash, since Gopher is supposed to be independent of the
host filesystem.)

-- 
---------------------------------- personal: http://www.armory.com/~spectre/ --
 Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * ckaiser@floodgap.com
-- FORTUNE: You learn from your mistakes. Today will be very educational. -----