Message-ID: <009401c335b6$24e89c50$43da5982@killspy>
From: "Ruliz Galaxor" <killspy@roelisch.tmfweb.nl>
To: <gopher@complete.org>
References: <200306181324.GAA12136@floodgap.com>
Subject: [gopher] Re: Veronica-2 and robot exclusion
Date: Wed, 18 Jun 2003 18:24:50 +0200
MIME-Version: 1.0
Content-type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org

> I re-read the long thread back in January '01 where we went over how to
> implement a robot exclusion standard for Gopherspace, and the solution
that
> had the widest support (except from yours truly at that time) was to
> re-use the existing robots.txt (for your convenience, the current 1994
> standard is available at
>
> gopher://gopher.floodgap.com/0/v2/robotstxt.txt

Should the robots.txt be visible in the index if you request it normally, or
is it also allowed to be invisible?

Invisible as in:
Requesting the index doesnt show any robots.txt -> gopher://foo.bar.com/
but requesting robots.txt works -> gopher://foo.bar.com/0/robots.txt

Is this 'invisible' use of robots.txt allowed?

greets,

Ruliz Galaxor
gopher://kabel218067.mobiel.utwente.nl/