Date: Thu, 19 Jun 2003 14:53:26 -0500
From: John Goerzen <jgoerzen@complete.org>
To: gopher@complete.org
Subject: [gopher] Re: Veronica-2 and robot exclusion
Message-ID: <20030619195326.GM6851@complete.org>
References: <200306181324.GAA12136@floodgap.com>
Mime-Version: 1.0
Content-type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <200306181324.GAA12136@floodgap.com>
User-Agent: Mutt/1.5.4i
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org

I'm on a train just now and away from the net (horrors!) so I can't check
your URLs, but a quick question:

Is this the same robots.txt format that is used for the Web?  If so, could
one potentially use a single robots.txt file in the root of a site that is
served up both as Gopher and over HTTP?  (As quux.org is, for instance)
That would be a very nice feature.

On Wed, Jun 18, 2003 at 06:24:18AM -0700, Cameron Kaiser wrote:
> Also, I was thinking of making the file ".robots.txt" since many Unix
> gophers don't serve dot-files, although there are a growing number of
> Windows-hosted gophers and I don't know if it will break these (I don't
> do x86 myself).

AFAIK, all Unix gophers *will* serve dot files, many just won't include them
in directory listings.  (Which I think is what you meant anyway)