To: gopher@complete.org
Subject: [gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)
References: <20010114002128.D4061@wonderland.linux.it>
	<200101140028.QAA13514@stockholm.ptloma.edu>
	<20010113193659.A20066@mothra>
From: John Goerzen <jgoerzen@complete.org>
Date: 14 Jan 2001 21:21:05 -0500
In-Reply-To: <20010113193659.A20066@mothra>
Message-ID: <873delr1wu.fsf@complete.org>
Lines: 31
User-Agent: Gnus/5.090001 (Oort Gnus v0.01) XEmacs/21.1 (Channel Islands)
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org


David Allen <s2mdalle@titan.vcu.edu> writes:

> Personally, I don't see any reason not to just lift the robots.txt
> verbatim and add it.  Minus the User-Agent part, which gopher doesn't

I was going to suggest that after I read up on the format of it :-)

> Questions on these items though:
> 
> 1.) Should "some_directory" be a selector string, or the portion of the URL
>   after the host?  I.e. on my system, I have a selector, 
> "1/Python Stuff".  Should it be listed as that, or as the portion of
>   the URL after the host, which would be: "/11/Python%20Stuff"?

Hrm, that's a good question.

> 2.) Should gopher servers hide files by the name of robot.txt from the
>   view of the client?  (i.e. should it be possible for a human user to
>   come into a directory and see a robots.txt entry, or should it be
>   automagically hidden?)

That should be at the discretion of the admin.  UMN gopherd has a type
"X" for this purpose.

-- John


-- 
John Goerzen <jgoerzen@complete.org>                       www.complete.org
Sr. Software Developer, Progeny Linux Systems, Inc.    www.progenylinux.com
#include <std_disclaimer.h>                     <jgoerzen@progenylinux.com>