Date: Sun, 14 Jan 2001 17:28:31 -0500
From: David Allen <s2mdalle@titan.vcu.edu>
To: gopher@complete.org
Subject: [gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)
Message-ID: <20010114172831.A25812@mothra>
References: <20010113193659.A20066@mothra>
 <200101140056.QAA12702@stockholm.ptloma.edu>
Mime-Version: 1.0
Content-type: text/plain; charset=us-ascii
User-Agent: Mutt/1.0.1i
In-Reply-To: <200101140056.QAA12702@stockholm.ptloma.edu>;
 from spectre@stockholm.ptloma.edu on Sat, Jan 13, 2001 at 04:56:21PM -0800
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org


On Sat, Jan 13, 2001 at 04:56:21PM -0800, Cameron Kaiser wrote:
> I'd like to see an approach on a per-directory (menu) basis, maybe a selector
> like this in menus not to be indexed:
> 
> iFFerror.hostF909
> 
> F, of course, is Gopherese for <TAB>. It's a null display string of item
> type i, so normal clients just display a blank line. You can encode any
> sort of pragma this way; I just encoded it in the port number for poor sods
> who use IE so that clicking on the link doesn't do anything useful. 909 as
> the port number is the pragma for the robot to not index anything in this
> directory/menu, so it leaves it alone.
> 
> Such a pragma, like I said, would be wide open for other applications and
> best of all is very simple to implement on both robot (my) side and servers.

That's not a bad idea...I'd prefer personally to put whatever
information that this null statement has in the gopher+ fields to
avoid overloading the meaning of the other fields (type, name, host,
port).  So maybe something like this to disallow the debian
subdirectory:

iFFerror.hostF909FDisallow: selector_for_debian_directory

I like the idea of using these null statements as pragmas of a sort,
so it may be better to use a keyword like Disallow or deny or
something like that, so that multiple pragmas can be used.  Just as a
random suggestion, you could use this to even define new types
previously unknown like:
iFFerror.hostF909FType "~" mimetype
to allow for the use of the ~ type to represent things of <mimetype>

For display purposes, just as a convention, it may be best to also put
these at the bottom of documents rather than the top or middle, so if it is
rendered as a blank line (older clients and clients that don't know
about these statements), it won't vertically indent the actual
content of the directory. 

-- 
David Allen
http://opop.nols.com/