From: Cameron Kaiser <spectre@stockholm.ptloma.edu>
Message-Id: <200101140056.QAA12702@stockholm.ptloma.edu>
Subject: [gopher] Re: Gopher "robots.txt" (was Re: New V-2 WAIS database)
In-Reply-To: <20010113193659.A20066@mothra> from David Allen at "Jan 13,
 1 07:36:59 pm"
To: gopher@complete.org
Date: Sat, 13 Jan 2001 16:56:21 -0800 (PST)
MIME-Version: 1.0
Content-type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org


> Personally, I don't see any reason not to just lift the robots.txt
> verbatim and add it.  Minus the User-Agent part, which gopher doesn't
> really support.  (Or, it could always be '*' in case web agents for
> some reason ended up reading the file) So maybe we could do something
> like this: 

I'm not fond of the robots.txt file for a number of reasons. Theoretically
because it separates the resource from its respective permissions (they
really should go together), but practically because since the robot has to
shuttle between servers it needs to cache the particular robots.txt of each
server or keep refetching it. This is even worse for me since V-2 isn't running
all the time, so it would have to cache all this persistently. I'm sure that
web bots have similar problems to worry about, but I'm a lazy bum.

I'd like to see an approach on a per-directory (menu) basis, maybe a selector
like this in menus not to be indexed:

iFFerror.hostF909

F, of course, is Gopherese for <TAB>. It's a null display string of item
type i, so normal clients just display a blank line. You can encode any
sort of pragma this way; I just encoded it in the port number for poor sods
who use IE so that clicking on the link doesn't do anything useful. 909 as
the port number is the pragma for the robot to not index anything in this
directory/menu, so it leaves it alone.

Such a pragma, like I said, would be wide open for other applications and
best of all is very simple to implement on both robot (my) side and servers.

Comments?

-- 
----------------------------- personal page: http://www.armory.com/~spectre/ --
 Cameron Kaiser, Point Loma Nazarene University * ckaiser@stockholm.ptloma.edu
-- Immigration is the sincerest form of flattery. -- Jack Paar ----------------