Date: Sun, 14 Jan 2001 22:34:27 -0500
From: David Allen <s2mdalle@titan.vcu.edu>
To: gopher@complete.org
Subject: [gopher] Re: Gopher "robots.txt"
Message-ID: <20010114223427.B26948@mothra>
References: <20010113193659.A20066@mothra>
 <200101140056.QAA12702@stockholm.ptloma.edu> <20010114172831.A25812@mothra>
 <20010114152537.B1974@success-info.com>
Mime-Version: 1.0
Content-type: text/plain; charset=us-ascii
User-Agent: Mutt/1.0.1i
In-Reply-To: <20010114152537.B1974@success-info.com>;
 from emanuel@heatdeath.org on Sun, Jan 14, 2001 at 03:25:37PM -0800
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org


On Sun, Jan 14, 2001 at 03:25:37PM -0800, emanuel at heatdeath organisation wrote:
> 
> On Sun, Jan 14, 2001 at 05:28:31PM -0500, David Allen wrote:
> > That's not a bad idea...I'd prefer personally to put whatever
> > information that this null statement has in the gopher+ fields to
> > avoid overloading the meaning of the other fields (type, name, host,
> > port).  So maybe something like this to disallow the debian
> > subdirectory:
> > 
> > iFFerror.hostF909FDisallow: selector_for_debian_directory
> 
> Why not put this information in the Gopher+ attribute information?  Have
> an attribute that means "ignore this selector" to the robot.  This way
> clients that don't know about it will never see it.  If it's set for a
> directory, the robot will ignore than directory and will never see
> anything liked by the directory.
> 
> Isn't this kind of thing what the Gopher+ attribute information is for?

I guess it depends on the robot.  It is sort of what the attribute
information is for, but should it be the case that a robot has to
fetch the attribute information for every single directory it ever
fetches?  That could cause a hit on performance/bandwidth since a
robot can't fetch just the resource and be safe with that.

-- 
David Allen
http://opop.nols.com/
----------------------------------------
"Be excellent to each other."
	- Bill and Ted's Excellent Adventure