Date: Wed, 12 Oct 2005 21:07:52 -0500
From: Benn Newman <newmanbe@sdf.lonestar.org>
To: gopher@complete.org
Subject: [gopher] Re: New Gopher Wayback Machine Bot
Message-ID: <20051013020752.GA20333@SDF.LONESTAR.ORG>
References: <20051012180132.GA19083@complete.org>
 <200510122345.QAA17070@floodgap.com>
Mime-Version: 1.0
Content-type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <200510122345.QAA17070@floodgap.com>
User-Agent: Mutt/1.4.2.1i
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org

On Wed, Oct 12, 2005 at 04:45:56PM -0700, Cameron Kaiser wrote:
> I'm a little concerned about your project since I do host a number of large
> subparts which are actually proxied services, and I think even a gentle bot
> going methodically through them would not be pleasant for the other side
> (especially if you mean to regularly update your snapshot).
I agree with Cameron and think that until it supports robots.txt, it shoudl be disabled.
I also think it would be a good idea to have it from a single, publically known
domain name.
-- 
Benn Newman
newmanbe@sdf.lonestar.org | gopher://igneous-rock.homeunix.net

Bradley's Bromide:                                                              
        If computers get too powerful, we can organize them into a              
committee -- that will do them in.