Date: Wed, 12 Oct 2005 14:57:21 -0500
From: John Goerzen <jgoerzen@complete.org>
To: gopher@complete.org
Subject: [gopher] Re: New Gopher Wayback Machine Bot
Message-ID: <20051012195721.GA18058@excelhustler.com>
References: <20051012180132.GA19083@complete.org>
 <20051012185141.GA21016@complete.org>
 <20051012192342.GA9832@SDF.LONESTAR.ORG>
MIME-Version: 1.0
Content-type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20051012192342.GA9832@SDF.LONESTAR.ORG>
User-Agent: Mutt/1.5.11
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org

On Wed, Oct 12, 2005 at 02:23:42PM -0500, Benn Newman wrote:
> I know that Veronica-2 obeys robots.txt. Does the bot have its own client name (or whatever it is that 
> robots.txt calls it? I have content on my server that is easily replaceable (a mirror) and there
> is no reason to do a full text of it. Last time I updated [my not publically avaliable] JUGHEAD, the 
> 'gophermap' file was already eight megabytes and I have added content since then. 

OK.  I can do robots.txt as well.  I'll use user agent name gopherbot.

For the moment, I have told it to pick a new host at random before each
connection, which should pretty much keep most people from even noticing
that there is a bot going on.