Received: with ECARTIS (v1.0.0; list gopher); Wed, 12 Oct 2005 14:57:35 -0500 (CDT) Received: from gatekeeper.excelhustler.com ([69.44.136.67] helo=wile.internal.excelhustler.com) by glockenspiel.complete.org with esmtps (with TLS-1.0:RSA_AES_256_CBC_SHA:32) (No TLS peer certificate) (Exim 4.50) id 1EPmjA-00060p-OK; Wed, 12 Oct 2005 14:57:35 -0500 Received: from jgoerzen by wile.internal.excelhustler.com with local (Exim 4.54) id 1EPmj3-0004lV-Pu; Wed, 12 Oct 2005 14:57:21 -0500 Date: Wed, 12 Oct 2005 14:57:21 -0500 From: John Goerzen <jgoerzen@complete.org> To: gopher@complete.org Subject: [gopher] Re: New Gopher Wayback Machine Bot Message-ID: <20051012195721.GA18058@excelhustler.com> References: <20051012180132.GA19083@complete.org> <20051012185141.GA21016@complete.org> <20051012192342.GA9832@SDF.LONESTAR.ORG> MIME-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20051012192342.GA9832@SDF.LONESTAR.ORG> User-Agent: Mutt/1.5.11 X-Spam-Status: No (score 0.0): AWL=0.001 X-Virus-Scanned: by Exiscan on glockenspiel.complete.org at Wed, 12 Oct 2005 14:57:35 -0500 Content-Transfer-Encoding: 8bit X-archive-position: 1111 X-ecartis-version: Ecartis v1.0.0 Sender: gopher-bounce@complete.org Errors-to: gopher-bounce@complete.org X-original-sender: jgoerzen@complete.org Precedence: bulk Reply-to: gopher@complete.org List-help: <mailto:ecartis@complete.org?Subject=help> List-unsubscribe: <mailto:gopher-request@complete.org?Subject=unsubscribe> List-software: Ecartis version 1.0.0 List-Id: Gopher <gopher.complete.org> X-List-ID: Gopher <gopher.complete.org> List-subscribe: <mailto:gopher-request@complete.org?Subject=subscribe> List-owner: <mailto:jgoerzen@complete.org> List-post: <mailto:gopher@complete.org> List-archive: <http://www.complete.org/mailinglists/archives/> X-list: gopher

On Wed, Oct 12, 2005 at 02:23:42PM -0500, Benn Newman wrote: > I know that Veronica-2 obeys robots.txt. Does the bot have its own client name (or whatever it is that > robots.txt calls it? I have content on my server that is easily replaceable (a mirror) and there > is no reason to do a full text of it. Last time I updated [my not publically avaliable] JUGHEAD, the > 'gophermap' file was already eight megabytes and I have added content since then.

OK. I can do robots.txt as well. I'll use user agent name gopherbot.

For the moment, I have told it to pick a new host at random before each connection, which should pretty much keep most people from even noticing that there is a bot going on.