Received: with ECARTIS (v1.0.0; list gopher); Wed, 12 Oct 2005 13:51:41 -0500 (CDT) Received: from jgoerzen by glockenspiel.complete.org with local (Exim 4.50) id 1EPlhV-0005TC-8i; Wed, 12 Oct 2005 13:51:41 -0500 Date: Wed, 12 Oct 2005 13:51:41 -0500 From: John Goerzen <jgoerzen@complete.org> To: gopher@complete.org Subject: [gopher] Re: New Gopher Wayback Machine Bot Message-ID: <20051012185141.GA21016@complete.org> References: <20051012180132.GA19083@complete.org> Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20051012180132.GA19083@complete.org> User-Agent: Mutt/1.5.9i Content-Transfer-Encoding: 8bit X-archive-position: 1108 X-ecartis-version: Ecartis v1.0.0 Sender: gopher-bounce@complete.org Errors-to: gopher-bounce@complete.org X-original-sender: jgoerzen@complete.org Precedence: bulk Reply-to: gopher@complete.org List-help: <mailto:ecartis@complete.org?Subject=help> List-unsubscribe: <mailto:gopher-request@complete.org?Subject=unsubscribe> List-software: Ecartis version 1.0.0 List-Id: Gopher <gopher.complete.org> X-List-ID: Gopher <gopher.complete.org> List-subscribe: <mailto:gopher-request@complete.org?Subject=subscribe> List-owner: <mailto:jgoerzen@complete.org> List-post: <mailto:gopher@complete.org> List-archive: <http://www.complete.org/mailinglists/archives/> X-list: gopher

On Wed, Oct 12, 2005 at 01:01:32PM -0500, John Goerzen wrote: > This bot is now running on my laptop and is spidering away. The IP that > you see connections from will vary depending on where my laptop is at a > given time ;-)

Speaking of this, do we have any equivolent of robots.txt? Cameron, do you have some sort of exclude list or anything?

-- John Goerzen Author, Foundations of Python Network Programming http://www.amazon.com/exec/obidos/tg/detail/-/1590593715