Received: with ECARTIS (v1.0.0; list gopher); Sun, 30 May 2004 18:08:13 -0500 (CDT) Return-Path: <tfraser@cs.umd.edu> X-Original-To: gopher@complete.org Delivered-To: gopher@complete.org Received: from localhost (localhost [127.0.0.1]) by glockenspiel.complete.org (Postfix) with ESMTP id 65BD52AA for <gopher@complete.org>; Sun, 30 May 2004 18:08:12 -0500 (CDT) Received: from glockenspiel.complete.org ([127.0.0.1]) by localhost (glockenspiel [127.0.0.1]) (amavisd-new, port 10025) with ESMTP id 18146-02 for <gopher@complete.org>; Sun, 30 May 2004 18:08:10 -0500 (CDT) Received: from junkmail.cs.umd.edu (junkmail.cs.umd.edu [128.8.128.69]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by glockenspiel.complete.org (Postfix) with ESMTP id BFE0AF9 for <gopher@complete.org>; Sun, 30 May 2004 18:08:05 -0500 (CDT) Received: from nerds.cs.umd.edu (nerds.cs.umd.edu [128.8.129.84]) by junkmail.cs.umd.edu (8.12.10/8.12.5) with ESMTP id i4UN806p013431 for <gopher@complete.org>; Sun, 30 May 2004 19:08:00 -0400 (EDT) Received: (from tfraser@localhost) by nerds.cs.umd.edu (8.12.10/8.12.5) id i4UN7x6P027575 for gopher@complete.org; Sun, 30 May 2004 19:07:59 -0400 (EDT) Date: Sun, 30 May 2004 19:07:59 -0400 From: Tim Fraser <tfraser@cs.umd.edu> To: gopher@complete.org Subject: [gopher] Re: Cicada Incomplete Gopher Census Message-ID: <20040530230758.GA27407@nerds.cs.umd.edu> References: <20040528022333.GA7147@nerds.cs.umd.edu> <200405280245.TAA06966@floodgap.com> Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200405280245.TAA06966@floodgap.com> User-Agent: Mutt/1.4.1i X-Virus-Scanned: by amavisd-new-20030616-p7 (Debian) at complete.org Content-Transfer-Encoding: 8bit X-archive-position: 929 X-ecartis-version: Ecartis v1.0.0 Sender: gopher-bounce@complete.org Errors-to: gopher-bounce@complete.org X-original-sender: tfraser@cs.umd.edu Precedence: bulk Reply-to: gopher@complete.org List-help: <mailto:ecartis@complete.org?Subject=help> List-unsubscribe: <mailto:gopher-request@complete.org?Subject=unsubscribe> List-software: Ecartis version 1.0.0 List-Id: Gopher <gopher.complete.org> X-List-ID: Gopher <gopher.complete.org> List-subscribe: <mailto:gopher-request@complete.org?Subject=subscribe> List-owner: <mailto:jgoerzen@complete.org> List-post: <mailto:gopher@complete.org> List-archive: <http://www.complete.org/mailinglists/archives/> X-list: gopher

ck> Actually, you can see the Floodgap census here

Thanks for updating the floodgap directory! It was browsing through this directory and cools sites like quux.org (to name just one) that got me interested in Gopher again. I think the "new gopher servers since 1999" directory is an especially interesting feature, since it highlights new growth.

ck> After the V-2 cleanup this weekend, it has pared itself down to ck> 255 unique hosts and a database of about 1.8 million selectors.

OK, I found only 154, so I clearly have a bug. My selector counts seem very low, too. I'm not sure it's worth debugging given that the floodgap index is updating again, but just in case I get bored: my spider is supposed to follow only selectors with type 1 or 11. Are there other directory types that I should follow?

tf> my primitive spider had been automatically banned ck> It was? I don't remember blocking any IP addresses ...

Perhaps I was mistaken. After using another machine to read point 4 in the floodgap terms of service (the one about automatically blocking the netblocks of spiders and robots), I just assumed that was the cause without any real proof and left it at that.

How does floodgap's Veronica-2 spider limit the load it places on sites? Does it check for a robots.txt file, or some similar mechanism?

- Tim Fraser