Received: with ECARTIS (v1.0.0; list gopher); Thu, 27 May 2004 21:23:43 -0500 (CDT) Return-Path: <tfraser@cs.umd.edu> X-Original-To: gopher@complete.org Delivered-To: gopher@complete.org Received: from localhost (localhost [127.0.0.1]) by glockenspiel.complete.org (Postfix) with ESMTP id 7C234148 for <gopher@complete.org>; Thu, 27 May 2004 21:23:43 -0500 (CDT) Received: from glockenspiel.complete.org ([127.0.0.1]) by localhost (glockenspiel [127.0.0.1]) (amavisd-new, port 10025) with ESMTP id 04859-01 for <gopher@complete.org>; Thu, 27 May 2004 21:23:41 -0500 (CDT) Received: from junkmail.cs.umd.edu (junkmail.cs.umd.edu [128.8.128.69]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by glockenspiel.complete.org (Postfix) with ESMTP id 02FCBC8 for <gopher@complete.org>; Thu, 27 May 2004 21:23:41 -0500 (CDT) Received: from nerds.cs.umd.edu (nerds.cs.umd.edu [128.8.129.84]) by junkmail.cs.umd.edu (8.12.10/8.12.5) with ESMTP id i4S2NY6p019781 for <gopher@complete.org>; Thu, 27 May 2004 22:23:34 -0400 (EDT) Received: (from tfraser@localhost) by nerds.cs.umd.edu (8.12.10/8.12.5) id i4S2NYj6007198 for gopher@complete.org; Thu, 27 May 2004 22:23:34 -0400 (EDT) Date: Thu, 27 May 2004 22:23:33 -0400 From: Tim Fraser <tfraser@cs.umd.edu> To: gopher@complete.org Subject: [gopher] Cicada Incomplete Gopher Census Message-ID: <20040528022333.GA7147@nerds.cs.umd.edu> Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.1i X-Virus-Scanned: by amavisd-new-20030616-p7 (Debian) at complete.org Content-Transfer-Encoding: 8bit X-archive-position: 919 X-ecartis-version: Ecartis v1.0.0 Sender: gopher-bounce@complete.org Errors-to: gopher-bounce@complete.org X-original-sender: tfraser@cs.umd.edu Precedence: bulk Reply-to: gopher@complete.org List-help: <mailto:ecartis@complete.org?Subject=help> List-unsubscribe: <mailto:gopher-request@complete.org?Subject=unsubscribe> List-software: Ecartis version 1.0.0 List-Id: Gopher <gopher.complete.org> X-List-ID: Gopher <gopher.complete.org> List-subscribe: <mailto:gopher-request@complete.org?Subject=subscribe> List-owner: <mailto:jgoerzen@complete.org> List-post: <mailto:gopher@complete.org> List-archive: <http://www.complete.org/mailinglists/archives/> X-list: gopher

I recently indulged my long-time fascination with Gopherspace by writing a spider program to determine how many Gophers are actually out there. Of course, now that I'm finished, I've found Cameron Kaiser's post in the gopher mailing list archive announcing that he'd finished updating the the much nicer V-2 database at floodgap.org, even as my little toy spider was chugging away. Seems I should have picked a different project to pass the time! ;^)

Fortunately, nobody ever said a fun hacking project had to be *useful*, so I'll post this note about my results nonetheless. While my program is not perfect, its results still seem interesting: After spidering intermittently between Sunday 23 May and Thursday 27 May 2004, I found 154 Gopher servers live and ready to serve useful data. I found an additional 16 live Gophers that could not serve any useful data due to an inability to access their log files, configuration files, data directories, or back-end NNTP servers. My spider found links to 1777 Gopher servers that were not operational at the time of my spidering.

I've somewhat pretentiously decided to call this the "Cicada Incomplete Gopher Census". "Cicada" is in honor of the 17-year brood of magicicadas that kept my computer room filled with their astonishing multidecibel flying-saucer wailing throughout the entire spidering process. (It's *really* loud!) "Incomplete" is in recognition of the fact that the census almost certainly missed some Gophers. The full results of this incomplete Gopher census and the spidering program I used to produce them are available via Gopher at:

sdf.lonestar.org/11/users/tfraser

Any feedback would be appreciated.

Perhaps the most egregious error in my survey is my failure to completely spider two large sites: Gopher.dna.affrc.go.jp:70 and bbs.nsysu.edu.tw:70. Near the end of my spidering, I realized that my primitive spider had been automatically banned from at least one site (floodgap.com) for behaving rudely in its quest to traverse all of the site's directories looking for links to other sites. Horrified that my little project to celebrate Gopher's longevity might be causing trouble for Gopher administrators, I cut the spidering of these last two sites short. My apologies to anyone I might have inconvenienced!

With these shortcomings in mind, here's some highlights:

The five Gophers with the most selectors might be:

74632 ftp.std.com:70 Software Tool & Die's The World (ISP) 29931 fas.sfu.ca:75 Faculty of Applied Sciences, Simon Fraser University 29799 nic.merit.edu:7043 Merit Network Information Center Services (MichNet) 28201 dongpo.math.ncu.edu.tw:70 Department of Mathematics, National Central University" 21775 osiris.wu-wien.ac.at:71 A Gopher at the University of Vienna

The five Gophers with links to the most other sites might be:

851 gopher.csie.nctu.edu.tw:70 Dept. of Comp Sci and Info Eng, National Chiao-Tung University" 211 gopher.floodgap.com:70 Floodgap systems, formerly gopher.ptloma.edu 185 www.polarhome.com:27070 Polarhome (public access UNIX and VMS) 180 sdf.lonestar.org:70 Super Dimension Fortress (public access UNIX) 178 iubio.bio.indiana.edu:70 Indiana University Bio-Archive

Many thanks to all who continue to support Gopherspace,

Tim Fraser