Date: Tue, 29 Nov 2005 17:20:06 -0600
From: John Goerzen <jgoerzen@complete.org>
To: gopher@complete.org
Subject: [gopher] Re: Bot update
Message-ID: <20051129232006.GP19727@complete.org>
References: <20051031034851.GA30223@katherina.lan.complete.org>
 <op.s0c9lfh2vbderv@wybotnnas01-pool0-a67.wybotn.tds.net>
Mime-Version: 1.0
Content-type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <op.s0c9lfh2vbderv@wybotnnas01-pool0-a67.wybotn.tds.net>
User-Agent: Mutt/1.5.10i
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org

On Wed, Nov 16, 2005 at 10:04:17PM -0600, Jeff wrote:
> On Sun, 30 Oct 2005 21:48:51 -0600, John Goerzen <jgoerzen@complete.org>  
> wrote:
> 
> > Here's an update on the gopher bot:
> >
> > There is currently 28G of data archived representing 386,315
> > documents.  1.3 million documents remain to be visited, from
> > approximately 20 very large Gopher servers.  I believe, then, that the
> > majority of gopher servers have been cached by this point.  3,987
> > different servers are presently represented in the archive.
> 
> Any news?

Not really.  The bot hit a point where its algorithm for storing page
information was getting to be too slow, and there was also a problem
with the database layer I'm using segfaulting.  When I get some time, I
will write a new layer.

In the meantime, I'd like to talk about how to get this data to others
that might be willing to host it, as well as how to store it out there
for the public.  Any ideas?