From: Cameron Kaiser <spectre@floodgap.com>
Message-Id: <200510122345.QAA17070@floodgap.com>
Subject: [gopher] Re: New Gopher Wayback Machine Bot
In-Reply-To: <20051012180132.GA19083@complete.org> from John Goerzen at "Oct
 12, 5 01:01:32 pm"
To: gopher@complete.org
Date: Wed, 12 Oct 2005 16:45:56 -0700 (PDT)
MIME-Version: 1.0
Content-type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org

> Cameron, floodgap.com seems to have some sort of rate limiting and keeps
> giving me a Connection refused error after a certain number of documents
> have been spidered.

I'm a little concerned about your project since I do host a number of large
subparts which are actually proxied services, and I think even a gentle bot
going methodically through them would not be pleasant for the other side
(especially if you mean to regularly update your snapshot).

Veronica-2 doesn't actually download content other than non-local selectors
in a directory to get around this problem since it doesn't index the
content in any case, just the titles and selector data.

I do support robots.txt, see

	gopher.floodgap.com/0/v2/help/indexer

-- 
---------------------------------- personal: http://www.armory.com/~spectre/ --
 Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * ckaiser@floodgap.com
-- "I'd love to go out with you, but I'm joining my split ends individually." -