From: Cameron Kaiser <spectre@floodgap.com>
Message-Id: <200306301521.IAA30688@floodgap.com>
Subject: [gopher] Re: bot's running
In-Reply-To: <004001c33ef0$e45c1920$43da5982@killspy> from Ruliz Galaxor at
 "Jun 30, 3 12:18:02 pm"
To: gopher@complete.org
Date: Mon, 30 Jun 2003 08:21:25 -0700 (PDT)
MIME-Version: 1.0
Content-type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 8bit
Sender: gopher-bounce@complete.org
Errors-to: gopher-bounce@complete.org
Precedence: bulk
Reply-to: gopher@complete.org

> Yes, I noticed... and I have a small question or actually favor to ask...
> because I'm currently using the system in which the type of request should
> also be present (e.g. 0/robots.txt).
> so could the bot both check for robots.txt and 0/robots.txt? or is that a
> problem?

I think it will probably be okay. This is how it will work though:

The bot will check for "robots.txt" first. If this works, fine, this is
accepted.
Next the bot will check for "0/robots.txt". If this works, fine, this is
accepted;
otherwise, no robots.txt is used for the site.

The reason this is worth bringing up is this could potentially map to
different selectors/files depending on the server, so the behaviour needs to
be known. Thus selector "robots.txt" always takes precedence if found.

If this is no problem to everyone, I'll take down the bot for a few minutes
this afternoon and add in the changes. Obviously whenever the bot restarts,
it refetches all robot exclusions; these are held in memory and not in
MySQL, since they're transient anyway.

> greets and keep on the great work,

Arigatoo :-)

If people want to look up stats while the bot is crawling,

	gopher://helsinki.floodgap.com/1/world/

Refresh and watch the numbers change. Great for those coffee breaks.

-- 
---------------------------------- personal: http://www.armory.com/~spectre/ --
 Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * ckaiser@floodgap.com
-- Greek tailor shop: "Euripedes?" "Yes -- Eumenides?" ------------------------