Received: with ECARTIS (v1.0.0; list gopher); Wed, 18 Jun 2003 12:35:19 -0500 (CDT) Return-Path: <spectre@floodgap.com> X-Original-To: gopher@complete.org Delivered-To: gopher@complete.org Received: by gesundheit.complete.org (Postfix, from userid 108) id 96B4D1832066; Wed, 18 Jun 2003 12:35:15 -0500 (CDT) X-Scanned-By: clamscan at complete.org Received: from floodgap.com (netblock-66-159-214-137.dslextreme.com [66.159.214.137]) by gesundheit.complete.org (Postfix) with ESMTP id DC84D183203D for <gopher@complete.org>; Wed, 18 Jun 2003 12:35:06 -0500 (CDT) Received: (from spectre@localhost) by floodgap.com (8.9.1/2003.05.26) id KAA12996 for gopher@complete.org; Wed, 18 Jun 2003 10:44:55 -0700 From: Cameron Kaiser <spectre@floodgap.com> Message-Id: <200306181744.KAA12996@floodgap.com> Subject: [gopher] Re: Veronica-2 and robot exclusion In-Reply-To: <009401c335b6$24e89c50$43da5982@killspy> from Ruliz Galaxor at "Jun 18, 3 06:24:50 pm" To: gopher@complete.org Date: Wed, 18 Jun 2003 10:44:55 -0700 (PDT) X-Mailer: ELM [version 2.4ME+ PL39 (25)] MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 8bit X-archive-position: 762 X-ecartis-version: Ecartis v1.0.0 Sender: gopher-bounce@complete.org Errors-to: gopher-bounce@complete.org X-original-sender: spectre@floodgap.com Precedence: bulk Reply-to: gopher@complete.org List-help: <mailto:ecartis@complete.org?Subject=help> List-unsubscribe: <mailto:gopher-request@complete.org?Subject=unsubscribe> List-software: Ecartis version 1.0.0 List-Id: Gopher <gopher.complete.org> X-List-ID: Gopher <gopher.complete.org> List-subscribe: <mailto:gopher-request@complete.org?Subject=subscribe> List-owner: <mailto:jgoerzen@complete.org> List-post: <mailto:gopher@complete.org> List-archive: <http://www.complete.org/mailinglists/archives/> X-list: gopher

> Should the robots.txt be visible in the index if you request it normally, or > is it also allowed to be invisible? > > Invisible as in: > Requesting the index doesnt show any robots.txt -> gopher://foo.bar.com/ > but requesting robots.txt works -> gopher://foo.bar.com/0/robots.txt > > Is this 'invisible' use of robots.txt allowed?

This would be the way I'd do it, and this is easy to do with a link list (like a Bucktooth gophermap file). There's probably no harm if the robots.txt file does get presented to the user, though.

In any case, itemtype=0 sel=robots.txt is what the crawler will ask for. (Note no leading slash, since Gopher is supposed to be independent of the host filesystem.)

-- ---------------------------------- personal: http://www.armory.com/~spectre/ -- Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * ckaiser@floodgap.com -- FORTUNE: You learn from your mistakes. Today will be very educational. -----