Received: with ECARTIS (v1.0.0; list gopher); Wed, 18 Jun 2003 12:43:18 -0500 (CDT) Return-Path: X-Original-To: gopher@complete.org Delivered-To: gopher@complete.org Received: by gesundheit.complete.org (Postfix, from userid 108) id 483DF1832066; Wed, 18 Jun 2003 12:43:16 -0500 (CDT) X-Scanned-By: clamscan at complete.org Received: from floodgap.com (netblock-66-159-214-137.dslextreme.com [66.159.214.137]) by gesundheit.complete.org (Postfix) with ESMTP id 23A9A183203F for ; Wed, 18 Jun 2003 12:43:12 -0500 (CDT) Received: (from spectre@localhost) by floodgap.com (8.9.1/2003.05.26) id KAA13648 for gopher@complete.org; Wed, 18 Jun 2003 10:53:01 -0700 From: Cameron Kaiser Message-Id: <200306181753.KAA13648@floodgap.com> Subject: [gopher] Re: Veronica-2 and robot exclusion In-Reply-To: <200306181744.KAA12996@floodgap.com> from Cameron Kaiser at "Jun 18, 3 10:44:55 am" To: gopher@complete.org Date: Wed, 18 Jun 2003 10:53:01 -0700 (PDT) X-Mailer: ELM [version 2.4ME+ PL39 (25)] MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 8bit X-archive-position: 763 X-ecartis-version: Ecartis v1.0.0 Sender: gopher-bounce@complete.org Errors-to: gopher-bounce@complete.org X-original-sender: spectre@floodgap.com Precedence: bulk Reply-to: gopher@complete.org List-help: List-unsubscribe: List-software: Ecartis version 1.0.0 List-Id: Gopher X-List-ID: Gopher List-subscribe: List-owner: List-post: List-archive: X-list: gopher > In any case, itemtype=0 sel=robots.txt is what the crawler will ask for. > (Note no leading slash, since Gopher is supposed to be independent of the > host filesystem.) (i.e., the request will be "robots.txt\r\n", just to make it mega-obvious.) -- ---------------------------------- personal: http://www.armory.com/~spectre/ -- Cameron Kaiser, Floodgap Systems Ltd * So. Calif., USA * ckaiser@floodgap.com -- Birth, n.: The first and direst of all disasters. -- Ambrose Bierce --------