Received: with ECARTIS (v1.0.0; list gopher); Thu, 19 Jun 2003 16:34:39 -0500 (CDT) Return-Path: <jgoerzen@complete.org> X-Original-To: gopher@complete.org Delivered-To: gopher@complete.org Received: by gesundheit.complete.org (Postfix, from userid 108) id A6CD218320B5; Thu, 19 Jun 2003 16:34:24 -0500 (CDT) X-Scanned-By: clamscan at complete.org Received: from heinrich.complete.org (gatekeeper.excelhustler.com [68.99.114.105]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "christoph.complete.org", Issuer "John Goerzen -- Root CA" (verified OK)) by gesundheit.complete.org (Postfix) with ESMTP id 377C518320DE; Thu, 19 Jun 2003 16:33:25 -0500 (CDT) Received: by heinrich.complete.org (Postfix, from userid 1000) id 6C6A233A; Thu, 19 Jun 2003 14:54:51 -0500 (CDT) Date: Thu, 19 Jun 2003 14:54:51 -0500 From: John Goerzen <jgoerzen@complete.org> To: gopher@complete.org Subject: [gopher] Re: Veronica-2 and robot exclusion Message-ID: <20030619195451.GN6851@complete.org> References: <009401c335b6$24e89c50$43da5982@killspy> <200306181744.KAA12996@floodgap.com> Mime-Version: 1.0 Content-type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200306181744.KAA12996@floodgap.com> User-Agent: Mutt/1.5.4i Content-Transfer-Encoding: 8bit X-archive-position: 770 X-ecartis-version: Ecartis v1.0.0 Sender: gopher-bounce@complete.org Errors-to: gopher-bounce@complete.org X-original-sender: jgoerzen@complete.org Precedence: bulk Reply-to: gopher@complete.org List-help: <mailto:ecartis@complete.org?Subject=help> List-unsubscribe: <mailto:gopher-request@complete.org?Subject=unsubscribe> List-software: Ecartis version 1.0.0 List-Id: Gopher <gopher.complete.org> X-List-ID: Gopher <gopher.complete.org> List-subscribe: <mailto:gopher-request@complete.org?Subject=subscribe> List-owner: <mailto:jgoerzen@complete.org> List-post: <mailto:gopher@complete.org> List-archive: <http://www.complete.org/mailinglists/archives/> X-list: gopher

On Wed, Jun 18, 2003 at 10:44:55AM -0700, Cameron Kaiser wrote: > > Is this 'invisible' use of robots.txt allowed? > > This would be the way I'd do it, and this is easy to do with a link list > (like a Bucktooth gophermap file). There's probably no harm if the robots.txt > file does get presented to the user, though.

It's pretty easy with UMN (and PyGopherd UMN module) as well:

Just add an entry in .names:

Path=./robots.txt Type=X

-- John