Received: with ECARTIS (v1.0.0; list gopher); Wed, 18 Jun 2003 11:25:28 -0500 (CDT) Return-Path: <killspy@roelisch.tmfweb.nl> X-Original-To: gopher@complete.org Delivered-To: gopher@complete.org Received: by gesundheit.complete.org (Postfix, from userid 108) id 6A7B91832049; Wed, 18 Jun 2003 11:25:21 -0500 (CDT) X-Scanned-By: clamscan at complete.org Received: from netlx010.civ.utwente.nl (netlx010.civ.utwente.nl [130.89.1.92]) by gesundheit.complete.org (Postfix) with ESMTP id DC422183203D for <gopher@complete.org>; Wed, 18 Jun 2003 11:24:59 -0500 (CDT) Received: from killspy (kabel218067.mobiel.utwente.nl [130.89.218.67]) by netlx010.civ.utwente.nl (8.11.4/HKD) with SMTP id h5IGMxu27254 for <gopher@complete.org>; Wed, 18 Jun 2003 18:22:59 +0200 Message-ID: <009401c335b6$24e89c50$43da5982@killspy> From: "Ruliz Galaxor" <killspy@roelisch.tmfweb.nl> To: <gopher@complete.org> References: <200306181324.GAA12136@floodgap.com> Subject: [gopher] Re: Veronica-2 and robot exclusion Date: Wed, 18 Jun 2003 18:24:50 +0200 MIME-Version: 1.0 Content-type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1158 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 X-UTwente-MailScanner: Found to be clean X-archive-position: 761 X-ecartis-version: Ecartis v1.0.0 Sender: gopher-bounce@complete.org Errors-to: gopher-bounce@complete.org X-original-sender: killspy@roelisch.tmfweb.nl Precedence: bulk Reply-to: gopher@complete.org List-help: <mailto:ecartis@complete.org?Subject=help> List-unsubscribe: <mailto:gopher-request@complete.org?Subject=unsubscribe> List-software: Ecartis version 1.0.0 List-Id: Gopher <gopher.complete.org> X-List-ID: Gopher <gopher.complete.org> List-subscribe: <mailto:gopher-request@complete.org?Subject=subscribe> List-owner: <mailto:jgoerzen@complete.org> List-post: <mailto:gopher@complete.org> List-archive: <http://www.complete.org/mailinglists/archives/> X-list: gopher

> I re-read the long thread back in January '01 where we went over how to > implement a robot exclusion standard for Gopherspace, and the solution that > had the widest support (except from yours truly at that time) was to > re-use the existing robots.txt (for your convenience, the current 1994 > standard is available at > > gopher://gopher.floodgap.com/0/v2/robotstxt.txt

Should the robots.txt be visible in the index if you request it normally, or is it also allowed to be invisible?

Invisible as in: Requesting the index doesnt show any robots.txt -> gopher://foo.bar.com/ but requesting robots.txt works -> gopher://foo.bar.com/0/robots.txt

Is this 'invisible' use of robots.txt allowed?

greets,

Ruliz Galaxor gopher://kabel218067.mobiel.utwente.nl/