Text File

RISKS-LIST: RISKS-FORUM Digest Wednesday 16 December 1987 Volume 5 : Issue 76

FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator

Contents: Designing for Failure (Don Wegeng) Computer MTBF and usage (Andy Freeman) Liability and software bugs (Nancy Leveson) Re: Need for Reporting Systems (Paul Garnet) Tom Swift and his Electric Jockstrap (Arthur Axelrod) Re: Expert Systems (Amos Shapir) The Saga of the Lost ATM Card (Scott E. Preece) Telephone Billing Risks (Fred Baube) Re: F4 in 'Nam (Reversed signal polarity causing accidents) (Henry Spencer) For Lack of a Nut (NASDAQ Power outage revisited) (Bill McGarry) Dutch Database Privacy Laws (Henk Cazemier)

The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, nonrepetitious. Diversity is welcome. Contributions to RISKS@CSL.SRI.COM, Requests to RISKS-Request@CSL.SRI.COM. For Vol i issue j, FTP SRI.COM, CD STRIPE:<RISKS>, GET RISKS-i.j. Volume summaries for each i in max j: (i,j) = (1,46),(2,57),(3,92),(4,97).

----------------------------------------------------------------------

Date: 16 Dec 87 15:59:38 EST (Wednesday) Subject: Designing for Failure To: RISKS@KL.SRI.COM From: Don Wegeng <Wegeng.Henr@Xerox.COM>

In RISKS 5.75 Wm Brown III makes some interesting observations about the reliability differences between microprocessors and relay logic (prompted by a discussion about using microprocessors to control railroad switching). What I would like to discuss here is how systems react when a portion of the system *does* fail. Since I am involved in the design of microprocessor-based real-time control systems, this area is of great interest to me.

In my opinion designers should always consider the worst situation that can result from the failure of a component. Part of this should include determining what events might be missed or ignored if a particular component fails. The system's reaction to the component failure must take these events into account. For example, a microprocessor might be used in a robot arm to monitor an electric eye that detects when something is in the path of the arm. If the microprocessor fails the system's reaction must include whatever it would do if the electric eye beam had been broken, for there is no way for the system to determine the state of the beam (and it must assume worse case). Note that all of this does little good if the system does not detect that the microprocessor failed (and detecting the failure may be a bigger problem than reacting to it).

Computer control has the potential to allow greater flexibility than previous technologies. This flexibility brings with it new risks (which have been discussed many times in RISKS). It is essential that such systems be designed so that their reaction to failures will be predictable and safe. Don

------------------------------

Date: Tue 15 Dec 87 19:20:17-PST From: Andy Freeman <ANDY@Sushi.Stanford.EDU> Subject: Computer MTBF and usage To: RISKS@KL.SRI.COM, Brown@[128.81.38.39]

Wm Brown III <Brown@GODZILLA.SCH.Symbolics.COM> writes: Computers typically have a fairly constant MTBF, regardless of what they are doing for those hours. Turning them on and off may substantially reduce this time, ...

Professor Ed McCluskey at Stanford has shown that the MTBF of computer systems does depend on usage/load. This isn't surprising for software, but I remember seeing something (in an abstract) about how load affects hardware failures as well. It was surprising enough to remember, but not interesting enough for me to investigate further. -andy

------------------------------

To: risks%csl.sri.com@ROME.UCI.EDU, lcc.ghb%seas.ucla.edu@ROME.UCI.EDU Subject: Liability and software bugs Reply-To: nancy@ics.UCI.EDU Date: Tue, 15 Dec 87 19:20:03 -0800 From: Nancy Leveson <nancy%murphy.uci.edu@ROME.UCI.EDU>

In Risks 5.75, George Bray writes with respect to the Therac accident:

>On the other hand, I believe that the bug was due to an unforeseen >interaction that would be very hard to eliminate. These kinds of bugs >probably exist in much of the software in thw [sic] world.

If a reasonable person would agree that bugs exist in much of the present software, then it seems that it is also reasonable that features be built into the system to protect against such bugs. Non-computerized versions of such machines usually contain interlocks to prevent such catastrophic behavior. These same interlocks can be built into the software (or into the hardware) to protect against software errors. If it is standard practice to include such features by hardware engineers, should it not be standard practice for software engineers? Shouldn't somebody have thought to include "reasonableness" checks in the AECL software? I have heard aeronautical engineers speak of a "safety envelope" and seen them include design features to detect when the safety envelope is violated and to recover from such events. It is often possible to determine the "safety envelope" of software behavior and include checks when it is being violated. Of course, checking routines can also have bugs in them, but they are often much simpler than the original software and provide an independent check that reduces the risk, probably significantly.

Do we teach programmers to expect software errors and to build the software to detect and handle erroneous or unsafe states resulting from software bugs? If manufacturers of hardware devices can be sued for not taking reasonable care by including safety features in their devices, couldn't manufacturers of software be similarly liable for not doing the same?

[By the way, for those interested, I have heard that AECL has settled out of court with the families of the victims. There is still one suit outstanding in which the hospital where two people were killed is suing for their money back. They do not believe that the problems have truly been fixed and do not want to use the machine anymore. One might note that AECL claimed the problem was fixed before a third person was killed a year ago in Yakima].

Nancy Leveson, UCI

------------------------------

Date: Wed, 9 Dec 87 14:09:01 est From: pgarnet@nswc-wo.ARPA To: RISKS@csl.sri.com Subject: Re: Need for Reporting Systems Cc: pgarnet@csl.sri.com

In Risks 5.71 Eugene Miya suggests

< If we are ever going to progress out of the software muck, we are going < to have to come up with mechanisms to replace all of our anecdotal < information with better information.

I agree. One part of the 'software muck' is illicit code, e.g., Trojan horses, viruses, etc. It is EXTREMELY difficult to obtain any examples of illicit code, as any organization which has been bitten by one of these bugs does not want to be responsible for exacerbating the situation by letting the illicit code out to possibly infect another system.

The software security community needs to study the diseases which we are trying to defend against, as potential defenses created in a vacuum of information will only work in a vacuum. A clearinghouse, repository, library, or whatever name one wants to give to such a function should be set up so that those of us who are trying to build defenses can have subjects to study.

There are, however, a number of sticky issues revolving about setting up such a clearinghouse.

1) How do you trust the repository? How does one know that information given to the repository will not be abused, nor will it be used against the giver?

2) How does the clearinghouse know who to disseminate which information to in order not to violate issue number 1? How does one decide on who has a legitimate need to see 'dangerous' information, e.g., details on viruses, trap doors, etc.

3) The clearinghouse must not be an information sink, sucking up information from anyone willing to donate their examples but never giving any information out. It must be clear that the purpose of the clearinghouse is to facilitate the sharing of information in a non- threatening way.

4) The clearinghouse must not be an organization that people are inherently scared of, "If I tell them what happened, what are they going to do to me?"

5) There must be some mechanism to validate the information coming to the clearinghouse to insure that it is correct. We do not want a repository of misleading, invalid data.

6) Who's going to pay for this service?

There are organizations which collect information about computer crime. One which immediately comes to mind is SRI. Maybe this or some other organization could be a starting point?

One issue which will certainly come up in trying to collect, organize, and disseminate this information will be classification. Is the data unclassified, classified (C,S,TS,...), sensitive, proprietary, ...? I believe that if issue number one, trust in the clearinghouse, is solved, the issue of putting information in its proper category is administratively solvable.

I could see great value in collecting examples of illicit code (and, of course, corresponding risks). What have people really done? How can we learn from the examples? What generic technical defenses can be developed which specific companies/organizations can then apply to their systems?

Anyone interested in discussing these issues in a workshop setting, either classified or unclassified, commercial or government - please contact me.

Maybe we can come up with a workable mechanism to not only replace all of our anecdotal information, but to persuade some of the hidden information to come out of the woodwork. Paul Garnett (pgarnet@NSWC-WO.ARPA)

------------------------------

Date: 14 Dec 87 13:48:43 PST (Monday) Subject: Tom Swift and his Electric Jockstrap From: "Arthur_Axelrod.WBST128"@Xerox.COM To: RISKS@csl.sri.com

From the Rochester (NY) Democrat and Chronicle, Sunday Dec. 13, 1987. Without permission.

GAMBLER WHO WIRED HIMSELF TO COMPUTER FACES TRIAL

Carson City [AP] -- The [Nevada] state Supreme Court upheld a ruling against a man who wired his athletic supporter to a hidden microcomputer to improve his odds of winning at blackjack. The ruling Thursday revived a charge of possessing a cheating device that had been filed against Philip Preston Anderson in Las Vegas.

According to the court, Anderson strapped a microcomputer to his calf. Wires ran to switches in his shoes that he could tap with his toes to keep track of the cards that had been played. The computer calculated Anderson's advantage or disadvantage with the house and sent "vibratory signals to a special receiver located inside an athletic supporter," the Supreme Court said.

[The gentleman must be very well endowed. Otherwise the state Supreme Court would have declined to hear the case, on the ancient legal principle, "de minimis non curat lex." -- ARA]

------------------------------

To: nsc!comp-risks@Sun.COM From: nsc!taux01!taux01.UUCP!amos@Sun.COM (Amos Shapir) Subject: Re: Expert Systems (RISKS DIGEST 5.75) Date: 16 Dec 87 14:35:05 GMT Organization: National Semiconductor (Israel) Ltd. Home of the 32532 Hdate: 25 Kislev 5748

It seems the main problem is blindly relying on expert systems, because of lack of time and expertise. A well designed expert system should therefore give not only the answers, but also the decision path by which it got at them. A country doctor may not have all the knowledge that a hospital system provides, but may well be qualified to judge whether a decision like 'the patient has blue eyes therefore it's pneumonia' is valid in a particular case.

Amos Shapir, National Semiconductor (Israel) 6 Maskit st. P.O.B. 3007, Herzlia 46104, Israel Tel. +972 52 522261 amos%taux01@nsc.com (used to be amos%nsta@nsc.com)

------------------------------

Date: Tue, 15 Dec 87 09:26:28 CST From: preece%fang@gswd-vms.Gould.COM (Scott E. Preece) To: RISKS@csl.sri.com Subject: The Saga of the Lost ATM Card

From: Alan Wexelblat <wex%SW.MCC.COM@MCC.COM> > They helpfully informed me that they had no way of retreiving my card, > but if I was willing to hang around for a while "...the machine will > probably spit it out." ... MBank returns them cut up, you see. ...

Well, that's better than having it hand the card to the next person to use the machine, anyway. Actually, my own bank's machine swallowed my card once, for no known reason (it never even acknowledged that the card had been inserted). The bank sent me a new card and said the machine cut the card when it claimed it. In retrospect, this is probably a good idea -- the machine probably claimed the card because it couldn't read the stripe; I would imagine the probability of a read failure on a card which has already failed once is higher than on a card picked at random. So I'd rather not have a known-to-be-flakey card, anyway.

Of course, I also have a wife who has a card, so I'm easier about the time delay than if that were my only access to the system...

scott preece, gould/csd - urbana uucp: ihnp4!uiucdcs!ccvaxa!preece

------------------------------

To: risks@csl.sri.com Subject: Telephone Billing Risks Date: Fri, 11 Dec 87 11:45:25 -0500 From: Fred Baube <fbaube@note.nsf.gov>

This is a follow-up to the article about the woman in West Germany who incorrectly returned her telephone handset to its cradle after a call to Africa, generating a phone bill of $1710 for 10 hours. It was reported here that the woman was told she could settle for one-third the amount, $570.

From _World_Weekly_News_, excerpted without permission:

"The stubborn old widow flatly refused to pay the bill. Then a judge ordered that she need only pay $570. Still she refused.

The judge threatened to toss her in the slammer.

"I said 'OK, go ahead and put me in prison,'" she declared .. " Well, that's what he did. They took me to jail .. I spent one night in jail. It was horrible .. The next day they said I could go if I would pay for only five minutes of the call.

"I said sure I would .. but not a penny more. Then I came home. But I'm still hopping mad !"

------------------------------

Date: Thu, 10 Dec 87 13:11:29 EST From: mnetor!utzoo!henry@uunet.UU.NET To: RISKS@kl.sri.com Subject: Re: F4 in 'Nam (Reversed signal polarity causing accidents)

> ...the low-tech means that the pilots developed to deal with the problem... > was to wire a pair of bayonets to the "rails" on either side of the ejection > seat so that the points projected above the pilot's head.

There are ejection-seat systems nowadays, in fact, that rely on such "canopy breakers" rather than using a canopy-ejection system. This does depend on having a relatively thin canopy; it wouldn't work on the thick one-piece canopies used on most new US fighters. But it's certainly simpler and more reliable than automatic canopy ejection.

Mind you, there is a negative side to having a relatively thin canopy. There was a recent accident in Britain, not yet explained in detail, which *might* have been due to the parachute-deployment system of an ejection seat firing *through* the canopy by accident (i.e. not as part of an ejection) and pulling the pilot out of the plane after it. The plane (a Harrier) unfortunately kept on flying and eventually ran out of fuel over deep ocean. Recovering it will be difficult, but may be tried because more information is badly needed.

(In case you're wondering why a parachute-deployment system should operate so violently: in an ejection at low altitude, getting the parachute out and inflated *immediately* is very important.)

Henry Spencer @ U of Toronto Zoology {allegra,ihnp4,decvax,pyramid}!utzoo!henry

------------------------------

From: decvax!bunker!wtm@ucbvax.Berkeley.EDU (Bill McGarry) To: CSL.SRI!NEUMANN@ucbvax.Berkeley.EDU Date: Sun, 13 Dec 87 23:23:00 EDT Subject: For Lack of a Nut (NASDAQ Power outage) [Refinement of RISKS-5.72]

[As noted in RISKS-5.72, the NASDAQ over-the-counter computer system was knocked out of service for several hours on December 9th]. A squirrel carrying a piece of ALUMINUM FOIL made contact with some electrical equipment at a electrical substation. Although there was a power outage in the area, apparently NASDAQ suffered only a power dip (I assume that they have some form of backup power) but this power dip was enough to shut the system down [... for] over 3 1/2 hours before most services were restored. Bill McGarry {philabs, decvax, fortune, yale}!bunker!wtm

[Curses, FOILED again? Maybe the squirrel was hungry and went for the power DIP (which was SPIKED). (By the way, this was at least the THIRD squirrel power case noted in RISKS -- see RISKS-4.2 for two others.) PGN]

------------------------------

Date: Thu, 10 Dec 87 12:02:41 EST From: cognos!henkc@zorac.ARPA (Henk Cazemier) To: RISKS@KL.SRI.COM Subject: Dutch Database Privacy Laws (Re: RISKS-5.68) Organization: Cognos Inc., Ottawa, Canada

>From: Robert Stanley <roberts%cognos%math.waterloo.edu@RELAY.CS.NET> >Subject: Dutch Database Privacy Laws >First of all, there is the classic problem of protected software. We use Sun/3 >workstations, and the first engineering response to problems is swap out the >processor board (our workstations are single-board). [...]

Actually, most replacement boards do NOT have an idprom when we receive them from SUN. SUN highly recommends that you retain your original idprom, which is what we do.

Henk Cazemier P.O. Box 9707 Cognos Incorporated 3755 Riverside Dr. VOICE: (613) 738-1440 FAX: (613) 738-0002 Ottawa, Ontario UUCP: decvax!utzoo!dciem!nrcaer!cognos!henkc CANADA K1G 3Z4

------------------------------

End of RISKS-FORUM Digest ************************