Text File

14-Aug-85 16:43:57-PDT,35455;000000000011 Return-Path: <uwmacc!myers@WISC-RSCH.ARPA> Received: from SRI-CSL.ARPA by SRI-CSLA.ARPA with TCP; Wed 14 Aug 85 16:39:30-PDT Received: from WISC-RSCH by SRI-CSL via DDN; 14 Aug 85 16:33:18-PDT Received: from uwmacc.UUCP by wisc-rsch.arpa; Wed, 14 Aug 85 18:33:00 cdt Received: by maccunix.UUCP (4.12/4.7) id AA12046; Wed, 14 Aug 85 18:08:57 cdt Date: Wed, 14 Aug 85 18:08:57 cdt From: uwmacc!myers@wisc-rsch.arpa (Latitudinarian Lobster) Message-Id: <8508142308.AA12046@maccunix.UUCP> To: uwvax!risks@sri-csl.arpa Subject: CPSR-Madison paper for an issue of risks?

The following may be reproduced in any form, as long as the text and credits remain unmodified. It is a paper especially suited to those who don't already know alot about computing. Please mail comments or corrections to:

Jeff Myers University of Wisconsin-Madison reflect the views of any other Madison Academic Computing Center person or group at UW-Madison. 1210 West Dayton Street Madison, WI 53706 ARPA: uwmacc!myers@wisc-rsch.ARPA UUCP: ..!{harvard,ucbvax,allegra,heurikon,ihnp4,seismo}!uwvax!uwmacc!myers BitNet: MYERS at MACCWISC

-------------------------------------------------------------------------------

C O M P U T E R U N R E L I A B I L I T Y

A N D

N U C L E A R W A R

Larry Travis, Ph.D., Professor of Computer Sciences, UW-Madison Daniel Stock, M.S., Computer Sciences, UW-Madison Michael Scott, Ph.D., Computer Sciences, UW-Madison Jeffrey D. Myers, M.S., Computer Sciences, UW-Madison James Greuel, M.S., Computer Sciences, UW-Madison James Goodman, Ph.D., Assistant Professor of Computer Sciences, UW-Madison Robin Cooper, Ph.D., Associate Professor of Linguistics, UW-Madison Greg Brewster, M.S., Computer Sciences, UW-Madison

Madison Chapter Computer Professionals for Social Responsibility

June 1984

Originally prepared for a workshop at a symposium on the Medical Consequences of Nuclear War Madison, WI, 15 October 1983

1. Computer Use in the Military Today James Greuel Greg Brewster

Computer use in the military, as in other areas, is widespread and grow- ing at a dramatic rate. It is estimated that in 1982 approximately 20% of the programmers in the United States were writing software for military applica- tions. This Section describes the extent to which current defense department policies are dependent upon computer technology.

1.1. Computerized Weapons

We first consider the direct use of computers in U.S. weapon systems. A prime example is a category of weapons called Precision Guided Munitions, or PGMs. These devices depend on computerized guidance systems to home in on their targets. The cruise missile, for example, can be programmed with a digitized map of the surface over which it is to fly. During flight, micro- electronic sensors scan the terrain, comparing it to the map. If the two views disagree, appropriate flight adjustments are made. The sensors also scan ahead for obstructions, allowing the missile to fly very low and avoid radar detection. The cruise missile has been tested successfully, though small changes in terrain appearance do cause problems for the guidance system. For example, the amount of change in light and dark terrain features intro- duced by a snowfall can require a change in the digitized map. In the event of a war, any craters from previous explosions in the missile path could cer- tainly cause navigational difficulties.

Currently under development are missiles that will use on-board computers to "recognize" their targets. These missiles can be fired in a general direc- tion and allowed to search for an appropriate target. Prototypes have had difficulties, however, in finding camouflaged targets. They have also had difficulty in distinguishing between objects with similar visual features: for example, between tanks and large boulders.

Computers are often used to assist human beings in the operation of sophisticated weapons systems. The F-15 fighter jet has 45 micro-computers on board. These are designed to improve maneuverability, locate targets at up to 100 miles, and aim missiles at those targets. The Air Force has had consider- able difficulty keeping the F-15 airborne. Field repairs require skilled technicians and computerized diagnostic equipment. There is currently a pro- posal under consideration to build large maintenance centers on the East and West coasts solely for the purpose of maintaining F-15s.

Computerized weapons are all susceptible to tactics known as Electronic Counter-Measures--ECMs. These are steps taken to confuse or disrupt enemy computer and radar systems. They can be as simple as a pilot dropping strips of aluminum (called chaff) to "foil" enemy radar, or they can be as sophisti- cated as the electronic equipment one finds aboard such aircraft as the stealth bomber. This equipment attempts to fool enemy radar into that indi- cating the plane is somewhere other than where it really is.

1.2. Weapons Design

A second use of computers in the military is in the design of weapons. An example is the development of self-forging fragments--disks of metal inside a bomb that are shaped into conical, armor-piercing projectiles by the force of the explosion. The idea for such weapons grew out of computer analysis of the kinetic processes that take place inside a detonating warhead. Similar work led to the development of area munitions, including so-called fuel-air explosives, which precisely distribute and ignite enough explosive gas to level a full city block.

1.3. Simulations

A third use of military computers is in the area of simulation. Comput- ers have been used not only to model conventional warfare, such as dogfights between jet fighters, but to simulate nuclear combat as well. For example, computers have been used to predict the effects of a nuclear exchange under various scenarios and to calculate the remaining war-making capabilities of each side. The results of these programs, which are highly susceptible to programmer error and user bias, have in part formed the basis of the government's claims of Soviet nuclear superiority.

1.4. Command, Control, Communications, and Intelligence

The fourth use of computer technology in the military is more vulnerable to computer error and ECMs than any other. An intricate system of computers, satellites, telephone lines, radar facilities, surveillance planes, and air- borne and underground control rooms constitutes the eyes, ears, and, in a frightening sense, the very brains of our national defense. This system is called Command, Control, Communication, and Intelligence--C3I.

In the context of nuclear war, the purpose of C3I is two-fold:

(1) To provide information to the right people quickly enough during a surprise nuclear attack for the "right" decisions to be made.

(2) To maintain a system of command, control, and communication once nuclear war begins.

Among other things, C3I monitors the world's air space; observes Soviet tests and launchings; provides navigational assistance to our missiles, air- craft, and submarines; controls the position and orientation of military satellites; and collects and analyzes the results of the military's electronic intelligence-gathering operations. There are currently proposals before Congress to add artificial intelligence capabilities to many computers in the C3I system, allowing it to make recommendations and, perhaps, eventually deci- sions on the firing of missiles when there is too little time for human decision-making.

The heart of C3I is a collection of Honeywell H6000 computers located at various sites across the country. These computers were built in the early '70s on the basis of designs created in the '60s. As many people know, any computer that old may be likened in a sense to an early horseless carriage. It would be easy to spend several billions of dollars to update those machines. The result might be able to accomplish more tasks more quickly than before, but it would not be more reliable: the sources of computer errors described in the remainder of this paper apply not only to obsolete computers, but to "state-of-the-art" machines as well. What we discuss are inherent sources of unreliability in all computer systems. No amount of money will change the fact that computers make mistakes. Only sensible human policies will keep those mistakes from starting World War III.

2. Causes of Unreliability Daniel Stock Michael Scott

Anyone who has done battle with a computerized billing system realizes that while computers are useful, they are by no means perfect. Computers have problems, and we can classify those problems into three general categories:

problems with data problems with hardware problems with software

We discuss each of these categories in turn, and illustrate them with histori- cal examples.

2.1. Data Errors

Data is the information fed into a computer for use in its calculations. Section 1 of this paper has already described the two most important causes of data unreliability in computerized weapons systems: electronic counter- measures and the general havoc of battlefield conditions.

Even a minor data error can have drastic consequences. A town in Rhode Island decided to computerize its tax records in 1972. A misplaced letter 'P' on a single punched card led the town to believe that its tax base was seven million dollars higher than it actually was. As a result of the mistake, the tax rate was set far too low, and the town found itself in a nasty financial bind.

A scarier mistake made the papers on November 9th, 1979. By accident, a war game simulation tape was fed into the computers monitoring American air- space at North American Air Defense (NORAD). Strategic Air Command went on immediate alert. B-52 bomber crews were sent to their planes, ten missile intercepting fighter planes were scrambled, and U.S. missiles were readied for launch. The mistake was discovered in only 6 minutes, but it took 20 minutes to return to normal status, and the Soviets had plenty of time to notice our alert and take action of their own.

2.2. Hardware Errors

By "hardware" we mean the actual physical components of a computer sys- tem. These components are the least important source of computer unreliabil- ity, but even so they cannot be ignored. Physical components wear out, and even brand new parts can be confused by small amounts of natural radiation that destroy the information they contain. Careful quality controls, backup components, and the storage of redundant information can reduce the likelihood of hardware errors, but they cannot prevent them.

Military computers are built to exacting specifications. They break down less often than their civilian counterparts, but they are not foolproof. On June 3rd, 1980, and again on June 6th, a faulty integrated circuit in a Chey- enne Mt. computer announced a Soviet attack. Again Strategic Air Command was placed on alert. Again human beings caught the mistakes in time.

The Air Force does not publicize such incidents. The three major alerts in 1979 and 1980 were leaked to the press. They prompted a congressional investigation. The investigation revealed that in the 18 month period ending June 30, 1980, there were 151 false alarms at NORAD. Most were prompted by missile tests in Russia. Five were serious enough to put us on alert status. The two not mentioned above were caused by a Soviet submarine test near Japan, and by an old rocket body falling out of orbit. Air Force officials quoted in the congressional study revealed that equipment failures produce two or three false alarms each year.

2.3. Software Errors

The most serious source of unreliability in computer systems is neither data nor hardware. It is software--the programs that tell computers what to do. Typical military programs amount to thousands of pages of code. Human beings are simply not capable of constructing anything that large and that complex without making mistakes. Many errors are detected in simulated tests, but the only practical way to find all mistakes is to put a program into use and wait until it misbehaves.

Programming a computer amounts to providing the machine, in advance, with instructions for every possible situation it may encounter. That isn't easy. Consider some examples:

In one of the simulated flights of the space shuttle, the astronauts decided to abort their flight, then changed their minds, then tried to abort the flight again. The program running in their on-board computer went into an "infinite loop," rendering itself useless. It had never occurred to the pro- grammers that anyone would try to abort the same shuttle flight twice.

Many readers will remember the terrible floods on the Colorado River in June of 1983. According to the governor of Nevada, those floods were the direct result of miscalculations by the Federal Bureau of Reclamation's com- puters. Programmers did not anticipate the bizarre weather caused by the tropical storm El Nino. Their programs kept too much water behind the dams all spring, and when the snow runoff came there wasn't enough room remaining to hold it all.

Even if programmers were smart enough to foresee all contingencies, they would still be left with the incredible task of explaining those contingencies to a computer, in excruciating detail. Slip-ups are inevitable. In March of 1979 the Nuclear Regulatory Commission discovered a bug in the programs that had been used to design five atomic power plants on the East Coast. Because of the bug, the plants would have been unable to survive earthquakes in their area.

There is an endless supply of these examples. Most readers probably remember the computer communications problem that delayed the first flight of the space shuttle. They may not remember that two Mariner space flights were lost completely because of programming errors. One program had a period where there should have been a comma. Another was missing the word 'NOT.'

A more amusing mistake was discovered in simulated tests of the F-16 fighter. Left uncorrected, it would not have been at all amusing to the first pilot to take his craft across the equator. On-board navigation computers would promptly have turned the plane upside down.

2.4. Electromagnetic Pulse

One final source of unreliability is a phenomenon encountered only in the presence of nuclear explosions. It is a problem so severe it threatens every piece of electronic equipment in North America and dashes any hope of waging a "limited" nuclear war. The phenomenon is know as EMP--ElectroMagnetic Pulse. When an atomic bomb is exploded above the earth's atmosphere, gamma rays are released. These gamma rays collide with air molecules in the upper reaches of the atmosphere to produce so-called Compton electrons that are captured by the earth's magnetic field and that lead to a massive jolt of electromagnetic energy blanketing thousands of square miles. A single large bomb detonated 300 miles above Omaha, Nebraska would produce an effect roughly equivalent to striking every medium-sized metal object in the continental United States with a bolt of lighting, all at once. Electric fields of between 25,000 and 50,000 volts per meter would wipe out the commercial communications and power grids, and cripple nearly all computers. Military C3I would be devastated. In the aftermath of EMP, there would be no hope of coordinating the strategy neces- sary to fight a "protracted," "limited" nuclear war. EMP, combined with our dependence on computerized controls, pushes military planners into a situation where they must fire all their bombs at once, or lose the ability to fire them at all.

3. Artificial Intelligence and the Military Robin Cooper

A good idea of the kind of technological research the military is involved in can be obtained by looking at the research program of DARPA, the Defense Advanced Research Projects Agency. This agency fulfills the central research function of the Department of Defense with an appropriation of 729.6 million dollars in 1983 (around 18% of DoD's total investment in science and technology). DARPA currently has five main focuses of interest:

(1) Strategic technology. This is mainly devoted to space-based strategies, working on the assumption that "existing technologies have already made space a potential battlefield."

(2) Tactical technology. This involves the development of weapons used in the air and on land and sea. For example, it includes the development of cruise missiles which rely on computers for their ability to find targets and avoid hazards, such as enemy radar.

(3) Directed Energy. This involves laser and particle beam technology for defense and communication.

(4) Defense Sciences. This involves basic research that will be useful in other projects, such as the development of new kinds of materials and electronic devices. It includes Systems Sciences, a project to improve man-machine systems and monitoring technology. It focuses on such things as software for command and control and computer-based training technol- ogy.

(5) Information processing. This involves developing technologies that can gather, receive and communicate information to human beings or other com- puters. A large component of the research in this program involves Artificial Intelligence.

The term "Artificial Intelligence" (AI) refers to techniques for using the representation of knowledge in computers in a sophisticated way in order to perform tasks which involve fairly complex reasoning--reasoning of such a kind that the machine is successfully imitating some aspect of the behavior of a human being. The imitated behavior may be very limited. Much successful current research is devoted to the writing of programs that the non-computer scientist would not consider intelligent at all--not in the normal sense of the word as applied to human beings.

AI has applications in a number of areas that are of direct interest to the military. For example:

(1) Natural language -- This would enable military personnel (or even presidents) to consult the computer directly without any training in com- puter techniques, perhaps even by voice so that they would not have to type. Pilots would be able to give oral commands to weapons systems while using both hands to fly their aircraft.

(2) Vision -- An intelligent vision system would be able to interpret the light or radar signals it receives and make inferences (like perspective) based on this information. Vision systems could be used for robots traversing strange terrain (such as the moon) or for surveillance satel- lites or missiles.

The reliance on AI is one of the scarier implications of current military research. "The goal is to make it possible for computers to assist and/or relieve military personnel in complex as well as routine decision-making tasks which are information or personnel intensive, tedious, dangerous, or in situa- tions which are rapidly changing." This goal is scary because of the particu- lar vulnerability of AI programs to the sorts of software errors described in Section 2.

One common AI technique for handling inference involves programming scripts into the computer. An expert system for restaurants, for example, would have in its knowledge base an account of the kinds of things that nor- mally happen in restaurants--a script for restaurants. Imagine the same tech- nique applied to nuclear war. Is it possible to imagine a script that antici- pates every eventuality? No matter how inventive the programmer, is it not likely that something will fail to go according to plan? The effects of EMP were overlooked for more than 20 years. What else have we failed to consider?

Many researchers are deeply convinced that machines will never be able to make really intelligent decisions in human terms unless we can build a machine that is exactly like a human and will go through all the learning experiences that humans go through. In the event of a nuclear attack a large number of entirely novel situations will be arising in rapid succession. It is meaning- less to expect that AI systems will ever be able to provide security in such an event.

Even if it were possible to design complete and accurate scripts, there would still be the problem of implementing the design in the form of a working program. AI systems are very large and complex. Moreover, many of the sys- tems the military wants to build will be distributed, that is they will use several different computers located in several different places and communi- cating with each other. No one programmer can ever have a complete overview of such a system. The programming task must be shared by a large number of people, each of whom writes a single module. Changes to improve one module may prevent it from interacting properly with other modules, despite the fact that every module does exactly what its programmer thought it should do. The resulting errors may require a large amount of practical use of the program before they surface and can be corrected.

Military planners are faced with a dilemma: either they keep a system at the state of the art, which means constantly making changes, or they keep it as it is for several years, only fixing bugs. The first approach leads to an unreliable system; the second approach leads to an outdated system.

With civilian computers, one can balance these considerations in a way that leads to sufficiently up-to-date systems with a tolerable level of relia- bility. It may be unpleasant to be stranded at O'Hare because the airline's computers lost a reservation, but it is not a disaster. It is a disaster, perhaps, if the computer on board a 747 malfunctions and causes the plane to crash, but even then it is not the end of the world. It may indeed be the end of the world if a computer error causes nuclear war.

4. Implications Larry Travis James Goodman

For many years, our stated national policy for deterrence was known as MAD--Mutual Assured Destruction--the promise that an attack by the Soviet Union would be answered by an all-out retaliatory strike by the United States. This policy was modified under the Carter Administration to include the possi- bility of a "limited nuclear war," and the idea was subsequently endorsed by the National Republican Platform in 1980. In this Section, we argue

(1) that limited nuclear war is not feasible, and

(2) that technical developments and policy decisions are rapidly increasing the likelihood of an accidental nuclear war.

It is tempting to pursue a policy of Mutual Assured Destruction for the indefinite future. After all, there have been no nuclear wars since MAD was adopted. By the same line of reasoning, however, a resident of New York City might decide that Central Park was safe at night after walking through it a couple of times without getting mugged. We're not sure we'd try it. It is not at all clear that MAD is actually responsible for the nuclear truce of the last 30 years. We may have just been lucky. Either way, there is consider- able evidence that MAD will not prove to be a viable policy for the next 30 years.

Two important trends are undermining MAD. The first is an increase in the effectiveness of modern weapons systems. New and more powerful armaments, coupled with delivery systems of remarkable precision, have made a first strike more effective than ever before. The theory of deterrence says that threatened retaliation can prevent such an attack. But to be effective, either the retaliation must be initiated before the first warheads have been detonated, or else the missiles and the communication systems must retain suf- ficient capacity to coordinate a massive attack afterwards. The latter option, "survivability," is becoming less and less likely. Powerful and accu- rate weapons can destroy all but the most thoroughly protected command centers. Moreover, as described in Section 2, electromagnetic pulse is almost certain to cripple most of what remains. The first option, known as "launch- on-warning," or "launch-under-attack," is emerging more and more as the stra- tegy of choice.

Unfortunately, launch-on-warning is complicated by a second important trend: a decrease in decision-making time. In the '50s, when nuclear weapons were delivered by bombers, it would have taken ten hours or more to deliver a weapon across the vast distance required. Either side could have detected such an attack by radar and had hours to establish readiness, possibly open communications with the other side, and select an appropriate response before the first bombs exploded. With the development of ICBMs in the late '50s, that warning time was reduced to about 25 or 30 minutes. With the installa- tion of Pershing II missiles in Europe, the time available for the Soviets to respond to a perceived attack on their capital has been reduced to somewhere in the range of 6 to 12 minutes. An EMP burst can be triggered with even less warning, since the warheads need not be directly on target, and need not fall to earth. Recent substantial improvements in the Soviet submarine fleet have put the United States in much the same situation as its adversary.

What are the consequences of these trends? Certainly the effectiveness of a first strike argues against the possibility of a "limited" nuclear war. The arms race in recent years has focused not so much on numbers of weapons or the size of warheads, but rather on the precision with which they can be delivered. Leaders can be annihilated quickly, or at least their leadership role can be destroyed by the lack of communication links. They need to be able to assume that their policies will be carried out in their absence. Under these conditions, the decision to launch missiles clearly resides with a very large number of people. Almost certainly, every submarine carrying nuclear weapons has on board a group of people who could collectively make such a decision. This delegation of authority alone would seem to preclude any notion of a "limited response." Combined with the fact that C3I can be incapacitated easily and quickly, it makes a "protracted" war impossible.

A even more serious consequence of recent trends is increased reliance on inherently unreliably computer systems. The effectiveness of a first strike appears to make launch-on-warning more or less essential. The lack of decision-making time makes it more or less impossible, at least for human beings. Six or twelve minutes is hardly enough time to confirm an attack, let alone alert leaders and allow them to make intelligent decisions. The incred- ibly short time from first warning until the first incoming weapons might explode makes it highly unlikely that a launch-on-warning policy could include human intervention in the decision-making process. Certainly not humans at the highest levels of command. The temptation to place much or all of the decision-making authority in the "hands" of automatic computer systems is all but irresistable. Computers have none of the time limitations of human beings. They also have no common sense.

We hope that the information in Section 2 has made it clear that comput- ers cannot be trusted with the future of the planet. The theory of deterrence depends totally and unconditionally on the rationality and good judgment of the superpowers. That rationality is suspect enough with humans in command. With computers in command, the theory falls apart.

Launch-on-warning, presumably under computer control, has been advocated by many people in policy-making positions. Harold Brown, Secretary of Defense during the Carter administration, is one example. Warnings he made to the Soviet Union during that administration suggested that the United States had actually adopted such a strategy. That suggestion appears to have been inac- curate. Many defense experts believe, however, that the Soviets have indeed adopted launch-on-warning, certainly since the Pershing II missiles were deployed, if not before. Whether or not the Russians currently employ such a strategy, there are a number of objective pressures which appear to be pushing both sides toward a policy of launch-on-warning.

(1) The increased accuracy of delivery systems makes a first strike poten- tially more effective, particularly against land-based missiles, and makes rapid response imperative. This is necessary because the land- based missiles may be destroyed before they are launched. The Soviets have greater concern in this regard than does the United States, because the Soviets place greater reliance on land-based missiles. However, both sides feel compelled to launch a retaliatory strike before the incoming missiles explode because the attack will also severely cripple communica- tions, making it difficult or impossible to coordinate retaliation.

(2) The decrease in decision-making time means that if weapons are to be launched before they are destroyed, they must be launched almost immedi- ately upon detection of an attack.

(3) The increased complexity of weapons and the greater number of resulting options means that much more time is necessary for human decisions. More strategies must be considered, and there is a greater likelihood of human miscalculation.

(4) The President's proposal for "Star Wars" missile defense will require decisions to be made with only seconds of warning, not minutes. Com- pletely autonomous computer control is a foregone conclusion. The installation of such a defense system would amount to an adoption of launch-on-warning, since its use would be an act of war.

Whenever a new application of computer technology is proposed, and a pol- itical decision must be made about its development, it is necessary to compare its benefits to its costs. In order to evaluate a cost which may or may not happen, the normal technique is to multiply the probability of its occurrence by its cost if it does occur. But what cost can we assign to the destruction of human civilization, or even the complete loss of the human species? If this cost is infinite, then any probability at all of such an event makes the cost of deploying such a system also infinite. Since 100 per cent reliability is unattainable, it is only reasonable to limit the use of computers to appli- cations in which we can tolerate an occasional mistake.

References

1. Computer Use in the Military Today

C. Simpson, "Computers in Combat," Science Digest, October 1982.

R. R. Everett, "Yesterday, Today, and Tomorrow in Command, Control, and Com- munications," Technology Review, January 1982.

"Command, Control Capability Upgraded," Aviation Week and Space Technology, 3 January 1983.

"An Upheaval in U.S. Strategic Thought," Science, 2 April 1982.

R. T. Smith, "They Have More EMT Than We," Science, 2 April 1982.

A. D. Frank, "Hello, Central, Give Me Bomb Control," Forbes, 23 November 1981.

W. Arkin and P. Pringle, "C3I : Command Post for Armageddon," Nation, 9 April 1983.

"The Conventional Weapons Fallacy," Nation, 9 April 1983.

L. Siegel, "Space Wars," The Progressive, June 1982.

2. Causes of Unreliability

"Computers and the U.S. Military Don't Mix," Science, 14 March 1980.

"False Alerts and Faulty Computers," Science, 5 June 1981.

Two articles by William J. Broad examining problems the military has had with computers.

"Electromagnetic Pulses: Potential Crippler," IEEE Spectrum, May 1981.

D. L. Stein, "Electromagnetic Pulse -- The Uncertain Certainty," Bulletin of the Atomic Scientists, March 1983.

Two good articles on the causes, effects, and implications for military strategy of EMP.

IEEE Spectrum, Volume 18, number 10 (October 1981).

IEEE Spectrum, Volume 19, number 10 (October 1982).

Special issues of a fairly easy-to-read journal for electrical engineers. The topics are "Technology in War and Peace" and "Reliability."

Letters from the Editor, ACM SIGSOFT Software Engineering Notes, Volumes 4-8 (1979-1983).

The editor of this journal, Peter G. Neumann, collects and publishes reports of intesting computer "bugs." Particularly recommended are Volume 4, number 2; Volume 5, numbers 2 and 3; and Volume 7, number 1.

G. J. Myers, Software Reliability, John Wiley and Sons, 1976.

An old but good textbook, peppered with suggestions on how to approach reliability and examples of software that didn't.

"Friendly Fire," Science 83, May 1983, page 10.

Exocets and the Sheffield.

"Nevada Governor Says Errors Led to Flooding," The New York Times, July 5, 1983, page I-10, column 6.

Brief note on computers and the flood.

3. Artificial Intelligence and the Military

Defense Advanced Research Projects Agency, "Fiscal Year 1984 Research and Development Program (U): A Summary Description," April 1983.

4. Implications

R. Thaxton, "Nuclear War by Computer Chip," The Progressive, August 1980.

R. Thaxton, "The Logic of Nuclear Escalation," The Progressive, February 1982.

J. Steinbruner, "Launch under Attack," Scientific American, January 1984.

M. Bundy, "Ending War Before It Starts," Review of two books in the New York Times Book Review, 9 October 1983.