Systems Engineering for Ethical Autonomous Systems
The transfer of responsibility for decisions and actions from humans to machines presents difficult problems for all those concerned with new concepts, their development and use. This book gives practical help by discussing the issues in the context of product design, and gives a methodology to solve them. The design cycle for autonomous systems is described, set in the context of human decisionmaking and the evolving ethical and legal environment. These are explained in separate chapters that will be invaluable to engineers and all the professions associated with autonomous systems. Systems engineering methods, used for weapon systems, are described. These are developed for both military and civil applications. A detailed worked example demonstrates the legal limits imposed on Lethal Autonomous Weapon Systems (LAWS) by current international law.
Inspec keywords: vehicles; military systems; law; systems engineering; ethical aspects; robots
Other keywords: robots; moral code; artificial intelligence; ethical autonomous systems; systems engineering; aerospace applications; legal framework; autonomous road vehicles
Subjects: Administration and management; Economic, social and political aspects of control; Transportation system control; General topics, engineering mathematics and materials science; Robotics; General and management topics; Military control systems; Policy, government and political aspects (military and defence)
- Book DOI: 10.1049/SBRA517E
- Chapter DOI: 10.1049/SBRA517E
- ISBN: 9781785613722
- e-ISBN: 9781785613739
- Page count: 510
- Format: PDF
-
Front Matter
- + Show details - Hide details
-
p.
(1)
-
1 The art of the acceptable, not the art of the possible
- + Show details - Hide details
-
p.
1
–14
(14)
This book covers a wider range of topics than is usual in an engineering text. Systems engineers normally anticipate looking at how a system interacts with those around it, including humans, but do not expect ethical and legal issues to play a central role. The ability of a system to make decisions and act on them does raise ethical issues about their design, not just their use. Early public debate centred on smart munitions and remote bombing using unmanned air vehicles (UAVs). Now ethical questions are raised about the acceptability of many other types of autonomous system whether they use artificial intelligence (AI) or not. Cross-disciplinary debates about autonomous systems involve users, regulators, lawyers, insurance companies and politicians as well as engineers. Engineers must understand the interaction between their work and that of these other communities. Conversely, these other communities need to understand how engineers produce sophisticated systems and the need for quantification and precision in setting requirements. Without mutual understanding and respect, progress will be difficult. This chapter gives an overview of the engineering processes and techniques, as well as the legal and human rights background described and developed in more detail elsewhere in the book. This should then provide the time-pressed and possibly non-specialist reader with some impressions of the whole, whilst enabling them to concentrate on their specific problem using the relevant chapter. Chapter 14 gives some final topics for consideration drawing on results from other chapters.
-
2 Decision-making
- + Show details - Hide details
-
p.
15
–39
(25)
The article examines the introduction of increased automation and autonomy into systems of humans and machines, so it is important that we understand how humans make decisions. Human cognition and decision-making are usually modelled as a three-part process of awareness, understanding and deliberation. The same three-part model is used to analyse requirements and design for autonomous systems. Individuals are allowed to take almost any action they choose, unless it is forbidden by law or regulation. Military commanders' authority is clearly defined with responsibilities set by national and international law. Firing or releasing a weapon against a military target with people in the area must be authorised with decisions based on reliable information. The requirements for authorisation and reliability of information also apply for actions by any autonomous systems that will interact with humans.The military targeting process is described in the article as this is based on human decision-making processes and illustrates the type of information and decisions made by any autonomous system that interacts with humans.
-
3 Automated control and autonomy
- + Show details - Hide details
-
p.
41
–81
(41)
Autonomy, when applied to humans, is taken to mean freedom to make choices and take actions based on those choices without constraints from others. There is an implicit assumption that the autonomous person will be subject to the laws and other ethical constraints which apply to every other person. It is generally accepted that the current developments of robots and highly automated systems will continue. However, concerns are raised about whether there are risks to individuals and to society as a whole from these developments. These concerns can only be alleviated by showing that the new type of product can be trusted to behave in an ethical manner. The book presents engineering approaches to the problem.The terms `robot', `intelligent system' and `Artificial Intelligence (AI)' are frequently used, but with little consensus about their exact meaning. Clear definitions of terms are essential in both engineering and law so various definitions are provided and discussed. An autonomous system is defined as `a system which has the ability to perform intended tasks based on current state, knowledge and sensing, without human intervention'. Autonomy level was a specialist concept, but is now gaining more widespread use, so the range of definitions of the relevant terms, with a selection of industry-specific definitions, is explained. The article reviews various issues relating to current thinking on automated and autonomous weapon systems.
-
4 Operational analysis to systems engineering
- + Show details - Hide details
-
p.
83
–122
(40)
This chapter covers the analysis and engineering processes which are used to move from the first idea, through the initial concept to a firm design which can be made by one or more companies, tested, delivered and meet the customer's expectations. Detailed design of the products making up the system is described in Chapter 5. The process is similar for the bid, development and design stages, but the time invested and detail involved will differ significantly. Detailed design, test and integration work will not be carried out during the scoping study but their costs and timescales must be estimated accurately.
-
5 Engineering design process
- + Show details - Hide details
-
p.
123
–156
(34)
This chapter, as with Chapter 4, gives the general principles common across the engineering sector. The application is for a subsystem which is a major part of a larger military infrastructure such as a major platform1 or an improved communication system. Attention is restricted to subsystems with increased autonomy in one or more or their major functions in order to identify the type of problem which can be anticipated for autonomous systems. It is assumed that even if the ethical aspects of the design are encapsulated in law and regulations, they may not all flow down the traditional engineering design processes. Chapters 6, 7, 10 and 13 explain the legal issues in more detail for both military and civilian applications. Those chapters also indicate that the legal frameworks will almost certainly change in the future.
-
6 Ethics, civil law and engineering
- + Show details - Hide details
-
p.
157
–177
(21)
New technologies which provide innovative products and service usually raise questions about the ethics of their use in many fields. There are many papers and books about the ethics of autonomous systems, especially those using Artificial Intelligence (AI). The fast pace of technical developments in AI and automation is matched by an extensive public and legislative debate with proposals for legislation. This book takes the pragmatic view that although ethical standards will eventually be implemented as laws and regulations, engineers currently engaged in autonomous systems projects will have to make design decisions now.
-
7 Introduction to military legal context and its relevance to engineering
- + Show details - Hide details
-
p.
179
–211
(33)
War is governed by its own set of rules, known as International Humanitarian Law (IIHL), often called the Laws of Armed Conflict (LOAC). Most of IHL relates to the conduct of warfare and its immediate aftermath, but IHL's evolution has been shaped by the technologies used in war. It accepts that war will happen but aims to minimise suffering so the results include both limitations on weapon design and requirements for improved performance. These affect many aspects of weapon design either directly or indirectly. This chapter describes the legal framework that applies to armed conflict. It shows how specific parts of international law set engineering requirements which must be met by a design.
-
8 Targeting
- + Show details - Hide details
-
p.
213
–237
(25)
Targeting is a formal process which needs to operate at a speed appropriate to the circumstances. An attack on a major installation such as a harbour or airfield is likely to take several days or weeks to prepare, so large amounts of data can be assimilated and even special surveillance operations undertaken to provide missing information. In contrast, an attack on a fleeting target of opportunity has to be completed in seconds but still comply with International Humanitarian Law (IHL). The targeting process is now generally considered to include assessing damage to the target, battle damage assessment and post operation reviews.
-
9 Influences on future military autonomous systems
- + Show details - Hide details
-
p.
239
–266
(28)
This chapter takes a different approach to that of roadmaps. It discusses how changes in the type of operations conducted in the last twenty years have affected the military's operating environment, as well as looking at longer term trends in politics and society. All these play a role in driving changes in materiel requirements. The actual changes will be driven by the interactions between many factors as well as the perceived military need so only a few speculative suggestions are made. As with roadmaps, these predictions should be treated with caution.
-
10 Systems engineering applied to International Humanitarian Law (IHL)
- + Show details - Hide details
-
p.
267
–300
(34)
The four basic tenets of International Humanitarian Law (IHL) are military necessity; humanity; proportionality; and distinction. They are used here to derive engineering requirements that are applicable to Autonomous Weapon Systems (AWS). Architectures are used to examine the most important functions in autonomy and help identify the problems faced in making technical developments compliant with IHL. The problem can be stated simply as, `what authority must an automated system have to act autonomously and be compliant with IHL?' The concept of authorised power is introduced. This gives a useful method of both limiting the behaviour of the weapon system and giving a basis for system tests to demonstrate adherence to Additional Protocol I. When used for weapon systems, it is based on the legal and regulatory requirements mandated by IHL and not based on one particular branch of applied ethics.
-
11 Systems engineering for a new military system
- + Show details - Hide details
-
p.
301
–340
(40)
This chapter takes an illustrative example of a hypothetical new air-to-ground targeting capability. The chapter is fairly detailed in order to show the range of complex issues and problems that arise in introducing a new system into an existing infrastructure, in this case a military one. Similar complexities will arise in most domains and must be solved by their systems engineers.This chapter considers solutions to the targeting problem which maximise the level of autonomy in critical functions and the limits which come from Articles 35 and 36 of API in the Geneva Conventions. It shows how the problem is analysed and the capability defined so that it can be delivered by an Unmanned Air Vehicle (UAV) system integrated into the nation's and coalition's existing Command and Control (C2) infrastructure. The UAV system is the system of a UAV, its ground station, the control and data links between the two and its operators.
-
12 Making military capabilities autonomous
- + Show details - Hide details
-
p.
341
–377
(37)
This chapter discusses autonomous functions in weapon systems. It identifies the `critical' functions in a system, i.e. the ones in the targeting chain that lead directly to a lethal decision, usually weapon release. They are generally taken to be the decisions and actions normally taken by a human during target identification, selection and attack. These can be, and are, described using the Observe, Orient, Decide and Act (OODA) process. The example used is the capability to detect and destroy a missile launcher in a scenario of a littoral operation against highly capable opposing forces. This chapter investigates the problems which arise when proposals are made to make the same capability autonomous, i.e. to develop an AWS with high levels of autonomy in its critical functions so that there will be no human decision-maker after the initial instruction to the system to find and destroy a particular target or type of target. A system of this type may not necessarily be illegal under IHL but would have some strong limitations on its use in operations.
-
13 Design of civilian autonomous systems using military methodologies
- + Show details - Hide details
-
p.
379
–417
(39)
This chapter shows how the techniques developed in earlier chapters can be applied to autonomous systems that have a physical component, in civilian applications. This does not restrict the discussion to robots, but it does exclude many aspects of purely algorithmic autonomous system and Artificial Intelligence (AI) applications. Ethical issues for physical autonomous system, in general, concern their safety and reliability as they affect humans, animals and the environment; both properties are the subject of national and international laws and regulations. This leads directly to ensuring that the actions taken by the autonomous system are legitimate, and that a person has responsibility for the consequences. One fundamental problem is that of designing an autonomous system that meets accepted norms for behaviour and proving it, especially for applications which do not currently have adequate regulations. Weapon systems are highly regulated, and the methodologies developed for Autonomous Weapon Systems (AWS), described in this book, show how it is practical to embed ethical principles in their design. This chapter summarises the military methods and examines their applicability for civilian applications which have, or will have, strict regulations. The chapter structure generally follows the steps given in Section 13.2 for the design process. Products and services that use new technologies usually raise questions about the ethics of their use as discussed briefly in Section 6.2. There are methods under development to address ethical issues for Information Technology (IT) systems in general. These are known as value-based or value-sensitive design and form the basis for the proposed IEEE P7000 series of standards explained in Section 6.5. However, it will be several years before these standards will be published, probably as advisory rather than mandatory. This series will be in addition to the current ones given in Section 6.4: ISO 8373; IEEE 1872; BS8611. Design teams for autonomous system products being considered for entry into the market before the evolution of adequate standards will still need to convince users and regulators that the autonomous system is ethical as well as legal. These teams may need some guidance in setting the capability, system and design requirements for their product to significantly reduce the risk that new regulations and standards will make their product obsolete. This has been achieved for AWS. It is shown in this chapter that many of the military techniques can be applied for civilian autonomous systems.
-
14 Final considerations for ethical autonomous systems
- + Show details - Hide details
-
p.
419
–428
(10)
It is almost inevitable that autonomous systems will play an increasing role in society. What matters are the actions that they perform without human intervention, their consequences and the human responsibilities for them. The professionals dealing with their immediate problems of delivering an autonomous system, whether in the civil or military domain, must find answers in their own, short commercial timescales. New regulations and standards are being developed, but may not be ready in these timescales. This book should provide guidance so that their products are ethically acceptable. This is mainly by using system engineering techniques as a methodology to derive capability requirements. These requirements can then be developed into engineering requirements for specific autonomous products. There are, and will continue to be, debates about acceptability criteria for autonomous systems and Artificial Intelligence (AI). Evolving standards, professional codes of ethics and initiatives such as the IEEE Global Initiative will eventually provide some formal guidance to the autonomous system and AI industries.
-
Appendix 1 - Red Cross Guide to Article 36 Reviews
- + Show details - Hide details
-
p.
429
–457
(29)
The article is a guide to the legal review of new weapons, means and methods of warfare measures to implement Article 36 of Additional Protocol I of 1977. The guide aims to assist States in establishing or improving procedures to determine the legality of new weapons, means and methods of warfare in accordance with Article 36 of Protocol I Additional to the 1949 Geneva Conventions. Article 36 of Additional Protocol I requires each State Party to determine whether the employment of any new weapon, means or method of warfare that it studies, develops, acquires or adopts would, in some or all circumstances, be prohibited by international law. All States have an interest in assessing the legality of new weapons, regardless of whether they are party to Additional Protocol I. Assessing the legality of new weapons contributes to ensuring that a State's armed forces are capable of conducting hostilities in accordance with its international obligations. Carrying out legal reviews of proposed new weapons is of particular importance today in light of the rapid development of new technologies.
-
Back Matter
- + Show details - Hide details
-
p.
(1)