“My notes and other stuff”

2023/03/10

Paper: On People and Computers in Joint Cognitive Systems at Work

A work discussion made me surface this set of notes from one of my favorite chapters ever because it explained limits of automation so well to me. It is On People and Computers in JCSs at Work, Chapter 11 of the book Joint Cognitive Systems: Patterns in Cognitive Systems Engineering (pp.143-165). And yeah, it's by David D Woods again.

The chapter introduces a concept called the Context Gap that I just want to refer to over and over again, write blog posts and talks about. It first starts by defining Joint Cognitive Systems (JCS), reminding people that people and machines are not in opposition (machines are better at this, people better at that) but part of a team that should work together.

They set that up through an example scenario with 3 characters:

  1. the problem-holder: this is an experienced practitioner who is able to self-reflect, and mostly is concerned with meeting needs in the field of practice. Machines and robots are tools to meat objectives, and he operates under pressure.
  2. a roboticist: this is someone who wants to make better machines. He can make interfaces for communication, guidance, robot interactions, and deal with the social/organisational consequences of machines. Challenges felt by the problem-holder are opportunities for this guy, and he wants to make his machine more autonomous to help.
  3. a cognitive systems engineer: this person thinks of joint cognitive systems, and wants to consider the adaptation and interactions between the humans and machines

They're put together in a scenario of emergency first responders, say, a chemical or biological incident (like a sarin gas attack). The roboticist will want to focus on more autonomous robots ("how can I get the robots to do things the problem-holder needs?"), the problem-holder will talk of pressing demands ("how do I enter a room that has both hostile people but civilians and assess the type of chemical damage they have?"), and the cognitive systems engineer will want to focus on adaptation needed and surprises ("how do they trade off time vs. energy constraints?")

Generally there's a tendency for each role to focus inwards: the cognitive system engineers will think of ways to use robot prototypes to explore human-robot coordination (how to gather information), the problem-holder will focus on learning about what the robots can do to know what they can do (what is reliable and deployable), and the roboticist will think of reconciling constraints (which capacities are worth prioritizing given budget and performance).

The chapter asks how can we cross-connect these perspectives to make the general framing less fragile and more adequate for insight generation:

Problem-holder: “What obstacles can it clear?”

Roboticist: “It can go over items 15 inches or less.”

Cognitive systems engineer: “How do (would) you tell?”

Practitioner: “We drive up to the obstacle and if it’s higher than the treads we know it cannot be scaled.”

Cognitive systems engineer: “The practitioner’s heuristic is an example of workarounds and inferences people develop to make up for the impoverished perceptual view through the robotic platform’s sensors. In contrast, when people are physically present in the environment being explored, they pick up these affordances immediately.”

Fusing the perspectives can yield new findings:

Making this fusion requires being able to translate across all 3 areas of expertise.

Responsibilities in JCSs

A basic premise is that Some human practitioners bear ultimate responsibility for operational goals. The pilot dies in the plane that goes down, the software operator gets paged and has to fix issues regardless of what it is, the dude remote-controlling the robot is considered to be in command. But what does it mean to be in command? NASA's flight director role defines control as:

So the automation and agents' activities must be comprehensible and predictable. Problem-holders are responsible for the consequences of decisions and actions; the person who holds responsibility for a problem is one who has some scope of authority to resolve the situation, and links authority and responsibility.

A critical part of this is dealing with goal conflicts. Multiple simultaneously active goals are the rule, not the exception. They produce tradeoffs and dilemmas which must be resolved under time pressure and through uncertainty. And sometimes, these are exacerbated by the way the authority-responsibility duality is handled.

For example, you may be responsible for the outcomes of a system but without the ability to influence or control the processes leading to them. This was common in nuclear power plants (following Three Mile Island) where operators had to strictly follow written procedure. However procedures are often incomplete and contradictory (brittle), and new circumstances for which no procedures exist arise, which demands adaptation and resilience, meaning you couldn't succeed by following procedures.

The double-bind is that if you follow procedures you can't meet productivity and safety objectives, but if you don't follow them and there was a problem, you could create safety or economic problems. But as operators, they did not have authority to adjust procedures, so you end up with risks of over- or under-adaptation that is invisible to management. This ends up with pithy informal rules like “Our policy is to do the right thing.”

But this sets up one of the core concepts around the Context Gap: all procedures and algorithms are brittle:

It is impossible to comprehensively list all possible situations and encode all appropriate responses because the world is too complex and fluid. [...] Thus the person in the situation is required to account for potentially unique factors in order to match or adapt algorithms and routines—whether embodied in computers, procedures, plans, or skills—to the actual situation at hand.

People adapt in one of the ways when they have responsibility (can be sanctioned) but lack authority to influence outcomes:

The latter meets immediate goals, but ends up degrading the system's own perception of itself over time. So instead, giving more control and reducing sanctions can help keep the system functioning without degradation of communication. This requires constant investment and renewal.

Literal-Minded Agents

Automation tends to do right by its model of the world. The issue is that the model of an automated agent is often limited, and the system can't tell if its model of the world is the world they're actually in or not:

As a result, the system will do the right thing [in the sense that the actions are appropriate given its model of the world], when it is in a different world [producing quite unintended and potentially harmful effects]. This pattern underlies all of the coordination breakdowns between people and automation

This is essentially the context gap: the gap between the situation assumed in the model, and the actual situation in the world. The context gap represents the need to test the assumptions in literal-minded agents are correct. Monitoring this gap is fundamental to avoiding solving the wrong problem.

These results point to a general pattern in technology change and JCSs at work: When a field of practice is about to experience an expanding role for automata, we can predict quite confidently that practitioners and organizations will adapt to develop means to align, monitor, and repair the context gap between the automata’s model of the world and the world. Limits to their ability to carry out these functions will mark potential paths to failure—breakdowns in resilience. In addition, we note that developers’ beliefs about the relationship of people and automation in complex and high consequence systems (substitution myth and other over-simplifications) lead designers to miss the need to provide this support and even to deny that such a role exists

The author points out that people are also vulnerable to being trapped in literal-mindedness where they correctly react to the wrong situation because their model of the world was inaccurate. However, practitioners are generally able to probe and test whether the situation they face is correct, and have an ability to repair their understanding that machines do not.

They introduce Norbert Weiner's contrast as a warning:

Artificial agents are literal minded and disconnected from the world while human agents are context sensitive and have a stake in outcomes.

The key is a comparison in how automata and people start from opposite points.

Automata:

On the other hand, people:

So you have this tension between context vs. general rule-based approaches, and automation and people sort of start from opposing points. People are contextualized and narrow down their understanding to create automation, and automation requires gradually expanding these models to work properly. But this is fundamentally iterative.

Paradoxically, literal-minded agents are less predictable because they are insensitive to context. When pilots in a cockpit are confused ("Why is it doing this? What will it do next?") it comes from that mismatch between them knowing context cues and the autopilot not doing so.

The computer starts from and defaults back to the position of a literal-minded agent. Being literal-minded, a **computer can’t tell if its model of the world is the world it is in**. This is a by-product of the limits of any model to capture the full range of factors and variations in the world. A model or representation, as an abstraction, corresponds to the referent processes in the world only in some ways. Good models capture the essential and leave out the irrelevant; the catch is that knowing what is essential and irrelevant depends on the goal and task context

As Ackoff (1979, p. 97) put it,

The optimal solution of a model is not an optimal solution of a problem unless the model is a perfect representation of the problem, which it never is.

There's always a need to revisit connections between the model deployed as algorithms, plans, procedures, routines, skills, and the actual conditions faced:

It is up to people situated in context to ground computer processing in the world, given that particular situations can arise to challenge the boundary conditions of the model behind the algorithm (the potential for surprise). For people in context there is an open future, while literal-minded agents are stuck within the walls of the model underlying their operation.

[...]

Closing the context gap is about knowing and testing what “rules” apply in what kind of situation. The key is to determine the kind of situation faced, and to recognize how situations can change. People can be sensitive to cues that enable them to switch rules or routines as they test whether the situation is different from what was originally construed. And despite not always performing well at this function [...], people provide the only model of competence at re-framing or re-conceptualizing that we can study for clues about what contributes to expert performance and how to support it.

Improving systems requires analyses of the brittleness of automata, but also of sources of resilience to determine how and how well people are supported in their roles.

But that's not all, because people have their own limits too:

On the other hand, people as context-bound agents adapt to what they experience in the particular. This is a local perspective that is open to the potential for unbounded particularity, despite regularities. This perspective is important in the sharp end of practice as people bound to a local situation pursue their goals, with their knowledge, given their mindset[...]. The danger is that the local agent responds too narrowly to the situation in front of them, given that they can call to mind only a limited set of the relevant material and entertain only a limited set of views of the situation, missing the side effects of their actions and decisions at broader scales and time frames.

[...]

Without continued effort, people tend to fall back to strategies that are dominated by local factors and context.

The abstractions of automation and procedures serve a purpose of connecting useful abstractions back to guided actions. There's always going to be a tension. People are limited by their local focus, and automation is limited by its brittleness and lack of contextualism. Together, they can work for better alignment.

Directions for Design

At this point we've covered the context gap which is the one thing I wanted to introduce, so I'll be a lot more terse here.

And with this, the chapter ends and the next one covers laws that cover joint cognitive systems.