“My notes and other stuff”

Paper: When mental models go wrong

Here are some older notes I had on When mental models go wrong. Co-occurrences in dynamic, critical systems by Denis Besnard, David Greathead, and Gordon Baxter. This is a bit of a lighter text, but hints at some interesting approaches around mental models, specifically in airline pilots although the lessons are applicable more broadly.

One of the patterns that is highlighted in many sorts of incidents is one where someone's mental model and understanding of the situation is wrong, and they end up repeatedly ignoring cues and events that contradict their understanding. So the paper looks into what causes this in someone who is trying to actually do a good job. The paper states:

Humans tend to consider that their vision of the world is correct whenever events happen in accordance with their expectations. However, two sequential events can happen as expected without their cause being captured. When this is the case, humans tend to treat the available evidence as exhaustively reflecting the world, erroneously believing that they have understood the problem at hand. These co-occurring events can seriously disrupt situation awareness when humans are using mental models that are highly discrepant to reality but nonetheless trusted.

There's a general issue of there being more signals to process than capacity to process them. So rather than building a mental model that handles all of the information we have in our environment, we build goal-directed abstractions. Their main aim is to understand the current state and future states of a situation, without necessarily having an in-depth awareness of all its subtleties. They're built by a) the things you know to achieve a goal, and b) some [but not all] data extracted from the environment. So the core features and concerns of a given problem are overemphasized, but the peripheral data is easy to overlook.

Another interesting aspect of this is that essentially, the more overloaded you are with limited ability to focus, the more likely you are to automatically simplify your model and deal with correlations of the strongest elements, at the cost of all the peripheral data. This is important because complex systems—such as an airplane cockpit during an emergency situation—increases the demands on the crew. The crew possibly has to deal with nervous passengers, change of plans with air traffic control, and keeping on flying the plane while it operates abnormally. So the at-rest capacity to fully reason through everything is likely to get reduced because you don't get more bandwidth but you do end up with more demanding tasks.

There's a reference to a great concept called Bounded Rationality, which states essentially that because of the above limitations, we tend to pick cheap adequate solutions (heuristics) over optimal solutions. We go for good-enough even if sub-optimal, because it is a compromise with the cognitive cost required.

Another aspect highlighted in the paper is regarding the validation and invalidation of mental models:

Flaws in mental models are detected when the interaction with the world reveals unexpected events. However, these inaccurate mental models do not always lead to accidents. Very often, they are recovered from. In this respect, error detection and compensation are significant features in human information processing. The weakness of mental models lies in their poor requirements in terms of validity: If the environmental stream of data is consistent with the operator’s expectations, that is enough for the operator to continue regarding the mental model as valid. The understanding of the mechanisms generating the data is not a necessary condition.

We are not concerned here with how operators could build exhaustive mental models, as their incompleteness reflects a strong need for information selection. The issue of interest is to understand the conditions in which operators believe they have a good picture of the situation whereas the underlying causal mechanisms have not been captured.

This is done through an analysis of The Kegworth air crash in 1989. This incident has to do with a plane that has two engine (one on the left, one on the right). A fan blade detached from one of the engines, causing major vibration, and smoke and fumes to enter the aircraft through the AC system. The captain asked the first officer which it was, the first officer was unsure, said it was the right one. The captain throttled that engine back, and the vibrations went away. So they thought the decision was right, for about 20 minutes. When they had to land, they added more power to the left engine, and vibration came back real strong. They tried to restart the right engine, but not in time to avoid a disaster.

So big thing there: you see a problem with vibration, you turn off an engine, vibration goes back to normal. Problem solved, mental model is pleased. This makes it different from fixation errors, which are patterns seen as in Chernobyl. The Chernobyl example is the one where the operators thought the power plant couldn't explode, and they came up with a different explanation. Even when graphite was visible and the whole thing had gone boom, it was hard for the operators to think it was an explosion anyway. Fixation occurs when you disregard increasing amounts of data to remain with your current explanation, whereas this incident is one where unrelated events (vibration stopping) was seen as an agreement with the current explanation and the mental model felt confirmed despite being wrong.

There are other interesting factors that contribute to this:

• while both the captain and the first officer were experienced (over 13,000 hours and over 3,200 hours flying time respectively), they had only 76 hours experience in the Boeing 737-400 series between them.
• There was a work overload (demands from air traffic control, the passengers, etc.)
• The captain mentioned not scanning the Engine Instrument System (EIS) for vibrations because they are often unreliable in other aircrafts
• The EIS on that airline model had moved away from physical gauges into digital ones; 64% of pilots said it was bad at getting their attention and 74% preferred the older (non-digital) style

Here's the EIS of a 737-400 cockpit, circled in black:

The secondary EIS is magnified on the right-hand side of the picture. The vibration indicators are circled in white.

So what the paper says here is that we had a great example of the cognitive workload of the crew growing out of control. Management of cognitive demands makes it so when we look for confirmation of existing models, we are okay with partial confirmation. But when we want to contradict our model, we wait for more consistent data to do so. This is related to confirmation bias. For this incident, turning off an engine and a noise reduction that follows was that partial confirmation.

The authors state that one of the reasons for this behavior is that dealing with discrepancies may mean a loss of control. If you have to stop what you're doing to correct your mental model, you can't spend as much energy keeping the plane flying. So this clash of priorities may explain why people focused on a more important concern (keep the plane in the air) have it take precedence over updating a mental model that is no longer entirely right nor adequate:

Provided they can keep the system within safe boundaries, operators in critical situations sometimes opt to lose some situation awareness rather than spend time gathering data at the cost of a total loss of control.

Critical situations can be caused by the combination of an emergency followed by some loss of control. When this happens, there is little room for recovery.

[...]

The emergency nature of the situation and the emerging workload delayed the revision of the mental model which ultimately was not resumed.

What are the implications for system design? Two avenues are mentioned. The first is operator training. The supposition is that if you know about these biases and mechanisms, you may end up aware of them when they take place, which should have a positive impact on system dependability. They mention catering to these possibilities by improving communication, better stress management, and more efficient distribution of decision-making.

Another one is the same one mentioned in a lot of papers: automation has to be able ot eventually cater to the cognitive needs of the user, and better plan and explain the state transitions it is going through and the objectives it is trying to attain. Essentially, find ways to give relevant data to the operator without them having to cognitively do all the work to filter it out and know its relevance.

This is, again and unsurprisingly, an open problem, because all of that stuff is contextual.