Paper: Rule- and Role-Retreat
This week's paper is Rule- and role-retreat: An empirical study of procedures and resilience, a paper written by Johan Bergström and N. Dahlstrom, and looks at simulation exercises in civilian Naval settings. This sort of studies comes specifically from the young discipline of Resilience Engineering:
Unexpected and escalating situations create a range of cognitive and coordinative demands. As the tempo of operations rises and becomes more (if not entirely) event-driven, there is more to consider, communicate and coordinate, i.e. more information to process, distribute and act upon. Goals can multiply, diversify and compete more steeply, and risk can increase dramatically.
A common regime in many operational worlds for absorbing those increased demands is proceduralization - the matching of situational symptoms with prepared scripts of coordinated action. [...] Proceduralization can also be indivisibly connected with role assignments.
Rule- and role-retreat are phenomena when faced with these escalating situations where rather than adapting and adjusting work, people fall back to whatever their established role and procedures are. If things aren't working, follow the procedure harder. In general, procedures are meant to help, but they're steeped in scientific management (Taylorism), where system designers can create order and stability through vertical control from top to bottom.
This influence was made bigger by the recent trend of cognitivism where humans are compared to computers as being fancy information-processing machines with "the idea of procedures as IF-THEN rule following, where procedural prompts serve as input signals to the human information processor." The problem with procedures:
They contain abstract descriptions of objects and actions that relate only loosely to particular objects and actions that are encountered in the actual situation, and a procedure often requires a whole set of cognitive and coordinative tasks to be executed that the procedure itself cannot specify or call for [...] Consequently, an over-reliance on procedures for safe operation may add additional layers of complexity rather than guide action and become part of the problem.
The folks at Lund are big on the idea of resilience though, and consider that the "ability to adapt is one of the key aspects of making a team work as a resilient organization, where resilience is defined as the ability to recognize, absorb and adapt to disruptions that fall outside a system’s design base." So the question they ask is whether experts can be considered resilient when they have been trained to follow rules, adhere to checklists, and believe there's a procedure for each situation.
The authors wanted to measure that effect, and so they set up a specific context for it. They used simulations during an emergency-management training course for ship officers in groups of 5 to 7, where everyone is given a role (captain, chief officer, chief engineer, chief steward, etc.) They have to navigate the ship in a stormy north Atlantic sea in a simulation where they are told "not to assume anything about the ship" (it is poorly maintained in the simulations). They mostly get reports from printer output and can order a "crew" around via order sheets. This includes things like sending misbehaving passengers to their cabins, evacuating and abandoning the ship, and have been given no limits as to what they can order or ask for.
They ran the experiments with multiple groups within 3 cohorts:
- novices in maritime settings (civil aviation student pilots) with no training
- maritime students with limited experience in maritime ops; they had some experience but not years of it on large ships
- experts who have been expected to know all standard procedures and have worked through some/many of them on large ships
They ran simulations first on each group, then gave training on emergency management, and ran them again. Their objective was to compare on two levels: outcomes (damage to ship and passenger casualties) and procedures (how they dealt with information overload and curve balls to standard procedures, or with idle time given to let them readjust).
Results
Novices
They were bad on all levels in the first runs; they lost the ship and most of the passengers. They quickly became overrun with information, never discussed much, monitored little, and things just went bad.
On their second run, after the training, they did better albeit not great. All groups lost their ship, but managed to save the majority of passengers and crew.
The paper is more interested in process-level stuff though, because it's more easily analyzed given the experiment design. This group reorganized their crew so that an officer role with low demands became moderator of signal, processes and dynamics, such that the captain was able to deal with high priority stuff while the moderator handled high-urgency stuff.
The following differences were noted:
Generally, they were hampered by their lack of understanding of maritime concepts, meaning they couldn't properly contextualize and extrapolate from existing concepts to generate creative solutions.
Maritime Students
These did moderately better in their first attempt: most groups lost the ship and a majority of crew and passengers.
These groups devised creative solutions which were difficult for the first type of group to come up with, since a basic understanding of ships is required to be able to devise such solutions. An example of this was when a group of this second type used a stream anchor to maneuver the ship after a breakdown of the steering engine. However, despite their creative problem solving capabilities, in comparison with the first type of group this type of group only performed marginally better in regards to aspects of process.
They were still mostly reactive. After receiving training however, these groups did very well by saving the ship and their passengers. Process-wise, they became proactive rather than reactive (eg. sending people to a higher deck earlier in case the fire wouldn't be contained and would spread). Particularly, they made few assumptions about things they couldn't verify or validate.
The following changes were noted:
These groups performed better than the first type of groups mainly because they had contextual knowledge that supported a more effective use of generic competencies. They devised creative solutions based on extrapolations of their nautical knowledge, e.g. reducing roll angle by course change and using auxiliary engines proactively
Experts
This is where things get interesting. This cohort was more successful than the two others in the first session, where the captain realized early they'd not fight a fire effectively and evacuated ship. In the second session though, the outcomes were marginally better. A lot of mistakes were repeated, but most of the weaknesses noted were in the process for this category.
Generally, they were more effective in the first session by virtue of having a well-established hierarchical role distribution, which maintained a clear structure. However, as the tempo kept increasing, they kept operating through procedures that would have helped in the real world but that the experiment designed to not work effectively. For example, life rafts were assumed to be in a good state and inspected as per regulation, but the experiment specifically had planned for some of them to have had bad maintenance.
For the fire:
a captain ordered the crew to go to their muster stations and only then it was discovered that these were not specified in the simulation. The participants were however unable to solve this problem by inventing an alternative to the concept of muster stations; instead there were repeated questions about them in the group as well as complaints to the facilitators about the lack of them.
The group tended to just be complaining about what things "should be" and not moving past these hurdles.
After their extra training, they didn't improve much. In one case it took a "hero" participants to ignore the hierarchy and suggest to the captain the he takes charge of evacuations while the captain was firefighting. This improved things, but wasn't a team dynamic so to say.
They point out that these crews' structures were robust more than flexible.
About the complaining of experts
There's a real interesting bit in the paper about the experts just being frustrated by the test situation being unrealistic, where things do not match their expectation of how a ship would work:
Overall the procedures, often unspoken and simply assumed, did not always work as anticipated, because the scenario of the simulation did not match entirely with their expectations based on their previous experiences. As a result the group was not able to advance much beyond their silent consensus on how things “should be”. There were also expectations on the behavior of the crew in the simulation which did not prove valid. The amount of expectations and inability to break out of them resulted in unanticipated failures and losses
There's a paragraph in the discussion part where they tackle that criticism more directly:
A point of criticism may be that the experienced crew did not do as well as the other type of groups during the simulation because the simulation did not exactly mirror real world conditions. This is however to some degree the main point that the participants of the simulation were meant to take home; in an emergency situation not everything will go according to plan, e.g. not everybody will report back and not all orders will be carried out. But under those circumstances groups need to find and mobilize resources and competences that will enable them to remain functioning. When procedures limit options they need to be able to find alternative ways to solve problems.
So they sort of argue "yeah this is the point, we made things unconventional to specifically see what they do when the training reaches its limits."
Interpretation
The paper continues with a discussion and interpretation of the results.
The first group couldn't do much because they had neither general nor specific knowledge to draw from. They're a sort of base comparison point.
The second group was able to continuously reorganize and readjust, and draw from general knowledge to come up with novel solutions.
The third group kept their structure intact even when it was no longer effective:
Relying on procedural knowledge can limit alternatives and may prevent potentially powerful non-presupposed solutions from being considered. This may mean that not even reviewing or reframing of a situation may occur. In short, relying on procedural knowledge can severely limit crews’ options to be resilient.
Other gotchas come from the third group, such as feeling overconfident and not validating assumptions. They assumed things were as they should be:
Procedures create reliability, i.e. expected events. Procedures are in place to fight foreseeable problems, whereas caution, forethought and inspection can give rise to resilience, to the ability to adapt to unexpected and escalating events.
[...]
When procedures limit options [groups] need to be able to find alternative ways to solve problems.
[...]
The experienced crews relied on tools and procedures they normally used. The maritime students on the other hand were able to look into the toolbox and select an appropriate (but not necessarily prescribed) tool for the situation.
They finish with a warning: a lot of industries have tended to assume that training means gaining more knowledge of procedures. While procedures should definitely remain part of training, they are not sufficient, and organizations must be able to recognize and adapt to deal with new perturbations that demand a shift in processes and challenge the established model of competence.
While unanticipated, it is industry’s strong commitment to and investment in safety and procedure that has left operators less able to respond to unexpected and escalating situations.
They should also aim to provide tools to manage situations going beyond what is anticipated, which would help prevent counter-productive role- and rule-retreat behaviours.