Paper: Voice Loops as Coordination Aids in Space Shuttle Mission Control
Yesterday when I posted Managing the Hidden Costs of Coordination by Dr. Laura Maguire, I realized that I made a reference to NASA's voice loops without mentioning them. When I myself didn't know what they were, I was always annoyed at seeing opaque references to it. So here's a paper review about it that I hijacked literally from an older YOSPOS forum post I made.
The paper is Voice Loops as Coordination Aids in Space Shuttle Mission Control by Emily S. Patterson and Jennifer Watts-Perotti. The paper is from 1999, where online voice communications for high-pace coordination weren't quite common place. But even then, there's something quite cool about it even by today's standards, especially if you've ever done live operations during outages in tech.
The voice loop design is sort of opaque-sounding from the outside. They're essentially a bunch of synchronous audio channels to allow group coordination. They're also used in air traffic management, aircraft carriers, and as is the case for this paper, the space shuttle mission control. The overall structure of the voice loops are matching the structure of mission control itself:
During missions, teams of flight controllers monitor spacecraft systems and activities 24 hours a day, 7 days a week. The head flight controller is the flight director, referred to as “Flight.” Flight is ultimately responsible for all decisions related to shuttle operations and so must make decisions that trade off mission goals and safety risks for the various subsystems of the shuttle. Directly supporting the flight director is a team of approximately sixteen flight controllers who are co-located in a single location called the “front room”. These flight controllers have the primary responsibility for monitoring the health and safety of shuttle functions and subsystems. [...] These controllers must have a deep knowledge of their own systems as well as know how their systems are interconnected to other subsystems (e.g., their heater is powered by a particular electrical bus) in order to recognize and respond to anomalies despite noisy data and needing to coordinate with other controllers.
Each of the flight controllers located in the front room has a support staff that is located in “back rooms.” The front room and back room controllers communicate with each other through the voice loop system by activating a voice loop channel through a touch screen and talking into a headset. The back room support staff are more specialized than the front room controllers on specific shuttle subsystems and monitor more detailed information sources.
This diagram is provided in the paper:
Controllers (people working in mission control) can listen on any loop they want at any time they want, even multiple at a time. They can also use a primary loop they can talk on; they typically listen to ~4 of them simultaneously as well.
So you can see that the flight director loop has all the top-level controllers able to talk on there. The flight director loop has all the critical core information broadcast to everyone, and pretty much everyone listens to it. The front-to-back loops are how the higher-level controllers can delegate comms to their subteams, for whom they communicate with the top level. The conference loops are pre-set loops where controllers from pre-defined peer groups can go and talk to each other. But all these loops, even the front-to-back and the conference loops, can be listened to and monitored by anyone:
By formal communication protocols in mission control, flight controllers have privileges to speak on only a subset of the loops they can listen in on. In the voice loop control interface, each channel can be set either to monitor or talk modes. Only one channel at a time can be set to the talk mode, although many channels can be monitored at the same time. In order to talk on a loop set to the talk mode, a controller presses a button on a hand unit or holds down a foot pedal and talks into a headset.
Each controller customizes the set of loops they monitor by manipulating the visual representation of the loops at their console. The controllers can save a configuration of multiple voice loops on ‘pages’ under their identification code. The most commonly used loops are grouped together onto a primary page. The controllers then reorganize and prioritize the loops to fit the particular operational situation going on at that time by changing the configuration of loops that are being monitored and by adjusting the relative volume levels on each loop.
The voice loop interface is generally considered to be easy to use and an appropriate communication tool for a dynamic environment like space shuttle mission control. The fundamental display units are visual representations of each auditory loop, which captures the way controllers think about the system. In addition, if individual loops are analogous to windows in a visual interface, then the pages of sets of loops are analogous to the ‘room’ concept in window management. Controllers are able to customize the interface by putting their most commonly used loops together on a single ‘page.’ Active loops on these pages can be dynamically reconfigured in response to the constantly changing environment. Dynamic allocations of which loops to listen to are done by directly selecting loops to turn off and on. Controllers increase or decrease the salience of particular loops by using loop volume controls to adjust relative loudness
What's interesting as a property of setting up loops like this comes from the ability to coordinate. A disturbance in one of the control systems is going to be detected in one of the back rooms, and discussed among people there, who may eventually escalate the issue to their controller. Their controller can then bring it up to other top-level controller or directly to flight control, at which point the information is broadcast everywhere. This approach ends up doing a few things:
- Each higher level in the loop turns from technical details at a low level to an event description that is high-level
- Controllers can monitor top-level loops of their most connected systems, and if they detect noise, they can start anticipating the need for diagnostic tests and hear what happens on these related subgroups; this lets them react faster through anticipation
- Higher-level loops have more formal communication patterns where a controller announces their department then who they intend to talk to, say there messages, etc.
- Being able to snoop on other people's loops lets people gauge their interruptibility to know when to escalate or wait for your turn to talk
- The conference rooms are pre-defined and allow for ad-hoc reorganization or diagnosis without overloading other channels
- The levels of the loops also represent levels of expertise. Each expert in the backroom has a higher level of specialization on subcomponents they're in charge of than the people representing them in the front room.
- The top-level loop acting as a broadcast mechanism ensures that most relevant information is heard by everyone and can help create alignment for the entire mission control
Other interesting properties mentioned:
When controllers hear about the failure on the Flight Director's loop, they can anticipate related questions from the flight director and prepare to answer them without delay. Controllers can also anticipate actions that will be required of them. For example, an anomaly in one subsystem might require diagnostic tests in another system. When the controller hears about the anomaly on the voice loops, he can anticipate the requirement of these tests, and prepare to conduct them when they are requested.
[...]
For example, if an event like a complex anomaly occurs in a shuttle subsystem, the event triggers diagnostic activity in all related subsystem teams. This activity generates more communication across teams over the voice loops. Therefore, it is possible for controllers to track the cascade of disturbances in shuttle systems by tracking the escalation of activities that occur in response to these disturbances. This general indication of activity tempo allows controllers to synchronize their processes and activities with rest of the flight control team.
[...]
In contrast to a system of direct communications where only invited parties are involved in a conversation, voice loops allow controllers to listen to communications without announcing their virtual presence. This ability allows controllers to better gauge the relevance of their communications in relation to what is happening on a loop before interrupting. Controllers are then better able to time their communications, either by speeding up or postponing communications in relation to spurts of activity or by waiting for a pause in the communications to interject.
The paper includes a sample log showing loop communications with annotations that describe intent and escalations. Things at the same horizontal level happen at the same time:
If you want explanations about that log, the paper contains them, but I'm eliding them here.
The authors decide that two main factors explain voice loops' success:
- Peripheral awareness of other discussions where you can drop in, without disrupting people having these conversations either nor requiring their attention. The burden of interaction rests with the people who benefit from the information.
- The voice loop structure reflects that of the organization. Members of a team share their front-to-back loop, controllers of a subsystem share conference loops, the flight director loop allows high-level/importance communications for central decision-making, and air-to-ground loop lets you know specifically what the astronauts are going through.
These characteristics make them a lot better than a single shared loop (a big conference call), and the fact that you don't get to create random loops also means you get to be more focused on existing structures and get to be predictable to everyone.