Paper: Can We Trust Best Practices?
I'm taking a couple of weeks off, during which I won't read papers, but I had older notes ready on one of the papers I keep sharing around in work contexts, so here is Can We Trust Best Practices? Six Cognitive Challenges of Evidence-Based Approaches by David D. Woods and Gary Klein.
The paper makes the argument that, in the context of evidence-based medicine within the healthcare industry, ideas such as "Best practices"—relying on data for treatment recommendations—are not necessarily a productive approach due to specific cognitive challenges. The authors instead suggest ways to improve their impact.
(note: I've been working in software for many years now, and these ideas sure look like they remain valid outside of healthcare)
They identify 6 cognitive challenges around best practices:
- Characterizing problems
most of the challenge is actually in figuring out what the problem is in the first place with all the variety of its context, whereas best practices tend to focus on finding which solution to apply based on generalized categories, which you still have to map the situation to. - Gauging confidence in the evidence
the quality and relevance of data behind best practice is often misleading, hard to replicate, and specific variables may make them irrelevant or inappropriate for the current context - Deciding what to do when the generally accepted best practices conflict with professional expertise
if the expertise of the clinician is credible (and the assumption is that it generally is), there may be a situation where they strongly believe that what the best practice recommends may be inappropriate at this point in time. This still represents a decision to be made. - Applying simple rules to complex situations
Rules are often built from population data, not specific cases; complex situations may contain many variables that are not visible or accounted for in the data, and therefore no set of rules can completely handle all situations. - Revising treatment plans that do not seem to be working
A statement made here is that evidence-based medicine is not well suited for plan adaptation. Practitioners have to start from early, subtle, and preliminary data, and apply the best treatment available. But as things change and evolve, they have to change and adjust the treatment as well, and gauge that against waiting for the current treatment to work. In short, the idea is that data-driven best practices tend to assume that the suggested approach works, and offer little in terms of support when it does not. - Considering remedies that are not best practices
Rarer situations are not necessarily well-documented; time pressure and constraint do not necessarily allow in-depth analysis; new ideas can provide useful for rare situations but would not be covered.
In short, best practices tend to oversimplify the world. They are good guidelines and models, but they can't on their own, account for all the work, and shouldn't be used as such. The paper concludes:
Best practices are an important opportunity for any community to shed outmoded traditions and unreliable anecdotal procedures. They provide an opportunity for scrutiny and debate and progress. They enable organizations to act in a consistent way. However, as we have argued, best practices come with their own challenges.
Cognitive engineering and NDM [Naturalistic Decision-Making] studies have shown some of the difficulties of using evidence in situations that have a great deal of variability, uncertainty, and risk. In effect, decision makers in domains such as health care need plans like best practices but also need to be effective at revising plans to fit the dynamics and variability of specific situations (e.g., patients and diseases) and to handle the changing knowledge about what is effective.
[…]
We should regard best practices as provisional, not optimal, as a floor rather than a ceiling. When we label an approach a best practice, it tends to become a ceiling that is hard to change even as more knowledge is gained. Instead, we can identify provisional best practices that serve as a floor while learning goes forward. It is a move from “best practices” to “better practices” that frees us from undocumented anecdotal approaches and forces a commitment to continual improvement.
I like this last paragraph a lot, hence the emphasis.