Case Studies: Uses and Limitations
September 14, 2008Earlier in the month, the health informaticist drew attention [1] to a post by marketeer Mark Earls on the dangers of reliance on case studies to make a point [2].
Earls argues that “case studies are bad at establishing truth” because:
1. Case study thinking excludes failures, the grey mush of moderate success or indifferent performance …
2. Case studies make it seem as if the success was inevitable …
3. Case studies force things into an oversimplified narrative arc … [3]
Personally, I’m not convinced by either of the first two points – or at least I’d see them as subsets of point three – in describing a single case study, it is easy to include in our narrative only the highlights (corresponding to Earls’ first point, perhaps) and misattributions of success to one or more factors we wish to highlight without discussing the wider circumstances (Earls’ point 2?).
And, as Lovell points out in his brief post, point three can be either a positive or a negative, depending on circumstance:
[Case studies] are nice for telling stories (invites the listener to place themselves in the story and imagine what they might do …), though while they add flavour and realism they don’t prove that the story is true. And secondly they can generate hypotheses worthy of further consideration: “Oh gosh, look at that, hmmm… interesting, I wonder if…” – that sort of thing. [4]
Earls’ ends his post by asking “What’s the data say?” [5] In an area as fraught with discussion as the hierarchy of evidence, we can’t reduce it down to simplistic data analysis, but a quick survey of what’s out there did turn up a clean summary of features of the different levels of the hierarchy in a paper by Daly et al [6]:
Single case study (level IV) – Features: Provides rich data on the views or experiences of one person. Can provide insights in unexplained contexts. Limitations: Does not analyze applicability to other contexts. Evidence for practice: Alerts practitioners to the existence of an unusual phenomenon.
Descriptive studies (level III) – Features: Sample selected to illustrate practical rather than theoretical issues. Record a range of illustrative quotes including themes from the accounts of “many”, “most” or “some” study participants. Limitations: Do not report full range of responses. Sample not diversified to analyze how or why differences occur. Evidence for practice: Demonstrate that a phenomenon exists in a defined group. Identify practice issues for further consideration. [7]
After discussing the top two levels in the hierarchy (“conceptual studies (level II)” and “generalizable studies (level III)”), the authors conclude that “not all research can reach this [ideal] standard” and go on to assert that “Understanding and evaluating claims to knowledge made by qualitative research is important in meeting the policy and practice needs of an increasing[ly] complex health environment.” [8]
So, perhaps don’t throw the baby out with the bathwater by abandoning the case study altogether … but use it as a springboard to further research (or an argument for further research funding) and, whether assessing the authority of a medical paper or sitting on the sofa watching a TV ad, be aware of the level of evidence with which we are presented before making our decisions …
Refs
[1] Alan Lovell. Beware the case study and other stories. the health informaticist, 9 September 2008.
[2] Mark Earls. Case studies bad, data good. Herd: the hidden truth about who we are, 8 September 2008.
[3] Ibid.
[4] Alan Lovell. Op Cit.
[5] Mark Earls. Op Cit.
[7] Ibid: 46.
[8] Ibid: 48.
Just a couple of things you might want to consider as evidence:
1. HERD is authored by me, Mark Earls; Mike Tyldesley is just a regular reader and commentator. Lots of evidence for this btw!
2. Of course, “data” is not enough but my point is that a series of anecdotes dressed up as case-studies – which is what much marketing and management science today is rooted in – is deeply misleading.
3. In addition, both disciplines suffer from a tendency to draw conclusions about how things work (“best practice”/”benchmarking”) by considering only success stories (the case studies touted around) and ignoring all the other cases. Any drug trial that excluded all but those subjects who demonstrated a large and tangible improvement in the targeted symptoms would be laughed out of town, but in these disciplines…
4. Moreover, each individual “successful” case is often used to “prove” the different things that diffferent authors want to prove – their own prejudices. In recent years, the success of Apple’s iPod has been explained by different authors by either distinctive product design, proprietary software, outsourced electrical engineering, charismatic leadership or purpose-led brand marketing.
Whichever way one looks at this, I just don’t think you say it’s very good science: we can’t rely on it to guide our decisions. And that’s why I ask about “data”
Bad referencing … there’s never any excuse for it … especially in a librarian. Apologies are never enough …
But, as a normally very good information professional, I certainly don’t say that single case studies or even descriptive studies should be relied upon in practice – I say caveat lector … and call for more, higher-level research. Systematic review, anyone???