How to read the medical news III: case reports and descriptive series

Here is another post about using medical evidence. My last one dealt with the usefulness (or not) of expert opinion as a basis for evaluating the validity of a medical claim. This post discusses the next highest item on the hierarchy of medical evidence — the case report.

The simple case description is the oldest tradition in medical research, dating back hundreds of years, long before biostatistics had even been thought of. The medical journals of the nineteenth and early twentieth centuries are filled with doctors’ descriptions of individual patients with particular diseases — how the problem began, how it evolved in the patient, and what happened to the patient. The writer usually then speculated about what the particular case taught us about the malady. He would usually compare his findings to whatever previous authors had written about the disease.

We still use this technique today, although such case reports are very weak tools for deciding anything important about diseases. Any nonphysician can see the pitfalls in this sort of medical evidence. It is totally dependent upon what happens to turn up in the doctor’s individual practice, and there is no way to tell if a particular case is representative of all, or even most, instances of the disorder. It is merely an anecdote of one occurrence.

Slightly better than a case report is a collection of case reports, or case series. There is also a very long tradition of these in medical writing, and a very large series does tell us things that are increasingly likely to be true as the series gets larger. But there remains a huge problem in using case series to understand whatever the problem is: they are not a random sample of all cases; rather, they were selected in ways that easily introduce bias. For example, a particular doctor may become known as an expert in a particular disease and so people with the problem come to see him. Or perhaps only patients with more severe, unusual forms find their way to him because the milder varieties are cared for by someone else or are never seen by a doctor at all.

One can imagine a large number of other circumstances that would introduce bias into whatever the doctor writing the case series would see. These issues make case reports and case series only slightly better than expert opinion as a tool for understanding medical causation and treatment. These pitfalls well illustrate a common saying among medical scientists: “The plural of anecdote is not data.”

The difficulties with isolated case reports are not unique to medicine. Consider something like an internet message board, most of which are devoted to specific topics. People posting on the board share their experiences and opinions about many things, such as particular products that work or do not work. Vigorous debate often ensues, some of it supported by evidence of some kind, some of it not. But the effect is that of a series of case reports, and readers of the messages have no idea how representative is the experience of an individual poster.

Case reports and series are an example of what is called selection bias — the possibility, or even the probability, that the individual cases have not been selected randomly from the total group they come from, which is all people who have the disease. They are therefore not representative of the entire group, and using them to draw any conclusions about the whole group is suspect. Consider again the example of the internet message board. If thousands of posters complain about a particular product, the odds get progressively higher there is some problem with it. But people with a complaint are probably more likely to post than are those satisfied with the product, so there is built-in selection bias. Additionally, the sample will include only people with internet access and familiarity with message boards. Only a survey of everybody who bought the product, or a properly randomized sample of them, can answer the question.