Explaining the Lancet to a lay person

Or an 'Independent' journalist (from Les Roberts, a world renowned epidemiologist and lead author of the report).

Dear Mr. Kirby and Ms. Dejevsky,

"I was disappointed to hear that you felt our study was in some way dismissed by Jack Straw's anemic response to our report in the Lancet last November. Serious reviews of our work and the criticisms of it were run in the Financial Times, the Economist, the Chronicle of Higher Education (attached above) and the WSJ [Wall Street Journal] Online on August 5th. Closer to home, John Rentoul of the Independent solicited a response to the Jack Straw letter last Nov. 21st and we responded with the attached letter [Not provided here]. I am told that it was printed by your paper.

"Many people, like Ms. Dejevsky, have used the word extrapolation to describe what we did. When I hear people use that word they mean what is described in my Webster's Unabridged: '1. Statistics. to estimate the value of a variable outside its tabulated or observed range.' By this definition and the one I hear used by everyone on this side of the Atlantic, we did not extrapolate. We did sample. We drew conclusions from within the confines of that universe from which we sampled. Aside from a few homeless and transient households that did not appear in the 2002 Ministry of Health figures or households who had been dissolved or killed since, every existing household in Iraq had an equal chance that we would visit them through our randomization process.

"I understand that you feel that the sample was small: this is most puzzling. 142 post-invasion deaths in 988 households is a lot of deaths, and for the setting, a lot of interviews. There is no statistical doubt mortality is up, no doubt that violence is the main cause, and no doubt that the coalition forces have caused far more of these violent deaths than the insurgents (p<.0000001).
"In essence this is an outbreak investigation. If your readers hear about a sample with 10 cases of mad cow disease in 1000 British citizens randomly tested, I am sure they would have no doubt there was an outbreak. In 1993, when the US Centers for Disease Control randomly called 613 households in Milwaukee and concluded that 403,000 people had developed

Cryptosporidium in the largest outbreak ever recorded in the developed world, no one said that 613 households was not a big enough sample. It is odd that the logic of epidemiology embraced by the press every day regarding new drugs or health risks somehow changes when the mechanism of death is their armed forces.

"The comments of Ms. Dejevsky regarding representativeness '(it seemed small from a lay perspective (i remember at the time) for the conclusions being drawn and there seemed too little account taken of the different levels of unrest in different regions. my main point, though, was less based on my impression than on the fact that this technique exposed the authors to the criticisms/dismissal that the govt duly made, and they had little to counter those criticisms with, bar the defence that their methods were standard for those sort of surveys.)' are also cause for concern because she seems to have not understood that this was a random sample.

"By picking random neighborhoods proportional to population, we are likely to account for the natural variability of ethnicity, income, and violence. Her words above strongly suggest that the Falluja numbers should be included, rather than being used to temper the results from the other 32 neighborhoods. Please understand how extremely conservative we were: we did a survey estimating that ~285,000 people have died due to the first 18 months of invasion and occupation and we reported it as at least ~100,000. "Finally, there are now at least 8 independent estimates of the number or rate of deaths induced by the invasion of Iraq. The source most favored by the war proponents (Iraqbodycount.org) is the lowest. Our estimate is the third from highest. Four of the estimates place the death toll above 100,000. The studies measure different things. Some are surveys, some are based on surveillance which is always incomplete in times of war. The three lowest estimates are surveillance based.

"The key issues are supported by all the estimates that attribute deaths to the various causes: violence is way up post-invasion and the Coalition is responsible for many times more deaths than are the insurgents. The exact number is less important that these two indisputable facts which helps us to understand why things are going badly and how to fix them.I hope these thoughts are helpful.Sincerely,Les Roberts"

Perhaps most damning in Roberts' reply - in light of media criticism of the Lancet's alleged exaggeration of civilian deaths - was his refutation of the claim that the uneven levels of violent unrest in Iraq compromised the accuracy of the figures. In fact the study not only accounted for this variability, it erred on the side of caution by excluding data from Fallujah where deaths were unusually high. Moreover, other violent hotspots - such as Ramadi, Tallafar and Najaf - were all passed over in the sample by random chance. This suggests that the actual total of civilian deaths is likely to be higher than 100,000. Indeed, it would make far more sense for the media to be criticising the report authors for under-estimating the number of deaths.

We wrote to Dejevsky asking if she had received Roberts' response. She replied on September 1:

"yes, and i understand the arguments. but i stick to my position that extrapolation, however scientific and well-thought through is no substitute for real figures. i know that the 'real' figures here do not exist, but i still think that extrapolation has obvious drawbacks which lay the resulting figures open to question - and therefore vulnerable to govt spokesmen who seek to discredit them. incidentally, my view on extrapolation is really neither here nor there. my chief objection to it is, as i have just said, that it lays the figures themselves open to question by those who have an interest in discrediting them.all the best, mary"

Full Article: Media Lens