Jabberwocky of the Day : Imaginary Data

I feel as if I have said this before, but science is all about data.  First you come up with an idea, then you go collect data, and finally you analyze the data to see if it supports your idea.

This process is formally known as the scientific method and, for the most part, it works well.  Sure, you will occasionally run across something where the researcher seems to have started out with the conclusion and, working backwards, selected only the data that fit the conclusion, but that doesn't happen all that often.

But, every once in a while, you will run across a paper where the researchers seemed to feel that the whole collecting data thing is overrated and went straight to the conclusion without bothering with any data.  I ran across an example of this last type recently and couldn't help but point it out.

The paper in question is "Prevalence of Pervasive Developmental Disorders Among Children at the English Montreal School Board" (open access, go read it) and it dealt primarily with measuring autism prevalence in a group of schools in Canada.  Overall, the paper wasn't bad and it showed what almost every recent study of autism prevalence has shown - the prevalence of autism increasing every year because of reasons unknown.  This part of the paper had real data behind in in the form of school records.

But then, there is this second part to the paper and that is where things get a little, well, strange.  Lets go through it from the top.

In the abstract of the study, you will find this objective listed -

Our objectives were to determine prevalence rates of PDDs among school-aged children, and to evaluate the impact of discontinuation of thimerosal use in 1996 in routine childhood vaccines on PDD rates.

Pay attention to that second part (yes, sorry, vaccines again), and what it implies.  The goal is to measure what impact - if any - the removal of thimerosal had on autism prevalence. To do determine that, you need some measure of exposure to thimerosal.

With that in mind, look at the "Conclusion" section of the abstract -

Our study provides additional evidence that the PDD rate is close to 1%. We estimate that at least 11 500 Canadian children aged 2 to 5 years suffer from a PDD. The reasons for the upward trend in prevalence could not be determined with our methods. Discontinuation of thimerosal use in vaccines did not modify the risk of PDD.

And, in the main body of the study, look at the "Interpretation" section -

As in other studies where the effect of the discontinuation of thimerosal in childhood vaccines was examined, no change in the underlying population trends for PDD rates could be observed in relation to thimerosal discontinuation.

I think it is obvious that the authors feel that their data shows that removing thimerosal didn't impact the rate of autism.  So, clearly, the paper should have measure of thimerosal exposure, right?

Wrong.  Look in the section of the paper entitled "Exposure to TCVs" -

Individual immunization data were not available for study subjects.

Wait, data about the actual exposure to vaccines and thus the exposure to thimerosal wasn't used?  Then how did the authors support their conclusion?

In Quebec, thimerosal was removed from vaccines used as part of the recommended childhood vaccine schedule in 1996. In previous years, exposure to thimerosal varied from 125 to 200 micrograms for birth cohorts included in our study.

Oh. So every child born before 1996 was assumed to have exposure to thimerosal but every child after that was assumed to have no exposure?

The prevalence in each individual birth cohort born in or after 1996 was consistently higher than that in cohorts born prior to 1996 (Table 1). ... Further, the inclusion of a dummy predictor variable indicating exposure or not to thimerosal (before or after 1996) to the model predicting prevalence with birth cohort did not improve the model and was not significant.

OK, no thimerosal data, just a "dummy predictor variable".  Hmm.  If thimerosal was completely removed from all vaccines in Quebec, then what should we make of this "Statement on Thimerosal" from the "National Advisory Committee on Immunization" in 2003 -

In Canada, the vaccines currently used in routine infant immunization do not contain thimerosal (see Table 1). Some hepatitis B vaccines licensed in Canada do, but one formulation with no thimerosal and another with only trace amounts are now available in Canada, and NACI recommends their use in infants preferentially. The two hepatitis B vaccines in which thimerosal is added as a preservative are gradually being phased out. Influenza vaccine also contains thimerosal but is only recommended for use in Canada for those infants > 6 months of age. The other vaccines licensed in Canada that contain thimerosal are primarily used for people travelling to developing countries and are not routinely administered to infants.

So, thimerosal was completely gone in 1996 - except that in 2003 it was still in some hepatitis B vaccines, the flu vaccines, and "other vaccines"?  I guess it is possible that Quebec did remove it completely from all vaccines in 1996 and it is only the rest of Canada that didn't.

But, even if that were the case, you would still have people moving from other parts of Canada or from other countries that would still have had exposure to thimerosal, so the "dummy" variable with the arbitrary cut-off date has no real relation to actual exposure.  And if there is no measure of exposure, how can the researchers conclude that removing that exposure had no effect?

Note to the researchers - next time actually have data to support your conclusion.