Lies, damned lies, and statistics

As I wrote about last time, a new survey has been released in the UK that seems to show that the rate of autism in adults is the same as in children - about 1 percent. These results are being pushed as some sort of final confirmation that autism is as common in adults as in children.

Yet, if you take the time to look at what the survey says, you will see that this result of 1 percent is rather flimsy and based in large part on statical models. I am going to walk through the numbers and how this survey has put together to show what I am talking about.

Here is the document that I will working from -

Autism Spectrum Disorders in adults living in households throughout England

A group of 13,171 households were selected from what was designed to be a representative population of people living in private households in England. The researchers contacted each of these households and 7,461 of them agreed to participate. From each household one person between the age of 16 and 74 was selected to be interviewed for the survey. This is referred to as phase I as of the survey.

These interviews took place between October 2006 and December 2007, which means that the latest birth year included in the survey should be 1990. As a side note for those people who think this survey disproves any relation between MMR and autism - keep in mind that the MMR was introduced in the UK in 1988. So the youngest people included in the survey most likely had the MMR.

Each person interviewed completed a series of short tests designed to determine the probability of Psychosis, ASD, Borderline personality disorder, or Antisocial personality disorder. Those that scored the highest in any of the four area were selected to be screened in phase II.

The autism section of this questionnaire has derived from the Autism Quotient screening test. But, the standard 50 question version of the test that was not used. The researchers thought that version was too long so they created their own version. They cut down the number of questions to 20 and changed the wording of some of the questions.

This custom test is one of the problems with this survey. Since this test has not been validated by other researchers (or even used anywhere else) it is difficult to know how well it finds individuals with autism. This becomes important later.

The exact method used to select for phase II is somewhat confusing. The simple way of looking at it is that the researchers converted each score in the four areas to a probability that this person would be selected for phase II. They then picked the highest single probability and used this as the chance that the person would be selected. Anyone who had a non-zero chance and agreed to participate in the second phase was included. This resulted in a set of 4,050 who were "eligible" for phase II.

The scales used to set the probabilities were very non-linear. For example, in the autism portion of the test the scores were grouped from 0-4 up to 12+. If your score was 0-4 you had no chance of being included while if your score was 9 you had 2.9 percent change, 10 was a 25 percent chance, 11 a 61 percent change, and 12+ was a 100 percent chance. There were similar scales for the other tests, and in general they worked the same way.

From this group of 4,050 the researchers selected those with the highest probabilities of being selected. This resulted in a group of 849 individuals that, out of the entire population, were MOST likely to have autism. Out of this group 630 agree to participate and were tested in a few ways, the most important being the ADOS. This test is considered to be the current "gold standard" for diagnosing autism.

In this group of 630 the ADOS ultimately identified 19 individuals with autism. So far, so good. The only real problem is that custom selection test.

If you look at the numbers at this point, the second phase of the survey covered the 8 percent of the population most likely to have autism and found about 3 percent of this group had autism. When put this back into terms of the entire population (19 in 7,461) you find that the rate would be close to 0.25 percent (about 25 in 10,000). This result is in line with past estimates of autism in adult populations.

This is where the survey gets strange. The researchers assert that, if they had examined the entire population of eligible for phase II, they would have found an additional 53 cases of autism. This would mean there would be 72 cases in all and the rate would jump to just under 1 percent. It is this assertion that is being widely quoted in the media.

Yet this figure is based on a complex statistical extrapolation that makes a lot of assumptions about the data. One of the main assumptions here is that anyone who had a score about 4 on the custom autism screening test had a non-zero chance of having autism. The problem is that we have no idea how good of a job this test does at finding people with autism. Is a score of 4, 7, 9, or highest actually associated with autism? This question has no answer because this test has not been validated. I am unsure that you can successfully extrapolate from a smaller group to a larger one when the properties of both groups are unknown.

I also find it hard to believe that, even though the segment most likely to have autism was examined, there are supposed to be more than double the the number of cases found in the remaining population.

If I gave a group of children a known screening test, such as the MCHAT, and used the results to select the 8 percent most likely to have autism, I would expect that this subset would contain most, if not all, of the children with autism. I would not think that there would be more than double the number of children in the other 92 percent of the population.

The other major problem with this survey is that we are not given any data about the age of the individuals with autism. All that we are told is that there were not enough individuals found to group into "narrow bands of 10 years", but, when combined into three broad bands of 20-30 years, there was a "slight downward" trend. The end result is that we have no real information about the ages of these individuals so it is entirely possible that they are mostly found at younger ages. But what we cannot say from what we know is that autism is evenly distributed across adults of all ages - we lack the data. But again, this is being claimed in the media.

The bottom line here is that you have to be careful when looking at the results of surveys or studies. It is way too easy to get the results that you want using statistics.

(P.S. If you made it all the way down to the bottom of this post you deserve an award)