Screening for Autism with the ASSQ

I want to talk about the recent study that estimated that the prevalence of autism in South Korea was 2.6 percent because I think that is a very interesting result.  But to set the state for that discussion, I think I first need to talk about screening tests for autism and the Autism Spectrum Screening Questionnaire (ASSQ) that was used in the study.

In general, the purpose of a screening test is separate out which individuals in a group of people are likely to have a condition from those that are not.  In a perfect world, a screening test would only select those individuals that have a condition while not selecting those individuals who do not have a condition.

These concepts are formally called sensitivity and specificity.  The sensitivity of a test tells you have good of a job the test does at finding the people who have the condition while the specificity tells you how good of a job the the test does at telling you which people don't have the condition.

In an ideal world, you would want a test to identify every person that has a condition (sensitivity of 100%) and exclude every person who doesn't (specificity of 100%).  But, in the real world, tests are never that accurate and we have to make do with what we can get.

What normally happens is that the people who develop the screening test take the time to test it against a real population.  They will give the screen to a (hopefully) large group of people and see what the scores look like for those that have the condition compared to those that don't.  They will use that data to come up with what is known as a cut-off score.  This score is just a magic number were the sensitivity and specificity of a test have been maximized.

Or in simple terms, the cut-off score is where the test picks up the higher number of people with the condition while excluding the most people without it.  I think it is worth noting that even below the cut-off score it is still possible to have the condition.

A secondary point, one that is often overlooked, is that a screening test should filter out individuals who have other, similar conditions.  This concept is tied up in the idea of sensitivity and specificity but is often not tested directly.

For example, if you look at the literature for the Autism Quotient (AQ), you will find that it has somewhat decent properties.  However, if you you give the AQ to people with schizophrenia - a population that was not necessarily included in the initial validity studies - you would see that the AQ has a hard time telling whether a person has schizophrenia or autism.  Hence, while the AQ might be decent at finding people with autism, it is also good at finding people with schizophrenia.

That might seem like a small point but if you have conditions where one is much more common than the others, it can make all of of the difference in the world in what your screening test picks up.

With that in mind, lets look at the Autism Spectrum Screening Questionnaire (ASSQ).  I want to cover the properties of this test because it was the screening test used in the South Korea study.

The ASSQ is a 27 question test meant to detect "autistic disorders" in "high-functioning" children, particularly Asperger syndrome.  Eleven of the questions have to do with social interactive, six cover communication problems, while 5 deal with restricted and repetitive behaviors.  Each question has three possible answers - No, Somewhat, and Yes - and each question has a score from 0 to 2 - 0 indicates "normality", 1 is "some abnormality", and 2 is "definite abnormality".  So the range of possible total scores is from 0 to 54.

If you want to see the actual questions on the test you can see them here, starting on page 11 in the document.  I don't want to comment on the test questions, but really, are the questions "is old-fashioned or precocious" and "is regarded as an eccentric professor by the other children" indicative of autism?  Anyways.

The real question, problems with the questions aside, is how good of a job that this screening test does.  It is one thing to question the questions (pun intended) but it another to talk about how good the test is.

Unfortunately, the answer is somewhat mixed.  The ASSQ is designated to be given to both parents and teachers and the results for the different groups are different.

If you look at the initial study for the screen (total of 109 people in the study), you would see that the a parental cut-off scope of 15 would yield a "true positive rate" of 76% and a "false positive rate" of 19%.  A teacher cut-off of 11 would yield a "true positive rate" of 90% and a "false positive rate" of 42%.

Or, in other words, assuming that you had a group of 1,000 children and 1% of the them had autism (10 out of 1,000), the parent test with a cut-off of 15 would correctly identify 8 of the 10 while misidentifying 188.  The teacher test would find 9 of the 10 while misidentifying 415.

Later studies in larger groups of children found somewhat different results.  In one (4,408 children), the authors found that a combined parental and teacher cut-off score of 30 or a teacher score of 22 is best, saying that -

 A valid cut-off for parents' single score could not been estimated. The clinicians are reminded that the ASSQ is a screening instrument, not a diagnosing instrument. The importance of using both parents' and teachers' ratings for screening in clinical settings is underlined.

While in another study (9,430 children), the authors suggested that a combined parent and teacher score of 17 or greater lead to a sensitivity of 0.91 and specificity of 0.86.  Using the above assumptions, that would mean that the test would find 9 of the ten people with autism while saying that 139 had autism who didn't.

The take away here is that with the ASSQ, using somewhat typical cut-off scores, a "positive" screen on the test translates into roughly a 5% chance of actually having autism and a 10% chance of missing someone who actually does have autism.  As the cut-off score gets lower, the change of missing someone with autism is decreased (better sensitivity) while the chance of including people who don't autism increases (worse sensitivity).

However, as I mentioned above, you have to watch out for other conditions that are also going to screen as "positive" on the test.  One such condition that the ASSQ has trouble with is ADD/ADHD.  This condition was included in the initial study plus a few other ones that I found.

In the initial study, the ADD group had a mean parental score of about 10 and a mean teacher score of about 12.  The autism group had a mean of about 22 on the parent score and 25 on the teacher report.  So if the respective cut-offs are placed high enough, the test should be able to distinguish between autism and ADD.

However, over results suggest that the difference between the two groups might not be as pronounced. One result suggested that only the social part of the ASSQ was different in autism and ADHD -

The ASSQ scores of the PDD group and the ADHD group were significantly higher than the control group. Furthermore, the PDD group scored higher than the ADHD group. Both groups also showed higher scores than the control group in all three domains, that is, restricted and repetitive behavior, social interaction, and communication problem. The PDD and the ADHD group showed no significant difference in the domains of communication problem, and restricted and repetitive behavior. The PDD group had a higher score than the ADHD group only in the social interaction domain.

Another result also showed that the mean scores of a modified ASSQ  for children with ADHD was 24 while the PDD group was 31.  Scaling those values back to the original test ranges would yield a mean of 16 and 21, respectively.

So an open question for me is how well the ASSQ can tell the difference between ADD/ADHD and autism, especially at lower cut-off scores.  You will see why this is important next time.

References

Ehlers, S, C Gillberg, and L Wing. 1999. “A screening questionnaire for Asperger syndrome and other high-functioning autism spectrum disorders in school age children.” Journal of autism and developmental disorders 29:129-41. Pubmed PMID: 10382133

Mattila, Marja-Leena et al. 2009. “When does the Autism Spectrum Screening Questionnaire (ASSQ) predict autism spectrum disorders in primary school-aged children?” European child & adolescent psychiatry 18:499-509. Pubmed PMID: 19597920

Posserud, Maj-Britt, Astri J Lundervold, and Christopher Gillberg. 2009. “Validation of the autism spectrum screening questionnaire in a total population sample.” Journal of autism and developmental disorders 39:126-34. Pubmed PMID: 18592364

Fujibayashi, Hiromi, Shinji Kitayama, and Masafumi Matsuo. 2010. “Score of inattention subscale of ADHD rating scale-IV is significantly higher for AD/HD than PDD.” The Kobe journal of medical sciences 56:E12-7. Pubmed PMID:  21063141

Hattori, Junri et al. 2006. “Are pervasive developmental disorders and attention-deficit/hyperactivity disorder distinct disorders?” Brain & development 28:371-4. Pubmed PMID: 16504439