Monday, May 16, 2011

Screening for Autism with the ASSQ

I want to talk about the recent study that estimated that the prevalence of autism in South Korea was 2.6 percent because I think that is a very interesting result.  But to set the state for that discussion, I think I first need to talk about screening tests for autism and the Autism Spectrum Screening Questionnaire (ASSQ) that was used in the study.

In general, the purpose of a screening test is separate out which individuals in a group of people are likely to have a condition from those that are not.  In a perfect world, a screening test would only select those individuals that have a condition while not selecting those individuals who do not have a condition.

These concepts are formally called sensitivity and specificity.  The sensitivity of a test tells you have good of a job the test does at finding the people who have the condition while the specificity tells you how good of a job the the test does at telling you which people don't have the condition.

In an ideal world, you would want a test to identify every person that has a condition (sensitivity of 100%) and exclude every person who doesn't (specificity of 100%).  But, in the real world, tests are never that accurate and we have to make do with what we can get.

What normally happens is that the people who develop the screening test take the time to test it against a real population.  They will give the screen to a (hopefully) large group of people and see what the scores look like for those that have the condition compared to those that don't.  They will use that data to come up with what is known as a cut-off score.  This score is just a magic number were the sensitivity and specificity of a test have been maximized.

Or in simple terms, the cut-off score is where the test picks up the higher number of people with the condition while excluding the most people without it.  I think it is worth noting that even below the cut-off score it is still possible to have the condition.

A secondary point, one that is often overlooked, is that a screening test should filter out individuals who have other, similar conditions.  This concept is tied up in the idea of sensitivity and specificity but is often not tested directly.

For example, if you look at the literature for the Autism Quotient (AQ), you will find that it has somewhat decent properties.  However, if you you give the AQ to people with schizophrenia - a population that was not necessarily included in the initial validity studies - you would see that the AQ has a hard time telling whether a person has schizophrenia or autism.  Hence, while the AQ might be decent at finding people with autism, it is also good at finding people with schizophrenia.

That might seem like a small point but if you have conditions where one is much more common than the others, it can make all of of the difference in the world in what your screening test picks up.

With that in mind, lets look at the Autism Spectrum Screening Questionnaire (ASSQ).  I want to cover the properties of this test because it was the screening test used in the South Korea study.

The ASSQ is a 27 question test meant to detect "autistic disorders" in "high-functioning" children, particularly Asperger syndrome.  Eleven of the questions have to do with social interactive, six cover communication problems, while 5 deal with restricted and repetitive behaviors.  Each question has three possible answers - No, Somewhat, and Yes - and each question has a score from 0 to 2 - 0 indicates "normality", 1 is "some abnormality", and 2 is "definite abnormality".  So the range of possible total scores is from 0 to 54.

If you want to see the actual questions on the test you can see them here, starting on page 11 in the document.  I don't want to comment on the test questions, but really, are the questions "is old-fashioned or precocious" and "is regarded as an eccentric professor by the other children" indicative of autism?  Anyways.

The real question, problems with the questions aside, is how good of a job that this screening test does.  It is one thing to question the questions (pun intended) but it another to talk about how good the test is.

Unfortunately, the answer is somewhat mixed.  The ASSQ is designated to be given to both parents and teachers and the results for the different groups are different.

If you look at the initial study for the screen (total of 109 people in the study), you would see that the a parental cut-off scope of 15 would yield a "true positive rate" of 76% and a "false positive rate" of 19%.  A teacher cut-off of 11 would yield a "true positive rate" of 90% and a "false positive rate" of 42%.

Or, in other words, assuming that you had a group of 1,000 children and 1% of the them had autism (10 out of 1,000), the parent test with a cut-off of 15 would correctly identify 8 of the 10 while misidentifying 188.  The teacher test would find 9 of the 10 while misidentifying 415.

Later studies in larger groups of children found somewhat different results.  In one (4,408 children), the authors found that a combined parental and teacher cut-off score of 30 or a teacher score of 22 is best, saying that -
 A valid cut-off for parents' single score could not been estimated. The clinicians are reminded that the ASSQ is a screening instrument, not a diagnosing instrument. The importance of using both parents' and teachers' ratings for screening in clinical settings is underlined.
While in another study (9,430 children), the authors suggested that a combined parent and teacher score of 17 or greater lead to a sensitivity of 0.91 and specificity of 0.86.  Using the above assumptions, that would mean that the test would find 9 of the ten people with autism while saying that 139 had autism who didn't.

The take away here is that with the ASSQ, using somewhat typical cut-off scores, a "positive" screen on the test translates into roughly a 5% chance of actually having autism and a 10% chance of missing someone who actually does have autism.  As the cut-off score gets lower, the change of missing someone with autism is decreased (better sensitivity) while the chance of including people who don't autism increases (worse sensitivity).

However, as I mentioned above, you have to watch out for other conditions that are also going to screen as "positive" on the test.  One such condition that the ASSQ has trouble with is ADD/ADHD.  This condition was included in the initial study plus a few other ones that I found.

In the initial study, the ADD group had a mean parental score of about 10 and a mean teacher score of about 12.  The autism group had a mean of about 22 on the parent score and 25 on the teacher report.  So if the respective cut-offs are placed high enough, the test should be able to distinguish between autism and ADD.

However, over results suggest that the difference between the two groups might not be as pronounced. One result suggested that only the social part of the ASSQ was different in autism and ADHD -
The ASSQ scores of the PDD group and the ADHD group were significantly higher than the control group. Furthermore, the PDD group scored higher than the ADHD group. Both groups also showed higher scores than the control group in all three domains, that is, restricted and repetitive behavior, social interaction, and communication problem. The PDD and the ADHD group showed no significant difference in the domains of communication problem, and restricted and repetitive behavior. The PDD group had a higher score than the ADHD group only in the social interaction domain.
Another result also showed that the mean scores of a modified ASSQ  for children with ADHD was 24 while the PDD group was 31.  Scaling those values back to the original test ranges would yield a mean of 16 and 21, respectively.

So an open question for me is how well the ASSQ can tell the difference between ADD/ADHD and autism, especially at lower cut-off scores.  You will see why this is important next time.

References

Ehlers, S, C Gillberg, and L Wing. 1999. “A screening questionnaire for Asperger syndrome and other high-functioning autism spectrum disorders in school age children.” Journal of autism and developmental disorders 29:129-41. Pubmed PMID: 10382133

Mattila, Marja-Leena et al. 2009. “When does the Autism Spectrum Screening Questionnaire (ASSQ) predict autism spectrum disorders in primary school-aged children?” European child & adolescent psychiatry 18:499-509. Pubmed PMID: 19597920

Posserud, Maj-Britt, Astri J Lundervold, and Christopher Gillberg. 2009. “Validation of the autism spectrum screening questionnaire in a total population sample.” Journal of autism and developmental disorders 39:126-34. Pubmed PMID: 18592364

Fujibayashi, Hiromi, Shinji Kitayama, and Masafumi Matsuo. 2010. “Score of inattention subscale of ADHD rating scale-IV is significantly higher for AD/HD than PDD.” The Kobe journal of medical sciences 56:E12-7. Pubmed PMID:  21063141

Hattori, Junri et al. 2006. “Are pervasive developmental disorders and attention-deficit/hyperactivity disorder distinct disorders?” Brain & development 28:371-4. Pubmed PMID: 16504439

7 comments:

  1. Enjoyed the post, many thanks. I think you touch on some very relevant issues here, not least what do screening instruments screen for in autism? and also what about co-morbidity as a potential 'influencing' variable? (I dare say that this second issue of co-morbidity is also perhaps a question that should be asked of intervention, in terms of what intervention actually targets and what it is reported to target symptom-wise).
    The Kim study is interesting when it comes to the ASSQ results: for the regular school group there was a response rate of 63% (23,234 out of 36,592). 7% (1,742) of these 23,234 screened positive on ASSQ but only 1,111 were sampled and then only 785 provided consent.
    I hope I am not stealing any thunder when I say that the cut-off values for a positive result used in the study were: teachers =>10, parents =>14.
    The paper makes no reference to ADHD or any other kind of co-morbidity (aside from telling us that Rett syndrome and Heller's syndrome were not detected in any cases).

    ReplyDelete
  2. Hi Paul,

    Glad you liked the post.

    I think you got the point of where I am headed next. You can't really fault and autism diagnosis that was made using the ADOS but the extrapolation of the results back through the screening test to the general population is somewhat questionable.

    Especially since the the number of children who screened positive and then had an autism diagnosis does not line up with what we know about the screen. The low cut-offs make this problem even worse - there should have been an extremely high percentage of false positives but there don't seem to be.

    I am still playing with the numbers from the study but either I am missing something basic or something isn't quite adding up.

    The co-morbid conditions is a second problem of the extrapolation (I assume - perhaps wrongly - that the ADOS could distinguish between ADD/ADHD and the various forms of autism).

    ReplyDelete
  3. Very interesting post, that raises all sorts of related questions about diagnosis of a condition that is identified by observable symptoms, rather than (say) a blood test or brain scan. Though similar issues arise for "internal" (and not just external observational) tests for identifying autism, such as this one I came across a while ago: http://www.guardian.co.uk/science/blog/2010/aug/12/autism-brain-scan-statistics . This is an area I think about a lot - the question of in/correct diagnosis and the consequences of making errors, as well as the shifting criteria that then affect the numbers of diagnoses.

    ReplyDelete
  4. Thanks MJ.
    The Kim study reported to use both ADOS and ADI (alongside some measures of cognitive functioning).
    ADI, despite being gold-standard, is perhaps the more 'flaky' of the instruments given its reliance on retrospective recall and report over direct observation and, particularly for those children older than 4-5 years old, where the so-called 'most abnormal' coding is used. Ask any parent of a 7-12 year old (irrespective of autism), what was your child like at 4-5 years old, and unless there is a good memory 'anchor', recall is going to be problematic at best. Cathy Lord and co recently talked about 'telescoping' based on the ADI-R data (Hus et al, 2011).
    ADOS by contrast is pretty specific to autism, certainly when using the module 1 and 2, where there is a lot of interaction between assessor and child using the various testing toys, games and books. Having said that I would assume that most of the kids in the Kim study from the school group would have been assessed using module 3 (child/adolescent with fluent speech) where far less material are used and more conversation/questioning is required.
    I've not actually seen anything discussing whether ADHD co-morbidity (or anything else) could affect the final outcome. I would perhaps be inclined to say that the criteria/algorithm is quite explicit when it comes to autism diagnostic criteria, and requires contribution from all clinical areas in order to reach cut-off points suggestive of a diagnosis.
    The Kim study actually says that DSM-IV critera were used to determine autism or ASD.
    As part of that diagnostic work-up, the data from ADI and ADOS were independently reviewed and diagnosis determined. This is perhaps where more detail is required from the study - did these reviewers use the video-taped ADI/ADOS sessions or merely use the notes taken when coming to a clinical judgement?

    ReplyDelete
  5. I think your comments about the ADI are pretty much on target. We actually just did an ADI for our youngest daughter (who is 4) for a research study and it was hard answering some of the questions.

    This is in spite of the facts that we were paying very close attention to her development because of her sisters' history, that most of the questions address things that happened relatively recently, and that both my wife and I were answering the questions.

    I can't imagine that parents are going to be able to properly answer the questions as their children get older. Maybe up to the teenage years but certainly not into the adult years.

    This wasn't likely a factor in this study because the children were younger but it certainly could be a factor in others.

    ReplyDelete
  6. Hey, thanks for this explanation of the ASSQ --- I had never heard of this particular measure before.

    (I am writing my own post about this study now, and am trying to figure out how on Earth they got from 0.36% of actual, confirmed autistic children to the 2.64% they think it implies).

    ReplyDelete
  7. Do you know how to interpate the results of this ASSQ ?

    ReplyDelete