Wednesday, October 27, 2010

Think You Have Autism? Don't Take a Quiz.

Over in the autism area on about.com, Lisa Jo Rudy posted something today where she talks about how taking the self-administered Autism Quotient (AQ) test might be a good first step if you think you have autism -
Of course, there is a disclaimer that these tests are not diagnostic, and they're only to be used for "genuine research." Still, for parents of children on the autism spectrum, adults wondering whether they might fall into an autism spectrum category, or anyone concerned about the symptoms of autism, these tests may be a useful first step in deciding whether or not to seek an evaluation.
This is very bad advice.

First of all, if you seriously think that you may have a disorder such as autism or any other mental disorder then you should seek professional help.  You should not try to take a test you found on the Internet to see if you might "fall into an autism spectrum category" or to give you a "hint".  None of the lists of signs symptoms on the Internet nor any of of the tests out there are going to be able to tell you whether you have autism.  Your "hint" should be that you think there is something wrong and that alone should spur you to seek a professional's opinion.

But lets say that you have decided to take Lisa's advise or that you, like many people before you, have decided that this test is an appropriate way to determine whether you have autism.   The question then becomes if you take the Autism Quotient test and score high enough, how likely are you to actually have autism?

It turns out that the answer is not very likely - the odds are about 1 in 10.

To say that the opposite way, if you had a group of ten people who all scored high (over 32), nine of them would not have autism.  So if you are claiming that you have autism based solely on a self-administered AQ test then you are most likely wrong.

To make matters worse, even if you score below that 32 number, you are still somewhat likely to have autism.  If you took a group of ten people who actually had autism, two of them would likely score below the 32 cutoff.

So either way the AQ isn't going to tell you that much.

To understand the reason the odds are so very much against you, you have to understand what the purpose of a test like the AQ is - and what it isn't.  A test like this is supposed to be used to help find people who are more likely to have a condition.  It is not supposed to be able to tell you that one specific person has the condition.

A test like this can be used as a screening tool in research to help researchers find the people who are more likely to have a condition so that they can focus on those people.  Or, if the test has been validated in earlier studies, researchers can use the results of the test to draw general conclusions about the people in their study.

In either case, to properly use the test you have to know how good of a job the test does at finding those who have a condition and eliminating those that don't.  These concepts are formally called sensitivity and specificity.  The sensitivity of a test tells you have good of a job the test does at finding the people who have the condition while the specificity tells you how good of a job the the test does at telling you which people don't have the condition.

In an ideal world, you would want every test to identify every person that has a condition (sensitivity of 100%) and exclude every person who doesn't (specificity of 100%).  But, in the real world, tests are never that accurate and we have to make do with what we can get.

When it comes the sensitivity and specificity, the AQ isn't the greatest but it isn't too bad.  According to a recent paper that I was reading, the AQ has been demonstrated to find 80% of the target group (sensitivity 80%) while only misidentifying 2% of the rest (specificity ~ 98%).

Which leads me to my next, point, what is it exactly that the AQ is supposed to be finding?  Here's a hint, it isn't what you normally think of as autism - "The Autism Spectrum Quotient (AQ) has been developed to measure the degree to which an adult with normal intelligence has autistic traits".  If you dig a little bit more, you would see that this test is meant to find adults who have "traits" of Asperger's or high-functioning autism.

Or in other words, this test is geared towards finding individuals who fall on a small portion of the autism spectrum but is not going to be very helpful in identifying the majority of people with autism.  Well, it won't be helpful  unless you believe that - all evidence to the contrary - the majority of people with autism are high functioning.

So far the AQ doesn't seem to bad - it should be capable of identifying the majority of its target audience.  But there is one last piece to the puzzle of understanding why the AQ isn't able to tell you much, and that piece is called the positive predictive value.  The idea is to take the sensitivity and specificity and put them in practical terms, and this value does that by telling you how likely it is that you have a condition if you have a positive value on the test.

This is where the AQ starts running into some serious problems - the positive predictive value is somewhere between 8% and 12%.  This means that even if you have a positive value on the test, you are only about 10% likely to have Asperger's or high-functioning autism.  The other 90% of the time the test is just wrong.

If that doesn't make any sense to you, perhaps an example would make the picture clearer.

Say you are looking at a group of 10,000 people and that we can assume that 1% of the people have any form of autism.  How many people will the AQ correctly identify, how many will it fail to identify, and how many will it falsely identify?

The first thing we need to know is how many of the cases of autism we would expect to be classified as Asperger's or high-functioning autism.  While there isn't a good answer to this question, I think it would be reasonable to assume that 25% of all of the people with autism fall into one of those two groups.  I suspect that the actual number would be less than this, but this number is good enough for this example.

So, with 10,000 people, we would expect 100 of them to have any form of autism and 25 of them to have Asperger's or high-functioning autism. Given what has been found in studies that looked at the validity of the AQ, we would expect that the AQ would correctly identity 20 of that 25 (80% sensitivity) and miss 5 out of the 25.  We would also expect that 2% of the rest of the population would be falsely identified as having Asperger's or high-functioning autism, or roughly 200 people.

If we put that all together, we would find that, out of the 220 people identified by this test, only 20 would actually be correctly identified and another 5 would be missed.  Hence the positive predictive value would be about 9.1%, which translates roughly to a 1 in 10 chance of being correctly identified.

Now, just to nitpick my own argument and be completely honest,  one possible source of error in the above analysis is what the AQ would show for the 75 other cases of autism.  Would it be able to label them as having autism or would it miss them as well?  The short answer is that we don't know because this test has never been validated in this population.  No published research has shown how well the AQ does at detecting classic autism, PDD-NOS, Retts, or CDD in a large group of people and, without that information, it is simply impossible to know how well it would perform.

But, for the sake of argument, lets assume that the AQ works the same for every person with autism rather than just the higher functioning ones.  If you use the same assumptions that I used above, you would find that a positive result in the test would give you a 1 in 3.5 chance (or about 30%) of actually having autism.

The bottom line is that even if you get a high score on the Autism Quotient you are not likely to autism.   So if you think you might have autism, don't take a random test on the Internet - go talk to a professional instead.

22 comments:

  1. My score on the AQ is 47, though I did not take that test as a "parlour game" on Facebook, which seems to me to be a total debasement of the principle.

    I took my test as part of the research cohort when the test was being devised, though it seems that my criticism of the test was not taken all that seriously.

    I do think the motives behind the test's origins and the actions of its devisors (in releasing it and the other tests in the Cambridge battery, for use copyright free by bona fide researchers) is noble enough because I have grave misgivings about various other protocols such as ADOS and DISCO, which are not available, expensive and highly proprietry.

    ReplyDelete
  2. Just to reiterate the material you've already quoted: this test is NOT diagnostic, but it can be an interesting tool for getting a sense of whether you do have some of the traits of high functioning autism/aspergers. Obviously if it were that easy to diagnose autism, we wouldn't need a whole army of specialists!

    Of course, I agree: if you do feel you may be diagnosable on the spectrum, it makes sense to see a professional. The problem is that there are not all that many professional psychologists, neurologists, etc. with real experience diagnosing adults with autism spectrum disorders.


    Lisa

    ReplyDelete
  3. Except of course that majority of professionals don't know what autism is either.

    ReplyDelete
  4. Author,

    It may not be a bad thing that the more comprehensive tests are less available as they really need to be administered by someone who has been properly trained. They are not the sort of thing that just anyone could or should pick up and use.

    That argument also applies to the AQ as well and I think that it being published in Wired and on Facebook was a bad thing. I am getting tired of running into people who think that they have autism simply because they are technically oriented and had a high score on the AQ.

    I am not saying that these tests should be completely unavailable but just that they should not be published in a way that encourages everyone to take them. I think that "typical" people will look at their high score on the AQ and wonder why people make a big deal about autism. They won't understand that this test doesn't tell you that you have autism (or even features of autism).

    ReplyDelete
  5. Lisa,

    The problem is that people won't go to see a professional and will self-diagnose on the basis of the AQ test alone. If you need proof of that just wander over the wrongplanet.net and start reading the discussion forums.

    I think we both know that there is a very large different between "traits" of HFA or asperger's and having the real thing. Although I have to wonder if it is even valid to talk about HFA/asperger having distinct "traits" because these "traits" are nothing more than typical human traits taken to extremes.

    Being fascinated by numbers or wanting to go to a museum doesn't make you autistic, it makes you a person.

    ReplyDelete
  6. It is all a matter of context.

    For some time it has been available via wired, as a sort of "in thing" amongst geeks and I have never been much impressed either by the strange behaviour of on line aspies who have gone to the test and compared scores, the same way as people compare the results of Meyers Briggs or whatever.

    It was intended as a serious research tool, where the context makes some sense, and to put it on facebook alongside such quizzes as "which superhero are you" or whatever is to me a travesty because it does devalue the notion of autistic difference in a trivial way as if it does not matter.

    It does matter.

    As for me even though I was professionally diagnosed I wasn't absolute prepared to accept it until I "peer validated" it, that is to say met and interacted with other people with the condition.

    It is of course possible to diagnose a range of things from questionnaires of various types. Just because they have never hitherto been recognised by a clinician doesn't mean they are not there and often because the clinician only has a narrow focus and fails to see a pattern.

    My GP, a trained psychiatrist in his other capacity, told me that he was grateful that I had been diagnosed because he began to see it in other patients after that.

    As with a lot of conditions, the even a trained GP only sees very small slices of your life, anything that helps put a wider context in is useful, such as the questionnaire having validity to sort out grounds for a referral. In that context the test is not thatdifferent from asking a series of questions of such as "where does it hurt?" "can you lift your arm above your head? "do you have a temperature?" of stuff to rule out or in a physical problem that needs further investigation, or indeed not different outside of the medical context from a dyslexia screening test to determine if one might need extra educational support.

    ReplyDelete
  7. "The question then becomes if you take the Autism Quotient test and score high enough, how likely are you to actually have autism? It turns out that the answer is not very likely - the odds are about 1 in 10. To say that the opposite way, if you had a group of ten people who all scored high (over 32), nine of them would not have autism."

    The above appears to imply a true positive rate of 10%, or a sensitivity of 10% for the AQ test.

    "According to a recent paper that I was reading, the AQ has been demonstrated to find 80% of the target group (sensitivity 80%) while only misidentifying 2% of the rest (specificity ~ 98%)."

    The above states a sensitivity of 80% for the AQ test.

    These statements appear contradictory. Could you please elaborate or cite the sources?

    ReplyDelete
    Replies
    1. I thought I went into all of that in the main post, but the short version is this.

      The 80% is the sensitivity and the 10% is the positive predictive value.

      The sensitivity is the number of people who have the condition that are correctly identified. So if you have a group of 1,000 people and 5 of them actually have HFA and the test picks up 4 of the 5, then the sensitivity is 80%.

      The positive predictive value, on the other hand, looks at the entire population and tells you how many of the population were correctly identified. So for that same 1,000 people, if the test told you that 20 people had HFA who didn't then your chance of being correctly identified as positive is about 16%.

      The above example assumes that the actual number of people with HFA is about 0.5% of the population which is a little on the high side. The positive predictive value is going to change as the condition becomes more common but the sensitivity isn't.

      There are links to the concepts in the main body of the post.

      Another problem with the AQ is that it can't reliably tell the difference between HFA and schizophrenia. Schizophrenia is more common in adults than HFA so that makes the AQ even worse. See here - http://autismjabberwocky.blogspot.com/2010/11/hazards-of-self-diagnosis.html

      The bottom line here is that you can't assume that you are on the spectrum simply because you get a high score on the AQ.

      Delete
    2. Thank you for the explanation. I had misread the definition of sensitivity vs. positive predictive value. Thanks again.

      Delete
  8. I'd be interested in the heredity aspect - like how many autistic children have parents who score high on the AQ, despite not having autism themselves. (I actually have ADD which residuals causes every self-diagnosing autism test for me to be positive!) Perhaps this is something Simon Baron Cohen has already looked at, though. Thanks for the informative post!

    ReplyDelete
    Replies
    1. That is actually a really interesting question! and a great way to make use of the AQ test.

      Delete
  9. Hi M.J.

    Just curious, what is your occupation? Are you a Statistician?

    (This isn't some sort of snarly underhand-comment, I am really curious)

    Apologise if I could find that info elsewhere on the blog, I couldn't find much info in the 'about' section and have only read a couple of posts.

    I don't agree with you about everything you write, but I like your number crunching and rigour.

    Ps. If I don't reply back again in a timely manner, then it is because my Wordpress comment feed don't keep track of comments on other blogging platforms, and I may forget to manually visit your blog to see if there is a reply.

    ReplyDelete
    Replies
    1. Hi Mados,

      No, I'm not a statistician but I have some education in statistics and I spend a lot of my professional life dealing with analysis and calculation. I don't really like to go into more detail than that for a variety of reasons.

      And I don't mind in the least when people disagree with me. In a way, I actually prefer it because you rarely learn anything new from people who agree with you.

      Delete
    2. Thank you for your answer. Ok then...

      I have a bit of education in Statistics too (a couple of subjects: Statistics and Market Analysis... and subjects that weren't Stats but which Stats was a prerequisite for, like Cultural Sociology, American Studies, British Studies, Globalization from an American Perspective, et.c.). Stats started with a crash course called 'how to lie with Statistics', about the misuse of Stats in news media etc. I don't use Stats at all. Except I work as a research interviewer (quant) and frequently need to explain basic principles of statistics to explain to respondent why I can't just replace them with the household next door. So that refreshes my mind a little bit, although I must admit that I have forgotten most of what I learned.

      Re. disagreeing, I agree with that:-) Disagreement is usually healthy, as long as it is based on mutual respect and willingness to try to understand others' perspectives.

      Delete
    3. I have one more curious question: have you taken the AQ test yourself? Just based on the writing style of the few of your posts I've read (rigour, structure, topic focus, perseverance, consistent adhering to chosen logic, black & white thinking et.c) I imagine you would score quite high. (no insult intended! as you say, a high score is not in any way a diagnosis, it just highlights a certain type of mindset, e.g. often an engineer/geeky/nerdy type of person)


      Delete
    4. Its been a long time since I took it but my AQ score wasn't that high, if I am remembering correctly.

      I find it interesting that the things that you mention aren't really associated with autism as a whole and shouldn't translate into a high score on an autism screening test. They are associated with one of the stereotypes about autism, the extreme male brain theory, but they aren't necessarily part of the larger spectrum of behaviors.

      For example, people with autism have restricted interests which can sometimes translate into perseverance but more often just takes the form of fixation or obsession.

      Delete
  10. Re. word verification: is it possible to turn that stuff off for repeat commentators? I have already proven that I am not a robot. The word verification process is difficult and tedious, and why punish people who want to interact with you on your blog with a straining visual exam every time they want to contribute.

    ReplyDelete
    Replies
    1. The word verification (captcha) isn't something that I can turn on or off, unfortunately. They are a part of the blogger platform.

      Delete
  11. Thank you for posting this...I am one of those helpless quiz takers.

    I scored a 22 on the AQ...but after reading this article, I still strongly believe that I have some sort of a mental disorder. I struggle greatly with verbal communication...it is almost like whenever I want to speak, I literally can't, because it almost feels like my brain shuts down, and I can't bring myself to talk aloud. I used to just think that maybe it was because I was extremely shy, but because of recent events in the past year, I wouldn't consider myself shy anymore. In fact, I would consider myself confident-it is my lack of the ability to communicate my feelings and thoughts aloud often that pulls me up short. What I am writing right now, I would never be able to say aloud, because my mind just wouldn't be able to handle the strain of it-I would start to stutter and then the words would disappear from my thoughts. It's sometimes almost like trying to remember that really important dream that you had before you woke up in the morning-it's nearly impossible! I do unusually well in school, my test scores are way above the expectations, I've read at college level since elementary school, and yet my mind puts up a wall whenever I hear people talk (unless I can look at their lips to read what they're saying)or when I try to speak. I'm desperate for this phenomenon to be explained, and after reading this article, hopefully I will finally get up the nerve to speak with my mom about seeking professional help about it. I'm really hoping that they will have answers...it would be a relief to know that I had autism or something similar; it would all finally be explained. :)

    Thank you.

    ReplyDelete
  12. Your blog is great so far ... I just wanted to say you are right that people are using these tests to self diagnose. I didn't realise the test had been publicised on Facebook / Wired. This explains why my doctor-shopping friend ended up finding out about it, and eventually going to the Cambridge centre to get "diagnosed". Knowing my friend as well as I do, they will certainly have read all the tests beforehand and done their best to get the diagnosis - and the researchers also had an agenda of needing high-functioning "aspies" to justify funding. It has been irritating the hell out of me as this person is not remotely autistic/asperger ... It is also making me sceptical about how many people there are out there who are self diagnosed or who have similar ropey diagnoses. I've noticed it seems like 99% of people who call themselves aspergers online are "self diagnosed", and that even a lot of people who aren't sound quite normal to someone who grew up as the class nerd in the 70s (ie me). Not saying that there aren't people who truly warrant these diagnoses, but I keep remembering that old adage about "no disorder without disability".

    Yes this is a personal rant. Sorry. Anyway, I like your blog. Just read your post about people with Aspergers not wanting to be called autistic and being hypocrites ... it was spot on. The friend I mentioned above has weaseled their way into an influential position in that community. I don't think they will like their label nearly as much when it changes - and also I wonder how quickly they would ditch it if forced medication was on the cards? Probably suddenly change to claiming to be normal and just misunderstood ...

    ReplyDelete
  13. QUESTION!!! It was brought to my attention of someone I know pretty well that I might be on the autistic spectrum due to some of the traits I have. I haven't taken any tests since ALL online tests seem to be bogus I just stay away. Any idea what type of health professional I need to see? A general doctor? A psychologist? what?! I dont have a primary doctor right now so I dont want to pay a $25 copay to go to the wrong doctor. any suggestions

    ReplyDelete
    Replies
    1. I would suggest finding a psychologist or psychiatrist in your area. But not all professionals are going to have experience with diagnosing autism in adults so you might want to talk to a local agency that deals with autism or your local chapter of autism speaks (or a similar organization) to see if they have any suggestions.

      Good luck.

      Delete