Revisiting The New York Times' DSM 5 Autism Coverage

If you remember, a few months back there was a big to-do about an article  in the New York Times on the upcoming DSM 5 autism criteria.  This article, which was based on unpublished data, breathlessly suggested that many people with a current definition of autism could lose the diagnosis under the new criteria.

The main claim from the article was basically this -

In the new analysis, Dr. Volkmar, along with Brian Reichow and James McPartland, both at Yale, used data from a large 1993 study that served as the basis for the current criteria. They focused on 372 children and adults who were among the highest functioning and found that overall, only 45 percent of them would qualify for the proposed autism spectrum diagnosis now under review.

Well, the study was finally published a few weeks back and, I have to say, the article in the NY Times was a flaming piece of horseshit.  Or perhaps a better way of putting that is the title of the article should have been "what happens when you map an ancient data set onto a new set of questions when the data for the new questions wasn't collected in the first place".  The subtitle should have been "What?  That one question was important?"

OK, that might be a little bit harsh, but there is just a little difference between "new definition of autism will exclude many" and what the study actually says.

To give you an example, see the line above that says "they focused on 372 children and adults who were among the highest functioning"?  Well, what that should have said was that they focused on the 237 "highest functioning" children and adults with autism but also included 96 children and adults who had developmental disabilities other than autism.  The "only 45 percent" referred to the 237 - not the 372.

OK, so they inflated the number a measly 56%, that doesn't change the result, right?  Well, other than make the higher functioning group appear to be much larger than it was no, it doesn't really.  Oh, but wait, there was this second group, a group of 420 "low-functioning" people with autism.  Does making the first group seem roughly the same size as this second group matter?

It doesn't really matter, no, not until you consider that the majority (about 70%) in this second group kept their diagnosis under the DSM 5.  In other words, the study found that the largest group mostly kept their diagnosis while this other, smaller group lost it about half of the time.  And, as I will show below, most of the .
people who "lost" their diagnosis in both groups did so because of just 1 criteria from the DSM 5 and the reason they didn't meet the criteria is likely because the original trial didn't collect enough data.

So, even though each little fact doesn't matter that much, when you put it all together it makes a great deal of difference.  But I guess calling the article "the majority of the majority keeps their diagnosis under the new autism criteria and those who lost it were probably just missing data" wouldn't have made such a good headline.

Sigh. Journalists never seem to let the facts get in the way of a good story.

The good news is that, now that the full study has been published, we have the opportunity to look at the data and see exactly what the claims in the NY Times article were based on.

As I suspected, the study is based on data that was originally collected for a 1994 field trial of the DSM IV. This earlier field trial was done to test how well the then new DSM IV criteria for autism compared to earlier criteria such as the DSM-III-R.  That last bit is quite important - this trial wasn't a general test of the proposed DSM IV criteria but rather a comparison of how well the criteria did compared to earlier ones in a group of people that were already "known" to have or not have autism.

The current study took the data collected in the earlier field trial for all of the criteria that were tested (DSM IV, DSM-III-R, ICD-10) and mapped them onto the proposed DSM 5 criteria to evaluate how well the DSM 5 worked.  This is the single largest problem with this study and why the results are pretty much meaningless.

The current study says its objective is to test the "potential impact of the proposed DSM 5 diagnostic criteria for autism spectrum disorder" but that is not what it really did.  In reality, the current study, like the earlier field trial, is a comparison of the proposed criteria to the current ones.  It is not a study about how well the new DSM 5 criteria identifies people with autism in the general population but rather a comparison of how much the old criteria and new criteria agree.

But the current study fails at even that task.  In the earlier field trial, data was collected for all of the criteria being tested and then the criteria were compared.  But since the original field trial didn't include the DSM 5 criteria, it didn't collect the data needed to see if someone met the DSM 5 criteria.  So the researchers came up with a system of mapping the items from the earlier criteria to the criteria in the DSM 5.

The problem is that the questions in the original field trial did not directly address the new criteria in the DSM  5.  There were areas that had a lot of coverage, areas that had extremely little coverage, and areas that had almost no coverage.  And, as I will show below, it is these areas of little to almost no coverage that are mostly responsible for "excluding" people under the DSM 5 criteria.

If the authors had actually wanted to test the "potential impact" of the DSM 5 criteria and how they stacked up to the current DSM IV, they should have done the same thing that the original field trial did.  They should have picked a large group of people and collected data for the DSM IV and the DSM 5 criteria and directly compared the two.

There are a few other problems with the study such as how the number of people with autism has grown in the time since the field trial was done and the possibility that the sort of person who would have gotten a autism diagnosis twenty years ago is likely to be different from the sort of person who would get a diagnosis today.  Also, since the world of autism has changed so much in the intervening years, it would really have helped to use more recent data.

To give you an example of how much the world of autism has changed, the original field trial used people who had "professional backgrounds and experience in the diagnosis of autism" to help rate the data.  About half of these people had "extensive experience" in diagnosing autism where "extensive experience" meant "had evaluated more than 25 cases of autism".  In other words, half of the people making the determinations in the earlier trial had made less than 25 autism diagnoses.

Compare that to today were even someone new to the field might will make that many diagnoses in their first year.  The current "experts" in the field have probably made hundreds of diagnoses and been involved in far more cases than that.

But, without further ado or commentary from me, here are the actual numbers from the study.

First, the paper took most of the original participants, broke them down by their specific diagnosis, and determined whether they met the DSM 5 criteria.  They also included the "Related Disorders" category from the initial study to test whether the new criteria would exclude them.

The table below shows the numbers from the original 1994 study, the number included in the current study, and the part of the group in the current study that met the DSM 5 criteria.


When you look at it this way, you can see that the majority of people with an "autism" label retained their diagnosis while the PDD-NOS and Asperger groups mainly lost them.  I will get into why this happened in a little bit.

Next, the study broke up the participants by "functioning level" and looked at how well the criteria fit the different groups.  "High Functioning" was defined to be children and adults with an IQ above 70.  Some of the participants were missing IQ data and were excluded from the "high functioning" group.  The numbers for the "low functioning & missing data" group are my own calculations from the data included in the study (Total - High Functioning).

The meaning of the columns in the table below are the same as the table above.


If you notice, more of the "high functioning" were excluded under then DSM 5 than the "lower functioning".  The authors felt that the DSM 5 fit the "lower functioning" side of the spectrum better than the higher functioning.  This finding was the basis of the NY Times article.

But, that isn't the complete story of what this data shows and there are some substantial weaknesses in this data.

As you can see, most of the participants from the original study fell into the low function group.  The "high functioning" group that was talked about in the NY Times made up only about 35% of the people with autism.  So, like normal, the NY Times article was focusing on the high functioning minority and ignoring the majority of people with autism.

However, the more important question is why some of the participants did not meet the new DSM 5 criteria.  In my opinion, the answer to this question is that the original study did not cover enough of the new DSM 5 criteria.  The study included some information about why participants might have been excluded so I was able to create the following summary.

There was not enough data in the study to break the groups down into high and low functioning, so I am using the groups from the paper.


The table above refers to the categories in the proposed DSM 5 criteria.  If you remember, the DSM 5 criteria combines DSM IV communication and social criteria and expands the repetitive and restrictive behavior criteria to include sensory issues.

The % Total Group column refers to the percent of the total group (all participant, n = 657 and High Functioning, n = 240) that could have been excluded because they failed to meet the criteria in this category.  So the first number, 27%, means that 27% of all participants could have been excluded for failing to meet the DSM 5 social communication criteria.

The Total Number column gives the number of participants that could have been excluded under this category while the % of Excluded puts the Total Number into terms of the excluded group.  Or in other words, 68% of the group that might lose their diagnosis under the DSM 5 could do so because they don't meet the social/communication criteria.

The important thing to keep in mind is that a given person could have been excluded for failing to meet more than one set of criteria.  So a person could been excluded for failing to meet the social communication criteria as well as the age of onset.

One thing that stands out for me here is that almost 10% of the excluded group could have been excluded because the symptoms didn't appear early enough (age of onset).  This is in spite of the fact that the DSM 5 doesn't have an age cut-off and actually is less restrictive when it comes to age than the DSM IV is.  This is in spite of the fact that the DSM IV PDD-NOS diagnosis does not have a strict age criteria.  Even though both systems were flexible with respect to age of onset and the DSM 5 is less restrictive than the DSM IV, almost 10% could have been excluded because of age of onset.

In other words, if a person met the onset requirements for the DSM IV, they almost certainly would meet the age requirements for the DSM 5 and should not have been excluded.  I don't know how many of this group were excluded for only missing the age requirement but this does serve as a good example of the disconnect between the original data and the DSM 5 criteria.

Moving along, when you dig down into the data further, it quickly becomes apparent that the vast majority of the exclusions came from three of the DSM 5 criteria.  All three of these criteria share the fact that they are mapped to only 1 of the items from the original field trial.  All of the other DSM 5 criteria are mapped to more than 1 items from the original study.

That simple fact speaks volumes about the final result - the DSM 5 checklist items with the least coverage in the original data are responsible for most of the exclusions.

The table below shows these three criteria and the part of each group that did not meet the criteria.  The columns are again the same as the last table.


All three of the criteria are interesting but I want to focus on just the one social communication criteria.  In the DSM 5, you have to meet all 3 of the criteria under the social/communication domain so failing to meet this one criteria could exclude you from getting an autism diagnosis.

As you can see from the table above, the majority of both groups (high functioning - 70%, all participants - 61%) could have been excluded because they failed just this one criteria.  The criteria isn't anything surprising or strange either, it is really one of the defining characteristics of any form of autism -

Deficits in nonverbal communicative behaviors used for social interaction; ranging from poorly integrated- verbal and nonverbal communication, through abnormalities in eye contact and body-language, or deficits in understanding and use of nonverbal communication, to total lack of facial expression or gestures.

It doesn't get any more basic than that and I can't really imagine anyone who would merit a DSM IV diagnosis of autism who didn't have problems in this area.  I think the problem here is that this criteria was poorly mapped and this one criteria alone could explain a large part of the results.

I can't say how much of the result is based on this one criteria for certain because there wasn't enough data included in the study.  But I can say that it was certainly a major contributor to the end result

The other two items had a lesser impact because the repetitive and restricted domains have 4 criteria so missing two of them doesn't automatically exclude you.  But it certainly would make it much harder to qualify unless you had the remaining two criteria.

Whew.  If you are still reading all the way down here then you certainly deserve a gold star.

The overall point here is that this paper doesn't really tell us anything.  It certainly doesn't tell us how many people with a current diagnosis of autism will be excluded under the proposed DSM 5 criteria nor does it tell us how the DSM 5 criteria compare to the DSM IV criteria.  And most of the result can be explained as just missing data.

All it really says to me is that the researchers had a bone to pick with the group of people who are writing the new DSM 5 autism criteria and decided to use an old data set to make a point.  I only wish they had kept their disagreement in the scientific world and not tried to take it to the court of public opinion.

The bottom line is that we are going to have to wait to see what impact the DSM 5 is going to have on people with an existing autism diagnosis.