Sunday, May 10, 2009

Harold Doherty's Fictitious ABA Study

You have to give Harold Doherty credit for trying. Apparently flummoxed by the lack of real autism science in support of applied behavioral analysis (ABA), Mr. Doherty has responded by concocting an ABA study pretty much out of thin air.

Mr. Doherty's technique this time was to lift some random words from a Washington Post article in order to create the following description of a study presented at this year's International Meeting for Autism Research (IMFAR): “a study was presented at an autism conference by University of Connecticut psychology professor Deborah Fein showing recovery of between 10 and 20% of subjects originally diagnosed as autistic who were later determined to have lost the autism diagnosis after years of intensive applied behavioral analysis.”

Except...

Both a perusal of the Washington Post article and an examination of search results from the IMFAR conference reveal that no such study exists. Deborah Fein did make a presentation at IMFAR regarding a study describing the various characteristics of children defined as “recovered,” but that study was not a long-term study of children under intensive ABA treatment, and that study did not draw any conclusions about the effectiveness of various forms of treatment. Nearly everything in Mr. Doherty's description turns out to be the product of his overactive and not very insightful imagination.

But here is the real irony: if Mr. Doherty's fictitious study were in fact an actual study, it would have devastating consequences for Mr. Doherty's own ABA advocacy. The problem, of course, is that a 10-20% recovery rate is a shockingly low number. The history of autism diagnosis, in which the vast majority of autistic individuals once went entirely unrecognized (and thus by Deborah Fein's definition, would have to be deemed as fully recovered) would suggest the corresponding recovery rate under no treatment at all runs significantly higher than a mere 10-20%. Thus, it would appear the use of ABA on the children in Mr. Doherty's fictitious study has had a tragically negative effect.

Here would be my suggestion to Mr. Doherty: the next time he decides to gild the ABA lily, perhaps he should do a little more gilding in favor of his own side.


[Update 05/16/2009: Be sure to read these observations and informative discussion from someone who actually attended Deborah Fein's presentation at IMFAR.]

5 comments:

Joseph said...

You might have already seen Michelle Dawson's response. The last part says:

"My second comment is that Dr Fein stated, in her IMFAR 2009 presentation, that nothing so far could be usefully concluded about the role, if any, of interventions or treatments in the findings she reported. And indeed she presented no information or data connecting any of her findings to any specific kind, quantity, etc., of intervention."

It would not be the first time something like that has been reported. E.g. see Eaves & Ho (2004).

Michelle Dawson said...

Joseph's comparison with Eaves and Ho (2004) is inaccurate.

Eaves and Ho (2004) reported data about interventions. They found that neither kind nor quantity of intervention was related to the outcomes they reported. Half their sample had received ABA-based interventions.

In constrast, at IMFAR 2009, Deborah Fein and colleagues did not report any data about interventions.

Marius Filip said...

"The history of autism diagnosis, in which the vast majority of autistic individuals once went entirely unrecognized (and thus by Deborah Fein's definition, would have to be deemed as fully recovered)"

Why such a statement? A person without a diagnosis is considered recovered? I don't think that any definition of recovery from autism would uphold that.

The rate of 10-20% of full recovery is actually pretty good. We are talking about full recovery, meaning losing virtually all the characteristics necessary to apply an autism diagnosis.

Alan Griswold said...

For Marius Filip, I'm not sure if I can explain the concept well in a short space, but I'll give it a try:

To me, one of the logical pitfalls underlying many autism studies dealing with diagnosis, intervention, treatment, recovery, etc. is that we do not consider the case of the undiagnosed autistic child. There must be such children, even today, because we have studies that decry the “late” average age of autism diagnosis (I think 5 or 6 is a commonly cited age), and it is not unreasonable to extrapolate from such studies that some children must be getting diagnosed so “late” that they are not getting diagnosed at all. And of course in previous years, before there was so much autism awareness, there undoubtedly would have been more such children going completely undiagnosed.

This group is, by definition, difficult to study, but they are an intriguing group to consider because in a sense they form the ideal control group. How better to judge an autism treatment than by comparing it to a group of autistic children who have received no treatment at all!

Let's take Mr. Doherty's fictitious study for example. Let's assume someone actually did such a study; that is, they followed a group of children who were diagnosed around the age of two, then received years of intensive behavioral therapy, were re-assessed at say the age of ten, and it was discovered that 10-20% were deemed as “fully recovered.” That number by itself means nothing, because there is nothing to compare it against. We could compare it to the recovery rate for a similar group of children who received a different form of treatment, and that would certainly be useful information, but the best information of all would be gathered by comparing results to a group of autistic children who received no treatment at all, which leads us right back to our group of undiagnosed autistic children.

It is of course difficult—indeed, speculative—to judge outcomes for undiagnosed autistic children. But my remark about the history of autism diagnosis is meant to remind us that autistic children who in the past were not diagnosed as autistic do not seem to be filling up our institutions, etc.; and so it seems reasonable to assume a certain percentage of those children simply made their way somehow into society and in many respects would be indistinguishable from children labeled as “fully recovered.” If that percentage turned out to be 50% for instance, then Mr. Doherty would stop feeling so grand about the 10-20% recovery rate of his fictitious study, because it would, by comparison, indicate that the years of intensive behavioral therapy had actually done some harm.

As I hope I have made clear throughout, none of this is science; it is just food for thought. My concern, however, is that no one ever stops to take a nibble.

Marius Filip said...

Thank you for your explanations.

I understood perfectly the logic of taking into consideration the undiagnosed children, yet I consider pernicious to link it to the notion of recovery, because this notion applies to children who have been clearly diagnosed with autism before intervention.

As far as I know, the study of Dr. Lovaas contained a control group. The group which received 40 hrs per week of intervention recovered with a rate of 47%. The control group, that received 10 hrs of intervention per week, recovered with a rate of 1/19.

But there is another method that can measure the efficacy of ABA, though not as good. It is to compare the rate of evolution of a child before and after the intervention. If the development quotient stagnates or goes down before intervention and it approaches 100 during the intervention, it is not hard to assume that the beneficial evolution is due to ABA.

For instance, if a child gains 2,5 years of development in 10 months, it is hard to disprove ABA.

You talk about the undiagnosed children and put them into this equation. It is comparing apples and pears. Children who get undiagnosed are either mild enough to go undetected for a long time (perhaps the entire life) or their families do not have access to quality evaluation.

The same applies to teenagers and adults. I know families with children diagnosed with Asperger's at 12 or 14. On YouTube there is a young woman diagnosed at age 21. Before the diagnosis, these children were catalogued as manic-depressive, as having some psychological (but not neural or psychiatric) problem or simply as being "weird". Teenagers without an autism diagnosis may transform into adults with the same fuzziness upon their fate - this does not mean they fit into the society the same way as a child who underwent years of intervention and was scrutinized for the same period.

The last but not the least, I give the case of my home country.

We have here a bizarre legislation which puts any individual with autism 18 years of age into the category of schizophrenics (the autism advocacy groups try to change the legislation, it will happen. hopefully, the next year).

So, in my land autism simply ceases to exist at age 18. Can we put this "miraculous disappearance" on the same level as the recovery resulted from years of ABA intervention?

I don't think so.

So, I consider that "ignoring" the undiagnosed autistics is not a flaw of the method because due to the very lack of diagnosis they cannot form a basis for anything, let alone for an objective study.