The Emergence of Early Behavioral Signs of an Autism Research Disorder

It looks as though I will to need to interrupt my blogging break before it has had much of a chance to begin:

The occasion for this interruption is the online publication of A Prospective Study of the Emergence of Early Behavioral Signs of Autism (Ozonoff et al., 2010). Note that I have recently posted my thoughts regarding a different paper from this same general group of researchers, Play and Developmental Outcomes in Infant Siblings of Children with Autism (Christensen et al., 2010), a paper I generally decried as containing too much researcher bias and too much reliance upon the use of concocted measures. But as luck would have it, Christensen et al. (2010) was apparently only the warm-up act: when it comes to researcher bias and concocted measures, surely nothing can hold a candle to Ozonoff et al. (2010).

And I am starting to worry. Observations of infants and toddlers who are at high risk for autism (because they have older siblings who have already been diagnosed with autism) carry the potential of providing some valuable insight into the nature of the condition; but this will only happen if those observations arrive mostly unfiltered. Based upon what I have seen so far in Christensen et al. (2010) and Ozonoff et al. (2010), and given that a good portion of the research wherewithal directed towards at-risk children has been entrusted to this one tight-knit, rather homogeneous group of researchers, it appears as though these observations are not only going to arrive filtered, they are going to arrive after having been passed through a very distorting lens.

Let me begin by summarizing my complaints and concerns about Ozonoff et al. (2010).

1. The study is based almost entirely on made-up measures—measures designed to give the appearance of science when in fact those measures are not scientific at all. And as if that were not bad enough, those measures are then used, quite falsely, to create the illusion of comparable data, when in fact no such comparison is warranted. The findings of Ozonoff et al. (2010), while not entirely without merit, are based far too much upon a constructed fiction.

2. This particular group of researchers has been displaying a consistent bias in how it regards autism—etiology, preferred treatments, etc.—a bias that is strongly coloring the group's research methodology, and more importantly, is causing the group to overlook and dismiss data that does not fit into its preconceived notions. The findings of Ozonoff et al. (2010), while not entirely without merit, have been rendered needlessly incomplete through researcher bias.

3. For a so-called prospective study, Ozonoff et al. (2010) seems to have had a good portion of its structure retrospectively decided. While this is not sinister in and of itself, given the background of the researchers and given their potential interest in having the findings of these studies turn out in certain ways, it would seem that a greater premium would be placed upon methodological transparency and fair-mindedness. The findings of Ozonoff et al. (2010), while not entirely without merit, raise questions about general approach and about potential conflicts of interest.

Made-up Measures. There is a reason that much of science has been built up around the consistency of the yardstick and stop watch. I realize not all experiments can be conducted with quite the same degree of measurement consistency as is provided by distance and time—including within the field of autism research—and thus some leeway towards the use of broader techniques can at times be tolerated. But that leeway should not extend to complete freedom in making up measurement tools on an as-needed basis.

The measures upon which Ozonoff et al. (2010) relies are spelled out in the section “Measures Used to Track Behavioral Symptom Emergence,” a section surely deserving of a creativity award, but just as surely not deserving of the name science. Although I cannot do justice to the section myself—it really needs to be read to understand just how much measurement construction is actually going on—let me say that in essence it lays out various observer count and judgment statistics that are re-grouped and massaged together into categories freely labeled as face gazes, social smiles and directed vocalizations—“yardsticks” that I doubt have ever been employed in quite this way before, and quite likely will never be used this way again. True, these methods do allow for some crude observational comparisons between autistic and non-autistic individuals at similar ages—and so they are not completely worthless—but think about trying to repeat this experiment. Think about another research group having to train a set of observers to count face gazes or different types of vocalizations in precisely the same way as in Ozonoff et al. (2010), and you will realize there are no legitimate means by which to replicate this study, because the study has been based almost entirely on measures more fuzzy than a cotton ball.

But it gets much worse. Note that these made-up measures are applied to the study subjects at 6 months of age, 12 months, 18 months, 24 months, and 36 months, with the researchers then implying, straight faced, that these measurements can then be directly compared across all these ages. This is just the height of folly! Take face gazes for instance. Think about what might be counted as a face gaze coming from a six month-old, and then think about what might be counted as a face gaze emanating from a 36 month-old. I am not an expert on infants and toddlers, but I do not hesitate to say that the quality and characteristics of a face gaze from a six month-old are not going to be anything like those from a 36 month-old. That does not stop the Ozonoff et al. (2010) researchers, however—not one bit. There you can find them, plotting out these measurements across all ages in the graphs of Figure 1, alongside corresponding conclusions about how autistic and non-autistic children have “gained” or “lost” skills over time; there is never the slightest hint that these across-age comparisons are instead a massive instance of placing 6 month-old apples next to 36 month-old oranges. And if you have the slightest doubt about the non-comparability of face gazes over time, think about how much more absurd are the comparisons of directed vocalizations. These are the summation of nonverbal vocalizations, word verbalizations and phrase verbalizations that can be corresponded to face gazes. But tell me, exactly how many word and phrase verbalizations do we expect from a 6 month-old, and in contrast, how many word and phrase verbalizations might we expect from a 36 month-old? Once again, the so-called directed vocalizations of 6 month-olds are demonstrably nothing like the directed vocalizations of 36 month-olds, and yet the researchers act as though giving these measurements the same name is all that is needed to justify their direct comparison.

This research behavior is inexcusable, because after all it was the Ozonoff et al. (2010) researchers who made up the measures in the first place. They as well as anybody would know that comparing these measurements across child ages is ludicrous, and I would have to seriously question the intelligence or integrity of anyone who would insist on doing so. But note that this is precisely what happens when scientists start relying on measurement schemes that stray too far from objectivity—it is not all that large a step from made-up measures to made-up use of those measures. As I said in my comments regarding Christensen et al. (2010), concocted measures are the calling card of a concocted science.

Researcher Bias. There is never much doubt about where this research group stands in its description of autism. Both Ozonoff et al. (2010) and Christensen et al. (2010) are literally littered with phrases describing autism as a social deficit disorder, one best approached through early intervention directed towards getting autistic children to adopt social behaviors more in line with those of non-autistic children. This philosophy precedes any attempt at observation.

Of course, there is nothing unusual or wrong about researchers having a point of view. But when that point of view colors nearly every aspect of their research methodology, and when that point of view causes the researchers to consistently overlook important pieces of information that do not fit neatly into their preconceived notions, then there is indeed a problem. My comments regarding Christensen et al. (2010) already noted that these researchers' dismissive attitude towards autistic-like behaviors in infants and toddlers has caused them to turn a blind eye to the characteristics of these behaviors, and thus the researchers are overlooking valuable information, namely that early autistic behaviors are indeed quite structured, predictable and purposeful, characteristics that would be obvious to anyone willing to take a closer look. But this theme of willful blindness is continued unabated right through Ozonoff et al. (2010). First, note the influence of the researchers' point of view on research design: all the measures these authors decide to use are directed solely towards their theory of autism as a social deficit disorder—face gazes, social smiles, directed vocalizations, examiner ratings of social engagement. This would be a lovely set of statistics if all we were interested in is what the authors want to tell us, but surely a much broader set of statistics would be more helpful if we what we are interested in is what the infants and toddlers have to tell us.

And then there is the curious case of the unused statistic. The researchers start out by measuring a category called gaze to objects, but note the problem already contained within the description of that measure—“infant's gaze is directed toward an object that the examiner is presenting to the child or to another object visible in the frame”—anyone with even a modicum of understanding about autism could tell you that there is a world of difference between attention paid to an object presented by another person and attention paid to an object through independent motivation. But this research group, so caught up in its social deficit model, fails to untangle that distinction, and when its further efforts to fit this already mangled statistic into its thesis fail to gain significance, the authors decide to drop all further mention of the measure. Thus orientation to objects and structure, potentially one of the more valuable pieces of information that might have been gathered from this study, ends up getting so messed up by researcher bias that there ends up being no information at all. This is clearly a disservice to science, and a disservice to autistic individuals.

It is easy to see what one wants to see, but the trick in science is to overcome this tendency long enough to see instead as broadly as one can. Ozonoff et al. (2010) does not rise to the level of that standard.

Questions about Approach. Although Ozonoff et al. (2010) does not spell out its overall approach in great detail or with much clarity, one can still piece together enough information from its pages to realize that what was prospective about this study was the gathering of much raw data from a rather large group of participants, while what was retrospective about this study was the harvesting of subsets of this data—as well as subsets of participants—from the initial study group. Of course, this approach raises some key questions about the timing and purpose of various study technique decisions, but these questions go largely unanswered.

I would not mention this but for the fact it must be realized that the circumstances of these at-risk infant studies, as well as the circumstances of the researchers who have been entrusted to conduct them, by necessity invite greater scrutiny. Lists of researchers given the means to conduct at-risk infant studies reveal a consistent and like-minded set of names: names such as S. Rogers, S. Ozonoff, M. Sigman and G. Dawson are associated with these studies again and again. These are researchers who are close colleagues, and who have built careers around a similar social-deficit view of autism, and who are holding mostly high-paying positions at organizations that espouse nearly identical views of autism, and who have been associated with a self-promoted and perhaps proprietary intervention technique (the Early Start Denver Model); so I do not think I am revealing any state secrets in suggesting that this group of researchers might have a vested interest in having the results of their at-risk infant studies turn out in a certain way. There is nothing necessarily sinister in this, and I certainly do not see any evidence of fraudulent results, but under these circumstances, and given the rather narrow focus of both study methodology and study results we have been seeing so far from this group, I think some healthy skepticism and a polite call for greater transparency are certainly warranted.

And I would also think the autism research community might want to reconsider the wisdom of entrusting such a new and potentially valuable line of research to such an homogeneous-minded set of researchers. While there is no easy way to eliminate conflicts of interest entirely from the autism research community, at the very least, if we had some competing interests engaged in conducting some of these studies, we might be more successful in broadening our view.

Allow me to borrow a page from these researchers' storybook and suggest that early diagnosis of their autism research disorder is actually a good thing, because it opens the door to some early intervention. The intervention need not be all that intensive in this case—I think some straightforward occupational therapy will do. For instance, we might try a few sessions where when the researchers attempt to invent novel and fuzzy measures by which to conduct their studies, they are immediately re-directed to consider measures a bit more objective, broadly scoped and possibly repeatable. When the researchers begin to perseverate on their biases, we might present them with a series of PECS cards, for instance, that demonstrate how data and information can get easily overlooked when scientists walk around with blinders on. And when the researchers insist on arranging their studies to suit only their particular interests, we might enroll them in some structured play dates—friendship classes, if you will—opportunities for these researchers to practice taking turns, sharing, playing by the rules, opportunities to experience the good feeling that comes from allowing others to express their interests too.

I am half tempted to package this form of intervention and market it under a catchy phrase—say, the Early Start Indianapolis Model. But of course I know I would never get away with such a scheme, everyone would see at once through my ruse. After all, I am only suggesting that these researchers associated with Ozonoff et al. (2010) and Christensen et al. (2010) merely follow what has actually been available to them all along, merely follow the well understood principles of logic, mathematics and science.

Ozonoff, S., Iosif, A., Baguio, F., Cook, I.C., Moore Hill, M., Hutman, T., Rogers, S.J., Rozga, A., Sangha, S., Sigman, M., Steinfeld, M.B., & Young, G.S. (2010). A Prospective Study of the Emergence of Early Behavioral Signs of Autism Journal of the American Academy of Child & Adolescent Psychiatry DOI: 10.1016/j.jaac.2009.11.009

Christensen, L., Hutman, T., Rozga, A., Young, G.S., Ozonoff, S., Rogers, S.J., Baker, B., & Sigman, M. (2010). Play and Developmental Outcomes in Infant Siblings of Children with Autism Journal of Autism and Developmental Disorders DOI: 10.1007/s10803-010-0941-y