HHS Report: Abstinence-Only Does Not Appear Effective

Man, I'll tell ya, there ought to be a job title "Research Digester." A couple of days ago it was the Heritage Foundation's faux-science report, today it's this monster from Mathematica, done for the Department of Health and Human Services (HHS). A hundred seventy eight pages of tables and numbers and graphs.

And you know what happens. Nobody reads this stuff. Then what happens is some media guys write whatever they want, knowing that nobody will double-check their work, and government guys will say whatever they were going to say anyway. So, for instance, here's how the Catholic World News put it:

A new study released on Tuesday by the Department of Health and Human Services reveals that abstinence education works. According to the interim report, teens who participated in abstinence programs had an increased awareness of the potential consequences of sexual activity before marriage, thought more highly of abstinent behaviors, and had less favorable opinions about sexual activity before marriage than did students who were not in abstinence programs. Federal study confirms abstinence education effective

and on the other hand, here's what Advocates for Youth read in it:

An objective reading of the study, conducted by Mathematica Policy Research Inc. for the Department of Health and Human Services calls into question the Bush Administration's support of abstinence-only as a national policy. Government Acting Like "Flat Earth Society," Promoting Abstinence-Only Programs

You might notice that these two summaries say opposite things. The study showed that abstinence-only programs work, or it shows that they don't work...

See what I mean? It oughta be somebody's job to read this stuff and summarize it for normal people.

This study is actually an experiment -- which is a good thing. Students were randomly assigned to participate in an abstinence program or a control group. There were four different abstinence programs, one in each of the geographical locations studied, and in each case it appears that the control group tended to take the regular health classes in their schools -- many had sex-ed training of some kind, it just wasn't the program being studied.

It is only by conducting an experiment, which is defined by random assignment to conditions, that causality can be assessed. The question here is whether the abstinence programs had any effect on subsequent attitudes and behaviors. Some measures were taken before the health classes, and some afterwards, so that change could be assessed.

I'm not going to go through this whole thing, which is generally well done (I have some quibbles with the Discussion section, but then I suppose Mathematica has a client to satisfy, don't they?). The question everybody wants to know is: do abstinence-only programs work?

The question is not answered in this study. The four cohorts interviewed here range in age from a mean of 10.3 years to 13.3. The question "Does abstinence-only education work," would require knowledge of whether the students had sexual intercourse before some criterion time, say in their teen years, or before marriage. You can't tell that by interviewing an eleven year old kid.

The substitute question, then, is whether abstinence-only education changed the students' attitudes and beliefs. And of course this form of the question has many components. The researchers report on five clusters of "intermediate variables": 1) views on abstinence, teen sex, and marriage, 2)peer influence and relations, 3)self-concept, refusal skills, and communication with parents, 4)perceived consequences of teen and nonmarital sex, and 5) expectations to refrain from sex. Some of these are attitude measures -- how the student feels about something -- and some are beliefs about how they will behave.

You are free to read the report online (LINK HERE), so let me jump to what is probably the most important variable, that is, the respondents' expectation of whether they will be abstinent through their teen years. On the questionnaire, the question was worded like this:

What is the chance you will have sexual intercourse as an unmarried teen? [Asked of non-sexually active youth only]
0 I definitely will do it
1 I might do it
2 I definitely will remain abstinent -- I will not have sexual intercourse

Do you think you will have sexual intercourse during the next year? [Asked of sexually active youth only]
0 I definitely will have sexual intercourse during the next year
0 I might have sexual intercourse during the next year
0 I definitely will not have sexual intercourse during the next year

(I hope that the zeroes for the codes on the last item are typos in the report!)

The results of this question are easy to report. There were no statistically significant differences, for any of the four programs, between experimental and control groups on the question of whether respondents expected to remain abstinent. When the two "older" groups were collapsed together, the authors report significance at the p<0.10 level, which is not usually considered "significant" in the social sciences, but the groups individually did not demonstrate significant differences. In all four cases, respondents in the abstinence-only programs reported slightly higher expectations, but in no case were these big enough to overpower the background noise.

But didn't Catholic World News say it worked? Didn't they say that teens were more likely to take abstinence vows after these classes? Well, yes they did, and the data support that. Respondents in three of the four programs pledged significantly more, at the p<0.05 level, than control group members. It is not said, but I have the suspicion that an abstinence pledge might be part of an abstinence-only class, and not part of a regular health class. So ... this is not really so surprising, is it?

One serious problem with this study has to do with confounded independent variables. Three of the four samples were made up of what the authors call "youth in high-risk communities." They were mostly black and poor, with unmarried parents. The fourth group was mostly middle-class white kids with two parents. These kinds of differences make it impossible to compare the effects of programs between groups. As each group received a different abstinence-only program, we cannot tell whether one was more effective than another, simply because program differences are confounded with demographic differences.

Effects are also obscured by the fact that the control groups were, well, uncontrolled, and not all students in the program groups actually attended the programs. Many students in the control groups took other health classes, including ones that talked about abstinence, and not all the kids in the abstinence programs did attend their assigned classes.

In three of the four programs, teens' attitudes were more supportive of abstinence and less supportive of teen sex than control group subjects. No difference was found in the level of support from friends for abstinence, or in dating and peer pressure estimates. Further, as the authors report, "Program and control group youth displayed no difference in their self-concept, refusal skills, or communication with parents." In all four programs, teens' perceptions of the potential adverse consequences of teen and nonmarital sex were increased significantly by the classes.

The authors also state that "There is limited evidence that the programs raised expectations to abstain from sex." This statement is not true, though, and would never survive a peer review process; expectations of abstinence did not differ significantly, at the p<0.05 level, between program and control groups. A p<0.10 was attained by pooling groups, though even that easy standard was not met by the programs individually.

The authors write:

There is little or no evidence that the first year of participation in these programs changed other intermediate outcomes that may be vehicles for changing behavior. These include views supportive of marriage; the extent to which youth's friends hold views supportive of abstinence; and self-concept, refusal skills, and communication with parents.

Now listen to what the government is saying (from the Catholic World News):

"Students who are in these [abstinence education] programs are recognizing that abstinence is a positive choice," HHS Assistant Secretary for Planning and Evaluation Michael O'Grady said. "Abstinence education programs that help our young people address issues of healthy relationships, self-esteem, decision-making, and effective communications are important to keeping them healthy and safe."

This is what I have been politely calling on this blog "bull-oney."

Looking back at the quotes at the top of this post, you've gotta think that the Catholic World News is reporting from the dream world, while Advocates for Youth wrapped it up pretty well: the government is acting like the Flat Earth Society, promoting programs even when their own research shows they don't work.