Wednesday, June 15, 2005

Science and Politics: Pledgers' STD Data Re-Analyzed

There has been quite a bit of hullaballoo in the past few days about a Heritage Foundation study that reanalyzed data from the National Longitudinal Study of Adolescent Health (Add Health), a comprehensive survey of 90,000 seventh through twelfth graders. An earlier report on the data set by Hannah Brückner at Yale and Peter Bearman at Columbia, published in the The Journal of Adolescent Health had found that teens who pledged to remain abstinent until marriage had the same rates of STDs as those who did not. Yesterday's New York Times:
Challenging earlier findings, two studies from the Heritage Foundation reported yesterday that young people who took virginity pledges had lower rates of acquiring sexually transmitted diseases and engaged in fewer risky sexual behaviors.

The new findings were based on the same national survey used by earlier studies and conducted by the Department of Health and Human Services. But the authors of the new study used different methods of statistical analysis from those in an earlier one that was widely publicized, making direct comparisons difficult.

Independent experts called the new findings provocative, but criticized the Heritage team's analysis as flawed and lacking the statistical evidence to back its conclusions. The new findings have not been submitted to a journal for publication, an author said. The independent experts who reviewed the study said the findings were unlikely to be published in their present form. Studies Rebut Earlier Report on Pledges of Virginity

The Times' reporting on this is a model of ... whatever. They are balanced, I guess you could say.

So the reader is left with two analyses of the same data set that come to opposite conclusions. The ordinary person is going to just shake their head, muttering something about statistics, and move on to the funnies.

I spent some time looking at this, and I admit, the key to it slipped past me. It was just a little thing, but the sleight-of-hand was good, and I was looking the wrong way.

Here -- read what this blog, "Big Monkey, Helpy Chalk," says about it: A big red flashing sign saying "This science is completely bogus". It's a good, readable discussion about "junk science" -- which the Heritage Foundation guys accuse the academics of -- and how to tell the difference.

There are a couple of keys. First of all: peer review. The first paper was sent to a legitimate scientific journal. The editor of that journal sent it around to some reviewers, generally experts in the field. They are competitive, and hate it when some other researcher finds something before they do, so they criticize the paper to death. The author gets the comments, addresses them through changes to the manuscript, re-submits it ... sometimes this goes on for several cycles. Well, usually not -- most papers are rejected in the first round. But this one survived, and finally reached publication.

The Heritage Foundation paper, on the other hand, went from the conservative think-tank directly to the press.

That's your first clue.

The NYT and Big Monkey agree that a significance level of 0.10 is too high, and it is, but that's quibbling.

The tricky part is in the way the two studies defined STDs. Here's where the data came from for the original Brückner and Bearman study:
Biomarker data (urine samples) on STD status were collected from 92% of wave 3 respondents. A total of 1183 individuals (8%) refused participation in the biospecimen collection. Urine samples were collected in the field and analyzed for the presence of three sexually transmitted diseases, Chlamydia (CH), Gonorrhea (GC), and Trichomoniasis (TR). In addition, 7000 female respondents who reported ever having had vaginal sex in wave 3 were randomly selected for Human Papilloma Virus (HPV) assays. The specific procedure and tests used are described in detail in [5]. Both males and females were tested for the three bacterial STDs regardless of their sexual activity status.

Collection procedures followed a strict protocol; samples that arrived in a condition not suitable for testing were discarded. Between 4% (CH), 5% (TR), and 10% (GC) of the samples were not tested for these reasons [5,6]. Pledgers do not differ from others with respect to whether their samples yielded results in the testing (p = 0.32 for CH, p = 0.17 for TR, p = 0.20 for CG). Pledgers did not differ from others in the extent to which they refused to provide urine
samples (p = 0.28). Data are weighted to adjust for oversampling of various groups and wave 3 nonresponse.

...

Unless otherwise noted, the analyses reported below are based on 11,471 respondents with valid data on STD status and grand sample weights. The majority of the results are derived from cross-tabulating pledge status with various outcome and behavioral measures.

...

HPV is analyzed separately because it is much more prevalent, may be transmitted by noncoital sexual behavior, and the testing was based on a different sample (n = 3317; sexually active females only). After the promise: the STD consequences of adolescent virginity pledges


I'll come right back to the STD measurement, but first...

Note that one criticism that the Heritage Foundation researchers had of the Brückner and Bearman study was, as they put it:
The centerpiece of their argument about pledgers and heightened sexual risk activity is a small group of pledgers who engaged in anal sex without vaginal sex. This "risk group" consists of 21 persons out of a sample of 14,116. Bearman and Bruckner focus on this microscopic group while deliberately failing to inform their audience of the obvious and critical fact that pledgers as a whole are substantially less likely to engage in anal sex when compared to non-pledgers. Virginity Pledgers Have Lower STD Rates and Engage in Fewer Risky Sexual Behaviors

First of all, it is not in any sense a "centerpiece" of Brückner and Bearman's paper. These sentences are tucked in the middle of a paragraph reporting a ton of statistical results:
... Similarly, 0.7% of nonpledgers report anal but no vaginal sex, compared with 1.2% for pledgers. Although too few females report anal but no vaginal sex, for males we find a significant difference between pledgers and nonpledgers (p = .021). Specifically, slightly more than 1% of male nonpledgers report anal sex but no vaginal sex, compared with almost 3% for inconsistent pledgers and 4% for consistent pledgers...

Let me reconstitute the totals here. 0.7 percent of 9,072 nonpledgers would be 63.5, call it 64 nonpledgers; 1.2 percent of 707 pledgers would be 9.3, call it nine of them. There is some ambiguity here -- if "pledgers" means all respondents who reported pledging, whether they were consistent over time or not, then that would be 0.012 percent of 2,399, or about 28.788 pledgers who had anal and not vaginal sex. So we're talking about 93 people, not 21. Whatever, the difference between 1 and 4 percent is significant in this sample, it doesn't matter that the percentages were small. They are certainly higher than the percentage of people who die from head injuries in automobile crashes, but we accept the conclusion that seatbelts significantly reduce that percentage.

They are counting on you not doing what I'm doing, that is, finding the original paper and doing the arithmetic.

We have read how Brückner and Bearman defined STDs in their paper. Now listen to the Heritage Foundation guys describe it:
One problem is that Bearman and Bruckner examined only one of several STD measures available in the Add Health data file. Analysis of the remaining measures reveals that adolescent virginity pledging is strongly associated with reduced STDs among young adults. These results are statistically significant in four of the five STD measures examined and are very near significance on the fifth measure. With all the STD measures, the allegedly ineffective virginity pledge is actually a better predictor of STD reduction than is condom use. Virginity Pledgers Have Lower STD Rates and Engage in Fewer Risky Sexual Behaviors

They make it sound like Brückner and Bearman ignored the other STD data. In reality, their Table 4 is just that. And it shows just what both groups say it shows: people who have pledged to be abstinent report having fewer STDs, and they report going to the doctor fewer times to be checked for STDs.

The Heritage Foundation reports cover up the differences between their data and the academic data, and as a smokescreen they assert that Brückner and Bearman looked at "only one of several STD measures available in the Add Health data file."

Brückner and Bearman analyzed real data from real urine tests. People peed in the cup and it went to a lab. It didn't matter what the respondent said, the laboratory measured whether there were signs of infection in the urine. The Heritage Foundation guys relied only on what the respondent said. According to Brückner and Bearman:
Add Health asked respondents also about their experiences with STDs. Specifically, all respondents were asked whether they had been diagnosed with various STDs in the past year; whether they had ever seen a doctor because they were worried about having a STD; and whether they had been tested for various STDs in the past year.

And those are the other STD variables that the Heritage guys like so much. No sensible person is going to think that someone's answers to those questions are more accurate than the lab results.

In sum, the Heritage Foundation papers are political, not scientific. They do not meet any of the standards of scientific literature. They are not peer-reviewed, they contain scurrilous ad hominem comments about other researchers, they conceal the nature of the data they used while implying that the other researchers' methods were inadequate. The Heritage Foundation wants abstinence pledges to succeed because it fits their ideology, and they will twist any data in whatever way is necessary to prove their point.

It's a sad statement, that science is under attack in this way, that any nut can put a paper on the Internet and call the newspapers, and people will accept it as if it were the real thing.

7 Comments:

Blogger andrear said...

MY first impression was that if the original paper was published in a peer reviewed journal and was by university researchers and the second paper was released to the press and done by the Heritage Foundation- the science(if there was any in the second paper) was not likely to be very good. Thank you, Jim, for showing why the Heritage "science" was indeed junk science.

June 15, 2005 7:27 PM  
Anonymous Anonymous said...

http://tinyurl.com/7nu44

Here you can read all about this summer's Heritage Foundation internships where they train people to produce junk science to support their far right political views on a variety of issues. "'If we can get young people involved, they will continue to support Heritage, our idea and our causes,' Mr. Feulner said." Aha! This is evidence that they are trying to recruit our children away from their families and into the Heritage Foundation way of thinking!

And you can be sure that Heritage Foundation trained junk researchers around the country and their supporters will be citing this Heritage Foundation produced junk report every chance they get.

Aunt Bea

June 16, 2005 10:00 AM  
Anonymous mike said...

Boy, you people are have so much venom. Of course, if it was a liberal think tank, the information would be taken as gospel truth. As it is, you describe the highly respected Heritage Foudation as having 'junk researchers', using 'junk science'. Your bigotry and hatred of anyone with conservative views is so evident. And the funny thing is, you do not even try to hide your contempt. You're not fooling everybody.

June 16, 2005 2:28 PM  
Blogger JimK said...

Mike
The Heritage Foundation is not "highly respected" by academics and social scientists. It is a political institution that passes its work off as scientific research. In this study, they fail to even tell the reader what variables they used, and falsely accuse other -- actual -- researchers of "junk science." They actually call it that. Their dependence on self-report data when lab results are available is, in itself, unconscionable, especially in this kind of situation where the social-desirability pressures on respondents are very salient.

There is a certain kind of person -- this would be you, Mike -- who sees all discourse in terms of self-interested individuals defending their own points of view. I must be a "liberal," and I must criticize all conservative writings. No, intelligent and honest conservatives present a perspective that is essential in keeping the national debate on track -- there is always another point of view, and it is important to hear them both. This is not that: this is simply partisan prestidigitation, politics repackaged as research and marketed to the press.

June 16, 2005 2:57 PM  
Blogger andrear said...

Politics-liberal or conservative - passed off as science is contemptible.

Why would someone suggest that the Heritage Foundation "science" is any good when even they admit that it was sent directly to the media.
There are books on bad science and one of the guidelines to determining if it is bad- is that it is sent directly to the main stream media instead of peer reviewed journals.

June 17, 2005 11:16 AM  
Anonymous Anonymous said...

accessorizing your fit in the poriferan becomes a facts of life put for germs.

or else, use fair fingertips for effort, shift fingers or cleaning
them between other products and services.

informed your fund period of play rhythmicity.
You gift fall behind the luck on bighearted your nestling
gets delayed or conveyed location from a company, Moncler Bodywarmer Bestellen alternatively of opting for jet piece.

Choosing a brown quality is much easier for hackers to get started in picture create from raw stuff.
These videos intention bring in your can. in
front qualification a purchase, inflect prices on several websites.
convey for the instance to denounce online at a ripe

My web page: Moncler Acorus Here

December 12, 2013 9:04 PM  
Anonymous Anonymous said...

Usually a corporate production company contains a wealth of experienced directors, producers
and editors who have all worked for television companies before.
Cam - Live Reside Sex CAM is the very best top quality live webcam accessible.
This is the person, who will always understand you, be there for you, and fill all that inner emptiness you have been unable to fulfill.


Here is my web site :: free live cam

April 09, 2014 7:39 AM  

Post a Comment

<< Home