Influential Errors | the Diet Heart Tale

Earlier this year, my colleagues and I were discussing the relationship between saturated fat and cardiovascular disease. One of us was writing an article on the topic and we were discussing an unusual trial often included in meta-analyses.

That trial is the Finnish Mental Hospital Study, a crossover study that compared patients on a control diet with a certain amount of saturated fat to patients on an intervention diet that replaced the saturated fat with polyunsaturated fats.

Here is a summary of the trial,

“A controlled intervention trial, with the purpose of testing the hypothesis that the incidence of coronary heart disease (CHD) could be decreased by the use of serum-cholesterol-lowering (SCL) diet, was carried out in 2 mental hospitals near Helsinki in 1959–71.

The subjects were hospitalized middle-aged men. One of the hospitals received the SCL diet, i.e. a diet low in saturated fats and cholesterol and relatively high in polyunsaturated fats, while the other served as the control with a normal hospital diet. Six years later the diets were reversed, and the trial was continued another 6 years.”

The study didn’t just include men, it also included women and is discussed in a separate paper by the same research group.

In total, the “two studies” (really just one study) had a sample size of 818 participants (for hard CVD events), so they often weigh quite a bit in meta-analyses.

I’d like to bring attention to one particular meta-analysis published eight years ago by Mozaffarian, Micha, & Wallace, 2010. It’s one of the most cited meta-analyses on this topic, with Google Scholar indicating that it’s been cited by over 900+ academic sources. Web of Science indicates that it’s been cited by 466 papers at the time of writing this post.

Web of Science metrics showing the number of times the Mozaffarian meta-analysis has been cited

Source: Web of Science

Clearly, it’s a well known study.

The meta-analysis of interest describes its inclusion and exclusion criteria as,

“We searched for all RCTs that randomized adults to increased total or n-6 PUFA consumption for at least 1 year without other major concomitant interventions (e.g., blood pressure or smoking control, other multiple dietary interventions, etc.), had an appropriate control group without this dietary intervention, and reported (or had obtainable from the authors) sufficient data to calculate risk estimates with standard errors for effects on occurrence of “hard” CHD events (myocardial infarction, CHD death, and/or sudden death). Studies were excluded if they were observational or otherwise nonrandomized;.”

So the authors state that the included studies must be randomized trials that are at least a year long and that they are excluding studies that are non-randomized or observational.

Here’s a list of the studies they included. Note the design of the Finnish studies (Turpeinen, 1979 & Miettinen, 1983), which I’ll touch upon below.

Characteristics of the included studies in the meta-analysis

What Were the Results?

Forest plot from the Mozaffarian meta-analysis showing a reduction in CVD events by replacing saturated fat intake with polyunsaturated fat intake

Mozaffarian D, Micha R, Wallace S (2010)

“Combining all trials, the pooled risk reduction for CHD events was 19% (RR = 0.81, 95% CI 0.70–0.95, p = 0.008)”

The 2010 meta-analysis found that replacing saturated fats in the diet with polyunsaturated fats had a notable, statistically significant reduction on CHD events. A 19% reduction is certainly, nothing to ignore and the confidence interval (CI) leans towards an effect. It seems promising as a dietary intervention. That could be a reason why the study is cited so widely. The quality of the trials included in the meta-analysis was low to moderate,

“Many of the trials had design limitations, such as single-blinding, inclusion of electrocardiographically defined clinical endpoints, or open enrollment. All trials utilized blinded endpoint assessment. Quality scores were in the modest range and relatively homogeneous: all trials had quality scores of either 2 or 3.”

And there was some suggestion of publication bias (could also be small-study effects),

Characteristics of the included studies in the meta-analysis

Mozaffarian D, Micha R, Wallace S (2010)

“Visual inspection of the resulting funnel plot indicated some potential for publication bias (Figure S1), with a borderline Begg’s test (continuity corrected p = 0.07), although such determinations are limited when the number of studies is relatively small.”

Regardless, the effects are quite interesting and worth exploring further.

What Went Wrong?

A major problem in this meta-analysis is that the two Finnish studies included in the quantitative analysis were not randomized. The authors made it clear with their inclusion criteria that they only wanted to include trials that were randomized.

The two Finnish Mental Hospital studies were labeled as “cluster randomized”, which you can see in the table of characteristics from above. When this meta-analysis was published, several individuals were critical that a “cluster-randomized trial” was being labeled as a randomized trial, especially when there were only two clusters (two hospitals). This is a valid criticism because a cluster-randomized trial with only one cluster per condition is invalid for any between-group statistical comparisons. Brown et al., 2015 explain in this comprehensive article,

A particularly pernicious and invalid design that requires recognition is the inclusion of only one cluster per condition… Such designs are unable to support any valid analysis for an intervention effect, absent strong and untestable assumptions (11, 12). In such designs, the variation that is due to the cluster is not identifiable apart from the variation due to the condition.

A one-cluster-per-condition design is analogous to assigning one person to the treatment and one person to the control in an ordinary (nonclustered) RCT, measuring each person’s outcome multiple times, treating the multiple observations per person like independent observations, and interpreting the results like a valid RCT. In such a situation, the observations on person A can be tested as to whether they are significantly different from those on person B but cannot support an inference about the effect of treatment per se.

So it is clear that a one-cluster-per-condition design is not valid to ascertain much about the intervention. However, many individuals (if not all) failed to notice that the Finnish Mental Hospital studies were not even cluster randomized! There is no indication in any of the five published papers from these two studies that there is any randomization. You can check all five papers here:

Journal Year Title
International Journal of Epidemiology 1983 Dietary Prevention of Coronary Heart Disease in Women: The Finnish Mental Hospital Study
Circulation 1979 Effect of Cholesterol-Lowering Diet on Mortality from Coronary Heart Disease and Other Causes
American Journal of Clinical Nutrition 1968 Dietary Prevention of Coronary Heart Disease: Long-Term Experiment: I. Observations on Male Subjects
International Journal of Epidemiology 1979 Dietary Prevention of Coronary Heart Disease: The Finnish Mental Hospital Study
The Lancet 1972 Effect of Cholesterol-Lowering Diet on Mortality from Coronary Heart-Disease and Other Causes a Twelve-Year Clinical Trial in Men and Women

Furthermore, cluster-randomized trials were not common when these studies were being conducted, which is why we should be skeptical of these being cluster-randomized trials.

Yet, these two studies were mistakenly labeled as being “cluster randomized” and therefore were included in the meta-analysis. Both of these studies contributed a total weight of 16% to the analysis.

And again, the authors found a pretty notable reduction in CVD events (RR: 0.81, 95% CI 0.70 – 0.95, p = 0.008)

Correcting the Error

So what happens to the results when you correct this mistake by removing the two studies?

Let’s open R and find out. If you’d like to reproduce the analysis on your own, you can find all the code at the bottom of this blog post. All the materials are also hosted on this repository.

My reanalysis of Mozaffarian's study with the exclusion of the Finnish study graphed onto a forest plot, showing a substantial reduction in the effect size

My reanalysis

As you can see above, rerunning the analysis after removing the Finnish studies results in the effect size shrinking from a 19% reduction to a 13% reduction (RR: 0.87, 95% CI 0.76 - 1.00). That’s a large difference!

If we’re concerned about statistical significance, the results are no longer significant. It’s worth noting that the upper bound of the confidence interval barely contains the null value (1) and the lower bound includes a value as low as 0.76. It’s clear that the CI still seems to lean towards an effect.

Regardless of your statistical philosophy, this was a noteworthy, objective mistake. It was a mistake in labeling two studies as meeting the inclusion criteria, and correcting for this mistake leads to a substantial change in the results. Yet, this error has not been corrected for in the journal. In fact, this study has been around for eight years with no corrections or retractions.

I reached out to both the authors and the editors of PLoS, but to date, there are no updates or corrections on the article itself. Therefore, I suspect several people who read the article or cite it, are not aware that the summary effects are incorrect and that some of the studies in the analysis should not be there!

It is very important to note that correcting for the errors in this study does not lead to completely different conclusions. Although the effect is no longer statistically significant, it is still there based on the effect size and coverage of the confidence intervals. However, the effect is reduced.

Systematic reviews by other groups including Cochrane did not include the Finnish studies in their meta-analyses because the authors didn’t believe that a “cluster randomized trial” with so few clusters (2) met the inclusion criteria for a randomized trial (also worth remembering, that there is no indication in any of the papers that this was even cluster randomized!). Some of these systematic reviews that exclude the Finnish studies still find a benefit to replacing saturated fats in the diet with polyunsaturated fats.

However, other meta-analyses have also found no statistically significant benefit to replacing saturated fats with polyunsaturated fats.

Clearly, there is quite a bit of disagreement on this topic. Regardless, the meta-analysis in question still made a large error and it is a problem for the following reasons (even if the overall conclusions of it were not to change after the correction):

  • Two prominent studies were misclassified
  • The studies did not meet the inclusion criteria but were included
  • Inclusion in the analysis leads to a substantially different effect size than without inclusion
  • The meta-analysis is constantly cited and misleading other readers

I’m certainly not suggesting that errors do not happen, especially when undertaking such large, comprehensive projects. In fact, I would probably be suspicious if there were never any errors when such large projects were conducted!

However, I believe that when such errors are pointed out, they should be corrected as quickly and transparently as possible. Hopefully, the authors and the editors address this issue soon to prevent any further confusion.

Thanks for reading. If you have any thoughts, I’d love to hear them below!

Reanalysis Script

Here is the dataset to load into R as a dataframe.

pufaMETA<-read.csv("PUFA.csv", header=TRUE)

## Meta-Analysis
dat <- escalc(measure="RR", ai=P_Events, n1i=P_Total,
ci=C_Events, n2i=C_Total, data=pufaMETA)
res <- rma(yi, vi, data=dat)


res <- rma(ai=P_Events, n1i=P_Total, ci=C_Events, 
        n2i=C_Total, data=dat, measure="RR",
           slab=paste(Study, sep=", "), method="DL")

## Data Visualization - Forest Plot Structure

forest(res, xlim=c(-16, 6), at=log(c(0.05, 0.25, 1, 2)), atransf=exp,
       ilab=cbind(dat$P_Events, dat$P_Total, dat$C_Events, dat$C_Total),
       ilab.xpos=c(-9.5,-8,-6,-4.5), cex=0.85, ylim=c(-1, 8.6),
       xlab="Risk Ratio", mlab="", psize=1.7, pch=16)

## Heterogeneity 

text(-16, -1, pos=4, cex=0.85, bquote(paste("Random Effects (Q = ",
   .(formatC(res$QE, digits=2, format="f")), ", df = ", .(res$k - res$p),
    ", p = ", .(formatC(res$QEp, digits=2, format="f")), "; ", I^2, " = ",
     .(formatC(res$I2, digits=1, format="f")), "%)")))

op <- par(cex=0.85, font=4)
### Bold Font

### Columns
text(c(-9.5,-8,-6,-4.5), 7.5, c("Events ", " Total", "Events ", " Total"))
text(c(-8.75,-5.25),     8.5, c("PUFA Diet", "Control Diet"))
text(-16,                7.5, "Study Name",  pos=4)
text(6,                  7.5, "Risk Ratio [95% CI]", pos=2)

Edit: Dr. Mozaffarian has replied to some of these criticisms in the comments of his paper and I have also responded below that.

See also:

comments powered by Disqus