November 2018

Results:

Quick Characteristics of Participants

• Median Age: 64 years old

• Females: 28.8%

• Median LDL: 75.0 mg/dL

• Median HDL: 40 mg/dL

• Median Triglycerides: 216.0 mg/dL

Changes in Lipids:

The median change in triglycerides from baseline to 1 year was a decrease of 18.3% (−39.0 mg per deciliter) for the Vascepa group and an increase of 2.2% (4.5 mg per deciliter).

A between-group comparison of change scores shows that the median group reduction was 19.7% higher (44.5 mg/dL, p<0.001) in the Vascepa group than it was in the placebo group.

LDL increased in both groups from baseline to year 1, however, the Vascepa group had a 6.6% (5.0 mg per deciliter, p<0.001) lower increase than the placebo group.

Differences in Clinical Endpoints (what we really care about):

A total of 1606 participants experienced one of the primary end-point events.

Approximately 17.2% of the participants in the Vascepa group experienced a primary end-point whereas 22.2% of the participants in the control group experienced a primary end-point.

Thus, the between-group difference is 4.8% (95% CI, 3.1 to 6.5). The results are: (hazard ratio, 0.75; 95% confidence interval [CI], 0.68 to 0.83; P<0.001).

So very clearly we can see that there was a statistically significant difference and the confidence interval leans towards an effect.

The authors calculated a number needed to treat for this between-group difference (NNT: 21 (95% CI, 15 to 33)) and its confidence interval which slightly makes me laugh. I’ve written about the NNT before and its flaws and the reality is that it doesn’t do what it claims to do and has very bizarre statistical properties.

With regards to secondary efficacy end-point events, 11.2% of the participants in the Vascepa group experienced an event compared to 14.8% of the participants in the placebo group.

The absolute between-group difference was 3.6% (95% CI, 2.1 to 5.0).

Again, very clearly a difference.

They also ran prespecified subgroup analyses to see whether the effects would be influenced. What was interesting to me was when they claimed this:

“Baseline triglyceride levels (≥150 vs. <150 mg per deciliter or ≥200 or <200 mg per deciliter) had no influence on the primary or key secondary efficacy end points (Figures 2 and Figure 3).”

This is interesting to me because there were so few patients that were enrolled into the trial with triglyceride levels that were beneath 150 or hovering around 150. This is very clearly reflected by the wide confidence intervals in that subgroup which can be seen in the figure above. This is because the protocol changed early in the trial to restrict participant inclusion to people with higher triglycerides,

“The first protocol amendment in May 2013 changed the lower limit of the acceptable triglyceride level from 150 mg per deciliter to 200 mg per deciliter (2.26 mmol per liter), with no allowance for variability.”

Whereas originally, it was,

“Eligible patients had a fasting triglyceride level of 150 to 499 mg per deciliter (1.69 to 5.63 mmol per liter)… because of the intraindividual variability of triglyceride levels, the initial protocol allowed for a 10% lower triglyceride level from the target lower limit, which permitted patients to be enrolled if they had a triglyceride level of at least 135 mg per deciliter (1.52 mmol per liter).”

Seems like more data is needed to say anything about this with confidence.

Hierarchical Tests

For their prespecified hierarchical tests, this is what they found:

• Rates of Composite Ischemic Endpoints + Rates of Cardiovascular Death: (Vascepa: 4.3% vs. Placebo: 5.2%; hazard ratio, 0.80; 95% CI, 0.66 to 0.98; P=0.03)

• Rate of Death From Any Cause: Vascepa: 6.7% vs. Placebo: 7.6% (hazard ratio, 0.87; 95% CI, 0.74 to 1.02).

• Doesn’t seem to be statistically significant, but the coverage of the intervals leans towards an effect.

• Rates of Adjudicated Sudden Cardiac Death: Vascepa: 1.5% vs. Placebo: 2.1% (hazard ratio, 0.69; 95% CI, 0.50 to 0.96)

• Rates of Cardiac Arrest: Vascepa: 0.5% vs. Placebo: 1.0% (hazard ratio, 0.52; 95% CI, 0.31 to 0.86)

So most seem to favor the Vascepa group.

Safety

What about its safety? A look at the supplementary appendix gives us this:

The analyses failed to show any statistically significant differences for serious treatment-emergent adverse events.

However, among other treatment-emergent adverse events, Peripheral Edema, Constipation, and Atrial Fibrillation were significantly more frequent in the Vascepa group.

There you have it.

Conclusions

The results of this large trial clearly suggest that Vascepa had a notable effect on triglycerides and clinical endpoints. These data along with the results from the JELIS trial should be considered as supporting evidence for the hypothesis that high doses of EPA (not your over-the-counter normal fish oil) may have an effect on cardiovascular disease events (even though JELIS was an open-label study).

Hopefully, future meta-analyses make high doses of EPA an inclusion criteria to avoid pooling null results from trials using smaller dosages, where it may be more difficult to see an effect. Currently, there are a few other ongoing trials that are also using high doses of EPA:

“Ongoing trials of moderate-to-high doses of pure EPA ethyl ester will provide further information on the effects of these agents. These trials include the Randomized Trial for Evaluation in Secondary Prevention Efficacy of Combination Therapy–Statin and EPA (RESPECT-EPA; UMIN Clinical Trials Registry number, UMIN000012069), a secondary prevention outcomes trial involving statin-treated patients in Japan, and the Effect of Vascepa on Improving Coronary Atherosclerosis in People with High Triglycerides Taking Statin Therapy (EVAPORATE; ClinicalTrials.gov number, NCT02926027), which is examining changes in coronary plaque over 9 to 18 months.”

Even though REDUCE-IT has been successful, it is only one trial, and it’s always worth remembering what the history of fish is, even though this trial didn’t look at fish oil but rather a pharmaceutical formulation of EPA.

A Brief Timeline of Fish Oil

• 1970s | A group of explorers noticed that people who consumed fish also had lower incidences of heart disease.

• 1980s-1990s | Early observational studies also saw a connection between fish consumption and heart disease risk

• 1990s | A few randomized trials were conducted to see the effects of fish oil supplementation and CVD outcomes

• 2002 | The American Heart Association put out a statement suggesting that fish oil supplementation was beneficial for fatal outcomes

• 2002 - 2016 | Several randomized trials and systematic reviews found inconsistent evidence for the effects of fish oil supplementation and CVD outcomes

• 2016 | The American Heart Association creates a scientific advisory group to look at all the evidence and concludes that despite the controversy, there was a small benefit from fish oil and it could be worth it for people who had a history of CVD events.

• 2018 | Large meta-analysis published in JAMA with 77,000 + participants found no benefit from fish oil supplementation on real clinical endpoints.

• Mid - 2018 | Most comprehensive meta-analysis and systematic review with 112,000 participants by Cochrane finds very small benefits to several CVD-related outcomes.

• Late 2018 | Announcement by the manufacturer of Vascepa exclaiming that high-doses of fish oil in the REDUCE-IT trial had a large effect (25% reduction) on CVD outcomes.

Edit: It came to my attention after writing this blog post that the placebo formulation included large amounts of mineral oil. Thus, the control arm was taking mineral oil + statins, while the intervention group was taking Vascepa + statins. This is problematic because mineral oil is a known laxative and could possibly interfere with the absorption of statins. Some data to suggest this is the change in LDL within the control arm. LDL increased after the trial, which doesn’t make much sense given that they were taking statins.

We have sufficient evidence from the literature to suggest that statins should lower LDL. It is also worth pointing out that side effects like diarrhea were far more prevalent within the control group in a 7,000+ clinical trial. Thus, I suspect that the control group may have been given enough mineral oil that it possibly interfered with the absorption of statins within the gut, thus reducing their efficacy, and making the difference between the Vascepa group and control group even more substantial.

Amarin, the manufacturer of Vascepa, has responded to some of these criticisms here.

Torcetrapib

A great example of this is the story of high-density lipoprotein (HDL) and myocardial infarction (heart attacks). Several studies had found associations between low levels of HDL, often considered “good cholesterol,” and heart attacks.^{Assmann1996-jd?,Curb2004-ee?,Gordon1977-le?,Gordon1989-mc?,Rahilly-Tierney2011-ox?,Sharrett2001-su?,Turner1998-gv?}

So, it shouldn’t come as a surprise that a drug (torcetrapib) was produced by Pfizer that attempted to increase the amount of HDL with the hopes that it would reduce the number of CVD events.

A group of researchers administered the drug to thousands of patients.¹ The drug was successful in changing the lipids of the participants to more favorable numbers. Patients who received the drug had a 24.9% decrease in LDL and a 72.1% increase in HDL. Seems pretty great. However, the number of deaths increased by 58% and the number of heart attacks increased by 21%.

A systematic review and meta-analysis^{Briel2009-yo?} published in the BMJ a few years later concluded the following after pooling studies that focused on interventions that primarily increased HDL and interventions that primarily decreased LDL,

“Available data suggest that simply increasing the amount of circulating high-density lipoprotein cholesterol does not reduce the risk of coronary heart disease events, coronary heart disease deaths, or total deaths. The results support reduction in low-density lipoprotein cholesterol as the primary goal for lipid modifying interventions.”

Svensson^{Svensson2013-vq?} provides us with a very lovely table showing other scenarios for which a drug had a favorable effect on a surrogate marker but had a negative impact on the clinical outcome.

What does this tell us? That focusing only on improving surrogate markers is not a very good idea if there is not much concordance between trials that focus on surrogates and trials that focus on hard endpoints. Until we’ve established a robust causal link between a surrogate marker and a hard endpoint, we shouldn’t be seduced by surrogates.

A Brief Timeline of Fish Oil

• 1970s | A group of explorers noticed that people who consumed fish also had lower incidences of heart disease.

• 1980s-1990s | Early observational studies also saw a connection between fish consumption and heart disease risk

• 1990s | A few randomized trials were conducted to see the effects of fish oil supplementation and CVD outcomes

• 2002 | The American Heart Association put out a statement suggesting that fish oil supplementation was beneficial for fatal outcomes

• 2002 - 2016 | Several randomized trials and systematic reviews found inconsistent evidence for the effects of fish oil supplementation and CVD outcomes

• 2016 | The American Heart Association creates a scientific advisory group to look at all the evidence and concludes that despite the controversy, there was a small benefit from fish oil and it could be worth it for people who had a history of CVD events.

• 2018 | Large meta-analysis published in JAMA with 77,000 + participants found no benefit from fish oil supplementation on real clinical endpoints.

• Mid - 2018 | Most comprehensive meta-analysis and systematic review with 112,000 participants by Cochrane finds very small benefits to several CVD-related outcomes.

• Late 2018 | Announcement by the manufacturer of Vascepa exclaiming that high-doses of fish oil in the REDUCE-IT trial had a large effect (25% reduction) on CVD outcomes.

Fish oil was accompanied by a lot of hype in the early days and little evidence, but eventually, like most things, it took a beating once several research groups started prodding the existing evidence.

The results of this trial (when they are presented in November at the AHA conference) and when they are published in a peer-reviewed journal, may potentially change the minds of skeptics everywhere.

That’s why I thought it would be a good time to review the preregistration protocol of the REDUCE-IT trial on ClinicalTrials.gov and a detailed paper that describes the rationale behind the study design. Let’s get into it.

So very clearly, the authors state what the study here is doing,

“AMR101 (icosapent ethyl [ethyl-EPA]) is a highly purified ethyl ester of eicosapentaenoic acid (EPA) being developed by Amarin Pharma Inc. for the treatment of hypertriglyceridemia.

The purpose of this study is to evaluate whether this drug, combined with a statin therapy, will be superior to the statin therapy alone, when used as a prevention in reducing long-term cardiovascular events in high-risk patients with mixed dyslipidemia.”

Comparing statins + highly purified, high doses of EPA to statins alone. Why are they using such high doses? The authors explain here,

“It is worth noting that the promising results from JELIS occurred with a high‐purity EPA preparation dosed at 1.8 g/d in a Japanese population, for whom the baseline EPA levels are higher than in western populations due to greater dietary intake of marine omega‐3 fatty acids.

Icosapent ethyl 12‐week dosing at 4 g/d in a high‐risk population similar to that within the Reduction of Cardiovascular Events with Icosapent Ethyl–Intervention Trial (REDUCE‐IT) who had persistent elevations of TG after treatment with statins resulted in significant reductions in TG and atherogenic lipoproteins, as well as comparable plasma EPA levels as the 1.8 g/d dosing group in JELIS.

Therefore, a dose of 4 g/d was selected as the dose for further study. In this context, REDUCE‐IT was designed to determine if treatment with icosapent ethyl 4 g/d vs placebo would reduce ischemic events in patients at increased CV risk already being treated with statins.”

Also “worth noting” that the JELIS trial was included in both meta-analyses mentioned at the beginning of the blog post.

The primary outcome is a composite endpoint that consists of CV death, MI, stroke, coronary revascularization, and hospitalization for unstable angina. Whenever I hear composite endpoint, I often become skeptical because I’ve seen composite endpoints abused or misused in several ways, resulting in very misleading results. There was a very nice post on composite endpoints by Hilda Bastian here.

The study (REDUCE-IT) enrolled 8,000 participants and began in 2011 and ended in 2018.

Why 8,000 participants?

The authors explain here,

“The sample‐size calculation was based on a hazard ratio assumption of 0.85. Therefore, 1612 events would be required to have approximately 90% power with a 1‐sided α‐level of 2.5% and with 2 interim analyses. This results in a total target sample size of 7990 patients.

Approximately 70% of randomized patients were to be in CV risk stratum 1 (established CVD) and approximately 30% of randomized patients were to be in CV risk stratum 2 (high‐risk primary prevention defined by diabetes mellitus and other risk factors). Randomization was stratified by CV risk strata, ezetimibe use, and by geographical region.”

The design power calculations seem like they were based on very optimistic expectations of the effect size, and apparently, justifiably so!

A bit unfortunate that there is no mention anywhere of what statistical analyses they planned on doing exactly.

In order for participants to be eligible for the trial:

• they had to be over the age of 45

• have high levels of triglycerides (hypertriglyceridemia)

• been on statins for at least a month

• they had to have a history of heart disease

There’s a bunch of exclusion criteria too that I don’t really feel the need to go over, but you can see them listed out on the registry.

Also, worth mentioning of course that the trial is funded by Amarin, AKA the manufacturer of Vascepa.

Even though my title mentions “fish oil” it really is important to point out that this is NOT the average fish oil supplement that you’ll get over the counter. It’s a highly purified substance, that’s currently being prescribed by doctors in high dosages for hypertriglyceridemia. It is a drug.

An extremely expensive one ($250+ for a 30 day supply… ) that most insurance companies will not cover unless your blood tests show an indication of high triglycerides.

So, while the substance technically is fish oil, a very purified, high-dose version of it, the results of the REDUCE-IT trial are not evidence of your average fish oil supplement having these same effects.

I look forward to hearing about the results of this study in November.

Edit: I’ve covered that study in this blog post.

A Summary of the JAMA Study

Last week, a nutritional epidemiological study (from now on nutri epi) published in JAMA posited that egg consumption was associated with an increased risk of cardiovascular disease incidence.¹ The results gained significant coverage from the media.

With that coverage came the usual criticism of “correlation not being causation,” and how nutri epi studies produce findings that are often contradicted by randomized trials, etc. Unlike many of these surface-level criticisms, I want to dive a little deeper into the findings of this study and the utility and limitations of studies like this. But first, a quick summary of the study.

Who’s Being Studied?

This study included data from six prospective cohorts and had a total sample size of 29,615 adults. That’s a lot of people. Those who follow nutrition studies should be familiar with many of the cohorts included such as:

• the Atherosclerosis Risk in Communities (ARIC) Study
• the Coronary Artery Risk Development in Young Adults (CARDIA) Study
• the Framingham Heart Study (FHS)
• the Framingham Offspring Study (FOS)
• the Jackson Heart Study (JHS)
• the Multi-Ethnic Study of Atherosclerosis (MESA)

Data collection started in early 1985 and halted in the summer of 2016 (nearly 31 years of data). The project was involved in answering the question of whether dietary cholesterol was associated with cardiovascular disease incidence.

What and How Were the Data Collected?

So how were dietary data collected? The authors give a very brief description in the methods,

Using a standardized protocol and data dictionary, diet data were harmonized cohort by cohort. Briefly, consumption frequencies were converted into estimated number per day using the middle value (eg, 3-4 times per week = 0.5 times per day).

However, a look at the supplementary files gives us more insight into how the data were collected:

So they mainly used various food frequency questionnaires which differed depending on the cohort.

The primary outcomes were…

The primary outcomes were incident CVD and all-cause mortality. Incident CVD was a composite end point of fatal and nonfatal coronary heart disease (0), stroke, heart failure, and CVD death from other causes.

Here are some of the covariates they collected data for

Standardized questionnaires and laboratory protocols were used to collect information on the following variables: age, sex, race/ethnicity, education, lifestyle factors (including smoking, alcohol intake, and physical activity), body mass index (BMI, calculated as weight in kilograms divided by height in meters squared), blood pressure, lipid profile, medication use, and medical conditions.

and what they adjusted for in their three statistical models. Let’s focus mainly on model 2 since they use that as a reference.

Models were sequentially adjusted for age, sex, race/ethnicity (white, black, Hispanic, Chinese), education (<high school, high school, some college or more) (model 1); plus total energy, smoking status (current, former, never), smoking pack-years (0, 0.1-4.9, 5.0-9.9, 10-19.9, 20-29.9, 30-39.9, ≥40), cohort-specific physical activity z score, alcohol intake (gram), use of hormone therapy (y/n) (model 2);

What Were the Results?

The authors graphed the hazard ratios by the amount of dietary cholesterol consumed per day (mg) for CVD incidence…

So what were the main results that the authors reported?

Based on model 2, each additional 300 mg of dietary cholesterol consumed per day was significantly associated with higher risk of incident CVD (adjusted HR, 1.17 [95% CI, 1.09-1.26]; adjusted ARD, 3.24% [95% CI, 1.39%-5.08%]; Figure 2A) and all-cause mortality (adjusted HR, 1.18 [95% CI, 1.10-1.26]; adjusted ARD, 4.43% [95% CI, 2.51%-6.36%]; Figure 2B).

So a 17% increase in the risk of incident CVD where a 9% increase all the way up to a 26% increase were reasonably compatible with the data and its assumptions. The results are practically the same for all-cause mortality.

The authors also adjusted for individual foods or combinations of foods to see how the effect deviated with model 2 being used as the reference.

• For incident CVD (shown above), the effect estimate increased from 17% to 20% when adjusted for eggs alone, and decreased to 13% when adjusted for eggs, unprocessed red meat, and processed meat.

• For all-cause mortality (graph not shown), the effect estimate dropped from a 17% increase to a 13% increase when adjusted for eggs alone and dropped to a 5% increase when adjusted for eggs, unprocessed red meat, and processed meat.

Here’s what the authors concluded:

The current study found that the significant associations of dietary cholesterol consumption with incident CVD and all-cause mortality were independent of fat amount and quality of the diet. Further, the significant association between egg consumption and incident CVD was fully accounted for by the cholesterol content in eggs…

Mechanistically, eggs and processed or unprocessed red meat are rich in other nutrients such as choline, iron, carnitine, and added sodium (for processed meat) that have been implicated in CVD risk via different pathways.

And then they discuss the limitations of their study:

First, appropriate interpretation of the study findings requires consideration of measurement error for self-reported diet data. Further, this study relied on single measurement of egg and dietary cholesterol consumption. Exposure misclassification may be of concern, but results were similar when censoring participants at different time points…

…Fifth, generalizing our results to non-US populations requires caution due to different nutrition and food environments and chronic disease epidemiology. Sixth, the study findings are observational and cannot establish causality.

Interestingly enough, right after saying the results cannot establish causality, they write this,

Among US adults, higher consumption of dietary cholesterol or eggs was significantly associated with higher risk of incident CVD and all-cause mortality in a dose-response manner. These results should be considered in the development of dietary guidelines and updates.

Well, I have a lot to say, but let’s first take a look at how the media interpreted and reported these results. For that, we shall consult the Google oracle.

How Did the Media Respond?

This is one of my favorite ones.

Okay, now that we’ve done that, we can discuss the measurements and analyses.

How Reliable Are the Measurements and Analyses?

As the authors mentioned in the methods, and supplementary files, most of the dietary data were collected via a food frequency questionnaire (FFQ). This is important for a couple of reasons, and I’m going to dive a little deeper than the usual “FFQ data are useless”.

FFQ data are known to suffer from serious measurement error, and there is a giant body of literature on this. For example, such data can be prone to classical measurement error (), systematic error, heteroscedastic error, and differential error.

Methods like regression calibration, moment reconstruction, multiple imputation, and graphical methods can be used to help address many of these errors, along with sensitivity analyses. For a more detailed read of this, see Keogh & White, 2014.²

Unfortunately, the authors of the JAMA study hardly bothered with this, and merely pointed out in their limitations section that their study was limited by measurement error.

This study has several limitations. First, appropriate interpretation of the study findings requires consideration of measurement error for self-reported diet data. Further, this study relied on single measurement of egg and dietary cholesterol consumption. Exposure misclassification may be of concern, but results were similar when censoring participants at different time points…

This isn’t something to take this lightly given how serious of a problem it is. There have been extensive criticisms published of FFQs with one of the most prominent of reviews stating the following six arguments as to why memory-based questionnaires should not be used.³

1. The use of memory-based methods is founded upon two inter-related logical fallacies: a category error and reification
2. Human memory and recall are not valid instruments for scientific data collection
3. The measurement errors of memory-based dietary assessment methods are neither quantifiable nor falsifiable; this renders these methods and data pseudoscientific
4. The post hoc process of pseudoquantification is impermissible and invalid
5. Memory-based dietary data were repeatedly demonstrated to be physiologically implausible (i.e., meaningless numbers)
6. The failure to cite or acknowledge contrary evidence and empirical refutations contributed to a fictional discourse on diet-disease relations

The authors don’t even believe that the measurement error from memory-based questionnaires can even be addressed! They conclude the following:

Our conclusion is that nutrition epidemiology is a degenerating research paradigm in which the use of expedient but nonfalsifiable anecdotal evidence impeded scientific progress and engendered a fictional discourse on diet-health relations. The continued funding and use of memory-based methods such as Food Frequency Questionnaires and 24-hour dietary interviews is anathema to evidence-based public policy.

Thus, our recommendation is simply that the use of memory-based methods must stop and all previous articles presenting memory-based methods data and conclusions must be corrected to include contrary evidence and address the consequences of a half century of pseudoscientific claims.

Ouch. Those are quite polarizing statements. Some researchers have argued that Archer et al have overstated their case against FFQs, but agree that FFQs are far from perfect.⁴ This discussion isn’t new though. FFQs have been criticized for so long that they’ve spurred parody videos like the one below:

Those are some of the issues with the measurements, now onto the analyses (I’ll be using many of the arguments from my nutri epi blog post). Unfortunately, the analytical plans of studies like this are rarely registered in advance, especially because data collection for many of these studies began decades ago, and most of the analyses are often viewed as being “exploratory.” This means that there is a great deal of analytical flexibility in what covariates to include in a model.

Some of the biggest critics of this have been statistician Stan Young and meta-researcher John Ioannidis. Young wrote the following in the journal Significance.⁵

For example, consider the use of linear regression to adjust the risk levels of two treatments to the same background level of risk. There can be many covariates, and each set of covariates can be in or out of the model. With ten covariates, there are over 1000 possible models. Consider a maze as a metaphor for modelling (Figure 3).

The red line traces the correct path out of the maze. The path through the maze looks simple, once it is known. Returning to a linear regression model, terms can be put into and taken out of a regression model. Once you get a p‐value smaller than 0.05, the model can be frozen and the model selection justified after the fact. It is easy to justify each turn.

Patel, Burford, and Ioannidis call this flexibility and its impact on the effect size, the “vibration of effects.”⁶ In their study (which I’ve mentioned several times in previous blog posts), they downloaded several variables from the NHANES dataset that were linked to all-cause mortality and were able to substantially change the effect size by including certain covariates and excluding others,

Ioannidis and colleagues downloaded 13 variables from the NHANES dataset that were linked to all-cause mortality, and that had a substantial number of participants associated with each variable (at least 1000 participants and 100 deaths). From those 13 variables, they were able to produce 8,192 different statistical models that all resulted in different hazard ratios.

Their recommendations were to report all possible statistical models from all possible combinations of covariates and to report the median of all these models, rather than selectively reporting a few models. And to note whether there was something called a “Janus effect”, where the effect size would go in both directions.

In the last pattern, as exemplified by α-tocopherol, the estimated HRs can be both greater and less than the null value (HR > 1 and HR ≤ 1) depending on what adjustments were made. We call this the Janus effect after the two-headed representation of the ancient Roman god.

For α-tocopherol, most of the HR and p-values were concentrated around 1 and non-significance, respectively… The Janus effect is common: 131 (31%) of the 417 variables had their 99th percentile HR>1 and their 1st percentile HR<1.

However, statistician/epidemiologist, Sander Greenland has been a critic of meta-research and epidemiological studies that lack biochemical sophistication and lump several compounds together as if they were the same and behaved the same in the body.

“For nutrition, the lack of biochemistry sophistication among the trial designers leads to a lot of dubious and noncomparable studies, while the meta-analysts and reviewers do a lot of distortive lumping, e.g., talking about “vitamin E” as if that were a single entity.

A recent review by prominent authors didn’t even notice that almost all trials used the racemic synthetic mixture, dl-alpha-tocopheryl, (which they misidentify with “alpha tocopherol”) with vastly diferent and unjustifed dosages, and which hardly resembles the eight or so natural d-tocopherols or d-tocotrienols that account for dietary intake.”⁶

Ioannidis and Young are also prominent critics of the fact that nutri epi studies and randomized trials have poor concordance. For example, in Young’s 2011 paper published in Significance (which Ioannidis cites often), he (Young) states the following,

We ourselves carried out an informal but comprehensive accounting of 12 randomised clinical trials that tested observational claims – see Table 1. The 12 clinical trials tested 52 observational claims. They all confirmed no claims in the direction of the observational claims. We repeat that figure: 0 out of 52. To put it another way, 100% of the observational claims failed to replicate.

In fact, five claims (9.6%) are statistically significant in the clinical trials in the opposite direction to the observational claim. To us, a false discovery rate of over 80% is potent evidence that the observational study process is not in control. The problem, which has been recognised at least since 1988, is systemic.

This seems pretty bad. I can agree with Young and Ioannidis that nutri epi studies can often produce much noise given the analytical flexibility and measurement error. However, I’m not particularly convinced by the poor concordance data for two reasons.

1. Most RCTs have designs that are likely to be underpowered for an effect size of interest. Contrast this with nutri epi studies which are often enormous and span decades. So fixating on statistical significance with this in mind makes little sense to me.

2. Statistical significance has far more utility in studies that have some random mechanism, whether that be random assignment or random sampling. Otherwise, they make little sense to fixate on, especially in nutri epi studies. Greenland, 1990 summarizes this well below.^{Greenland1990-ay?}

Randomization provides the key link between inferential statistics and causal parameters. Inferential statistics, such as P values, confidence intervals, and likelihood ratios, have very limited meaning in causal analysis when the mechanism of exposure assignment is largely unknown or is known to be nonrandom. It is my impression that such statistics are often given a weight of authority appropriate only in randomized studies

I’m reasonably confident that Greenland has changed his mind since then on the appropriate weight of authority such statistics should be given, even in randomized studies. Especially given the recent Nature commentary he published on the problems with statistical significance and dichotomous thinking.⁷

In the same 1990 paper, Greenland also points out the problems with several probabilistic arguments that are used to justify classical statistics in areas where the exposure is unknown or unlikely to be random. He proposes some possible solutions:

In causal analysis of observational data, valid use of inferential statistics as measures of compatibility, conflict, or support depends crucially on randomization assumptions about which we are at best agnostic and more usually doubtful. Among the possible remedies are:

• Restrain our interpretation of classical statistics by explicating and criticizing any randomization assumptions that are necessary for probabilistic interpretations;
• train our students and retrain ourselves to focus on nonprobabilistic interpretations of inferential statistics;
• deemphasize inferential statistics in favor of pure data descriptors, such as graphs and tables;
• expand our analytic repertoire to include more elaborate techniques that depend on assumptions in the “agnostic” rather than the “doubtful” realm, and subject the results of these techniques to influence and sensitivity analysis.

These are neither mutually exclusive nor exhaustive possibilities, but I think any one of them would constitute an improvement over much of what we have done in the past.

Now, with all of these nuances in mind, how convincing is this observational study that has produced a hazard ratio of 1.17 with a 95% CI ranging from 1.09 - 1.26? I don’t think it’s that convincing. I’m not saying that dietary cholesterol or eggs don’t impact CVD incidence and serum cholesterol, but just that I don’t find this study’s data to be very convincing.

Perry Wilson also has an interesting argument where he utilized graphical methods to assess the plausibility of the results,

Does Dietary Cholesterol Impact Serum Cholesterol?

This has been a very controversial subject ever since dietary cholesterol was first implicated in cardiovascular disease in the 1900s with one of the most famous epidemiological studies being the Seven Countries Study.⁸ I’m not going to touch on that study since I wouldn’t be doing it justice, but there’s plenty of reading material out there.

But here’s a very brief timeline of dietary cholesterol:

Brief Timeline of Dietary Cholesterol

• 1950s | Dietary cholesterol is implicated in heart disease.⁹
• Late 1900s | Seven Countries Study is published.⁸
• Late 1900s | Metabolic ward experiments find that dietary cholesterol impacts serum cholesterol.¹⁰
• Late 1900s | Other trials fail to find an effect.^{bowmanEffectDietaryFat1988?}
• Late 1900s - Early 2000s | Government agencies and medical organization recommend limiting dietary cholesterol.¹¹
• 2013 | American Heart Association states that there’s insufficient evidence whether removing cholesterol is beneficial.¹²
• 2015 | Dietary guidelines drop arbitrary recommendations on limiting dietary cholesterol, but still recommend little intake.

Wow, that can be pretty confusing and frustrating… Wait till you see the next study I talk about.

What Do Reviews of Randomized Trials Say?

A few months ago, a meta-regression of only randomized controlled trials that were at least two weeks long (and a million other inclusion/exclusion criteria) was published in The American Journal of Clinical Nutrition.¹³ This study was interested in assessing the relationship between dietary cholesterol and serum cholesterol. Here are some characteristics of the study:

• Included 55 randomized trials (2,652 subjects), with 44 being crossover trials.
• Used Bayesian methods with uninformed priors.
• Used the Mensink equation to subtract the impact of fatty acids on cholesterol to isolate the impact of dietary cholesterol better.
• Explored both linear and nonlinear models.

So what were the results?

The nonlinear models better fit the data than the linear model and at 100 mg/d of dietary cholesterol change, there was an associated 4.58 mg/dL change in LDL cholesterol. The linear model estimated this change to be 1.90 mg/dL, which is similar to estimates found by the meta-analysis of metabolic ward studies.¹⁰ At 200-400 mg/d of dietary cholesterol change, the nonlinear and linear models had substantial differences in their estimations (see the table below and then the graph).

Wow. So dietary cholesterol does indeed impact serum cholesterol (LDL)! Here’s what the authors write in their conclusion…

The meta-regression analyses presented herein indicate that, after providing a theoretical control for intakes of SFAs, MUFAs, PUFAs, and when possible, TFAs, there is a positive, dose-related trend between changes in cholesterol intake and changes in circulating LDL cholesterol, a finding that is consistent with the published literature.

The nonlinear dose-response shapes (MM and Hill) best fit the data. For a 100-mg/d increase in dietary cholesterol intake, circulating LDL-cholesterol concentrations are predicted to increase by 1.90 mg/dL (linear model), 4.46 mg/dL (MM model), or 4.58 mg/dL (Hill model).

Then they discuss the practical significance of this.

The results from these meta-analyses have potential implications for dietary recommendations. For comparison, the equation given by Mensink et al. predicts a change of 1.23 mg/dL for each 1% increase in SFAs in exchange for carbohydrate.

Therefore, based on the nonlinear models, increasing dietary cholesterol by 100 mg/d would be predicted to have an effect comparable to that of increasing dietary SFAs by 3.7%, and increasing dietary cholesterol by 200 mg/d is predicted to be comparable to increasing dietary SFAs by 5.5%.

A large egg contains ∼185 mg cholesterol and would therefore be expected to increase the LDL-cholesterol concentration by 6.5 mg/dL based on the nonlinear models. However, the majority of dietary cholesterol is not attributable to egg intake in the United States, except for individuals in the highest quartile of TC intake.

According to a recent publication reporting dietary sources of cholesterol in US adults aged ≥20 y based on NHANES 2013–2014 data, mean dietary cholesterol intake was 293 mg/d and the primary dietary cholesterol source in the overall population was meat (defined as poultry, mixed dishes, red meat, processed meat, and seafood), which accounted for 42% of dietary cholesterol.

Wait, so Should We Be Worried About Eggs?

I’m going to argue that we probably shouldn’t. The reason is that just because a food contains dietary cholesterol doesn’t mean it will be absorbed efficiently in the gastrointestinal tract. Foods are not homogenous and do not contain the same nutrients, anti-nutrients, and minerals. Consuming eggs in a sitting is not the same as consuming pure dietary cholesterol nor is it the same as consuming other foods high in cholesterol (poultry, red meat, and processed meat).

This also seems to be supported by a few randomized crossover trials that have shown that the cholesterol in eggs doesn’t seem to be well absorbed.^{kimDietaryCholesterolContained2018?} Several other trials have also failed to show an effect on serum cholesterol.^{14,15,bowmanEffectDietaryFat1988?} Is it possible that these trials were just underpowered in the first place? Very possible.

This all needs to be taken into account with the fact that in addition to the randomized trials included in the meta-regression mentioned above,¹³ there exists a plethora of metabolic ward studies suggesting an effect of dietary cholesterol on serum cholesterol.¹⁰ However, many of the metabolic ward trials included in the prominent meta-analysis by Clarke et al¹⁰ were also nonrandomized,

Solid food diets were assessed in 72 of these reports among 129 groups of subjects in 395 experiments with various designs (109 randomised crossover, 57 randomised or matched parallel, 77 non-randomised Latin square, and 152 non-randomised sequential).

So while there is plenty of evidence suggesting that a diet high in dietary cholesterol impacts serum cholesterol, the evidence to suggest that eggs in particular have this effect seems far less concrete. There also may be a number of nutrients found in eggs that inhibit cholesterol absorption.^16,17 Here are some excerpts from Kuang et al¹⁷

Yang et al. reported that dietary PUFAs, PC, and SM significantly inhibit the uptake of cholesterol in Caco-2 monolayer, which may have potential therapeutic effect on reducing cholesterol absorption as functional food ingredients [44, 45]….

Ezetimibe reduces the absorption of cholesterol in the small intestine by inhibiting the activity of NPC1L1 protein [52].

Therefore, consuming eggs may inhibit the intestinal absorption of cholesterol via the phospholipids in eggs. A fundamental question is how this biological effect is regulated.

How convincing is this evidence? I’m still not sure. Much of it seems to be very preliminary, but I’m leaning towards the idea that most people probably don’t have to worry about eggs, especially since Americans don’t even get most of their dietary cholesterol from eggs.¹³ I don’t find the existing data for eggs (including those from this study) to be convincing. More better research is needed.

We need less research, better research, and research done for the right reasons.¹⁸ - Doug Altman

See my other articles on nutritional epidemiology:

• How Useful Is Nutritional Epidemiology?

• Is Moderate Carbohydrate Intake the Best?

• Does Protein Increase the Risk of Heart Failure?

Stata Examples

Stata code can be included for syntax highlighting:

* ============================================
* Example: Linear Regression Analysis
* ============================================

* Load example dataset
sysuse auto, clear

* Descriptive statistics
summarize price mpg weight foreign

* Create a scatter plot
scatter price mpg, title("Price vs. MPG")

* Run linear regression
regress price mpg weight foreign

* Store estimates
estimates store model1

* Display results with confidence intervals
regress price mpg weight foreign, level(95)

* Test joint significance
test mpg weight

* Predict fitted values
predict price_hat

* Calculate residuals
predict residuals, residuals

* Plot residuals
scatter residuals price_hat, ///
  yline(0) ///
  title("Residual Plot")

* ============================================
* Panel Data Example
* ============================================

* Load panel dataset
webuse nlswork, clear

* Declare panel structure
xtset idcode year

* Summary statistics by panel
xtsum ln_wage age

* Fixed effects regression
xtreg ln_wage age tenure, fe

* Random effects regression
xtreg ln_wage age tenure, re

* Hausman test
hausman fe re

Julia Examples

Julia code for statistical computing:

# ============================================
# Example: Linear Regression in Julia
# ============================================

using DataFrames, Statistics, GLM, Plots

# Create sample data
n = 100
x = randn(n)
y = 2.0 .+ 3.0 .* x .+ randn(n) .* 0.5

# Create DataFrame
df = DataFrame(x = x, y = y)

# Fit linear model
model = lm(@formula(y ~ x), df)

# Display results
println(model)

# Plot data and fitted line
scatter(x, y, label="Data", alpha=0.6)
plot!(x, predict(model), label="Fitted Line", linewidth=2)
xlabel!("X")
ylabel!("Y")
title!("Linear Regression")

# ============================================
# Bootstrap Example
# ============================================

using Distributions, StatsBase

# Define a function to bootstrap
function bootstrap_mean(data, n_bootstrap=1000)
    n = length(data)
    bootstrap_means = zeros(n_bootstrap)

    for i in 1:n_bootstrap
        # Resample with replacement
        sample = rand(data, n)
        bootstrap_means[i] = mean(sample)
    end

    return bootstrap_means
end

# Generate sample data
data = randn(100) .+ 5.0

# Perform bootstrap
bootstrap_samples = bootstrap_mean(data)

# Calculate confidence interval
ci_lower = quantile(bootstrap_samples, 0.025)
ci_upper = quantile(bootstrap_samples, 0.975)

println("95% Bootstrap CI: [", ci_lower, ", ", ci_upper, "]")

# Plot bootstrap distribution
histogram(bootstrap_samples,
         bins=30,
         label="Bootstrap Distribution",
         xlabel="Sample Mean",
         ylabel="Frequency")
vline!([ci_lower, ci_upper],
       label="95% CI",
       linewidth=2,
       color=:red)

Pseudocode Examples

ALGORITHM: Linear Regression via Gradient Descent
INPUT: X (n × p design matrix), y (n × 1 response vector), learning rate α
OUTPUT: β (p × 1 coefficient vector)

1. Initialize β = 0, iteration = 0, maxiter = 1000
2. REPEAT until convergence OR iteration > maxiter:
   a. Compute predictions: ŷ = Xβ
   b. Compute residuals: r = y - ŷ
   c. Compute gradient: ∇L = -2X'r/n
   d. Update parameters: β = β - α∇L
   e. iteration = iteration + 1
3. RETURN β

Comparing Approaches

Here’s how the same analysis looks across languages:

data <- rnorm(100, mean = 5, sd = 2)
mean_val <- mean(data)
ci <- t.test(data)$conf.int

using Statistics, HypothesisTests
data = randn(100) .* 2 .+ 5
mean_val = mean(data)
ci = confint(OneSampleTTest(data))

clear
set obs 100
generate data = rnormal(5, 2)
mean data

import numpy as np
from scipy import stats
data = np.random.normal(5, 2, 100)
mean_val = np.mean(data)
ci = stats.t.interval(0.95, len(data)-1,
                      loc=mean_val,
                      scale=stats.sem(data))

Best Practices

1. Use syntax highlighting for display-only code - Most readers just need to see the code
2. Execute code sparingly - Only execute when output is essential
3. Document your setup - Note required packages and versions
4. Test before publishing - Ensure code runs without errors
5. Provide context - Explain what the code does and why

Resources

• Quarto Stata Documentation
• Quarto Julia Documentation
• Highlight.js Language Support

Conclusion

With support for multiple languages, you can now share statistical methods and algorithms in the language that best suits your analysis and audience.

The Power of the Bootstrap

The bootstrap^{1, 2} is an incredibly powerful method of approximation. Indeed, frequentists are even able to approximate posterior distributions using it, In an excerpt from the paper below, Efron describes how these procedures are similar and gives examples using the parametric bootstrap.

The point of this brief note is to say that in some situations the bootstrap, in particular the parametric bootstrap, offers an easier path toward the calculation of Bayes posterior distributions. An important but somewhat under-appreciated article by Newton and Raftery (1994) made the same point using nonparametric bootstrapping. By “going parametric,” we can illustrate more explicitly the bootstrap/MCMC connection. The arguments here will be made mainly in terms of a simple example, with no attempt at the mathematical justifications seen in Newton and Raftery.

It is not really surprising that the bootstrap and MCMC share some overlapping territory. Both are general-purpose computer-based algorithmic methods for assessing statistical accuracy, and both enable the statistician to deal effectively with nuisance parameters, obtaining inferences for the interesting part of the problem. On the less salubrious side, both share the tendency of general-purpose algorithms toward overuse.

Of course the two methodologies operate in competing inferential realms: frequentist for the bootstrap, Bayesian for MCMC. Here, as in Newton and Raftery, we leap over that divide, bringing bootstrap calculations to bear on the Bayesian world. The working assumption is that we have a Bayesian prior in mind and the only question is how to compute its posterior distribution. Arguments about the merits of Bayesian versus frequentist analysis will not be taken up here, except for our main point that the two camps share some common ground.

However, they can not only approximate posterior distributions, but confidence distributions too. Confidence distributions are an incredibly dense and technical topic and is practically an umbrella term to refer to multiple concepts, for example, they can refer to confidence curves, confidence densities and deviance functions. For a definitive source to learn more about what they are, see the following^{3, 4}

In a paper prepared for Efron’s Fisher lecture, he writes that confidence distributions may provide the key to syncretism between frequentist and bayesian motivations:

Fiducial distributions, or confidence densities, of- fer a way to finesse this difficulty. A good argument can be made that the confidence density is the posterior density for the parameter of interest, after all of the nuisance parameters have been integrated out in an objective way. If this argument turns out to be valid, then our progress in constructing approximate confidence intervals, and approximate confidence densities, could lead to an easier use of Bayesian thinking in practical problems. This is all quite speculative, but here is a safe prediction for the 21st century: statisticians will be asked to solve bigger and more complicated prob- lems. I believe that there is a good chance that objective Bayes methods will be developed for such problems, and that something like fiducial infer- ence will play an important role in this develop- ment. Maybe Fisher’s biggest blunder will become a big hit in the 21st century!

Indeed, some authors have shown that the bootstrap distribution is equal to the confidence distribution because it meets the definition of a confidence distribution.^5–7. Some have even referred to posteriors as a quick and dirty approximation to confidence distributions.⁸

Properties of Confidence Distributions

Xie and Singh, as summarized by Fraser define these properties as:

The first is the classical definition CL and defines the confidence distribution function as the distribution function version of the confidence quantile function ̃, where is a level confidence interval.

The second definition (2.1) identifies a confidence distribution function as a distribution function in θ for each given y and as a pivot with a uniform distribution for eachθ. But this also closely identifies with what Fisher (1930) offered: fiducial but with Fisher’s promotion replaced by a claim that the argument is pure frequentist. Of course it is pure frequentist just as Fisher (1930) was pure frequentist, except for Fisher’s accompanying claim to having purer probabilities, which was then a direct confrontation to the Bayes aficionados of the time.

The third definition (2.2) essentially gives just the quantile equivalent say ) of the distribution function where u is Uniform (0,1). Aren’t confidence quantile and confidence upper bound just different labelling for the same object? Confidence distributions can have many properties: the distribution function should of course be , but also it should inherit continuity when present in the model, should use all available information, and should generally be sensible. These properties aren’t really addressed in the authors’ proposal. A promotion of confidence distributions should acknowledge these inherent issues and also mention marginal and conditional conflicts as discussed in the literature.

The bootstrap distribution and the asymptotic consonance distribution would be defined as:

Efron writes,

As the authors point out, all of this has something to do with the bootstrap. Let represent B bootstrap replications ofθˆ, an estimator of parameter θ(possibly in the presence of nuisance parameters). The -th empirical quantile of the θˆ∗’s is then theupper endpoint of a first-order accurate-levelαconfidence interval. In this sense, the bootstrap distribution is an approximate confidence distribution. The BCa density, Efron & Tibshirani (1998), improves the confidence accuracy by reweighting the values. Let Gˆbe this empirical cdf, and and a be the bias correction and acceleration constants in my 1987 paper. Rather than equal weights 1/B, the BCa density puts weight proportional to . The reweighted bootstrap distribution then becomes a second-order accurate confidence distribution. Efron (2012) discusses this construction in the context of objective Bayes inference.

Fraser, however, argues that the connection is nothing special

The bootstrap.The bootstrap as in Section 2.3 provides an approximation to the distributions described by a model, and in doing this the bootstrap can also eliminate the influence of nuisance parameters. It can be applied to statistics or to pivots, with faster effect using suitable pivots. It can be used with least squares, or with maximum like lihood statistics, or with statistical quantities, or anywhere where distributions are wanted. Ofcourse confidence calculations are just one such use but there are many others including of course testing. So there is no particular attachment of the bootstrap to confidence distribution functions other than providing an approximate means of calculation for such.

Certain bootstrap methods such as the BCa method and t-bootstrap method also yield second order accuracy of consonance distributions.

Here, I demonstrate how to use these particular bootstrap methods to arrive at consonance curves and densities.

We’ll use the Iris dataset and construct a function that’ll yield a correlation coefficient and we will try to estimate the confidence intervals for them.

The Nonparametric Bootstrap

iris <- datasets::iris
foo <- function(data, indices) {
  dt <- data[indices, ]
  c(cor(dt[, 1], dt[, 2], method = "p"))
}

We can now use the curve_boot() method to construct a function. The default method used for this function is the “Bca” method provided by the bcaboot package (also written by Efron).

I will suppress the output of the function because it is unnecessarily long. But we’ve placed all the estimates into a list object called y.

The first item in the list will be the consonance distribution constructed by typical means, while the third item will be the bootstrap approximation to the consonance distribution.

ggplot2::qplot(boot::boot(iris, foo, 1000)$t, geom = "histogram") +
  theme_light()

ggcurve(data = y[[2]], type = "cd", nullvalue = TRUE, fill= "red") +
  theme_light()

We can also print out a table for TeX documents

ztable(gg <- curve_table(data = y[[1]]))

More bootstrap replications will lead to a smoother function. But for now, we can compare these two functions to see how similar they are.

result <- curve_from_se(estimate = 2.5, se = 0.8, measure = "mean")
ggcurve(result[[1]], type = "c", nullvalue = TRUE)

# Odds ratio with SE (on log scale internally)
result <- curve_from_se(estimate = 1.5, se = 0.2, measure = "ratio")
#> [1] 0.133
ggcurve(result[[1]], type = "c", nullvalue = TRUE)

If we wanted to look at the bootstrap standard errors, we could do so by loading the fifth item in the list

where in the top row, theta is the point estimate, and sdboot is the bootstrap estimate of the standard error, sdjack is the jacknife estimate of the standard error. z0 is the bias correction value and a is the acceleration constant.

The values in the second row are essentially the internal standard errors of the estimates in the top row.

One can also construct the confidence density, here I provide an example from the pvaluefunction R package The consonance curve and density are nearly identical. With more bootstrap replications, they are very likely to converge. [aadland2015ajcn]

library("pvaluefunctions")
result <- bcaboot::bcajack(
  x = iris, B = 10000, func = foo,
  alpha = c((1:99) / 100), verbose = FALSE
)

bca <- data.frame(result[["lims"]][, 1])

ggplot2::qplot(bca$result...lims......1., geom = "density") +
  theme_less()

The Parametric Bootstrap

For the examples above, we mainly used nonparametric bootstrap methods. Here I show an example using the parametric Bca bootstrap and the results it yields.

data(diabetes, package = "bcaboot")
X <- diabetes$x
y <- scale(diabetes$y, center = TRUE, scale = FALSE)
lm.model <- lm(y ~ X - 1)
mu.hat <- lm.model$fitted.values
sigma.hat <- stats::sd(lm.model$residuals)
t0 <- summary(lm.model)$adj.r.squared
y.star <- sapply(mu.hat, rnorm, n = 1000, sd = sigma.hat)
tt <- apply(y.star, 1, function(y) summary(lm(y ~ X - 1))$adj.r.squared)
b.star <- y.star %*% X

Now, we’ll use the same function, but set the method to bcapar for the parametric method.

bcadf <- (bcaboot::bcapar(
  t0 = t0, alpha = c((1:99) / 100),
  tt = tt, bb = b.star, cd = 1
)[["lims"]][, 1])

Now we can look at our outputs.

ggplot2::qplot((unlist(bcadf)), geom = "density") +
  theme_less()

That concludes our demonstration of the bootstrap method to approximate consonance functions.

References

Statistical Models as Approximations

Scientific inference is a highly complex process often accompanied by numerical outputs from statistical models. To some, such outputs are not convincing as they believe that anyone can lie with numbers, but the reality is that it is a lot easier to lie without them, making them crucial because if deployed correctly, they may be close approximations to reality. The presence of a statistical model can allow others to view an answer to a complex question from a numerical standpoint that many may be able to verify.

Unfortunately, there are rampant issues in the building, reporting, and interpretation of such statistical models, and such issues can lead to devastating policies. We have elsewhere reviewed proposals to reform terminology and improve interpretations of conventional statistics by emphasizing logical and information concepts over probability concepts (1–3), as the latter has continued to confuse even well-credentialed statisticians familiar with the misinterpretations.

What Goes into a Model?

We here focus on how common descriptions of statistical outputs can be technically accurate yet still mislead when there is considerable uncertainty about what are known as background or auxiliary assumptions. Such uncertainty is typical in many contexts, where assumptions may include both mathematical and methodological ones:

For example, in the popular regression book by Gelman, the following assumptions are given for regression models in decreasing order of importance:

• validity

• additivity and linearity

• independence of errors

• equal variance of errors

• normality of errors

This is typical to see in many regression modeling books, but important methodological assumptions underlying the model are rarely discussed such as:

• patient outcomes are independent

• interactions and trends follow the regression model used for analysis (e.g., linear or logistic)

• no uncontrolled confounding

• subject-selection bias

• measurement error

• sparse-data bias

• no selective reporting of results* based on their P-values, interval estimates, or any other output – or if there is such selection, any bias it could produce has been adjusted for (or blocked) in the computation and presentation of the final results.

In addition to these general statistical and methodological assumptions, many assume that the statistical model being fit is answering a proper scientific question that has been translated into a statistical one, with assumptions that match or approximate reality. When there is a disconnect between the scientific question and the statistical one being answered by the model, the analyst is committing a type-III error, and will produce statistical outputs with of little use, unless the assumption violations of the chosen and reported are interrogated/varied and considered in the overall analysis.

Many are not aware of what a type-III error is due to the focus on type-I and type-II errors. I paste the following definition from the APA dictionary, which I found myself agreeing with:

Type-III Error: the error that occurs when there is a discrepancy between the research focus and the hypothesis actually tested. For example, a Type III error would have happened if a researcher collected data on individual differences within a sample and determined the causes of variation but the question of interest concerned differences between populations. In other words, a Type III error involves providing the right answer for the wrong question. Also called Type 0 error.

Violation of Assumptions

Misleading interpretations arise because the descriptions of such statistical outputs are conditional on such assumptions (i.e., they treat them as given/being true), and so do not factor uncertainty regarding the assumptions into their assessments. While various risk-assessment methods can incorporate additional sources in quantified uncertainties, such methods demand considerably more skilled user input than do conventional regression methods (5–9) and still do not account for unquantified sources of uncertainty. We thus present a more direct and nontechnical approach based on logic and associated language to honor assumption uncertainty, called deconditioning.

For example, suppose that a model assumes independence, a common assumption in many statistical models, but this violated due to the presence of a correlation, this will lead to P-values that are invalid, demonstrable by their failure to achieve asymptotic uniformity.

For example, a model that is approximately correct will generate P-values that are asymptotically uniform (a characteristic used to assess the validity of P-values)

#' @title Simulation of valid P-values where test hypothesis is true
#' @param X The first variable we are simulating
#' @param Y The second variable we are simulating
#' @param n.sim # The number of simulations
#' @param t The object storing the t-test results
#' @param t.sim # Empty numeric vector to contain values
#' @param n.samp # Sample size in each group
#' @NOTE The null hypothesis does not have to be 0, it can be any value.

n.sim <- 10000
t.sim <- numeric(n.sim)
n.samp <- 1000

for (i in 1:n.sim) {
  X <- rnorm(n.samp, mean = 0, sd = 1)
  Y <- rnorm(n.samp, mean = 0, sd = 1)
  df <- data.frame(X, Y)
  t <- t.test(X, Y,
    mu = 0, paired = FALSE,
    var.equal = TRUE, data = df)
  t.sim[i] <- t[[3]]}

#' @title Simulation of invalid P-values where test hypothesis is false
#' @param n_simulations # The number of simulations
#' @param p_values The object storing the t-test results
#' @param p_value_data # Empty numeric vector to contain values
#' @param n_subjects # Sample size in each group
#' @param correlation # correlation to violate independence

library(MASS)
n_simulations <- 1000
n_subjects <- 50
correlation <- 0.5

generate_correlated_data <- function(n_subjects, correlation) {
    Sigma <- matrix(correlation, n_subjects, n_subjects) +
        diag(1 - correlation, n_subjects)
    data <- mvrnorm(n = 2, mu = rep(0, n_subjects),
        Sigma = Sigma)
    return(data)
}
run_simulation <- function(n_subjects, correlation) {
    data <- generate_correlated_data(n_subjects, correlation)
    t_test_result <- t.test(data[1, ], data[2, ])
    t_test_p_value <- t_test_result$p.value
    return(t_test_p_value)
}
p_values <- replicate(n_simulations, run_simulation(n_subjects,
    correlation))
p_value_data <- data.frame(PValue = p_values)

Deconditioning shifts the logical status of assumptions in the description of results from what is assumed as given (conditioned on) to an integral part of what is being checked by common statistics. This shift makes clear that the results might be explained by violations of background assumptions rather than, or in addition to, failure of a targeted hypothesis. Such deconditioning does not change calculations; hence it does not change any P-value, interval estimate, or posterior distribution. Instead, it provides a different interpretation of these statistics, one arising naturally in thorough contextual discussions, but which deserves more emphasis in basic statistical practice, including teaching and presentations.

In philosophical accounts of theory testing, it is well recognized that an apparent violations of a hypothesis may instead be due to violation of background or auxiliary assumptions (10–13). Among these assumptions are researcher competence and integrity in the conduct and reporting of statistical analyses which have come under scrutiny in the “replication crisis”) as well as common statistical assumptions such as linearity, outcome independence, and effect homogeneity, and methodological assumptions such as no uncontrolled confounding or selection bias. Perhaps because of this generality, we have found that our recommendation to decondition inferences (present inferences using an unconditional interpretation) (1,3) is often difficult for statistically sophisticated readers to comprehend or accept, and is even resisted, misunderstood, and misrepresented by some reviewers

Using the cifunction Command for Confidence Interval Functions

An alternative approach to producing confidence interval functions in Stata is to use the cifunction command, which computes and graphically displays all possible confidence intervals around a point estimate.

The cifunction command is an immediate command that allows you to specify a point estimate and standard error, and it will produce a confidence interval function (also called a confidence curve, P-value function, or consonance interval).

cifunction #b , se(#) [ df(#) eform figure[(twoway_options)] saving(filename, replace) ]

getregstats 2.0, p(0.05) mod(rr)
#> Can not set Stata directory, try using -statapath()- option
#> r(601);
#> 
#> 
#> 
#> 
#> ------------------------------------------------------------------------------
#>              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
#> -------------+----------------------------------------------------------------
#>    Estimates |          2    .707306    -1.96   0.050            1           4
#> ------------------------------------------------------------------------------

cifunction 2.0, se(.707306) eform fig(xscale(log) xlab(0.5 1 2 5 10) xtitle("Risk Ratio (Log Scale)"))
#> Can not set Stata directory, try using -statapath()- option
#> r(601);

webuse dollhill3, clear
poisson deaths smokes if agecat==1, exposure(pyears) irr
poisson deaths smokes if agecat==2, exposure(pyears) irr
poisson deaths smokes if agecat==3, exposure(pyears) irr
poisson deaths smokes if agecat==4, exposure(pyears) irr
#> Can not set Stata directory, try using -statapath()- option
#> r(601);
#> 
#> 
#> (Doll and Hill (1966))
#> 
#> 
#> Iteration 0:  Log likelihood = -4.0705386  
#> Iteration 1:  Log likelihood = -3.9613823  
#> Iteration 2:  Log likelihood = -3.9612634  
#> Iteration 3:  Log likelihood = -3.9612634  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   9.73
#>                                                         Prob > chi2   = 0.0018
#> Log likelihood = -3.9612634                             Pseudo R2     = 0.5511
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |   5.736638   4.181258     2.40   0.017     1.374811    23.93712
#>        _cons |   .0001064   .0000753   -12.94   0.000     .0000266    .0004256
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.
#> 
#> 
#> Iteration 0:  Log likelihood = -5.4793565  
#> Iteration 1:  Log likelihood = -5.4104407  
#> Iteration 2:  Log likelihood =   -5.41027  
#> Iteration 3:  Log likelihood =   -5.41027  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   7.59
#>                                                         Prob > chi2   = 0.0059
#> Log likelihood = -5.41027                               Pseudo R2     = 0.4123
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |   2.138812   .6520701     2.49   0.013     1.176691    3.887609
#>        _cons |   .0011243   .0003246   -23.52   0.000     .0006385    .0019798
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.
#> 
#> 
#> Iteration 0:  Log likelihood = -6.2456963  
#> Iteration 1:  Log likelihood = -6.1713863  
#> Iteration 2:  Log likelihood =  -6.171298  
#> Iteration 3:  Log likelihood =  -6.171298  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   4.01
#>                                                         Prob > chi2   = 0.0453
#> Log likelihood = -6.171298                              Pseudo R2     = 0.2450
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |    1.46824    .295728     1.91   0.057     .9893522     2.17893
#>        _cons |   .0049037   .0009267   -28.14   0.000     .0033858    .0071021
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.
#> 
#> 
#> Iteration 0:  Log likelihood = -6.1640322  
#> Iteration 1:  Log likelihood = -6.1203079  
#> Iteration 2:  Log likelihood = -6.1202768  
#> Iteration 3:  Log likelihood = -6.1202768  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   2.43
#>                                                         Prob > chi2   = 0.1189
#> Log likelihood = -6.1202768                             Pseudo R2     = 0.1658
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |    1.35606   .2748845     1.50   0.133     .9114509    2.017551
#>        _cons |   .0108317    .002047   -23.95   0.000     .0074789    .0156877
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.

cifunction 5.736638 2.138812 1.46824 1.35606, ///
  se(4.181258 .6520701 .295728 .2748845) eform ///
  fig(xscale(log) xlabel(.3 .5 1 2 5 10 20 50 100) ///
      legend(label(1 "35-44") label(2 "45-54") label(3 "55-64") ///
             label(4 "65-74") title("Age Group", size(small))) ///
      xtitle("Rate Ratio (Log Scale)"))
#> Can not set Stata directory, try using -statapath()- option
#> r(601);

#> Can not set Stata directory, try using -statapath()- option
#> r(601);
#> 
#> 
#> 
#>                                                          Proportion
#>                  |   Exposed   Unexposed  |      Total      exposed
#> -----------------+------------------------+------------------------
#>            Cases |        10          36  |         46       0.2174
#>         Controls |         5          40  |         45       0.1111
#> -----------------+------------------------+------------------------
#>            Total |        15          76  |         91       0.1648
#>                  |                        |
#>                  |      Point estimate    |    [95% conf. interval]
#>                  |------------------------+------------------------
#>       Odds ratio |         2.222222       |      .69377    7.118025 (Woolf)
#>  Attr. frac. ex. |              .55       |      -.4414    .8595116 (Woolf)
#>  Attr. frac. pop |         .1195652       |
#>                  +-------------------------------------------------
#>                                chi2(1) =     1.87  Pr>chi2 = 0.1719
#> 
#> 
#>                                                          Proportion
#>                  |   Exposed   Unexposed  |      Total      exposed
#> -----------------+------------------------+------------------------
#>            Cases |        18          78  |         96       0.1875
#>         Controls |         7          86  |         93       0.0753
#> -----------------+------------------------+------------------------
#>            Total |        25         164  |        189       0.1323
#>                  |                        |
#>                  |      Point estimate    |    [95% conf. interval]
#>                  |------------------------+------------------------
#>       Odds ratio |         2.835165       |    1.123936    7.151794 (Woolf)
#>  Attr. frac. ex. |         .6472868       |    .1102698    .8601749 (Woolf)
#>  Attr. frac. pop |         .1213663       |
#>                  +-------------------------------------------------
#>                                chi2(1) =     5.18  Pr>chi2 = 0.0228
#> 
#> 
#> 
#> ------------------------------------------------------------------------------
#>              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
#> -------------+----------------------------------------------------------------
#>    Estimates |   2.835165   1.338429     2.21   0.027     1.123936    7.151794
#> ------------------------------------------------------------------------------
#> 
#> 
#> 
#> ------------------------------------------------------------------------------
#>              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
#> -------------+----------------------------------------------------------------
#>    Estimates |   2.222222   1.319891     1.34   0.179     .6937698    7.118025
#> ------------------------------------------------------------------------------

#> Can not set Stata directory, try using -statapath()- option
#> r(601);

Cite R Packages

about
#> Can not set Stata directory, try using -statapath()- option
#> r(601);
#> 
#> 
#> 
#> StataNow/MP 19.5 for Mac (Apple Silicon)
#> Revision 28 Jan 2026
#> Copyright 1985-2025 StataCorp LLC
#> 
#> Total physical memory: 48.01 GB
#> 
#> Stata license: Single-user 2-core , expiring  6 Feb 2027
#> Serial number: 501909358563
#>   Licensed to: Zad Rafi
#>                Hunter

Session info

si <- sessionInfo()
print(si, RNG = TRUE, locale = TRUE)
#> R version 4.5.2 (2025-10-31)
#> Platform: aarch64-apple-darwin20
#> Running under: macOS Tahoe 26.3
#> 
#> Matrix products: default
#> BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
#> LAPACK: /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.1
#> 
#> Random number generation:
#>  RNG:     Mersenne-Twister 
#>  Normal:  Inversion 
#>  Sample:  Rejection 
#>  
#> locale:
#> [1] C.UTF-8/C.UTF-8/C.UTF-8/C/C.UTF-8/C.UTF-8
#> 
#> time zone: America/New_York
#> tzcode source: internal
#> 
#> attached base packages:
#>  [1] splines   grid      stats4    parallel  stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#>  [1] cli_3.6.5             texPreview_2.1.0      tinytex_0.58          rmarkdown_2.30        brms_2.23.0          
#>  [6] bootImpute_1.3.0      knitr_1.51            boot_1.3-32           gtsummary_2.5.0       reshape2_1.4.5       
#> [11] ProfileLikelihood_1.3 ImputeRobust_1.3-1    gamlss_5.5-0          gamlss.dist_6.1-1     gamlss.data_6.0-7    
#> [16] mvtnorm_1.3-3         performance_0.15.3    summarytools_1.1.5    tidybayes_3.0.7       htmltools_0.5.9      
#> [21] Statamarkdown_0.9.6   car_3.1-3             carData_3.0-6         qqplotr_0.0.7         ggcorrplot_0.1.4.1   
#> [26] mitml_0.4-5           pbmcapply_1.5.1       Amelia_1.8.3          Rcpp_1.1.1            blogdown_1.23        
#> [31] doParallel_1.0.17     iterators_1.0.14      foreach_1.5.2         lattice_0.22-7        bayesplot_1.15.0     
#> [36] wesanderson_0.3.7     VIM_7.0.0             colorspace_2.1-2      here_1.0.2            progress_1.2.3       
#> [41] loo_2.9.0             mi_1.2                Matrix_1.7-4          broom_1.0.12          yardstick_1.3.2      
#> [46] svglite_2.2.2         Cairo_1.7-0           cowplot_1.2.0         mgcv_1.9-4            nlme_3.1-168         
#> [51] xfun_0.56             broom.mixed_0.2.9.6   reticulate_1.44.1     kableExtra_1.4.0      posterior_1.6.1      
#> [56] checkmate_2.3.3       parallelly_1.46.1     miceFast_0.8.5        randomForest_4.7-1.2  missForest_1.6.1     
#> [61] miceadds_3.18-36      quantreg_6.1          SparseM_1.84-2        MCMCpack_1.7-1        MASS_7.3-65          
#> [66] coda_0.19-4.1         latex2exp_0.9.8       rstan_2.32.7          StanHeaders_2.32.10   lubridate_1.9.4      
#> [71] forcats_1.0.1         stringr_1.6.0         dplyr_1.1.4           purrr_1.2.1           readr_2.1.6          
#> [76] tibble_3.3.1          ggplot2_4.0.1         tidyverse_2.0.0       ggtext_0.1.2          concurve_3.0.0       
#> [81] showtext_0.9-7        showtextdb_3.0        sysfonts_0.8.9        future.apply_1.20.1   future_1.69.0        
#> [86] tidyr_1.3.2           magrittr_2.0.4        mice_3.19.0           rms_8.1-0             Hmisc_5.2-5          
#> 
#> loaded via a namespace (and not attached):
#>   [1] dichromat_2.0-0.1       nnet_7.3-20             TH.data_1.1-5           vctrs_0.7.1             digest_0.6.39          
#>   [6] png_0.1-8               shape_1.4.6.1           proxy_0.4-29            magick_2.9.0            fontLiberation_0.1.0   
#>  [11] withr_3.0.2             ggpubr_0.6.2            survival_3.8-6          doRNG_1.8.6.2           emmeans_2.0.1          
#>  [16] MatrixModels_0.5-4      systemfonts_1.3.1       ragg_1.5.0              zoo_1.8-15              V8_8.0.1               
#>  [21] ggdist_3.3.3            DEoptimR_1.1-4          Formula_1.2-5           prettyunits_1.2.0       rematch2_2.1.2         
#>  [26] httr_1.4.7              otel_0.2.0              rstatix_0.7.3           globals_0.19.0          ps_1.9.1               
#>  [31] rstudioapi_0.18.0       extremevalues_2.4.1     pan_1.9                 generics_0.1.4          processx_3.8.6         
#>  [36] base64enc_0.1-6         curl_7.0.0              mitools_2.4             lgr_0.5.2               desc_1.4.3             
#>  [41] xtable_1.8-4            svUnit_1.0.8            pracma_2.4.6            evaluate_1.0.5          hms_1.1.4              
#>  [46] glmnet_4.1-10           rcartocolor_2.1.2       lmtest_0.9-40           palmerpenguins_0.1.1    robustbase_0.99-6      
#>  [51] matrixStats_1.5.0       svgPanZoom_0.3.4        class_7.3-23            pillar_1.11.1           caTools_1.18.3         
#>  [56] compiler_4.5.2          stringi_1.8.7           paradox_1.0.1           jomo_2.7-6              minqa_1.2.8            
#>  [61] plyr_1.8.9              crayon_1.5.3            abind_1.4-8             metadat_1.4-0           sp_2.2-0               
#>  [66] mathjaxr_2.0-0          rapportools_1.2         twosamples_2.0.1        sandwich_3.1-1          whisker_0.4.1          
#>  [71] codetools_0.2-20        multcomp_1.4-29         textshaping_1.0.4       bcaboot_0.2-3           openssl_2.3.4          
#>  [76] flextable_0.9.10        QuickJSR_1.9.0          e1071_1.7-17            gridtext_0.1.5          lme4_1.1-38            
#>  [81] fs_1.6.6                itertools_0.1-3         listenv_0.10.0          Rdpack_2.6.5            pkgbuild_1.4.8         
#>  [86] estimability_1.5.1      ggsignif_0.6.4          callr_3.7.6             tzdb_0.5.0              pkgconfig_2.0.3        
#>  [91] tools_4.5.2             rbibutils_2.4.1         viridisLite_0.4.2       DBI_1.2.3               numDeriv_2016.8-1.1    
#>  [96] fastmap_1.2.0           scales_1.4.0            officer_0.7.3           opdisDownsampling_1.0.1 insight_1.4.5          
#>  [ reached 'max' / getOption("max.print") -- omitted 64 entries ]

Setting up the Function

#' @title Simulation of a Two-Group Parallel-Arm Trial With Interim Analyses
#' @docType Custom function for simulation from the rpact package
#' @author Andrew Althouse with edits by Zad Rafi
#' NOTE: If you want to confirm "type 1 error" under different stopping rules,
#' make death = in the two treatment arms (e.g. no treatment effect)
#' NOTE: I have set this one up to test the power for a treatment that would reduce mortality
#' from 40% in control group (1) to 30% in treatment group (2)
#' NOTE: Trial Design Parameters - Part 1
#' Here we will specify the basics: total N patients to enroll, and death rate for each treatment arm
#' NOTE: Trial Design Parameters - Part 2
#' Here we will define the interim analysis strategy and stopping rules
#' For this trial we will include provisions for efficacy stopping only (no pre-specified futility stopping)
#' We will use the rpact package to compute the stopping/success thresholds at the interim and final analysis
#' NOTE: Required packages: rpact and stringr
#' @param nSims # The number of simulations, the default is 1000
#' @param nPatients # here is where you specify the planned max number of patients you want included in each RCT
#' @param death1 # here is where you specify the event rate for patients receiving 'treatment 1' in these trial
#' @param death2 # here is where you specify the event rate for patients receiving 'treatment 2' in these trials
#' @param nLooks # here is where you put the number of looks that will take place (INCLUDING the final analysis)
#' @param analyses_scheduled # schedule of interim analyses
#' @param sided # Whether the test is 1-sided or 2-sided
#' @param alpha # Specified alpha level, the default is 0.05
#' @param informationRates #
#' @param trials # The total number of trials you wish to load in the table results.
#' @param typeOfDesign # The type of design.
#' @param seed # Argument to set the seed for the simulations
#' @return list of dataframes and plots
#' @example See below this code block
interim_sim <- function(nPatients = 1000, death1 = 0.4, death2 = 0.3,
                        nLooks = 4, analyses_scheduled = c(0.25, 0.50, 0.75, 1),
                        sided = 1, alpha = 0.05,
                        informationRates = analyses_scheduled,
                        typeOfDesign = "asOF", nSims = 1000, trials = 10,
                        seed = 1031) {

efficacy_thresholds <- numeric(nLooks)
design <- getDesignGroupSequential(
  sided = sided, alpha = alpha,
  informationRates = analyses_scheduled,
  typeOfDesign = typeOfDesign)
design_2 <- getDesignGroupSequential(typeOfDesign = "P") # Pocock
design_3 <- getDesignGroupSequential(typeOfDesign = "asP") # Alpha-spending Pocock
design_4 <- getDesignGroupSequential(typeOfDesign = "OF") # O'Brien-Fleming
designSet <- getDesignSet(designs = c(design, design_2,
                                      design_3, design_4),
                          variedParameters = "typeOfDesign")

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed(seed)
for (j in 1:nLooks) {
  efficacy_thresholds[j] <- design$stageLevels[j]
}

analyses_nPatients <- analyses_scheduled * nPatients
efficacy_thresholds
pb <- txtProgressBar(min = 0, max = nSims, initial = 0, style = 3)
trialnum <- numeric(nSims)
or <- data.frame(matrix(ncol = nLooks, nrow = nSims))
lcl <- data.frame(matrix(ncol = nLooks, nrow = nSims))
ucl <- data.frame(matrix(ncol = nLooks, nrow = nSims))
pval <- data.frame(matrix(ncol = nLooks, nrow = nSims))
success <- data.frame(matrix(ncol = nLooks, nrow = nSims))

strings <- c("OR_%d", "LCL_%d", "UCL_%d",
             "Pval_%d", "Success_%d")

colnames(or) <- sprintf("OR_%d", (1:nLooks))
colnames(lcl) <- sprintf("LCL_%d", (1:nLooks))
colnames(ucl) <- sprintf("UCL_%d", (1:nLooks))
colnames(pval) <- sprintf("Pval_%d", (1:nLooks))
colnames(success) <- sprintf("Success_%d", (1:nLooks))
overall_success <- numeric(nSims)
df <- data.frame(trialnum, or, lcl, ucl,
                 pval, success, overall_success)

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed(seed)
time <- system.time(
for (i in 1:nSims) {
  trialnum[i] <- i
  pid <- seq(1, nPatients, by = 1)
  treatment <- rep(1:2, nPatients / 2)
  deathprob <- numeric(nPatients)
  deathprob[treatment == 1] <- death1
  deathprob[treatment == 2] <- death2
  death <- rbinom(nPatients, 1, deathprob)
  trialdata <- data.frame(cbind(pid, treatment, death))

  for (j in 1:nLooks) {
    analysisdata <- subset(trialdata, pid <= analyses_nPatients[j])
    model <- glm(death ~ treatment,
                 family = binomial(link = "logit"),
                 data = analysisdata)
    or[i, j] <- exp(summary(model)$coefficients[2])
    lcl[i, j] <- exp(confint.default((model))[2, 1])
    ucl[i, j] <- exp(confint.default((model))[2, 2])
    pval[i, j] <- summary(model)$coefficients[8]
    success[i, j] <- ifelse(or[i, j] < 1 & pval[i, j] < efficacy_thresholds[j], 1, 0)
  }
  overall_success[i] <- 0
  for (j in 1:nLooks) {
    if (success[i, j] == 1) {
      overall_success[i] <- 1
    }
  }
    setTxtProgressBar(pb, i)
})

df <- data.frame(trialnum, or, lcl, ucl, pval,
                 success, overall_success)
simulation_results <- data.frame(matrix(vector(),
  nrow = nPatients, ncol = (length(df))))
colnames(simulation_results) <- c("trialnum", (do.call(
  rbind,
  lapply(1:length(strings),
    FUN = function(j) {
      (do.call(rbind, lapply(1:nLooks,
        FUN = function(i) ((sprintf(strings, i)))
      )[]))[, j]
    }
  )
)), "overall_success")
simulation_results[intersect(names(df), names(simulation_results))] <- (df[intersect(
  names(df),
  names(simulation_results)
)])
simulation_results <- as.data.frame(simulation_results)

outputs <- simulation_results[, c(-1, -length(simulation_results))]

cols <- as.character(1:nLooks)
rows <- as.character(1:length(strings))
interim <- matrix(
  nrow = length(strings), ncol = nLooks,
  dimnames = list((rows), (cols)),
  data = rep(0, (length(strings) * nLooks)))

for (i in cols) {
  interim[, i] <- str_subset(colnames(outputs), i)
}

colnames(interim) <- sprintf("Interim_Look_%d", (1:nLooks))
colnames(simulation_results)[1] <- c("TrialNum")
colnames(simulation_results)[length(simulation_results)] <- c("Overall_Success")
simulation_results_trials <- head(simulation_results, trials)
results <- list(
  summary(design),
  summary(time),
  plot(design, 1) +
    theme_light() +
    theme(
      plot.title = element_text(size = 14, face = "bold"),
      axis.title = element_text(size = 12, face = "bold")
    ),
  plot(designSet, type = 1) +
    theme_light() +
    theme(
      plot.title = element_text(size = 14, face = "bold"),
      axis.title = element_text(size = 12, face = "bold")
    ),
  table(overall_success),
  simulation_results_trials)

names(results) <- c(
  "Design Summary", "Time to complete simulation",
  "Main Design Plot", "Plot of Various Designs",
  "Table of Overall Success", "Simulation Results")

return(results)
}

A Simulated Example

results <- interim_sim(nPatients = 1000, death1 = 0.4, death2 = 0.3,
                       nLooks = 4, analyses_scheduled = c(0.25, 0.50, 0.75, 1),
                       sided = 1, alpha = 0.025, typeOfDesign = "asOF",
                       informationRates =  c(0.25, 0.50, 0.75, 1),
                       nSims = 1000, trials = 20, seed = 1031)

Examining the Results

results[1:5]
#> $`Design Summary`
#> 
#> $`Time to complete simulation`
#>    user  system elapsed 
#>   5.865   0.703   6.744 
#> 
#> $`Main Design Plot`
#> NULL
#> 
#> $`Plot of Various Designs`
#> NULL
#> 
#> $`Table of Overall Success`
#> overall_success
#>   0   1 
#> 134 866

table_results <- results[[6]]

table_results[ , 2:21] <- round((table_results[ , 2:21]), 3)

table_results$TrialNum <- row.names(table_results)
row.names(table_results) <- NULL

for (i in colnames(table_results)){
  table_results[,i] <- cell_spec(table_results[,i], color = "#777")
}

table_results <- table_results[c(colnames(table_results))]

To examine the results, I used the kableExtra package, though this is not necessary, and simply using the following script will suffice

table_results <- results[[6]]
View(table_results)

Statistical Environment

The analyses were run on:

#> ─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.5.2 (2025-10-31)
#>  os       macOS Tahoe 26.3
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  C.UTF-8
#>  ctype    C.UTF-8
#>  tz       America/New_York
#>  date     2026-01-07
#>  pandoc   3.8.3 @ /opt/homebrew/bin/ (via rmarkdown)
#>  quarto   1.8.26 @ /Applications/quarto/bin/quarto
#> 
#> ─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  ! package           * version    date (UTC) lib source
#>    abind               1.4-8      2024-09-12 [1] CRAN (R 4.5.0)
#>    Amelia            * 1.8.3      2024-11-08 [1] CRAN (R 4.5.0)
#>    arm                 1.14-4     2024-04-01 [1] CRAN (R 4.5.0)
#>    arrayhelpers        1.1-0      2020-02-04 [1] CRAN (R 4.5.0)
#>    askpass             1.2.1      2024-10-04 [1] CRAN (R 4.5.0)
#>    backports           1.5.0      2024-05-23 [1] CRAN (R 4.5.0)
#>    base              * 4.5.2      2025-11-01 [?] local
#>    base64enc           0.1-3      2015-07-28 [1] CRAN (R 4.5.0)
#>    bayesplot         * 1.15.0     2025-12-12 [1] CRAN (R 4.5.2)
#>    bcaboot             0.2-3      2021-05-09 [1] CRAN (R 4.5.0)
#>    bitops              1.0-9      2024-10-03 [1] CRAN (R 4.5.0)
#>    blogdown          * 1.22.2     2026-01-06 [1] Github (rstudio/blogdown@18ea707)
#>    boot              * 1.3-32     2025-08-29 [1] CRAN (R 4.5.0)
#>    bootImpute        * 1.3.0      2025-12-15 [1] CRAN (R 4.5.2)
#>    bridgesampling      1.2-1      2025-11-19 [1] CRAN (R 4.5.2)
#>    brms              * 2.23.0     2025-09-09 [1] CRAN (R 4.5.0)
#>    Brobdingnag         1.2-9      2022-10-19 [1] CRAN (R 4.5.0)
#>    broom             * 1.0.11     2025-12-04 [1] CRAN (R 4.5.2)
#>    broom.mixed       * 0.2.9.6    2024-10-15 [1] CRAN (R 4.5.0)
#>    Cairo             * 1.7-0      2025-10-29 [1] CRAN (R 4.5.0)
#>    car               * 3.1-3      2024-09-27 [1] CRAN (R 4.5.0)
#>    carData           * 3.0-5      2022-01-06 [1] CRAN (R 4.5.0)
#>    caTools             1.18.3     2024-09-04 [1] CRAN (R 4.5.0)
#>    checkmate         * 2.3.3      2025-08-18 [1] CRAN (R 4.5.0)
#>    class               7.3-23     2025-01-01 [1] CRAN (R 4.5.0)
#>    cli               * 3.6.5      2025-04-23 [1] CRAN (R 4.5.0)
#>    clipr               0.8.0      2022-02-22 [1] CRAN (R 4.5.0)
#>    cluster             2.1.8.1    2025-03-12 [1] CRAN (R 4.5.0)
#>    coda              * 0.19-4.1   2024-01-31 [1] CRAN (R 4.5.0)
#>    codetools           0.2-20     2024-03-31 [1] CRAN (R 4.5.0)
#>    colorspace        * 2.1-2      2025-09-22 [1] CRAN (R 4.5.0)
#>  P compiler            4.5.2      2025-11-01 [1] local
#>    concurve          * 2.7.7      2026-01-04 [1] Github (zadrafi/concurve@088c0ca)
#>    cowplot           * 1.2.0      2025-07-07 [1] CRAN (R 4.5.0)
#>    crayon              1.5.3      2024-06-20 [1] CRAN (R 4.5.0)
#>    curl                7.0.0      2025-08-19 [1] CRAN (R 4.5.0)
#>    data.table          1.18.0     2025-12-24 [1] CRAN (R 4.5.2)
#>  P datasets          * 4.5.2      2025-11-01 [1] local
#>    DBI                 1.2.3      2024-06-02 [1] CRAN (R 4.5.0)
#>    DEoptimR            1.1-4      2025-07-27 [1] CRAN (R 4.5.0)
#>    desc                1.4.3      2023-12-10 [1] CRAN (R 4.5.0)
#>    details             0.4.0      2025-02-09 [1] CRAN (R 4.5.0)
#>    dichromat           2.0-0.1    2022-05-02 [1] CRAN (R 4.5.0)
#>    digest              0.6.39     2025-11-19 [1] CRAN (R 4.5.2)
#>    distributional      0.5.0      2024-09-17 [1] CRAN (R 4.5.0)
#>    doParallel        * 1.0.17     2022-02-07 [1] CRAN (R 4.5.0)
#>    doRNG               1.8.6.2    2025-04-02 [1] CRAN (R 4.5.0)
#>    dplyr             * 1.1.4      2023-11-17 [1] CRAN (R 4.5.0)
#>    e1071               1.7-17     2025-12-18 [1] CRAN (R 4.5.2)
#>    emmeans             2.0.1      2025-12-16 [1] CRAN (R 4.5.2)
#>    estimability        1.5.1      2024-05-12 [1] CRAN (R 4.5.0)
#>    evaluate            1.0.5      2025-08-27 [1] CRAN (R 4.5.0)
#>    extremevalues       2.4.1      2024-12-17 [1] CRAN (R 4.5.0)
#>    farver              2.1.2      2024-05-13 [1] CRAN (R 4.5.0)
#>    fastmap             1.2.0      2024-05-15 [1] CRAN (R 4.5.0)
#>    flextable           0.9.10     2025-08-24 [1] CRAN (R 4.5.0)
#>    fontBitstreamVera   0.1.1      2017-02-01 [1] CRAN (R 4.5.0)
#>    fontLiberation      0.1.0      2016-10-15 [1] CRAN (R 4.5.0)
#>    fontquiver          0.2.1      2017-02-01 [1] CRAN (R 4.5.0)
#>    forcats           * 1.0.1      2025-09-25 [1] CRAN (R 4.5.0)
#>    foreach           * 1.5.2      2022-02-02 [1] CRAN (R 4.5.0)
#>    foreign             0.8-90     2025-03-31 [1] CRAN (R 4.5.0)
#>    Formula             1.2-5      2023-02-24 [1] CRAN (R 4.5.0)
#>    fs                  1.6.6      2025-04-12 [1] CRAN (R 4.5.0)
#>    furrr               0.3.1      2022-08-15 [1] CRAN (R 4.5.0)
#>    future            * 1.68.0     2025-11-17 [1] CRAN (R 4.5.2)
#>    future.apply      * 1.20.1     2025-12-09 [1] CRAN (R 4.5.2)
#>    gamlss            * 5.5-0      2025-08-19 [1] CRAN (R 4.5.0)
#>    gamlss.data       * 6.0-7      2025-09-04 [1] CRAN (R 4.5.0)
#>    gamlss.dist       * 6.1-1      2023-08-23 [1] CRAN (R 4.5.0)
#>    gdtools             0.4.4      2025-10-06 [1] CRAN (R 4.5.0)
#>    generics            0.1.4      2025-05-09 [1] CRAN (R 4.5.0)
#>    ggcorrplot        * 0.1.4.1    2023-09-05 [1] CRAN (R 4.5.0)
#>    ggdist              3.3.3      2025-04-23 [1] CRAN (R 4.5.0)
#>    ggplot2           * 4.0.1      2025-11-14 [1] CRAN (R 4.5.2)
#>    ggpubr              0.6.2      2025-10-17 [1] CRAN (R 4.5.0)
#>    ggsignif            0.6.4      2022-10-13 [1] CRAN (R 4.5.0)
#>    ggtext            * 0.1.2      2022-09-16 [1] CRAN (R 4.5.0)
#>    glmnet              4.1-10     2025-07-17 [1] CRAN (R 4.5.0)
#>    globals             0.18.0     2025-05-08 [1] CRAN (R 4.5.0)
#>    glue                1.8.0      2024-09-30 [1] CRAN (R 4.5.0)
#>  P graphics          * 4.5.2      2025-11-01 [1] local
#>  P grDevices         * 4.5.2      2025-11-01 [1] local
#>  P grid              * 4.5.2      2025-11-01 [1] local
#>    gridExtra           2.3        2017-09-09 [1] CRAN (R 4.5.0)
#>    gridtext            0.1.5      2022-09-16 [1] CRAN (R 4.5.0)
#>    gtable              0.3.6      2024-10-25 [1] CRAN (R 4.5.0)
#>    gtsummary         * 2.5.0      2025-12-05 [1] CRAN (R 4.5.2)
#>    here              * 1.0.2      2025-09-15 [1] CRAN (R 4.5.0)
#>    Hmisc             * 5.2-4      2025-10-05 [1] CRAN (R 4.5.0)
#>    hms                 1.1.4      2025-10-17 [1] CRAN (R 4.5.0)
#>    htmlTable           2.4.3      2024-07-21 [1] CRAN (R 4.5.0)
#>    htmltools         * 0.5.9      2025-12-04 [1] CRAN (R 4.5.2)
#>    htmlwidgets         1.6.4      2023-12-06 [1] CRAN (R 4.5.0)
#>    httr                1.4.7      2023-08-15 [1] CRAN (R 4.5.0)
#>    ImputeRobust      * 1.3-1      2018-11-30 [1] CRAN (R 4.5.0)
#>    inline              0.3.21     2025-01-09 [1] CRAN (R 4.5.0)
#>    insight             1.4.4      2025-12-06 [1] CRAN (R 4.5.2)
#>    iterators         * 1.0.14     2022-02-05 [1] CRAN (R 4.5.0)
#>    itertools           0.1-3      2014-03-12 [1] CRAN (R 4.5.0)
#>    jomo                2.7-6      2023-04-15 [1] CRAN (R 4.5.0)
#>    jsonlite            2.0.0      2025-03-27 [1] CRAN (R 4.5.0)
#>    kableExtra        * 1.4.0      2024-01-24 [1] CRAN (R 4.5.0)
#>    km.ci               0.5-6      2022-04-06 [1] CRAN (R 4.5.0)
#>    KMsurv              0.1-6      2025-05-20 [1] CRAN (R 4.5.0)
#>    knitr             * 1.51       2025-12-20 [1] CRAN (R 4.5.2)
#>    laeken              0.5.3      2024-01-25 [1] CRAN (R 4.5.0)
#>    latex2exp         * 0.9.6      2022-11-28 [1] CRAN (R 4.5.0)
#>    lattice           * 0.22-7     2025-04-02 [1] CRAN (R 4.5.0)
#>    lifecycle           1.0.4      2023-11-07 [1] CRAN (R 4.5.0)
#>    listenv             0.10.0     2025-11-02 [1] CRAN (R 4.5.0)
#>    lme4                1.1-38     2025-12-02 [1] CRAN (R 4.5.2)
#>    lmtest              0.9-40     2022-03-21 [1] CRAN (R 4.5.0)
#>    loo               * 2.9.0      2025-12-23 [1] CRAN (R 4.5.2)
#>    lubridate         * 1.9.4      2024-12-08 [1] CRAN (R 4.5.0)
#>    magick              2.9.0      2025-09-08 [1] CRAN (R 4.5.0)
#>    magrittr          * 2.0.4      2025-09-12 [1] CRAN (R 4.5.0)
#>    MASS              * 7.3-65     2025-02-28 [1] CRAN (R 4.5.0)
#>    mathjaxr            2.0-0      2025-12-01 [1] CRAN (R 4.5.2)
#>    Matrix            * 1.7-4      2025-08-28 [1] CRAN (R 4.5.0)
#>    MatrixModels        0.5-4      2025-03-26 [1] CRAN (R 4.5.0)
#>    matrixStats         1.5.0      2025-01-07 [1] CRAN (R 4.5.0)
#>    mcmc                0.9-8      2023-11-16 [1] CRAN (R 4.5.0)
#>    MCMCpack          * 1.7-1      2024-08-27 [1] CRAN (R 4.5.0)
#>    metadat             1.4-0      2025-02-04 [1] CRAN (R 4.5.0)
#>    metafor             4.8-0      2025-01-28 [1] CRAN (R 4.5.0)
#>  P methods           * 4.5.2      2025-11-01 [1] local
#>    mgcv              * 1.9-4      2025-11-07 [1] CRAN (R 4.5.0)
#>    mi                * 1.2        2025-09-02 [1] CRAN (R 4.5.0)
#>    mice              * 3.19.0     2025-12-10 [1] CRAN (R 4.5.2)
#>    miceadds          * 3.18-36    2025-09-12 [1] CRAN (R 4.5.0)
#>    miceFast          * 0.8.5      2025-02-03 [1] CRAN (R 4.5.0)
#>    minqa               1.2.8      2024-08-17 [1] CRAN (R 4.5.0)
#>    missForest        * 1.6.1      2025-10-26 [1] CRAN (R 4.5.0)
#>    mitml             * 0.4-5      2023-03-08 [1] CRAN (R 4.5.0)
#>    mitools             2.4        2019-04-26 [1] CRAN (R 4.5.0)
#>    multcomp            1.4-29     2025-10-20 [1] CRAN (R 4.5.0)
#>    mvtnorm           * 1.3-3      2025-01-10 [1] CRAN (R 4.5.0)
#>    nlme              * 3.1-168    2025-03-31 [1] CRAN (R 4.5.0)
#>    nloptr              2.2.1      2025-03-17 [1] CRAN (R 4.5.0)
#>    nnet                7.3-20     2025-01-01 [1] CRAN (R 4.5.0)
#>    numDeriv            2016.8-1.1 2019-06-06 [1] CRAN (R 4.5.0)
#>    officer             0.7.2      2025-12-04 [1] CRAN (R 4.5.2)
#>    opdisDownsampling   1.0.1      2024-04-15 [1] CRAN (R 4.5.0)
#>    openssl             2.3.4      2025-09-30 [1] CRAN (R 4.5.0)
#>    otel                0.2.0      2025-08-29 [1] CRAN (R 4.5.0)
#>    pan                 1.9        2023-12-07 [1] CRAN (R 4.5.0)
#>    pander              0.6.6      2025-03-01 [1] CRAN (R 4.5.0)
#>  P parallel          * 4.5.2      2025-11-01 [1] local
#>    parallelly        * 1.46.0     2025-12-12 [1] CRAN (R 4.5.2)
#>    pbmcapply         * 1.5.1      2022-04-28 [1] CRAN (R 4.5.0)
#>    performance       * 0.15.3     2025-12-01 [1] CRAN (R 4.5.2)
#>    pillar              1.11.1     2025-09-17 [1] CRAN (R 4.5.0)
#>    pkgbuild            1.4.8      2025-05-26 [1] CRAN (R 4.5.0)
#>    pkgconfig           2.0.3      2019-09-22 [1] CRAN (R 4.5.0)
#>    plyr                1.8.9      2023-10-02 [1] CRAN (R 4.5.0)
#>    png                 0.1-8      2022-11-29 [1] CRAN (R 4.5.0)
#>    polspline           1.1.25     2024-05-10 [1] CRAN (R 4.5.0)
#>    posterior         * 1.6.1      2025-02-27 [1] CRAN (R 4.5.0)
#>    pracma              2.4.6      2025-10-22 [1] CRAN (R 4.5.0)
#>    prettyunits         1.2.0      2023-09-24 [1] CRAN (R 4.5.0)
#>    ProfileLikelihood * 1.3        2023-08-25 [1] CRAN (R 4.5.0)
#>    progress          * 1.2.3      2023-12-06 [1] CRAN (R 4.5.0)
#>    proxy               0.4-29     2025-12-29 [1] CRAN (R 4.5.2)
#>    pryr                0.1.6      2023-01-17 [1] CRAN (R 4.5.0)
#>    purrr             * 1.2.0      2025-11-04 [1] CRAN (R 4.5.0)
#>    qqconf              1.3.2      2023-04-14 [1] CRAN (R 4.5.0)
#>    qqplotr           * 0.0.7      2025-09-05 [1] CRAN (R 4.5.0)
#>    quantreg          * 6.1        2025-03-10 [1] CRAN (R 4.5.0)
#>    QuickJSR            1.8.1      2025-09-20 [1] CRAN (R 4.5.0)
#>    R6                  2.6.1      2025-02-15 [1] CRAN (R 4.5.0)
#>    ragg                1.5.0      2025-09-02 [1] CRAN (R 4.5.0)
#>    randomForest      * 4.7-1.2    2024-09-22 [1] CRAN (R 4.5.0)
#>    ranger              0.17.0     2024-11-08 [1] CRAN (R 4.5.0)
#>    rappdirs            0.3.3      2021-01-31 [1] CRAN (R 4.5.0)
#>    rapportools         1.2        2025-02-28 [1] CRAN (R 4.5.0)
#>    rbibutils           2.4        2025-11-07 [1] CRAN (R 4.5.0)
#>    RColorBrewer        1.1-3      2022-04-03 [1] CRAN (R 4.5.0)
#>    Rcpp              * 1.1.0      2025-07-02 [1] CRAN (R 4.5.0)
#>    RcppParallel        5.1.11-1   2025-08-27 [1] CRAN (R 4.5.0)
#>    Rdpack              2.6.4      2025-04-09 [1] CRAN (R 4.5.0)
#>    readr             * 2.1.6      2025-11-14 [1] CRAN (R 4.5.2)
#>    reformulas          0.4.3      2025-12-17 [1] CRAN (R 4.5.2)
#>    rematch2            2.1.2      2020-05-01 [1] CRAN (R 4.5.0)
#>    reshape2          * 1.4.5      2025-11-12 [1] CRAN (R 4.5.0)
#>    reticulate        * 1.44.1     2025-11-14 [1] CRAN (R 4.5.2)
#>    rlang               1.1.6      2025-04-11 [1] CRAN (R 4.5.0)
#>    rmarkdown         * 2.30       2025-09-28 [1] CRAN (R 4.5.0)
#>    rms               * 8.1-0      2025-10-14 [1] CRAN (R 4.5.0)
#>    rngtools            1.5.2      2021-09-20 [1] CRAN (R 4.5.0)
#>    robustbase          0.99-6     2025-09-04 [1] CRAN (R 4.5.0)
#>    rpact             * 4.3.0      2025-12-16 [1] CRAN (R 4.5.2)
#>    rpart               4.1.24     2025-01-07 [1] CRAN (R 4.5.0)
#>    rprojroot           2.1.1      2025-08-26 [1] CRAN (R 4.5.0)
#>    rstan             * 2.32.7     2025-03-10 [1] CRAN (R 4.5.0)
#>    rstantools          2.5.0      2025-09-01 [1] CRAN (R 4.5.0)
#>    rstatix             0.7.3      2025-10-18 [1] CRAN (R 4.5.0)
#>    rstudioapi          0.17.1     2024-10-22 [1] CRAN (R 4.5.0)
#>    S7                  0.2.1      2025-11-14 [1] CRAN (R 4.5.2)
#>    sandwich            3.1-1      2024-09-15 [1] CRAN (R 4.5.0)
#>    scales              1.4.0      2025-04-24 [1] CRAN (R 4.5.0)
#>    sessioninfo         1.2.3      2025-02-05 [1] CRAN (R 4.5.0)
#>    shape               1.4.6.1    2024-02-23 [1] CRAN (R 4.5.0)
#>    showtext          * 0.9-7      2024-03-02 [1] CRAN (R 4.5.0)
#>    showtextdb        * 3.0        2020-06-04 [1] CRAN (R 4.5.0)
#>    sp                  2.2-0      2025-02-01 [1] CRAN (R 4.5.0)
#>    SparseM           * 1.84-2     2024-07-17 [1] CRAN (R 4.5.0)
#>  P splines           * 4.5.2      2025-11-01 [1] local
#>    StanHeaders       * 2.32.10    2024-07-15 [1] CRAN (R 4.5.0)
#>    Statamarkdown     * 0.9.6      2026-01-06 [1] Github (Hemken/Statamarkdown@dc936d8)
#>  P stats             * 4.5.2      2025-11-01 [1] local
#>  P stats4            * 4.5.2      2025-11-01 [1] local
#>    stringi             1.8.7      2025-03-27 [1] CRAN (R 4.5.0)
#>    stringr           * 1.6.0      2025-11-04 [1] CRAN (R 4.5.0)
#>    summarytools      * 1.1.4      2025-04-29 [1] CRAN (R 4.5.0)
#>    survival            3.8-3      2024-12-17 [1] CRAN (R 4.5.0)
#>    survminer           0.5.1      2025-09-02 [1] CRAN (R 4.5.0)
#>    survMisc            0.5.6      2022-04-07 [1] CRAN (R 4.5.0)
#>    svglite           * 2.2.2      2025-10-21 [1] CRAN (R 4.5.0)
#>    svgPanZoom          0.3.4      2020-02-15 [1] CRAN (R 4.5.0)
#>    svUnit              1.0.8      2025-08-26 [1] CRAN (R 4.5.0)
#>    sysfonts          * 0.8.9      2024-03-02 [1] CRAN (R 4.5.0)
#>    systemfonts         1.3.1      2025-10-01 [1] CRAN (R 4.5.0)
#>  P tcltk               4.5.2      2025-11-01 [1] local
#>    tensorA             0.36.2.1   2023-12-13 [1] CRAN (R 4.5.0)
#>    texPreview        * 2.1.0      2024-01-24 [1] CRAN (R 4.5.0)
#>    textshaping         1.0.4      2025-10-10 [1] CRAN (R 4.5.0)
#>    TH.data             1.1-5      2025-11-17 [1] CRAN (R 4.5.2)
#>    tibble            * 3.3.0      2025-06-08 [1] CRAN (R 4.5.0)
#>    tidybayes         * 3.0.7      2024-09-15 [1] CRAN (R 4.5.0)
#>    tidyr             * 1.3.2      2025-12-19 [1] CRAN (R 4.5.2)
#>    tidyselect          1.2.1      2024-03-11 [1] CRAN (R 4.5.0)
#>    tidyverse         * 2.0.0      2023-02-22 [1] CRAN (R 4.5.0)
#>    timechange          0.3.0      2024-01-18 [1] CRAN (R 4.5.0)
#>    tinytex           * 0.58       2025-11-19 [1] CRAN (R 4.5.2)
#>  P tools               4.5.2      2025-11-01 [1] local
#>    twosamples          2.0.1      2023-06-23 [1] CRAN (R 4.5.0)
#>    tzdb                0.5.0      2025-03-15 [1] CRAN (R 4.5.0)
#>  P utils             * 4.5.2      2025-11-01 [1] local
#>    uuid                1.2-1      2024-07-29 [1] CRAN (R 4.5.0)
#>    V8                  8.0.1      2025-10-10 [1] CRAN (R 4.5.0)
#>    vcd                 1.4-13     2024-09-16 [1] CRAN (R 4.5.0)
#>    vctrs               0.6.5      2023-12-01 [1] CRAN (R 4.5.0)
#>    VIM               * 6.2.6      2025-09-18 [1] CRAN (R 4.5.0)
#>    viridisLite         0.4.2      2023-05-02 [1] CRAN (R 4.5.0)
#>    wesanderson       * 0.3.7      2023-10-31 [1] CRAN (R 4.5.0)
#>    whisker             0.4.1      2022-12-05 [1] CRAN (R 4.5.0)
#>    withr               3.0.2      2024-10-28 [1] CRAN (R 4.5.0)
#>    xfun              * 0.55       2025-12-16 [1] CRAN (R 4.5.2)
#>    xml2                1.5.1      2025-12-01 [1] CRAN (R 4.5.2)
#>    xtable              1.8-4      2019-04-21 [1] CRAN (R 4.5.0)
#>    yaml                2.3.12     2025-12-10 [1] CRAN (R 4.5.2)
#>    yardstick         * 1.3.2      2025-01-22 [1] CRAN (R 4.5.0)
#>    zip                 2.3.3      2025-05-13 [1] CRAN (R 4.5.0)
#>    zoo                 1.8-15     2025-12-15 [1] CRAN (R 4.5.2)
#> 
#>  [1] /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/library
#> 
#>  * ── Packages attached to the search path.
#>  P ── Loaded and on-disk path mismatch.
#> 
#> ─ External software ────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting        value
#>  cairo          1.17.6
#>  cairoFT
#>  pango          1.50.14
#>  png            1.6.44
#>  jpeg           9.5
#>  tiff           LIBTIFF, Version 4.5.0
#>  tcl            8.6.13
#>  curl           8.7.1
#>  zlib           1.2.12
#>  bzlib          1.0.8, 13-Jul-2019
#>  xz             5.6.3
#>  deflate        1.23
#>  PCRE           10.44 2024-06-07
#>  ICU            76.1
#>  TRE            TRE 0.8.0 R_fixes (BSD)
#>  iconv          Apple or GNU libiconv 1.11 /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libR.dylib
#>  readline       5.2
#>  BLAS           /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
#>  lapack         /Library/Frameworks/R.framework/Versions/4.5-arm64/Resources/lib/libRlapack.dylib
#>  lapack_version 3.12.1
#> 
#> ─ Python configuration ─────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  Python is not available
#> 
#> ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> $RNGkind
#> [1] "L'Ecuyer-CMRG" "Inversion"     "Rejection"

Article Citation

¹

Valid -values Are Uniform Under the Null Model

Here we show that valid -values have specific properties, when the null model is true. We first generate two variables (, ) that come from the same normal distribution with a of 0 and of 1, each with a total of 1000 observations. We assume that there is no relationship between these two variables. We run a simple t-test between and and iterate this 100000 times and compute 100000 -values to see the overall distribution of the -values, which we then plot using a histogram./

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
n.sim <- 100
t.sim <- numeric(n.sim)
n.samp <- 1000

for (i in 1:n.sim) {
    X <- rnorm(n.samp, mean = 0, sd = 1)
    Y <- rnorm(n.samp, mean = 0, sd = 1)
    df <- data.frame(X, Y)
    t <- t.test(X, Y, data = df)
    t.sim[i] <- t[[3]]
}

ggplot(NULL, aes(x = t.sim)) + geom_histogram(bins = 30, col = "black",
    fill = "#99c7c7", alpha = 0.25) + labs(title = "Distribution of P-values Under the Null",
    x = "P-value") + scale_x_continuous(breaks = seq(0, 1, 0.1)) +
    theme_bw()

This can also be shown using the TeachingDemos R package, which has a function dedicated to showing this phenomenon.

library("TeachingDemos")
RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
obs_p <- Pvalue.norm.sim(n = 1000, mu = 0, mu0 = 0, sigma = 1,
    sigma0 = 1, test = "t", alternative = "two.sided", alpha = 0.05,
    B = 1e+05)

ggplot(NULL, aes(x = obs_p)) + geom_histogram(bins = 30, col = "black",
    fill = "#99c7c7", alpha = 0.25) + labs(title = "Distribution of P-values Under the Null",
    x = "P-value") + scale_x_continuous(breaks = seq(0, 1, 0.1)) +
    theme_bw()

As you can see, when the null model is true, the distribution of -values is uniform. Valid -values are uniform under the null hypothesis and their corresponding -values are exponentially distributed. We run the same simulation as before, but then convert the obtained -values into -values, to see how they are distributed.

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
n.sim <- 100
t.sim <- numeric(n.sim)
n.samp <- 1000

for (i in 1:n.sim) {
    X <- rnorm(n.samp, mean = 0, sd = 1)
    Y <- rnorm(n.samp, mean = 0, sd = 1)
    df <- data.frame(X, Y)
    t <- t.test(X, Y, data = df)
    t.sim[i] <- t[[3]]
}

ggplot(NULL, aes(x = -log2(t.sim))) + geom_histogram(bins = 30,
    col = "black", fill = "#d46c5b", alpha = 0.5) + labs(title = "Distribution of S-values Under the Null",
    x = "S-value (Bits of Information)") + theme_bw()

Posterior Predictive -values

Despite the name, posterior predictive -values are not considered valid frequentist -values because they do not meet the uniformity criterion, instead they are pulled towards the parameter value 0.5. For further discussion, see our technical supplement along with Greenland (2019) for comprehensive theoretical discussions.^{2, 3}

A quick excerpt from our main paper and the technical supplement explains why this is not the case,

As discussed in the Supplement, in Bayesian settings one may see certain -values that are not valid frequentist -values, the primary example being the posterior predictive -value⁴;⁵; unfortunately, the negative logs of such invalid -values do not measure surprisal at the statistic given the model, and so are not valid -values.

The decision rule “reject H if ” will reject H 100% of the time under sampling from a model M obeying H (i.e., the Type-1 error rate of the test will be ) provided the random variable corresponding to is valid (uniform under the model M used to compute it), but not necessarily otherwise⁶. This is one reason why frequentist writers reject invalid -values (such as posterior predictive -values, which highly concentrate around 0.50) and devote considerable technical coverage to uniform -values⁶;⁵;⁴. A valid -value (“U-value”) translates into an exponentially distributed -value with a mean of 1 nat or bits where is the base of the natural logs.

Uniformity is also central to the “refutational information” interpretation of the -value used here, for it is necessary to ensure that the -value from which is derived is in fact the percentile of the observed value of the test statistic in the distribution of the statistic under M, thus making small surprising under M and making the corresponding degree of surprise. Because posterior predictive -values do not translate into sampling percentiles of the statistic under the hypothesis (in fact, they are pulled toward 0.5 from the correct percentiles)⁵;⁴, the resulting negative log does not measure surprisal at the statistic given M, and so is not a valid -value in our terms.

And indeed, we can show this phenomenon below with simulations. Here we fit a simple Bayesian regression model with a weakly informative prior normal(0, 10) using rstanarm, where both the predictor and response variable come from the same distribution and have the same location and scale parameters (, . We then calculate the observed test statistics, along with their distributions, and convert them into posterior predictive -values and plot them using Bayesplot functions. Then, we iterate this 1000 times to examine the distribution of posterior predictive -values and compare them to standard -values that are known to be uniform.

But first, we’ll generate the distribution of the test statistic from one model.

library("bayesplot")
library("rstan")
library("rstanarm")
rstan_options(auto_write = TRUE)
options(mc.cores = 4)
teal <- c("#99c7c7")
color_scheme_set("teal")
RNGkind(kind = "L'Ecuyer-CMRG")
n.samp <- 100
X <- rnorm(n.samp, mean = 0, sd = 1)
Y <- rnorm(n.samp, mean = 0, sd = 1)
df <- data.frame(X, Y)
mod1 <- stan_glm(Y ~ X, data = df, chains = 1, cores = 4, refresh = 0,
    prior = normal(0, 10))
#> Error in `if (!prior_dist_name %in% unlist(ok_dists)) ...`:
#> ! argument is of length zero
yrep <- posterior_predict(mod1)
#> Error:
#> ! object 'mod1' not found

h <- ppc_stat(Y, yrep, stat = "median", freq = TRUE, binwidth = 0.01) +
    labs(title = "Distribution of Posterior Test Statistic") +
    theme_bw() + yaxis_text(on = TRUE)
#> Error:
#> ! object 'yrep' not found

values_all <- h[["plot_env"]][["T_yrep"]]
#> Error:
#> ! object 'h' not found
prob_to_find <- h[["plot_env"]][["T_y"]]
#> Error:
#> ! object 'h' not found
quantInv <- function(distr, value) ecdf(distr)(value)
posterior_p_value <- quantInv(values_all, prob_to_find)
#> Error:
#> ! object 'values_all' not found

l <- ppc_stat_2d(Y, yrep) + theme_bw() + labs(title = "Distribution of Posterior Test Statistic")
#> Error:
#> ! object 'yrep' not found

plot(h)
#> Error in `h()`:
#> ! error in evaluating the argument 'x' in selecting a method for function 'plot': object 'h' not found
plot(l)
#> Error in `h()`:
#> ! error in evaluating the argument 'x' in selecting a method for function 'plot': object 'l' not found

print(prob_to_find)
#> Error in `h()`:
#> ! error in evaluating the argument 'x' in selecting a method for function 'print': object 'prob_to_find' not found

The above is the posterior predictive test statistic for one model, along with it’s distribution. Now let’s iterate this process, convert them into posterior predictive -values, and generate their distribution.

rstan_options(auto_write = TRUE)
options(mc.cores = 4)
RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
n.sim <- 100
ppp <- numeric(n.sim)
n.samp <- 1000

quantInv <- function(distr, value) ecdf(distr)(value)

for (i in 1:length(ppp)) {
    X <- rnorm(n.samp, mean = 0, sd = 1)
    Y <- rnorm(n.samp, mean = 0, sd = 1)
    df <- data.frame(X, Y)
    mod1 <- stan_glm(Y ~ X, data = df, chains = 1, cores = 4,
        iter = 1000, refresh = 0, prior = normal(0, 10))
    yrep <- posterior_predict(mod1)
    h <- ppc_stat(Y, yrep, stat = "median")
    values_all <- h[["plot_env"]][["T_yrep"]]
    prob_to_find <- h[["plot_env"]][["T_y"]]
    posterior_p_value <- quantInv(values_all, prob_to_find)
    ppp[i] <- posterior_p_value
}
#> Error in `if (!prior_dist_name %in% unlist(ok_dists)) ...`:
#> ! argument is of length zero

ggplot(NULL, aes(x = ppp)) + geom_histogram(bins = 30, col = "black",
    fill = "#99c7c7", alpha = 0.25) + labs(title = "Distribution of Posterior Predictive P-values",
    subtitle = "From Simulation of 1000 Simple Linear Regressions",
    x = "Posterior Predictive P-value") + scale_x_continuous(breaks = seq(0,
    1, 0.1)) + theme_bw()

We can also show this in Stata 16.1, using their new bayesstats ppvalues command which generates posterior predictive -values.

set obs 100

generate x = rnormal(0, 1)
generate y = rnormal(0, 1)

parallel initialize 8, f
bayesmh y x, likelihood(normal({var})) prior({var}, normal(0, 10))
prior({y:}, normal(0, 10)) rseed(1031) saving(coutput_pred, replace) mcmcsize(1000)

bayespredict (mean:@mean({_resid})) (var:@variance({_resid})),
rseed(1031) saving(coutput_pred, replace)
bayesstats ppvalues {mean} using coutput_pred

As we can see from the R plot above, the test statistics are are generally concentrated around 0.5 and the distribution (as shown above via the R code) hardly resembles the uniform distribution of -value under the null model. Further, the base-2 log transformation of posterior predictive -values is not exponentially distributed, making them invalid -values in our terms.

ggplot(NULL, aes(x = (-log2(ppp)))) + geom_histogram(bins = 30,
    col = "black", fill = "#d46c5b", alpha = 0.25) + labs(title = "Distribution of -log2(Posterior Predictive P-values)",
    subtitle = "S-values Produced From Posterior Predictive P-values",
    x = "-log2(Posterior Predictive P-value)") + theme_bw()
#> Error in `geom_histogram()`:
#> ! Problem while computing stat.
#> ℹ Error occurred in the 1st layer.
#> Caused by error in `seq_len()`:
#> ! argument must be coercible to non-negative integer

Table 1: Some -values, -values, Maximum Likelihood Ratios, and Likelihood Ratio Statistics

Here, we look at some common -values, and how they correspond to other information statistics and likelihood measures.

pvalue <- c(0.99, 0.9, 0.5, 0.25, 0.1, 0.05, 0.025, 0.01, 0.005,
    1e-04, paste("5 sigma (~ 2.9 in 10 million)"), paste("1 in 100 million (GWAS)"),
    paste("6 sigma (~ 1 in a billion)"))

svalue <- round(c(-log2(c(0.99, 0.9, 0.5, 0.25, 0.1, 0.05, 0.025,
    0.01, 0.005, 1e-04)), 21.7, 26.6, 29.9), 2)

pvalue[1:10] <- formatC(pvalue[1:10], format = "e", digits = 2)

mlr <- round(c(1, 1.01, 1.26, 1.94, 3.87, 6.83, 12.3, 27.6, 51.4,
    1935, (5.2 * (10^5)), (1.4 * (10^7)), (1.3 * (10^8))), 2)

# likelihood ratio statistic

lr <- round(c(0.00016, 0.016, 0.45, 1.32, 2.71, 3.84, 5.02, 6.63,
    7.88, 15.1, 26.3, 32.8, 37.4), 2)

table1 <- data.frame(pvalue, svalue, mlr, lr)

colnames(table1) <- c("P-value (compatibility)", "S-value (bits)",
    "Maximum Likelihood Ratio", "Deviance Statistic 2ln(MLR)")

kbl(table1, format = "html", padding = 45, longtable = TRUE,
    color = "#777") %>%
    row_spec(row = 0, color = "#777", bold = T) %>%
    column_spec(column = 1:4, color = "#777", bold = F) %>%
    kable_classic(full_width = TRUE, html_font = "Cambria", lightable_options = "basic",
        font_size = 17, fixed_thead = list(enabled = F)) %>%
    footnote(general_title = "Abbreviations: ", title_format = "underline",
        general = c("Table 1 $P$-values                  and binary $S$-values, with corresponding maximum-likelihood ratios (MLR) and deviance (likelihood-ratio) statistics for a simple test hypothesis H under background assumptions A"))

For further discussion of 5 and 6 sigma cutoffs, see⁷.

Frequentist Interval Estimate Percentiles

Refer to Long-Run Coverage

Here we simulate a study where one group with 100 participants has an of 100 with a of 20 and the second group has the same number of participants but an average of 80 and a standard deviation of 20. We compare them using a Welch’s -test and generate 95% compatibility intervals several times, specifically 100 times, and then plot them. Since we know the mean difference is 20, we wish to see how often the interval estimates cover this true parameter value.

This shows that the intervals tend to vary simply as a result of randomness, and that the proper interpretation of the attached percentile is about long run coverage of the true parameter. However, this does not preclude interpretation of a single interval estimate from a study.⁸ As we write in the Replace unrealistic “confidence” claims with compatibility measures section of our paper.¹

The fact that “confidence” refers to the procedure behavior, not the reported interval, seems to be lost on most researchers. Remarking on this subtlety, when Jerzy Neyman discussed his confidence concept in 1934 at a meeting of the Royal Statistical Society, Arthur Bowley replied, “I am not at all sure that the ‘confidence’ is not a confidence trick.”⁹. And indeed, 40 years later, Cox and Hinkley warned, “interval estimates cannot be taken as probability statements about parameters, and foremost is the interpretation ‘such and such parameter values are consistent with the data.’”⁸. Unfortunately, the word “consistency” is used for several other concepts in statistics, while in logic it refers to an absolute condition (of noncontradiction); thus, its use in place of “confidence” would risk further confusion.

To address the problems above, we exploit the fact that a 95% CI summarizes the results of varying the test hypothesis H over a range of parameter values, displaying all values for which > 0.05¹⁰ and hence < 4.32 bits³;¹¹. Thus the CI contains a range of parameter values that are more compatible with the data than are values outside the interval, under the background assumptions³;¹². Unconditionally (and thus even if the background assumptions are uncertain), the interval shows the values of the parameter which, when combined with the background assumptions, produce a test model that is “highly compatible” with the data in the sense of having less than 4.32 bits of information against it. We thus refer to CI as compatibility intervals rather than confidence intervals¹³;³;¹¹; their abbreviation remains “CI.” We reject calling these intervals “uncertainty intervals,” because they do not capture uncertainty about background assumptions¹³.

Indeed, this also has to do with our paper on deemphasizing the model assumptions that are often behind common statistical outputs, and why it is necessary to treat them as uncertain, rather than given.¹⁴ For example, a 95% interval estimate is assumed to be free of bias in its construction, however, it’s coverage claims are no longer so, once there are systematic errors in play. Many of these common, classical statistical procedures are designed to deal with random variation, and less so with bias (a relatively new field, with shiny new methods).

Brown et al. (2017) Reanalysis

Taken from the Brown et al. data.^{15, 16}

Here we take the reported statistics from the Brown et al. studies in order to run statistical tests of different alternative test hypotheses and use those results to construct various functions.

We calculate the standard errors from the point estimate and the confidence (compatibility) limits.

se <- log(2.59/0.997)/3.92

logUL <- log(2.59)
logLL <- log(0.997)

logpoint <- log(1.61)

logpoint + (1.96 * se)
#> [1] 0.954
logpoint - (1.96 * se)
#> [1] -0.0011

testhypothesis <- c("Halving of hazard", "No effect (null)",
    "Point estimate", "Doubling of hazard", "Tripling of hazard",
    "Quintupling of hazard")

hazardratios <- c(0.5, 1, 1.61, 2, 3, 5)
pvals <- c(1.6e-06, 0.05, 1, 0.37, 0.01, 3.2e-06)
svals <- round(-log2(pvals), 2)
mlr2 <- c((1 * 10^5), 6.77, 1, 1.49, 26.2, (5 * 10^4))
lr <- round(c(exp((((log(0.5/1.61))/se)^2)/(2)), exp((((log(1/1.61))/se)^2)/(2)),
    exp((((log(1.61/1.61))/se)^2)/(2)), exp((((log(2/1.61))/se)^2)/(2)),
    exp((((log(3/1.61))/se)^2)/(2)), exp((((log(5/1.61))/se)^2)/(2))),
    3)

LR <- formatC(lr, format = "e", digits = 2)

table2 <- data.frame(testhypothesis, hazardratios, pvals, svals,
    mlr2, LR)

colnames(table2) <- c("Test Hypothesis", "HR", "P-values", "S-values",
    "MLR", "LR")

kbl(table2, format = "html", padding = 45, longtable = TRUE,
    color = "#777") %>%
    row_spec(row = 0, color = "#777", bold = T) %>%
    column_spec(column = 1:6, color = "#777", bold = F) %>%
    kable_classic(full_width = TRUE, html_font = "Cambria", lightable_options = "basic",
        font_size = 15, fixed_thead = list(enabled = F))

label <- c("Brown et al. (2017)\n JAMA", "Brown et al. (2017)\n J Clin Psychiatry")

point <- c(1.61, 1.7)
lower <- c(0.997, 1.1)
upper <- c(2.59, 2.6)

df <- data.frame(label, point, lower, upper)

ggplot(data = df, mapping = aes(x = label, y = point, ymin = lower,
    ymax = upper, group = label)) + geom_pointrange(color = "#007C7C",
    size = 1.5, alpha = 0.4) + geom_hline(yintercept = 1, lty = 3,
    color = "#d46c5b", alpha = 0.5) + coord_flip() + scale_y_log10(breaks = scales::pretty_breaks(n = 10),
    limits = c(0.8, 4)) + labs(title = "Association Between Serotonergic Antidepressant Exposure
               \nDuring Pregnancy and Child Autism Spectrum Disorder",
    subtitle = "Reported 95% Compatibility Intervals From Primary Results",
    x = "Study", y = "Hazard Ratio (JAMA) +
            Adjusted Pooled Odds Ratio (J Clin Psychiatry)") +
    annotate(geom = "text", x = 1.5, y = 1, size = 5, label = "Null parameter value of 1",
        color = "#d46c5b", alpha = 0.75) + annotate(geom = "text",
    x = 2.2, y = 1.8, size = 5, label = "Point Estimate = 1.61,
           Lower Limit = 0.997, Upper Limit = 2.59",
    color = "#000000", alpha = 0.75) + annotate(geom = "text",
    x = 1.2, y = 1.8, size = 5, label = "Point Estimate = 1.7,
           Lower Limit = 1.1, Upper Limit = 2.6",
    color = "#000000", alpha = 0.75) + theme_light() + theme(axis.text.y = element_text(angle = 90,
    hjust = 1, size = 13)) + theme(axis.text.x = element_text(size = 13)) +
    theme(title = element_text(size = 13))

library("concurve")

curve1 <- curve_rev(point = 1.61, LL = 0.997, UL = 2.59, measure = "ratio")

kbl(curve_table(curve1[[1]]), format = "html", padding = 45,
    longtable = TRUE, color = "#777", row.names = F, col.names = colnames(curve_table(curve1[[1]]))) %>%
    row_spec(row = 0, color = "#777", bold = T) %>%
    column_spec(column = 1:7, color = "#777", bold = F) %>%
    kable_classic(full_width = TRUE, html_font = "Cambria", lightable_options = "basic",
        font_size = 17, fixed_thead = list(enabled = F)) %>%
    footnote(general_title = c("Table of Statistics for Various Interval Estimate Percentiles"),
        general = " ")

ggcurve(curve1[[1]], type = "c", measure = "ratio", nullvalue = c(1),
    levels = c(0.5, 0.75, 0.95)) + labs(title = expression(paste(italic(P),
    "-value (Compatibility) Function")), subtitle = expression(paste(italic(P),
    "-values for a range of hazard ratios (HR)")), x = "Hazard Ratio (HR)",
    y = expression(paste(italic(P), "-value"))) + geom_vline(xintercept = 1.61,
    lty = 1, color = "gray", alpha = 0.2) + geom_vline(xintercept = 2.59,
    lty = 1, color = "gray", alpha = 0.2) + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11))

curve1 <- curve_rev(point = 1.61, LL = 0.997, UL = 2.59, measure = "ratio")

curve2 <- curve_rev(point = 1.7, LL = 1.1, UL = 2.6, measure = "ratio")

ggcurve(curve1[[2]], type = "cdf", measure = "ratio", nullvalue = c(1)) +
    labs(title = "P-values for a range of hazard ratios (HR)",
        subtitle = "Association Between Serotonergic Antidepressant Exposure
       \nDuring Pregnancy and Child Autism Spectrum Disorder",
        x = "Hazard Ratio (HR)", y = "Cumulative Compatibility") +
    theme(plot.title = element_text(size = 12), plot.subtitle = element_text(size = 11),
        axis.title.x = element_text(size = 12), axis.title.y = element_text(size = 12),
        text = element_text(size = 11))

lik2 <- curve_rev(point = 1.7, LL = 1.1, UL = 2.6, type = "l",
    measure = "ratio")

lik1 <- curve_rev(point = 1.61, LL = 0.997, UL = 2.59, type = "l",
    measure = "ratio")

plot_compare(data1 = lik1[[1]], data2 = lik2[[1]], type = "l1",
    measure = "ratio", nullvalue = TRUE, title = "Brown et al. 2017. J Clin Psychiatry. vs.
              \nBrown et al. 2017. JAMA.",
    subtitle = "J Clin Psychiatry: OR = 1.7, 1/6.83 LI: LL = 1.1, UL = 2.6
                         \nJAMA: HR = 1.61, 1/6.83 LI: LL = 0.997, UL = 2.59",
    xaxis = "Hazard Ratio / Odds Ratio") + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11))

plot_compare(data1 = curve1[[1]], data2 = curve2[[1]], type = "c",
    measure = "ratio", nullvalue = TRUE, title = "Brown et al. 2017. J Clin Psychiatry. vs.
              \nBrown et al. 2017. JAMA.",
    subtitle = "J Clin Psychiatry: OR = 1.7, 1/6.83 LI: LL = 1.1, UL = 2.6
              \nJAMA: HR = 1.61, 1/6.83 LI: LL = 0.997, UL = 2.59",
    xaxis = "Hazard Ratio / Odds Ratio") + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11))