Less Likely https://lesslikely.com/blog.html Statistical science, applied to medicine and nutrition — the quiet arithmetic of being wrong less often. https://lesslikely.com/images/Archives.jpg Less Likely https://lesslikely.com/blog.html quarto-1.10.18 Wed, 16 Sep 2026 04:00:00 GMT The P-value Is Not a Verdict: An Interactive Consonance Curve Zad Rafi https://lesslikely.com/statistics/consonance-explorer.html A single P-value reports one number about one hypothesis. The P-value function reports the same test statistic applied to every candidate parameter value at once — and it contains every compatibility interval at every level as horizontal slices.1 Nothing below is new mathematics. It is the same normal approximation you already use, drawn in full instead of evaluated at one point.

1 Slice the curve at height and read off the x-coordinates: those are the 95% limits. Slice at and you have the 99% limits. The interval was never a separate calculation.

The display is old. Poole called for it in 19871, Sullivan and Foster catalogued its uses in 19902, and it has sat in Modern Epidemiology ever since. What has changed is that it now costs nothing to render, and that we have a better vocabulary for the vertical axis: compatibility rather than confidence, and surprisal rather than significance3.

TipDefinitions, once

A P-value function (equivalently, a consonance or compatibility curve) plots, for every candidate parameter value , the P-value from testing against the data under the full set of analysis assumptions.

An S-value is the Shannon transform : the number of consecutive heads from a fair coin that would be exactly as surprising as the observed test statistic, if and every other assumption were correct.

A 95% compatibility interval is the set of whose P-value exceeds 0.05 — a horizontal slice through the curve, not a separate object.

The explorer

Move the sliders. Everything recomputes in the browser — no server, no round trip.

Calibrating bits

The S-value exists because “P = 0.05” carries no intuition and “significant” carries the wrong one. Bits do carry intuition, because everyone has a physical model of coin flipping. Drag μ above and watch the coins.

The S-value measures surprise at the whole model, not at alone. Every bit is also a bit against the assumed absence of uncontrolled confounding, selection effects, measurement error, and model form. A large S-value says the data are hard to reconcile with the conjunction of and everything else you assumed; it does not tell you which conjunct to abandon. In observational work, the “everything else” is usually the weaker link.

A worked example: one data set, four analyses

Brown and colleagues studied serotonergic antidepressant use in pregnancy and autism spectrum disorder in 35,906 Ontario births, of which 2,837 pregnancies (7.9%) were exposed4. They reported four hazard ratios from the same cohort under progressively stronger confounding control. This is the example reanalysed at length in3, and the working code for that paper lives here.

Look at the point estimates before you look at anything else.

Three of the four point estimates are 1.59, 1.60 and 1.61. By the conventional reading, one of them is a finding and two of them are nothing. What actually changed between them is not the estimate — it is the width of the interval, which grows from a ratio of 1.9 to 5.4 across the three adjusted analyses as each trades precision for stronger confounding control. The verdicts flip; the estimate does not move.

The published conclusion was that exposure “was not associated with” autism. That sentence is doing work the arithmetic will not support: the IPT-weighted analysis puts against the null, 4.31 bits of surprise, and its interval runs to 2.59.

Uncheck the crude analysis and the remaining three curves are near-concentric: same peak, different width. That is the shape of a precision problem, not a disagreement about effect size. Now test a value other than the null:

Set the slider to 1.5 and all four analyses agree: = 6.6, 0.5, 0.4 and 0.2 bits. Three of the four find a 50% increase in hazard less surprising than a single coin flip. The analysis that was “not significant” and the analysis that was “significant” are, against this hypothesis, saying nearly the same thing.

Testing HR = 1 is a legitimate question. Reporting only that test, and only its dichotomised outcome, discards the answer to every other question the same data address — including questions the reader may care about more, such as whether an effect large enough to change prescribing can be ruled out. On these data it cannot: the IPT-weighted 95% interval reaches 2.59, and even the 50% interval runs from 1.37 to 1.90.

What more data would do

Precision and parameter value are separate things that a single P-value fuses into one number. Hold the IPT-weighted estimate fixed and vary only the information: the standard error scales as when the sample size is multiplied by .

The dashed ghost is the study as published. Push to 4 and the null falls well outside the 95% interval without the estimate having moved a hair. Pull down to 0.3 and the same estimate becomes compatible with almost everything between 0.7 and 3.9.

This is worth sitting with, because it is the mechanism behind most “failure to replicate” reporting. A study that reports and a study that reports can have identical point estimates and differ only in how many people they enrolled. Neither number is a property of the effect.

ImportantThe one thing this widget cannot show

Scaling the standard error models random error only. Multiplying the sample size by 16 does nothing to confounding, selection, or measurement error — those biases do not shrink with , and past some sample size they dominate entirely. A very narrow curve centred in the wrong place is the most confident way to be wrong. If you want that quantified rather than assumed away, the tool is bias analysis, not more subjects.

Do two estimates conflict?

“Study A was significant, study B was not, so they disagree” is a comparison of two verdicts, not of two estimates. The comparison that answers the question is a curve in its own right: under independence, the log ratio-of-ratios has standard error , and the null of no difference sits at a ratio of 1.

Study A is the explorer at the top of this page. Set study B here.

With the defaults — A at 1.61 () and B at 1.05 () — the two studies land on opposite sides of the conventional threshold. Their ratio of ratios is 1.53, 95% interval 0.90 to 2.61, and the difference carries 3.1 bits of information: fewer than the 4.3 bits the 0.05 convention itself demands, and fewer than four coin flips. Two verdicts that disagree, one comparison that does not.

WarningThis does not apply to the four Brown analyses

The overlay in the previous section shows four analyses of the same subjects. Their estimates are strongly correlated, so badly overstates the standard error of their difference and the resulting P-value is not interpretable. Use this widget for genuinely independent studies. For nested analyses of one data set, the honest summary is the overlay itself: same peak, different width.

A P-value and an S-value are the same quantity on different scales, so a dual axis is technically defensible. It is still a bad idea: readers reliably misread which curve belongs to which axis, and the visual slope of the P-value curve near the tails is uninformative precisely where the S-value is most informative. Two stacked panels sharing an x-axis cost one extra inch of page and remove the ambiguity.

  • No shading of “significant” regions. The point of the curve is that there is no cliff at 0.05.
  • No default null at 1. The μ slider starts at 1 because that is the conventional test value, not because it is privileged.
  • No P-value below about 1e-300. Double precision runs out. The implementation stays accurate to roughly z = 37, which is far past the point where the normal approximation itself is the binding constraint.
  • No normality diagnostics. Every curve here assumes the log estimate is approximately normal with the reported standard error. That is the same assumption the published interval already made; the curve does not add it, it only makes it visible across the whole range.

Interactive table

The same idea applied to tabular results — searchable and sortable by clicking column headers, which is most of what makes a long results table usable. These are the Brown analyses again, with each row carrying its own P-value against a hypothesis you choose.

How to report this

Nothing above requires a new statistic, a new threshold, or a new software stack. It requires four sentences you can paste into a results section:

  1. Report the estimate and interval without the word “significant” — “HR 1.61, 95% compatibility interval 0.997 to 2.59”.
  2. Report the P-value as a number, not as an inequality against 0.05, and say what hypothesis it tests.
  3. Report at least one P-value for a non-null hypothesis that matters clinically or practically. If the data cannot distinguish the null from a 60% increase, that is the finding.
  4. Say that the interval and P-value are conditional on the entire model, not just on the parameter.

If a reviewer asks for the significance verdict back, the curve is the reply: it contains their verdict as one horizontal slice, plus every slice they did not ask for.

Reader infrastructure that is already on

Three things people associate with Gwern’s pages are Quarto defaults and need no code at all:

  • Citation popups. Hover any @citekey reference.
  • Footnote popups. Hover the marker on 2 above.
  • Cross-reference popups. Hover any @fig- or @tbl- reference.

2 The convention that 0.05 marks a boundary is worth pricing in bits: it is 4.32 bits of information against the tested hypothesis, or roughly the surprise of four consecutive heads. Few people would abandon a belief on four coin flips.

All three are controlled by citations-hover, footnotes-hover and crossrefs-hover, which default to true. The margin placement of footnotes here comes from reference-location: margin in this document’s front matter.

References

1. Poole C. (1987). “Beyond the confidence interval.” American Journal of Public Health. 77:195–199. doi: 10.2105/ajph.77.2.195.
2. Sullivan KM, Foster DA. (1990). “Use of the confidence interval function.” Epidemiology. 1:39–42. doi: 10.1097/00001648-199001000-00009.
3. Rafi Z, Greenland S. (2020). “Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise.” BMC Medical Research Methodology. 20:244. doi: 10.1186/s12874-020-01105-9.
4. Brown HK, Ray JG, Wilton AS, Lunsky Y, Gomes T, Vigod SN. (2017). “Association between serotonergic antidepressant use during pregnancy and autism spectrum disorder in children.” JAMA. 317:1544–1552. doi: 10.1001/jama.2017.3415.
Back to top ]]> statistics interactive https://lesslikely.com/statistics/consonance-explorer.html Wed, 16 Sep 2026 04:00:00 GMT Constructing Likelihood Functions in R, Stata, Python, Julia, MATLAB, and SQL Zad Rafi https://lesslikely.com/statistics/likelihood-polyglot.html

Three constructions, six languages, one inferential object. The likelihood function, its deviance transform, the P-value (compatibility) function, and the S-value function are coordinate views of the same thing; a script that can build one can build them all. Each language section below carries the same self-checks:

  • the from-scratch MLE must reproduce the packaged estimator,
  • the likelihood peak must land on the known estimator, and
  • the support interval must reproduce the 95% interval, since .

A silent run means every check passed. The running example for the summary-statistic construction is the Brown et al. (2017) JAMA HDPS-adjusted hazard ratio, HR (0.997, 2.59).

However, there are also variations of likelihood which, of which one we touch on. To see a clear expalanation of the differences between these various likelihood, see1, 2

Maximum likelihood from scratch

Fit a normal linear model and a logistic regression by writing the log-likelihood yourself and maximizing it, then check against the built-in estimator. The one trick used everywhere: parameterize with so the scale parameter stays positive during optimization.

Base R needs no helper package: optim on a hand-written negative log-likelihood, checked against lm and glm.

set.seed(1031)

# ---- Part A: normal linear model ----
n <- 200
x <- rnorm(n)
y <- 2 + 0.75*x + rnorm(n, sd = 1.5)
X <- cbind(1, x)

negloglik_normal <- function(theta, y, X) {
  beta <- theta[1:2]; log_sigma <- theta[3]
  -sum(dnorm(y, mean = X %*% beta, sd = exp(log_sigma), log = TRUE))
}

fit <- optim(c(0, 0, 0), negloglik_normal, y = y, X = X, method = "BFGS")
b_ml <- fit$par[2]; sigma_ml <- exp(fit$par[3])

ols <- lm(y ~ x)
sigma_ols_nscaled <- sqrt(sum(residuals(ols)^2) / n)   # ML sigma uses n, not n-k

stopifnot(abs(b_ml - coef(ols)[["x"]]) < 1e-4,
          abs(sigma_ml - sigma_ols_nscaled) < 1e-3)
cat("Part A validation PASSED: optim MLE reproduces lm\n")
#> Part A validation PASSED: optim MLE reproduces lm

# ---- Part B: logistic regression ----
n2 <- 300
x2 <- rnorm(n2)
y2 <- as.numeric(runif(n2) < plogis(-0.4 + 0.9*x2))
X2 <- cbind(1, x2)

negloglik_logit <- function(beta, y, X) {
  xb <- X %*% beta
  sum(log1p(exp(-xb))*y + log1p(exp(xb))*(1 - y))
}

fit2 <- optim(c(0, 0), negloglik_logit, y = y2, X = X2, method = "BFGS")
glmfit <- glm(y2 ~ x2, family = binomial())

stopifnot(abs(fit2$par[2] - coef(glmfit)[["x2"]]) < 1e-4,
          abs(-fit2$value - as.numeric(logLik(glmfit))) < 1e-4)
cat("Part B validation PASSED: optim MLE reproduces glm\n")
#> Part B validation PASSED: optim MLE reproduces glm

The lf evaluator receives one linear predictor per equation and fills in the observation-level log likelihood (see [R] ml).

#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> Number of observations (_N) was 0, now 200.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Initial:      Log likelihood =  -806.4418
#> Alternative:  Log likelihood = -450.70073
#> Rescale:      Log likelihood = -429.09515
#> Rescale eq:   Log likelihood = -374.61873
#> Iteration 0:  Log likelihood = -374.61873  
#> Iteration 1:  Log likelihood = -348.89609  
#> Iteration 2:  Log likelihood = -343.02889  
#> Iteration 3:  Log likelihood = -342.99237  
#> Iteration 4:  Log likelihood = -342.99235  
#> 
#>                                                         Number of obs =    200
#>                                                         Wald chi2(1)  =  72.77
#> Log likelihood = -342.99235                             Prob > chi2   = 0.0000
#> 
#> ------------------------------------------------------------------------------
#>            y | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#> mu           |
#>            x |   .8119551   .0951852     8.53   0.000     .6253954    .9985147
#>        _cons |   1.868002   .0954381    19.57   0.000     1.680947    2.055057
#> -------------+----------------------------------------------------------------
#> lnsigma      |
#>        _cons |   .2960232        .05     5.92   0.000      .198025    .3940214
#> ------------------------------------------------------------------------------
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Part A validation PASSED: ml reproduces regress
#> 
#> 
#> Number of observations (_N) was 0, now 300.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Initial:      Log likelihood = -207.94415
#> Alternative:  Log likelihood =  -205.2231
#> Rescale:      Log likelihood = -204.28183
#> Iteration 0:  Log likelihood = -204.28183  
#> Iteration 1:  Log likelihood = -187.36811  
#> Iteration 2:  Log likelihood = -187.34499  
#> Iteration 3:  Log likelihood = -187.34499  
#> 
#>                                                         Number of obs =    300
#>                                                         Wald chi2(1)  =  28.32
#> Log likelihood = -187.34499                             Prob > chi2   = 0.0000
#> 
#> ------------------------------------------------------------------------------
#>            y | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>            x |   .7507449   .1410829     5.32   0.000     .4742275    1.027262
#>        _cons |  -.3434135    .124075    -2.77   0.006     -.586596    -.100231
#> ------------------------------------------------------------------------------
#> 
#> 
#> 
#> 
#> Part B validation PASSED: ml reproduces logit
import numpy as np
from scipy import optimize
from scipy.stats import norm
import statsmodels.api as sm

rng = np.random.default_rng(1031)

# ---- Part A: normal linear model ----
n = 200
x = rng.normal(0, 1, n)
y = 2 + 0.75 * x + rng.normal(0, 1.5, n)
X = sm.add_constant(x)

def negloglik_normal(theta, y, X):
    beta, log_sigma = theta[:-1], theta[-1]
    return -np.sum(norm.logpdf(y, loc=X @ beta, scale=np.exp(log_sigma)))

fit = optimize.minimize(negloglik_normal, np.zeros(3), args=(y, X), method="BFGS")
b_ml, sigma_ml = fit.x[1], np.exp(fit.x[2])

ols = sm.OLS(y, X).fit()
sigma_ols_nscaled = np.sqrt(ols.ssr / n)   # ML sigma uses n, not n-k

assert abs(b_ml - ols.params[1]) < 1e-4
assert abs(sigma_ml - sigma_ols_nscaled) < 1e-3
print("Part A validation PASSED: scipy MLE reproduces OLS")
#> Part A validation PASSED: scipy MLE reproduces OLS

# ---- Part B: logistic regression ----
n = 300
x2 = rng.normal(0, 1, n)
y2 = (rng.uniform(size=n) < 1/(1 + np.exp(-(-0.4 + 0.9*x2)))).astype(float)
X2 = sm.add_constant(x2)

def negloglik_logit(beta, y, X):
    xb = X @ beta
    return np.sum(np.logaddexp(0, -xb)*y + np.logaddexp(0, xb)*(1 - y))

fit2 = optimize.minimize(negloglik_logit, np.zeros(2), args=(y2, X2), method="BFGS")
logit = sm.Logit(y2, X2).fit(disp=0)

assert abs(fit2.x[1] - logit.params[1]) < 1e-4
assert abs(-fit2.fun - logit.llf) < 1e-4
print("Part B validation PASSED: scipy MLE reproduces Logit")
#> Part B validation PASSED: scipy MLE reproduces Logit

Note: current Optim.jl rejects the old autodiff = :forward symbol (it now wants an ADTypes object); the default finite-difference gradient is version-robust and plenty here. The trailing semicolons are load-bearing for the output, not the math: Julia echoes the value of every top-level expression, so without them each randn(rng, n) would dump a 200-element vector into the page. Three kinds of line are deliberately left unsuppressed, because what they echo is worth reading – the DataFrames (which self-truncate with ⋮ and a rows-omitted note), the fitted models (lm/glm print a coefficient table with standard errors and intervals, which is what the assertions below check), and the bare p on a plotting chunk’s last line, which is what emits the figure.

using Distributions, Optim, GLM, DataFrames, Random, Statistics

rng = MersenneTwister(1031);

# ---- Part A: normal linear model ----
n = 200;
x = randn(rng, n);
y = 2 .+ 0.75 .* x .+ 1.5 .* randn(rng, n);
X = hcat(ones(n), x);

function negloglik_normal(theta)
    beta, log_sigma = theta[1:2], theta[3]
    -sum(logpdf.(Normal.(X * beta, exp(log_sigma)), y))
end;

fit = optimize(negloglik_normal, zeros(3), BFGS());
b_ml, sigma_ml = Optim.minimizer(fit)[2], exp(Optim.minimizer(fit)[3]);

df = DataFrame(y = y, x = x)
#> 200×2 DataFrame
#>  Row │ y          x
#>      │ Float64    Float64
#> ─────┼────────────────────────
#>    1 │  2.51681    1.53733
#>    2 │  2.49614    0.67567
#>    3 │  3.53854    0.529915
#>    4 │  2.40127   -0.469345
#>    5 │  2.47939   -0.344731
#>    6 │  1.25291    0.00596521
#>    7 │ -0.12643   -1.52318
#>    8 │  2.20185    0.390072
#>   ⋮  │     ⋮           ⋮
#>  194 │  1.6941     1.13584
#>  195 │  0.352342   1.1385
#>  196 │  1.16013   -0.839467
#>  197 │  4.71371    1.00806
#>  198 │  0.659784  -1.24059
#>  199 │  1.914     -0.300155
#>  200 │  0.488223  -0.759049
#>               185 rows omitted
ols = lm(@formula(y ~ x), df)
#> StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Vector{Float64}}, GLM.DensePredChol{Float64, CholeskyPivoted{Float64, Matrix{Float64}, Vector{Int64}}}}, Matrix{Float64}}
#> 
#> y ~ 1 + x
#> 
#> Coefficients:
#> ────────────────────────────────────────────────────────────────────────
#>                 Coef.  Std. Error      t  Pr(>|t|)  Lower 95%  Upper 95%
#> ────────────────────────────────────────────────────────────────────────
#> (Intercept)  2.06123     0.111722  18.45    <1e-44   1.84091     2.28154
#> x            0.820263    0.111287   7.37    <1e-11   0.600803    1.03972
#> ────────────────────────────────────────────────────────────────────────
sigma_ols_nscaled = sqrt(sum(residuals(ols) .^ 2) / n);

@assert abs(b_ml - coef(ols)[2]) < 1e-4
@assert abs(sigma_ml - sigma_ols_nscaled) < 1e-3
println("Part A validation PASSED: Optim MLE reproduces lm")
#> Part A validation PASSED: Optim MLE reproduces lm

# ---- Part B: logistic regression ----
n2 = 300;
x2 = randn(rng, n2);
y2 = Float64.(rand(rng, n2) .< 1 ./ (1 .+ exp.(-(-0.4 .+ 0.9 .* x2))));
X2 = hcat(ones(n2), x2);

log1pexp(u) = u > 35 ? u : log1p(exp(u));
negloglik_logit(beta) =
    sum(log1pexp.(-(X2 * beta)) .* y2 .+ log1pexp.(X2 * beta) .* (1 .- y2));

fit2 = optimize(negloglik_logit, zeros(2), BFGS());
df2 = DataFrame(y = y2, x = x2)
#> 300×2 DataFrame
#>  Row │ y        x
#>      │ Float64  Float64
#> ─────┼─────────────────────
#>    1 │     1.0   0.64402
#>    2 │     0.0  -0.883275
#>    3 │     0.0  -0.746231
#>    4 │     0.0   2.32986
#>    5 │     1.0   1.89985
#>    6 │     0.0  -0.931408
#>    7 │     1.0  -0.316581
#>    8 │     1.0   0.801148
#>   ⋮  │    ⋮         ⋮
#>  294 │     1.0  -0.198019
#>  295 │     0.0  -0.0022127
#>  296 │     1.0   0.10344
#>  297 │     0.0  -0.831018
#>  298 │     1.0   0.177091
#>  299 │     0.0  -1.0159
#>  300 │     0.0  -0.478225
#>            285 rows omitted
logit = glm(@formula(y ~ x), df2, Binomial(), LogitLink())
#> StatsModels.TableRegressionModel{GeneralizedLinearModel{GLM.GlmResp{Vector{Float64}, Binomial{Float64}, LogitLink}, GLM.DensePredChol{Float64, CholeskyPivoted{Float64, Matrix{Float64}, Vector{Int64}}}}, Matrix{Float64}}
#> 
#> y ~ 1 + x
#> 
#> Coefficients:
#> ─────────────────────────────────────────────────────────────────────────
#>                  Coef.  Std. Error      z  Pr(>|z|)  Lower 95%  Upper 95%
#> ─────────────────────────────────────────────────────────────────────────
#> (Intercept)  -0.385155    0.127502  -3.02    0.0025  -0.635055  -0.135256
#> x             0.881431    0.144892   6.08    <1e-08   0.597447   1.16541
#> ─────────────────────────────────────────────────────────────────────────

@assert abs(Optim.minimizer(fit2)[2] - coef(logit)[2]) < 1e-4
@assert abs(-Optim.minimum(fit2) - loglikelihood(logit)) < 1e-4
println("Part B validation PASSED: Optim MLE reproduces glm")
#> Part B validation PASSED: Optim MLE reproduces glm

fminunc is the Optimization Toolbox analogue of optim/minimize; with no toolbox, fminsearch (Nelder–Mead, base MATLAB) reaches the same optimum more slowly. Same parameterization as everywhere else.

rng(1031, 'twister');

% ---- Part A: normal linear model ----
n = 200;
x = randn(n, 1);
y = 2 + 0.75*x + 1.5*randn(n, 1);
X = [ones(n,1) x];

% theta = [b0; b1; log_sigma]
negloglik_normal = @(th) -sum(log(normpdf(y, X*th(1:2), exp(th(3)))));

opts  = optimset('TolX', 1e-10, 'TolFun', 1e-10, 'MaxFunEvals', 1e4);
fit   = fminsearch(negloglik_normal, [0; 0; 0], opts);
b_ml     = fit(2);
sigma_ml = exp(fit(3));

b_ols = X \ y;                        % backslash = least squares
resid = y - X*b_ols;
sigma_ols_nscaled = sqrt(sum(resid.^2) / n);   % ML sigma uses n, not n-k

assert(abs(b_ml - b_ols(2))            < 1e-4)
assert(abs(sigma_ml - sigma_ols_nscaled) < 1e-3)
disp('Part A validation PASSED: fminsearch MLE reproduces backslash')

% ---- Part B: logistic regression ----
n2 = 300;
x2 = randn(n2, 1);
y2 = double(rand(n2,1) < 1 ./ (1 + exp(-(-0.4 + 0.9*x2))));
X2 = [ones(n2,1) x2];

% log1p(exp(u)) via a numerically safe branch, as in the other languages
log1pexp = @(u) (u > 35).*u + (u <= 35).*log1p(exp(min(u, 35)));
negloglik_logit = @(b) sum(log1pexp(-(X2*b)).*y2 + log1pexp(X2*b).*(1 - y2));

fit2 = fminsearch(negloglik_logit, [0; 0], opts);

% glmfit needs the Statistics and Machine Learning Toolbox; it prepends the
% intercept itself, so pass x2 rather than X2.
b_glm = glmfit(x2, y2, 'binomial', 'link', 'logit');

assert(abs(fit2(2) - b_glm(2)) < 1e-4)
disp('Part B validation PASSED: fminsearch MLE reproduces glmfit')
#> Part A validation PASSED: fminsearch MLE reproduces backslash
#> Part B validation PASSED: fminsearch MLE reproduces glmfit

In SQL the closed forms are the ML estimates – REGR_SLOPE and REGR_INTERCEPT – and grid search plays the role of the optimizer. The chunks below execute at render time against an in-process DuckDB seeded in the setup chunk, so every number on this page is query output rather than a transcription. The dialect is ANSI where it counts; the Snowflake originals differ only in how rows get generated (GENERATOR/SEQ4 for generate_series, UNIFORM for random).

-- The data is simulated in the setup chunk above (n = 200,
-- y = 2 + 0.75x + N(0, 1.5)) via Box-Muller, since neither DuckDB nor
-- Snowflake ships a normal deviate. Note the two-stage CTE: you cannot write
-- AVG(POWER(y - REGR_INTERCEPT(y,x) OVER () ..., 2)) -- an aggregate may not
-- contain a window call. Compute the coefficients first, then the residual
-- variance against them.
WITH coef AS (
  SELECT regr_slope(y, x) AS b1, regr_intercept(y, x) AS b0
  FROM mytable
)
SELECT c.b1 AS b1_mle,
       c.b0 AS b0_mle,
       -- ML sigma^2 (divides by n, not n-k):
       avg(power(d.y - c.b0 - c.b1 * d.x, 2)) AS sigma2_mle
FROM mytable d CROSS JOIN coef c
GROUP BY c.b1, c.b0;
1 records
b1_mle b0_mle sigma2_mle
0.595 1.84 2.46

The estimates are real query output. Pulling the same rows into R confirms SQL is computing the maximum likelihood fit and not something adjacent to it:

d   <- DBI::dbGetQuery(con, "SELECT * FROM mytable")
ols <- lm(y ~ x, data = d)
sql <- DBI::dbGetQuery(con, "
  WITH coef AS (SELECT regr_slope(y,x) AS b1, regr_intercept(y,x) AS b0 FROM mytable)
  SELECT c.b1, c.b0, avg(power(d.y - c.b0 - c.b1*d.x, 2)) AS s2
  FROM mytable d CROSS JOIN coef c GROUP BY c.b1, c.b0;")

stopifnot(abs(sql$b1 - coef(ols)[["x"]]) < 1e-12,
          abs(sql$s2 - sum(residuals(ols)^2)/nrow(d)) < 1e-12)
cat(sprintf("SQL b1 = %.12f\n lm b1 = %.12f\n", sql$b1, coef(ols)[["x"]]))
#> SQL b1 = 0.595368434861
#>  lm b1 = 0.595368434861
cat("validation PASSED: REGR_SLOPE == lm, to machine precision\n")
#> validation PASSED: REGR_SLOPE == lm, to machine precision

Likelihood function from summary statistics

A point estimate and a 95% interval on the ratio scale are enough to reconstruct the whole function: recover the standard error from the interval width, lay down a grid, and transform one per grid point.

point <- 1.61; LL <- 0.997; UL <- 2.59
se <- log(UL / LL) / (2 * 1.96)

hr <- seq(0.5, 4.0, length.out = 600)
z  <- (log(hr) - log(point)) / se

support  <- exp(-(z^2) / 2)
deviance <- z^2
pvalue   <- 2 * (1 - pnorm(abs(z)))
svalue   <- -log2(pvalue)

inside <- hr[support >= 1/6.83]
stopifnot(abs(min(inside) - LL) < 0.02, abs(max(inside) - UL) < 0.02)
cat("validation PASSED: 1/6.83 LI == 95% CI\n")
#> validation PASSED: 1/6.83 LI == 95% CI

plot(hr, support, type = "l", log = "x", col = "#007C7C", lwd = 2,
     xaxt = "n", xlab = "Hazard Ratio (HR)", ylab = "Relative likelihood",
     main = "Relative Likelihood Function")
axis(1, at = c(0.5, 0.75, 1, 1.5, 2, 3, 4))
abline(v = 1, lty = 3, col = "#d46c5b")
abline(h = 1/6.83, col = "gray", lty = 2)

#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> Number of observations (_N) was 0, now 200.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Initial:      Log likelihood =  -806.4418
#> Alternative:  Log likelihood = -450.70073
#> Rescale:      Log likelihood = -429.09515
#> Rescale eq:   Log likelihood = -374.61873
#> Iteration 0:  Log likelihood = -374.61873  
#> Iteration 1:  Log likelihood = -348.89609  
#> Iteration 2:  Log likelihood = -343.02889  
#> Iteration 3:  Log likelihood = -342.99237  
#> Iteration 4:  Log likelihood = -342.99235  
#> 
#>                                                         Number of obs =    200
#>                                                         Wald chi2(1)  =  72.77
#> Log likelihood = -342.99235                             Prob > chi2   = 0.0000
#> 
#> ------------------------------------------------------------------------------
#>            y | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#> mu           |
#>            x |   .8119551   .0951852     8.53   0.000     .6253954    .9985147
#>        _cons |   1.868002   .0954381    19.57   0.000     1.680947    2.055057
#> -------------+----------------------------------------------------------------
#> lnsigma      |
#>        _cons |   .2960232        .05     5.92   0.000      .198025    .3940214
#> ------------------------------------------------------------------------------
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> check: ML slope  =  0.811955  OLS slope  =  0.811955
#> 
#> check: ML sigma  =  1.344501  OLS sigma (n-scaled) =  1.344501
#> 
#> 
#> 
#> Part A validation PASSED: ml reproduces regress
#> 
#> 
#> Number of observations (_N) was 0, now 300.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Initial:      Log likelihood = -207.94415
#> Alternative:  Log likelihood =  -205.2231
#> Rescale:      Log likelihood = -204.28183
#> Iteration 0:  Log likelihood = -204.28183  
#> Iteration 1:  Log likelihood = -187.36811  
#> Iteration 2:  Log likelihood = -187.34499  
#> Iteration 3:  Log likelihood = -187.34499  
#> 
#>                                                         Number of obs =    300
#>                                                         Wald chi2(1)  =  28.32
#> Log likelihood = -187.34499                             Prob > chi2   = 0.0000
#> 
#> ------------------------------------------------------------------------------
#>            y | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>            x |   .7507449   .1410829     5.32   0.000     .4742275    1.027262
#>        _cons |  -.3434135    .124075    -2.77   0.006     -.586596    -.100231
#> ------------------------------------------------------------------------------
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> check: ml slope =  0.750745  logit slope =  0.750745
#> 
#> check: ml ll    =  -187.344987  logit ll    =  -187.344987
#> 
#> 
#> 
#> Part B validation PASSED: ml reproduces logit
#> 
#> 
#> All maximum likelihood constructions validated.

The inverse route: sweeping levels instead of parameters with cifunction

Everything above builds the curve by sweeping the parameter axis: fix a grid of candidate values, compute one (or one refit) per value, and read off a P-value. There is a second route to the same object that works from the other side: sweep the confidence level from 0 to 99.99% and compute the interval limits at each level. Plotting the limits against the levels traces exactly the same curve – the two constructions are inverse functions of each other, since the P-value function evaluated at a level’s limit returns that level’s .

A confidence interval function (also called a confidence curve, P-value function, or consonance function) displays every confidence interval around an estimate simultaneously. Ariel Linden’s cifunction Stata module implements the level-sweeping construction directly; the R equivalent is concurve. Install once from SSC:

ssc install cifunction
ssc install getregstats   // helper: recover SEs from p-values or CI limits

One convention to respect: with eform, cifunction takes se() on the ratio scale (the delta-method SE, ), does its computation on the log scale, and back-transforms. Same Brown et al. example as above, with the saved output asserted against both the reported interval and the manual z-grid construction: ## Python

import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt

point, LL, UL = 1.61, 0.997, 2.59
se = np.log(UL / LL) / (2 * 1.96)

hr = np.linspace(0.5, 4.0, 600)
z = (np.log(hr) - np.log(point)) / se

support  = np.exp(-(z**2) / 2)
deviance = z**2
pvalue   = 2 * (1 - norm.cdf(np.abs(z)))
svalue   = -np.log2(pvalue)

inside = hr[support >= 1/6.83]
assert abs(inside.min() - LL) < 0.02 and abs(inside.max() - UL) < 0.02
print("validation PASSED: 1/6.83 LI == 95% CI")
#> validation PASSED: 1/6.83 LI == 95% CI

fig, ax = plt.subplots(figsize=(7, 4.5))
ax.plot(hr, support, color="#007C7C")
ax.axvline(1, ls=":", color="#d46c5b"); ax.axhline(1/6.83, color="gray", alpha=0.4)
ax.set_xscale("log"); ax.set_xticks([0.5, 0.75, 1, 1.5, 2, 3, 4])
ax.set_xlabel("Hazard Ratio (HR)"); ax.set_ylabel("Relative likelihood")
ax.set_title("Relative Likelihood Function")
plt.show()

using Distributions, Plots
gr();

point, LL, UL = 1.61, 0.997, 2.59;
se = log(UL / LL) / (2 * 1.96);

hr = range(0.5, 4.0; length = 600);
z = (log.(hr) .- log(point)) ./ se;

support  = exp.(-(z .^ 2) ./ 2);
deviance = z .^ 2;
pvalue   = 2 .* (1 .- cdf.(Normal(), abs.(z)));
svalue   = .-log2.(pvalue);

inside = hr[support .>= 1/6.83];
@assert abs(minimum(inside) - LL) < 0.02 && abs(maximum(inside) - UL) < 0.02
println("validation PASSED: 1/6.83 LI == 95% CI")
#> validation PASSED: 1/6.83 LI == 95% CI

p = plot(hr, support; xscale = :log10, legend = false,
         xlabel = "Hazard Ratio (HR)", ylabel = "Relative likelihood",
         title = "Relative Likelihood Function");
hline!(p, [1/6.83]; color = :gray, alpha = 0.4);
vline!(p, [1.0]; linestyle = :dot, color = :red);
p

normcdf needs the Statistics Toolbox; the erfc form below is base MATLAB and gives the identical two-sided tail, so this tab runs either way.

point = 1.61; LL = 0.997; UL = 2.59;
se = log(UL / LL) / (2 * 1.96);

hr = linspace(0.5, 4.0, 600)';
z  = (log(hr) - log(point)) / se;

support  = exp(-(z.^2) / 2);
deviance = z.^2;
% two-sided p; erfc form avoids the Statistics Toolbox dependency
pvalue   = erfc(abs(z) / sqrt(2));
svalue   = -log2(pvalue);

inside = hr(support >= 1/6.83);
assert(abs(min(inside) - LL) < 0.02 && abs(max(inside) - UL) < 0.02)
disp('validation PASSED: 1/6.83 LI == 95% CI')

figure;
semilogx(hr, support, 'LineWidth', 2, 'Color', [0 0.486 0.486]);
xticks([0.5 0.75 1 1.5 2 3 4]);
xline(1,      ':', 'Color', [0.831 0.424 0.357]);
yline(1/6.83, '--', 'Color', [0.5 0.5 0.5]);
xlabel('Hazard Ratio (HR)'); ylabel('Relative likelihood');
title('Relative Likelihood Function');
#> validation PASSED: 1/6.83 LI == 95% CI

Neither Snowflake nor DuckDB ships erf(), so uses the Abramowitz–Stegun 26.2.17 polynomial ( per tail). SQL does not plot; it returns a table, and something else draws it – Power BI, or in this case R.

WITH params AS (
  SELECT 1.61::DOUBLE AS point,
         ln(2.59 / 0.997) / (2 * 1.96) AS se
),
grid AS (
  SELECT 0.5 + (i - 1) * (4.0 - 0.5) / 599 AS hr
  FROM generate_series(1, 600) AS t(i)
),
lik AS (
  SELECT g.hr,
         (ln(g.hr) - ln(p.point)) / p.se                     AS z,
         exp(-power((ln(g.hr) - ln(p.point)) / p.se, 2) / 2) AS support,
         power((ln(g.hr) - ln(p.point)) / p.se, 2)           AS deviance
  FROM grid g CROSS JOIN params p
),
pv AS (
  SELECT hr, z, support, deviance,
         2 * ( exp(-z*z/2) / sqrt(2*pi()) *
               ( 0.319381530 * (1/(1 + 0.2316419*abs(z)))
               - 0.356563782 * power(1/(1 + 0.2316419*abs(z)), 2)
               + 1.781477937 * power(1/(1 + 0.2316419*abs(z)), 3)
               - 1.821255978 * power(1/(1 + 0.2316419*abs(z)), 4)
               + 1.330274429 * power(1/(1 + 0.2316419*abs(z)), 5) ) ) AS pvalue
  FROM lik
)
SELECT hr, support, deviance, pvalue, -ln(pvalue)/ln(2) AS svalue
FROM pv ORDER BY hr LIMIT 5;
5 records
hr support deviance pvalue svalue
0.500 0 23.1 0 19.3
0.506 0 22.6 0 18.9
0.512 0 22.2 0 18.6
0.518 0 21.7 0 18.3
0.523 0 21.3 0 17.9

All four coordinate views come out of that one query. The interval it returns is against the reported , and the Abramowitz–Stegun polynomial tracks R’s pnorm to :

cur <- DBI::dbGetQuery(con, "
  WITH params AS (SELECT 1.61::DOUBLE AS point, ln(2.59/0.997)/(2*1.96) AS se),
  grid AS (SELECT 0.5 + (i-1)*(4.0-0.5)/599 AS hr FROM generate_series(1,600) AS t(i)),
  lik AS (SELECT g.hr, (ln(g.hr)-ln(p.point))/p.se AS z,
                 exp(-power((ln(g.hr)-ln(p.point))/p.se,2)/2) AS support,
                 power((ln(g.hr)-ln(p.point))/p.se,2) AS deviance
          FROM grid g CROSS JOIN params p),
  pv AS (SELECT hr, z, support, deviance,
           2 * ( exp(-z*z/2)/sqrt(2*pi()) *
             ( 0.319381530*(1/(1+0.2316419*abs(z)))
             - 0.356563782*power(1/(1+0.2316419*abs(z)),2)
             + 1.781477937*power(1/(1+0.2316419*abs(z)),3)
             - 1.821255978*power(1/(1+0.2316419*abs(z)),4)
             + 1.330274429*power(1/(1+0.2316419*abs(z)),5) ) ) AS pvalue
         FROM lik)
  SELECT hr, support, deviance, pvalue, -ln(pvalue)/ln(2) AS svalue
  FROM pv ORDER BY hr;")

inside <- cur$hr[cur$support >= 1/6.83]
exact  <- 2 * (1 - pnorm(abs((log(cur$hr) - log(1.61)) /
                             (log(2.59/0.997)/(2*1.96)))))
stopifnot(abs(min(inside) - 0.997) < 0.02, abs(max(inside) - 2.59) < 0.02,
          max(abs(cur$pvalue - exact)) < 1e-6)
cat(sprintf("1/6.83 interval: [%.4f, %.4f]\n", min(inside), max(inside)))
#> 1/6.83 interval: [1.0025, 2.5918]
cat(sprintf("max |Abramowitz-Stegun - pnorm| = %.2e\n",
            max(abs(cur$pvalue - exact))))
#> max |Abramowitz-Stegun - pnorm| = 1.49e-07
cat(sprintf("S-value at HR = 1: %.2f bits\n",
            cur$svalue[which.min(abs(cur$hr - 1))]))
#> S-value at HR = 1: 4.27 bits

op <- par(mfrow = c(2, 2), mar = c(4, 4, 2.5, 1))
panel <- function(y, ylab, main, href = NA) {
  plot(cur$hr, y, type = "l", log = "x", col = "#007C7C", lwd = 2,
       xaxt = "n", xlab = "Hazard Ratio", ylab = ylab, main = main)
  axis(1, at = c(0.5, 1, 2, 4))
  abline(v = 1, lty = 3, col = "#d46c5b")
  if (!is.na(href)) abline(h = href, col = "gray", lty = 2)
}
panel(cur$support,  "Relative likelihood", "Support",   1/6.83)
panel(cur$deviance, "Deviance",            "Deviance",  3.84)
panel(cur$pvalue,   "P-value",             "P-value (compatibility)", 0.05)
panel(cur$svalue,   "Bits of information", "S-value",   4.32)

par(op)

The dotted vertical line marks HR . Reading the bottom-right panel: the S-value there is 4.27 bits – about as much evidence against the null as five straight heads, which is not much.

Profile likelihood for a regression coefficient

The offset trick: to fix the coefficient of at , move into an offset and refit the model without . The refit re-maximizes the intercept and the other covariates at every grid point – that is what makes it a profile likelihood rather than a slice through the likelihood at the MLEs.

set.seed(1031)
n <- 300
x <- rnorm(n)
w <- rnorm(n)
y <- as.numeric(runif(n) < plogis(-0.3 + 0.8*x + 0.4*w))

full  <- glm(y ~ x + w, family = binomial())
bhat  <- coef(full)[["x"]]
sehat <- sqrt(diag(vcov(full)))[["x"]]

# offset trick: fix the coefficient of x at b and refit without x, so the
# intercept and w are re-maximized at every grid point
grid <- seq(bhat - 4*sehat, bhat + 4*sehat, length.out = 121)
ll <- vapply(grid, function(b)
  as.numeric(logLik(glm(y ~ w, family = binomial(), offset = b*x))),
  numeric(1))

support  <- exp(ll - max(ll))
deviance <- -2*(ll - max(ll))

stopifnot(abs(grid[which.max(support)] - bhat) < diff(grid)[1])
cat("validation PASSED: profile peak == full-model MLE\n")
#> validation PASSED: profile peak == full-model MLE

si <- range(grid[support >= 1/6.83])
cat(sprintf("1/6.83 support interval: [%.4f, %.4f]   Wald 95%% CI: [%.4f, %.4f]\n",
            si[1], si[2], bhat - 1.96*sehat, bhat + 1.96*sehat))
#> 1/6.83 support interval: [0.5593, 1.1148]   Wald 95% CI: [0.5459, 1.1090]

plot(grid, support, type = "l", col = "#007C7C", lwd = 2,
     xlab = "Coefficient of x (log-odds scale)",
     ylab = "Relative likelihood",
     main = "Profile Likelihood Function")
abline(v = bhat, lty = 2, col = "gray40")
abline(h = 1/6.83, col = "gray", lty = 2)

#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> 
#> Number of observations (_N) was 0, now 300.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> (file /tmp/S_70849.000001 not found)
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> profile peak =  0.748198   full-model MLE =  0.748198
#> 
#> 
#> 
#> 1/6.83 support interval: [0.4730, 1.0430]   Wald 95% CI: [0.4593, 1.0371]
#> 
#> 
#> 
#> Part B validation PASSED: profile peak and support interval check out
#> 
#> 
#> 
#> 
#> file profile_likelihood_logit.svg saved as SVG format
#> 
#> 
#> All likelihood-function constructions validated.

import numpy as np
import statsmodels.api as sm
import matplotlib.pyplot as plt

rng = np.random.default_rng(1031)
n = 300
x = rng.normal(0, 1, n)
w = rng.normal(0, 1, n)
y = (rng.uniform(size=n) < 1/(1 + np.exp(-(-0.3 + 0.8*x + 0.4*w)))).astype(float)

X_full = sm.add_constant(np.column_stack([x, w]))
full = sm.GLM(y, X_full, family=sm.families.Binomial()).fit()
bhat, sehat = full.params[1], full.bse[1]
X_nuis = sm.add_constant(w)

grid = np.linspace(bhat - 4*sehat, bhat + 4*sehat, 121)
ll = np.array([
    sm.GLM(y, X_nuis, family=sm.families.Binomial(), offset=b*x).fit().llf
    for b in grid
])
support = np.exp(ll - ll.max())

assert abs(grid[np.argmax(support)] - bhat) < grid[1] - grid[0]
print("validation PASSED: profile peak == full-model MLE")
#> validation PASSED: profile peak == full-model MLE

fig, ax = plt.subplots(figsize=(7, 4.5))
ax.plot(grid, support, color="#007C7C")
ax.axvline(bhat, ls="--", color="gray")
ax.axhline(1/6.83, color="gray", alpha=0.4)
ax.set_xlabel("Coefficient of x (log-odds)")
ax.set_ylabel("Relative likelihood")
ax.set_title("Profile Likelihood Function")
plt.show()

using Distributions, GLM, DataFrames, Random, Plots

rng = MersenneTwister(1031);
n = 300;
x = randn(rng, n);
w = randn(rng, n);
y = Float64.(rand(rng, n) .< 1 ./ (1 .+ exp.(-(-0.3 .+ 0.8 .* x .+ 0.4 .* w))));
df = DataFrame(y = y, x = x, w = w)
#> 300×3 DataFrame
#>  Row │ y        x            w
#>      │ Float64  Float64      Float64
#> ─────┼──────────────────────────────────
#>    1 │     0.0   1.53733      0.212606
#>    2 │     1.0   0.67567      2.18329
#>    3 │     0.0   0.529915    -0.411984
#>    4 │     0.0  -0.469345     0.243801
#>    5 │     0.0  -0.344731     0.181979
#>    6 │     1.0   0.00596521   0.102878
#>    7 │     0.0  -1.52318     -0.446261
#>    8 │     1.0   0.390072    -1.61528
#>   ⋮  │    ⋮          ⋮           ⋮
#>  294 │     1.0  -1.21183      0.133128
#>  295 │     1.0   0.35636     -0.373798
#>  296 │     1.0   0.125155     0.561362
#>  297 │     0.0  -1.63194     -0.0784788
#>  298 │     0.0  -0.464039    -0.214666
#>  299 │     1.0   1.21094     -0.392618
#>  300 │     1.0  -1.2263      -0.228791
#>                         285 rows omitted

full = glm(@formula(y ~ x + w), df, Binomial(), LogitLink())
#> StatsModels.TableRegressionModel{GeneralizedLinearModel{GLM.GlmResp{Vector{Float64}, Binomial{Float64}, LogitLink}, GLM.DensePredChol{Float64, CholeskyPivoted{Float64, Matrix{Float64}, Vector{Int64}}}}, Matrix{Float64}}
#> 
#> y ~ 1 + x + w
#> 
#> Coefficients:
#> ─────────────────────────────────────────────────────────────────────────
#>                  Coef.  Std. Error      z  Pr(>|z|)  Lower 95%  Upper 95%
#> ─────────────────────────────────────────────────────────────────────────
#> (Intercept)  -0.378416    0.133353  -2.84    0.0045  -0.639782  -0.117049
#> x             1.07021     0.160245   6.68    <1e-10   0.756132   1.38428
#> w             0.475207    0.130953   3.63    0.0003   0.218545   0.73187
#> ─────────────────────────────────────────────────────────────────────────
bhat, sehat = coef(full)[2], stderror(full)[2];

grid = range(bhat - 4*sehat, bhat + 4*sehat; length = 121);
ll = [loglikelihood(glm(@formula(y ~ w), df, Binomial(), LogitLink();
                        offset = b .* x)) for b in grid];
support = exp.(ll .- maximum(ll));

@assert abs(grid[argmax(support)] - bhat) < grid[2] - grid[1]
println("validation PASSED: profile peak == full-model MLE")
#> validation PASSED: profile peak == full-model MLE

p = plot(grid, support; legend = false,
         xlabel = "Coefficient of x (log-odds)",
         ylabel = "Relative likelihood",
         title = "Profile Likelihood Function");
vline!(p, [bhat]; linestyle = :dash, color = :gray);
hline!(p, [1/6.83]; color = :gray, alpha = 0.4);
p

glmfit takes an 'Offset' name–value pair, so the offset trick carries over directly: fix the coefficient of at , push into the offset, and refit on alone so the intercept and are re-maximized at every point.

rng(1031, 'twister');
n = 300;
x = randn(n, 1);
w = randn(n, 1);
y = double(rand(n,1) < 1 ./ (1 + exp(-(-0.3 + 0.8*x + 0.4*w))));

% full fit: MLE and Wald SE define the grid
[b_full, ~, stats] = glmfit([x w], y, 'binomial', 'link', 'logit');
bhat  = b_full(2);
sehat = stats.se(2);

grid = linspace(bhat - 4*sehat, bhat + 4*sehat, 121)';
ll   = zeros(size(grid));
for k = 1:numel(grid)
    % offset fixes b*x; only the intercept and w are re-estimated
    [~, dev] = glmfit(w, y, 'binomial', 'link', 'logit', ...
                      'Offset', grid(k)*x);
    ll(k) = -dev / 2;   % glmfit returns deviance = -2*loglik (up to a constant)
end

support  = exp(ll - max(ll));
deviance = -2*(ll - max(ll));

[~, imax] = max(support);
assert(abs(grid(imax) - bhat) < grid(2) - grid(1))
disp('validation PASSED: profile peak == full-model MLE')

si = [min(grid(support >= 1/6.83)), max(grid(support >= 1/6.83))];
fprintf('1/6.83 support interval: [%.4f, %.4f]   Wald 95%% CI: [%.4f, %.4f]\n', ...
        si(1), si(2), bhat - 1.96*sehat, bhat + 1.96*sehat);

figure;
plot(grid, support, 'LineWidth', 2, 'Color', [0 0.486 0.486]);
xline(bhat,   '--', 'Color', [0.4 0.4 0.4]);
yline(1/6.83, '--', 'Color', [0.5 0.5 0.5]);
xlabel('Coefficient of x (log-odds scale)');
ylabel('Relative likelihood');
title('Profile Likelihood Function');
#> validation PASSED: profile peak == full-model MLE
#> 1/6.83 support interval: [0.4246, 0.9635]   Wald 95% CI: [0.4116, 0.9579]

For the binomial family glmfit’s deviance is up to an additive constant that does not depend on the parameters, so it cancels in ll - max(ll) – which is all the relative likelihood needs.

SQL cannot iteratively refit a GLM (no IRLS), so a logistic profile is out of scope. But for the normal linear model the conditional MLEs of the intercept and given a fixed slope are closed-form, so profiling collapses to one AVG per candidate slope – a GROUP BY, not a refit loop. The profile-grid MLE matches REGR_SLOPE exactly here – the peak and the closed form agree to the last printed digit, since both are computed from the same rows. (Dialect landmine: Snowflake rejects correlated scalar subqueries here; the CROSS JOIN + GROUP BY below is the working idiom.)

-- One AVG per candidate slope: profiling collapses to a GROUP BY because the
-- conditional MLEs of the intercept and sigma^2 given a fixed slope are
-- closed-form. No refit loop, no IRLS.
WITH closed AS (
  SELECT regr_slope(y, x) AS b1, count(*) AS n FROM mytable
),
m AS (
  SELECT avg(x) AS xbar, avg(y) AS ybar FROM mytable
),
grid AS (
  SELECT c.b1 - 0.3 + (g.i - 1) * 0.6 / 400 AS b_cand
  FROM generate_series(1, 401) AS g(i) CROSS JOIN closed c
),
prof AS (
  SELECT g.b_cand,
         -(c.n / 2.0) *
         ln(avg(power(d.y - (m.ybar - g.b_cand * m.xbar)
                          - g.b_cand * d.x, 2))) AS prof_loglik
  FROM grid g
  CROSS JOIN mytable d
  CROSS JOIN m
  CROSS JOIN closed c
  GROUP BY g.b_cand, c.n
)
SELECT b_cand,
       exp(prof_loglik - max(prof_loglik) OVER ()) AS support
FROM prof ORDER BY b_cand LIMIT 5;
5 records
b_cand support
0.295 0.012
0.297 0.013
0.298 0.013
0.300 0.014
0.301 0.014

That returns all 401 grid points (truncated to five above). SQL has no plotting layer, so hand the result set to whatever draws your charts – here, R:

prof <- DBI::dbGetQuery(con, "
  WITH closed AS (SELECT regr_slope(y,x) AS b1, count(*) AS n FROM mytable),
  m AS (SELECT avg(x) AS xbar, avg(y) AS ybar FROM mytable),
  grid AS (SELECT c.b1 - 0.3 + (g.i-1) * 0.6/400 AS b_cand
           FROM generate_series(1,401) AS g(i) CROSS JOIN closed c),
  prof AS (
    SELECT g.b_cand,
           -(c.n/2.0) * ln(avg(power(d.y - (m.ybar - g.b_cand*m.xbar)
                                         - g.b_cand*d.x, 2))) AS prof_loglik
    FROM grid g CROSS JOIN mytable d CROSS JOIN m CROSS JOIN closed c
    GROUP BY g.b_cand, c.n)
  SELECT b_cand, exp(prof_loglik - max(prof_loglik) OVER ()) AS support
  FROM prof ORDER BY b_cand;")

b1_sql <- sql$b1
peak   <- prof$b_cand[which.max(prof$support)]
si     <- range(prof$b_cand[prof$support >= 1/6.83])
ci     <- confint(ols, "x", level = 0.95)

# the two self-checks from the top of the post
stopifnot(abs(peak - b1_sql) < diff(prof$b_cand)[1])
cat(sprintf("profile peak = %.10f   REGR_SLOPE = %.10f\n", peak, b1_sql))
#> profile peak = 0.5953684349   REGR_SLOPE = 0.5953684349
cat(sprintf("1/6.83 support interval: [%.4f, %.4f]\n", si[1], si[2]))
#> 1/6.83 support interval: [0.3989, 0.7919]
cat(sprintf("lm 95%% CI:               [%.4f, %.4f]\n", ci[1], ci[2]))
#> lm 95% CI:               [0.3974, 0.7933]

plot(prof$b_cand, prof$support, type = "l", col = "#007C7C", lwd = 2,
     xlab = "Candidate slope", ylab = "Relative likelihood",
     main = "Profile Likelihood Computed Entirely in SQL")
abline(v = b1_sql, lty = 2, col = "gray40")
abline(h = 1/6.83, col = "gray", lty = 2)
segments(si[1], 1/6.83, si[2], 1/6.83, col = "#d46c5b", lwd = 3)
text(b1_sql, 0.55, "REGR_SLOPE", cex = 0.75, col = "gray30", pos = 4)
text(mean(si), 1/6.83 - 0.055, "1/6.83 support interval",
     cex = 0.75, col = "#d46c5b")

The support interval drawn in orange and the Wald interval from lm agree to about three decimals, which is the same identity the R, Python, and Julia sections check – here with every number produced by a GROUP BY.

Existing R tools as alternatives

Everything above builds the likelihood by hand. In R you usually do not have to: several packages will do the maximizing and profiling for you, and their output reshapes into the same five-column object concurve plots. The one helper below is all the glue needed. See the concurve vignettes on likelihood tools and profile likelihoods for the longer treatment.

library(concurve)

# Reshape any (parameter grid, log-likelihood) pair into concurve's format.
as_concurve_lik <- function(values, loglik) {
  loglik_rel <- loglik - max(loglik)
  support    <- exp(loglik_rel)
  df <- data.frame(
    values        = values,
    likelihood    = support,
    loglikelihood = loglik_rel,
    support       = support,
    deviancestat  = -loglik_rel
  )
  class(df) <- c("data.frame", "concurve")
  df
}

set.seed(123)
x <- rnorm(50, mean = 9.3, sd = 3.2)
c(mean = mean(x), sd = sd(x))
#> mean   sd 
#> 9.41 2.96

Base R, no dependencies. optim finds the MLE and the Hessian gives Wald standard errors. To profile , substitute the conditional MLE of at each grid point rather than holding fixed.

nll_normal <- function(theta, x) {
  mu <- theta[1]; sigma <- theta[2]
  -sum(dnorm(x, mean = mu, sd = sigma, log = TRUE))
}

fit <- optim(c(mu = 8, sigma = 3), nll_normal, x = x,
             method = "BFGS", hessian = TRUE)
fit$par
#>    mu sigma 
#>  9.41  2.93
sqrt(diag(solve(fit$hessian)))
#>    mu sigma 
#> 0.415 0.293

n <- length(x)
mu_grid <- seq(fit$par["mu"] - 3, fit$par["mu"] + 3, length.out = 4000)
loglik_mu <- vapply(mu_grid, function(m) {
  sigma2_hat <- mean((x - m)^2)          # conditional MLE of sigma^2 given mu
  -(n / 2) * (log(2 * pi) + log(sigma2_hat) + 1)
}, numeric(1))

lik_optim <- as_concurve_lik(mu_grid, loglik_mu)
stopifnot(abs(mu_grid[which.max(loglik_mu)] - fit$par[["mu"]]) < 1e-2)

ggcurve(lik_optim, type = "l1", nullvalue = round(fit$par[["mu"]], 2),
        xaxis = expression(mu),
        title = "Likelihood for the Mean (via optim profiling)")

maxLik maximizes the log-likelihood directly, with no sign flipping, and prints a summary with standard errors.

library(maxLik)

loglik_normal <- function(theta, x) {
  sum(dnorm(x, mean = theta[1], sd = sqrt(theta[2]), log = TRUE))
}
theta_mle <- maxLik(loglik_normal, start = c(mu = 8, sig2 = 9), x = x)
summary(theta_mle)
#> --------------------------------------------
#> Maximum Likelihood estimation
#> Newton-Raphson maximisation, 7 iterations
#> Return code 8: successive function values within relative tolerance limit (reltol)
#> Log-Likelihood: -125 
#> 2  free parameters
#> Estimates:
#>      Estimate Std. error t value Pr(> t)    
#> mu      9.410      0.415   22.66 < 2e-16 ***
#> sig2    8.602      1.712    5.02 5.1e-07 ***
#> ---
#> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#> --------------------------------------------

loglik_mu2 <- vapply(mu_grid, function(m) {
  loglik_normal(c(m, mean((x - m)^2)), x = x)
}, numeric(1))

stopifnot(abs(coef(theta_mle)[[1]] - fit$par[["mu"]]) < 1e-3)

ggcurve(as_concurve_lik(mu_grid, loglik_mu2), type = "l1",
        nullvalue = round(coef(theta_mle)[[1]], 2),
        xaxis = expression(mu),
        title = "Likelihood for the Mean (via maxLik)")

bbmle::mle2 is the only route here that profiles for you. profile() returns the signed square-root deviance , so the relative log-likelihood is .

library(bbmle)

nll <- function(mu, sigma) -sum(dnorm(x, mean = mu, sd = sigma, log = TRUE))
m <- mle2(nll, start = list(mu = 8, sigma = 3), data = list(x = x))

pr    <- profile(m)
pr_mu <- as.data.frame(pr)
pr_mu <- pr_mu[pr_mu$param == "mu", ]

mu_vals <- pr_mu[["par.vals.mu"]]
if (is.null(mu_vals)) mu_vals <- pr_mu[["mu"]]

lik_bbmle <- as_concurve_lik(mu_vals, -0.5 * pr_mu$z^2)
stopifnot(abs(coef(m)[["mu"]] - fit$par[["mu"]]) < 1e-3)

ggcurve(lik_bbmle, type = "l1", nullvalue = round(coef(m)[["mu"]], 2),
        xaxis = expression(mu),
        title = "Profile Likelihood for the Mean (via bbmle)")

For a binary outcome carrying many nuisance parameters, cond gives exact conditional likelihood inference: it conditions the nuisance parameters out instead of profiling them.

library(cond)
data(babies)

mod_babies <- glm(cbind(r1, r2) ~ day + lull - 1,
                  family = binomial, data = babies)
cond_fit <- cond(mod_babies, offset = lullyes)
summary(cond_fit)
#> 
#>  Formula:  cbind(r1, r2) ~ day + lull - 1
#>  Family:  binomial
#>  Offset:  lullyes
#> 
#>           Estimate   Std. Error 
#> uncond.        1.43        0.734
#> cond.          1.27        0.689
#> 
#> Confidence intervals
#> --------------------
#>  level = 95 %
#>                                            lower two-sided upper
#> Wald pivot                              -0.00651            2.87
#> Wald pivot (cond. MLE)                  -0.08020            2.62
#> Likelihood root                          0.12300            3.09
#> Modified likelihood root                 0.01070            2.76
#> Modified likelihood root (cont. corr.)  -0.15200            3.10
#> 
#> Diagnostics:
#> ----------- 
#>   INF    NP 
#> 0.076 0.289 
#> 
#>  Approximation based on 20 points
plot(cond_fit, which = 2)

The pitfall: slicing is not profiling

Fixing a nuisance parameter instead of re-maximizing it is the most common way to get this wrong. Holding here yields a likelihood far too narrow, because the real is about 3.2 — the interval it implies is roughly a third of the honest width:

loglik_slice <- vapply(mu_grid, function(m) -0.5 * sum((x - m)^2), numeric(1))
lik_slice    <- as_concurve_lik(mu_grid, loglik_slice)

w <- function(o) {
  s <- curve_support(o, 6.83)
  s$upper.limit - s$lower.limit
}
c(slice_width = w(lik_slice), profile_width = w(lik_optim))
#>   slice_width profile_width 
#>         0.554         1.658
stopifnot(w(lik_slice) < w(lik_optim))

ggcurve(lik_slice, type = "l1", nullvalue = round(fit$par[["mu"]], 2),
        xaxis = expression(mu),
        title = "WRONG: mean likelihood with sigma^2 fixed at 1",
        subtitle = "Too narrow -- the real sigma is about 3.2, not 1")

Profile likelihood from ProfileLikelihood

profilelike.glm profiles one coefficient of a GLM over a grid, and curve_lik turns that straight into a concurve object — which then gives all four coordinate views from the top of this post off a single fit. The self-check: the profile’s peak must reproduce the coefficient from the ordinary glm.

library(ProfileLikelihood)
data(dataglm)

xx <- profilelike.glm(y ~ x1 + x2, data = dataglm, profile.theta = "group",
                      lo.theta = -1.5, hi.theta = 4.2,
                      family = binomial(link = "logit"), length = 500, round = 2)
lik_pl <- curve_lik(xx, data = dataglm)

si <- curve_support(lik_pl[[1]], 6.83)
si
k support.level lower.limit upper.limit mle
6.83 0.146 0.098 2.91 1.36 

# the profile peak must land on the full-model MLE
mod_pl <- glm(y ~ x1 + x2 + group, data = dataglm,
              family = binomial(link = "logit"))
stopifnot(abs(si$mle - coef(mod_pl)[["group"]]) < 0.01)
cat("validation PASSED: profile peak == glm coefficient\n")
#> validation PASSED: profile peak == glm coefficient

The same object, four ways — relative likelihood, log-likelihood, likelihood, and deviance:

ggcurve(lik_pl[[1]], type = "l1", nullvalue = TRUE)

ggcurve(lik_pl[[1]], type = "l2")

ggcurve(lik_pl[[1]], type = "l3")

ggcurve(lik_pl[[1]], type = "d")

Profile versus Wald, and where the 1/6.83 interval lands

curve_lik_glm profiles a coefficient directly. Comparing it against the Wald normal approximation on the same grid shows the skew the Wald interval discards — and the Wald support interval reproduces its own consonance interval, which is the identity from the top of this post:

mod <- glm(am ~ mpg, family = binomial, data = mtcars)
est <- coef(mod)[["mpg"]]
se  <- sqrt(diag(vcov(mod)))[["mpg"]]

lik_prof <- curve_lik_glm(mod, "mpg", steps = 200)
grid     <- lik_prof[[1]]$values
lik_wald <- as_concurve_lik(grid, dnorm(grid, mean = est, sd = se, log = TRUE))

cons <- curve_analytic(estimate = est, se = se, dist = "z")
ci95 <- cons[[1]][which.min(abs(cons[[1]]$intrvl.level - 0.95)),
                  c("lower.limit", "upper.limit")]

out <- rbind(
  `profile 1/6.83 support` = unlist(curve_support(lik_prof[[1]], 6.83)[c("lower.limit", "upper.limit")]),
  `Wald    1/6.83 support` = unlist(curve_support(lik_wald,      6.83)[c("lower.limit", "upper.limit")]),
  `Wald    95% consonance` = unlist(ci95)
)
out
#>                        lower.limit upper.limit
#> profile 1/6.83 support      0.1218       0.588
#> Wald    1/6.83 support      0.0818       0.532
#> Wald    95% consonance      0.0819       0.532

# Wald support interval and Wald consonance interval agree to ~1e-3
stopifnot(max(abs(out[2, ] - out[3, ])) < 1e-3)
cat("validation PASSED: Wald 1/6.83 support == Wald 95% consonance\n")
#> validation PASSED: Wald 1/6.83 support == Wald 95% consonance

The profile curve leans right of the symmetric Wald curve, and that lean is precisely what a Wald interval throws away:

plot_compare(lik_prof[[1]], lik_wald, type = "l1",
             xaxis = expression(beta[mpg]))

Profile likelihood for a regression coefficient

The offset trick: to fix the coefficient of at , move into an offset and refit the model without . The refit re-maximizes the intercept and the other covariates at every grid point – that is what makes it a profile likelihood rather than a slice through the likelihood at the MLEs.

version 16
clear all
set seed 1031

set obs 300
generate x = rnormal(0, 1)
generate w = rnormal(0, 1)                       // nuisance covariate
generate y = runiform() < invlogit(-0.3 + 0.8*x + 0.4*w)

* full fit: the MLE and Wald SE define the grid
quietly logit y x w
scalar bhat  = _b[x]
scalar sehat = _se[x]

* grid: MLE +/- 4 SEs, 121 points (each point costs one -logit- refit)
local npts  = 121
local blo   = bhat - 4*sehat
local bstep = 8*sehat / (`npts' - 1)

tempname results
tempfile profile
postfile `results' double(beta ll) using "`profile'", replace

forvalues i = 1/`npts' {
    local b = `blo' + (`i' - 1)*`bstep'
    capture drop off
    quietly generate double off = `b'*x
    quietly logit y w, offset(off)
    post `results' (`b') (e(ll))
}
postclose `results'

use "`profile'", clear

* normalize into support / deviance
quietly summarize ll
generate support  = exp(ll - r(max))
generate deviance = -2*(ll - r(max))

* --- validation 1: profile peak sits at the full-model MLE ---
quietly summarize beta if support == 1
display as text _n "profile peak = " as result %9.6f r(mean)  ///
    as text "   full-model MLE = " as result %9.6f bhat
assert abs(r(mean) - bhat) < `bstep'   // within one grid step

* --- validation 2: 1/6.83 support interval ~ Wald 95% CI ---
quietly summarize beta if support >= 1/6.83
display as text "1/6.83 support interval: [" as result %6.4f r(min)   ///
    as text ", " as result %6.4f r(max) as text "]"                    ///
    as text "   Wald 95% CI: [" as result %6.4f bhat - 1.96*sehat      ///
    as text ", " as result %6.4f bhat + 1.96*sehat as text "]"
* profile and Wald agree closely here (smooth likelihood, moderate n);
* they diverge exactly as much as the likelihood is skewed
assert abs(r(min) - (bhat - 1.96*sehat)) < 3*`bstep'
assert abs(r(max) - (bhat + 1.96*sehat)) < 3*`bstep'
display as result "validation PASSED: profile peak and support interval check out"

* --- plots ---
twoway (line support beta),                                            ///
    xline(`=bhat', lpattern(dash) lcolor(gs10))                        ///
    xline(0, lpattern(dot) lcolor(red))                                ///
    yline(0.1464, lcolor(gs12))                                        ///
    title("Profile Likelihood Function for a Logistic Coefficient")    ///
    subtitle("Nuisance parameters re-maximized at each grid point")    ///
    xtitle("Coefficient of x (log-odds scale)")                        ///
    ytitle("Relative Likelihood") name(prof_lik, replace)

twoway (line deviance beta),                                           ///
    xline(`=bhat', lpattern(dash) lcolor(gs10))                        ///
    yline(3.84, lcolor(gs12))                                          ///
    title("Profile Deviance Function")                                 ///
    subtitle("Horizontal line at 3.84; crossings = 95% profile CI")    ///
    xtitle("Coefficient of x (log-odds scale)")                        ///
    ytitle("Profile Deviance") name(prof_dev, replace)

graph combine prof_lik prof_dev, cols(2)
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> Number of observations (_N) was 0, now 300.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> (file /tmp/S_76071.000001 not found)
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> profile peak =  0.748198   full-model MLE =  0.748198
#> 
#> 
#> 
#> 1/6.83 support interval: [0.4730, 1.0430]   Wald 95% CI: [0.4593, 1.0371]
#> 
#> 
#> 
#> validation PASSED: profile peak and support interval check out

Validation status

Language Runtime validated Notes
Python yes – all asserts pass scipy/statsmodels/matplotlib
Julia yes – all asserts pass (numerics) Plots.jl output not exercised
MATLAB yes – executes at render via matlab -batch fminsearch is base MATLAB; glmfit/normcdf need the Statistics Toolbox
SQL (DuckDB, live) yes – executes at render; asserts vs lm/pnorm agreement to machine precision; plotting delegated to R
SQL (Snowflake) yes – all three parts run live there same logic, dialect differs in row generation
Stata syntax verified against [R] ml, [P] postfile run once locally; asserts are the safety net
Stata cifunction asserts against reported CI + z-grid construction SSC module; conventions per the cifunction guide

Package Citations

citation("Statamarkdown")
#> To cite package ‘Statamarkdown’ in publications use:
#> 
#>   Hemken D, Palmer T (2026). _Statamarkdown: 'Stata' Markdown_. doi:10.32614/CRAN.package.Statamarkdown
#>   <https://doi.org/10.32614/CRAN.package.Statamarkdown>. R package version 1.0.0,
#>   <https://CRAN.R-project.org/package=Statamarkdown>.
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {Statamarkdown: 'Stata' Markdown},
#>     author = {Doug Hemken and Tom Palmer},
#>     year = {2026},
#>     note = {R package version 1.0.0},
#>     url = {https://CRAN.R-project.org/package=Statamarkdown},
#>     doi = {10.32614/CRAN.package.Statamarkdown},
#>   }
citation("reticulate")
#> To cite package ‘reticulate’ in publications use:
#> 
#>   Kalinowski T, Ushey K, Allaire J, Tang Y (2026). _reticulate: Interface to 'Python'_.
#>   doi:10.32614/CRAN.package.reticulate <https://doi.org/10.32614/CRAN.package.reticulate>. R package version 1.47.0,
#>   <https://CRAN.R-project.org/package=reticulate>.
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {reticulate: Interface to 'Python'},
#>     author = {Tomasz Kalinowski and Kevin Ushey and JJ Allaire and Yuan Tang},
#>     year = {2026},
#>     note = {R package version 1.47.0},
#>     url = {https://CRAN.R-project.org/package=reticulate},
#>     doi = {10.32614/CRAN.package.reticulate},
#>   }
citation("JuliaCall")
#> To cite package ‘JuliaCall’ in publications use:
#> 
#>   Li C (2019). “JuliaCall: an R package for seamless integration between R and Julia.” _The Journal of Open Source
#>   Software_, *4*(35), 1284. doi:10.21105/joss.01284 <https://doi.org/10.21105/joss.01284>.
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Article{JuliaCall,
#>     author = {Changcheng Li},
#>     title = {{JuliaCall}: an {R} package for seamless integration between {R} and {Julia}},
#>     journal = {The Journal of Open Source Software},
#>     publisher = {The Open Journal},
#>     year = {2019},
#>     volume = {4},
#>     number = {35},
#>     pages = {1284},
#>     doi = {10.21105/joss.01284},
#>   }

Statistical Environments

R Environment

#> ─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.6.1 (2026-06-24)
#>  os       macOS Golden Gate 27.0
#>  system   aarch64, darwin25.4.0
#>  ui       unknown
#>  language (EN)
#>  collate  C
#>  ctype    C.UTF-8
#>  tz       America/New_York
#>  date     2026-09-13
#>  pandoc   3.11 @ /opt/homebrew/bin/ (via rmarkdown)
#>  quarto   1.10.18 @ /usr/local/bin/quarto
#> 
#> ─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  package           * version    date (UTC) lib source
#>  abind               1.4-8      2024-09-12 [1] CRAN (R 4.6.1)
#>  Amelia            * 1.8.3      2024-11-08 [1] CRAN (R 4.6.1)
#>  arm                 1.15-3     2026-04-15 [1] CRAN (R 4.6.1)
#>  arrayhelpers        1.1-2      2026-07-24 [1] CRAN (R 4.6.1)
#>  backports           1.5.1      2026-04-03 [1] CRAN (R 4.6.1)
#>  base              * 4.6.1      2026-06-24 [2] local
#>  base64enc           0.1-6      2026-02-02 [1] CRAN (R 4.6.1)
#>  bayesplot         * 1.16.0     2026-08-25 [1] CRAN (R 4.6.1)
#>  bbmle             * 1.0.26     2026-08-31 [1] CRAN (R 4.6.1)
#>  bdsmatrix           1.3-7      2024-03-02 [1] CRAN (R 4.6.1)
#>  bitops              1.1-0      2026-07-30 [1] CRAN (R 4.6.1)
#>  blogdown          * 1.24       2026-06-19 [1] CRAN (R 4.6.1)
#>  boot              * 1.3-32     2025-08-29 [1] CRAN (R 4.6.1)
#>  bootImpute        * 1.3.0      2025-12-15 [1] CRAN (R 4.6.1)
#>  bridgesampling      1.2-1      2025-11-19 [1] CRAN (R 4.6.1)
#>  brms              * 2.23.0     2025-09-09 [1] CRAN (R 4.6.1)
#>  Brobdingnag         1.2-9      2022-10-19 [1] CRAN (R 4.6.1)
#>  broom             * 1.0.13     2026-05-14 [1] CRAN (R 4.6.1)
#>  broom.mixed       * 0.2.9.7    2026-02-17 [1] CRAN (R 4.6.1)
#>  Cairo             * 1.7-0      2025-10-29 [1] CRAN (R 4.6.1)
#>  car               * 3.1-5      2026-02-03 [1] CRAN (R 4.6.1)
#>  carData           * 3.0-6      2026-01-30 [1] CRAN (R 4.6.1)
#>  caTools             1.18.4     2026-07-20 [1] CRAN (R 4.6.1)
#>  checkmate         * 2.3.4      2026-02-03 [1] CRAN (R 4.6.1)
#>  class               7.3-24     2026-08-03 [1] CRAN (R 4.6.1)
#>  cli               * 3.6.6      2026-04-09 [1] CRAN (R 4.6.1)
#>  clipr               0.8.1      2026-05-25 [1] CRAN (R 4.6.1)
#>  cluster             2.1.8.3    2026-07-30 [1] CRAN (R 4.6.1)
#>  coda              * 0.19-4.1   2024-01-31 [1] CRAN (R 4.6.1)
#>  codetools           0.2-20     2024-03-31 [1] CRAN (R 4.6.0)
#>  colorspace        * 2.1-3      2026-07-12 [1] CRAN (R 4.6.1)
#>  compiler            4.6.1      2026-06-24 [2] local
#>  concurve          * 3.0.5      2026-09-13 [1] local
#>  cond              * 1.2-4      2025-05-25 [1] CRAN (R 4.6.1)
#>  cowplot           * 1.2.0      2025-07-07 [1] CRAN (R 4.6.1)
#>  crayon              1.5.3      2024-06-20 [1] CRAN (R 4.6.1)
#>  curl                8.0.0      2026-08-25 [1] CRAN (R 4.6.1)
#>  data.table          1.18.6.1   2026-08-24 [1] CRAN (R 4.6.1)
#>  datasets          * 4.6.1      2026-06-24 [2] local
#>  DBI               * 1.3.0      2026-02-25 [1] CRAN (R 4.6.1)
#>  DEoptimR            1.2-1      2026-08-20 [1] CRAN (R 4.6.1)
#>  desc                1.4.3      2023-12-10 [1] CRAN (R 4.6.1)
#>  details             0.4.0      2025-02-09 [1] CRAN (R 4.6.0)
#>  dichromat           2.0-1      2026-07-22 [1] CRAN (R 4.6.1)
#>  digest              0.6.39     2025-11-19 [1] CRAN (R 4.6.1)
#>  distributional      0.9.0      2026-09-08 [1] CRAN (R 4.6.1)
#>  doParallel        * 1.0.17     2022-02-07 [1] CRAN (R 4.6.1)
#>  doRNG               1.8.6.3    2026-02-05 [1] CRAN (R 4.6.1)
#>  dplyr             * 1.2.1      2026-04-03 [1] CRAN (R 4.6.1)
#>  duckdb              1.5.5      2026-07-25 [1] CRAN (R 4.6.1)
#>  e1071               1.7-17     2025-12-18 [1] CRAN (R 4.6.1)
#>  emmeans             2.0.4      2026-07-15 [1] CRAN (R 4.6.1)
#>  estimability        2.0.0      2026-06-26 [1] CRAN (R 4.6.1)
#>  evaluate            1.0.5      2025-08-27 [1] CRAN (R 4.6.1)
#>  extremevalues       2.4.1      2024-12-17 [1] CRAN (R 4.6.1)
#>  farver              2.1.2      2024-05-13 [1] CRAN (R 4.6.1)
#>  fastmap             1.2.0      2024-05-15 [1] CRAN (R 4.6.1)
#>  forcats           * 1.0.1      2025-09-25 [1] CRAN (R 4.6.1)
#>  foreach           * 1.5.2      2022-02-02 [1] CRAN (R 4.6.1)
#>  foreign             0.8-91     2026-01-29 [1] CRAN (R 4.6.1)
#>  Formula             1.2-6      2026-08-03 [1] CRAN (R 4.6.1)
#>  fs                  2.1.0      2026-04-18 [1] CRAN (R 4.6.1)
#>  furrr               0.4.0      2026-03-31 [1] CRAN (R 4.6.1)
#>  future            * 1.75.0     2026-07-20 [1] CRAN (R 4.6.1)
#>  future.apply      * 1.20.2     2026-02-20 [1] CRAN (R 4.6.1)
#>  gamlss            * 5.5-0      2025-08-19 [1] CRAN (R 4.6.1)
#>  gamlss.data       * 6.0-7      2025-09-04 [1] CRAN (R 4.6.1)
#>  gamlss.dist       * 6.1-11     2026-09-10 [1] CRAN (R 4.6.1)
#>  generics            0.1.4      2025-05-09 [1] CRAN (R 4.6.1)
#>  ggcorrplot        * 0.3.0      2026-07-24 [1] CRAN (R 4.6.1)
#>  ggdist              3.3.3      2025-04-23 [1] CRAN (R 4.6.1)
#>  ggplot2           * 4.0.3      2026-04-22 [1] CRAN (R 4.6.1)
#>  ggtext            * 0.2.0      2026-08-28 [1] CRAN (R 4.6.1)
#>  glmnet              5.0        2026-05-04 [1] CRAN (R 4.6.1)
#>  globals             0.19.1     2026-03-13 [1] CRAN (R 4.6.1)
#>  glue                1.8.1      2026-04-17 [1] CRAN (R 4.6.1)
#>  graphics          * 4.6.1      2026-06-24 [2] local
#>  grDevices         * 4.6.1      2026-06-24 [2] local
#>  grid              * 4.6.1      2026-06-24 [2] local
#>  gridExtra           2.3.1      2026-06-25 [1] CRAN (R 4.6.1)
#>  gridtext            0.1.6      2026-02-19 [1] CRAN (R 4.6.0)
#>  gtable              0.3.6      2024-10-25 [1] CRAN (R 4.6.1)
#>  gtsummary         * 2.6.1      2026-09-04 [1] CRAN (R 4.6.1)
#>  here              * 1.0.2      2025-09-15 [1] CRAN (R 4.6.1)
#>  Hmisc             * 5.3-0      2026-09-06 [1] CRAN (R 4.6.1)
#>  hms                 1.1.4      2025-10-17 [1] CRAN (R 4.6.1)
#>  htmlTable           2.5.0      2026-04-22 [1] CRAN (R 4.6.1)
#>  htmltools         * 0.5.9      2025-12-04 [1] CRAN (R 4.6.1)
#>  htmlwidgets         1.6.4      2023-12-06 [1] CRAN (R 4.6.1)
#>  httr                1.4.9      2026-09-01 [1] CRAN (R 4.6.1)
#>  ImputeRobust      * 1.3-1      2018-11-30 [1] CRAN (R 4.6.1)
#>  inline              0.3.21     2025-01-09 [1] CRAN (R 4.6.1)
#>  insight             1.5.4      2026-09-05 [1] CRAN (R 4.6.1)
#>  iterators         * 1.0.14     2022-02-05 [1] CRAN (R 4.6.1)
#>  itertools           0.1-3      2014-03-12 [1] CRAN (R 4.6.1)
#>  jomo                2.7-6      2023-04-15 [1] CRAN (R 4.6.1)
#>  jsonlite            2.0.0      2025-03-27 [1] CRAN (R 4.6.1)
#>  JuliaCall           0.17.6     2024-12-07 [1] CRAN (R 4.6.1)
#>  kableExtra        * 1.4.1      2026-07-08 [1] CRAN (R 4.6.1)
#>  knitr             * 1.52       2026-09-06 [1] CRAN (R 4.6.1)
#>  labeling            0.4.3      2023-08-29 [1] CRAN (R 4.6.1)
#>  laeken              0.5.3      2024-01-25 [1] CRAN (R 4.6.1)
#>  latex2exp         * 0.9.8      2026-01-09 [1] CRAN (R 4.6.1)
#>  lattice           * 0.23-1     2026-08-12 [1] CRAN (R 4.6.1)
#>  lifecycle           1.0.5      2026-01-08 [1] CRAN (R 4.6.1)
#>  listenv             1.0.0      2026-06-22 [1] CRAN (R 4.6.1)
#>  lme4                2.0-6      2026-07-16 [1] CRAN (R 4.6.1)
#>  lmtest              0.9-40     2022-03-21 [1] CRAN (R 4.6.1)
#>  loo               * 2.10.1     2026-07-24 [1] CRAN (R 4.6.1)
#>  lubridate         * 1.9.5      2026-02-04 [1] CRAN (R 4.6.1)
#>  magick              2.9.1      2026-02-28 [1] CRAN (R 4.6.1)
#>  magrittr          * 2.0.5      2026-04-04 [1] CRAN (R 4.6.1)
#>  MASS              * 7.3-66     2026-07-15 [1] CRAN (R 4.6.1)
#>  Matrix            * 1.7-6      2026-07-25 [1] CRAN (R 4.6.1)
#>  MatrixModels        0.5-4      2025-03-26 [1] CRAN (R 4.6.1)
#>  matrixStats         1.5.0      2025-01-07 [1] CRAN (R 4.6.1)
#>  maxLik            * 1.5-2.2    2025-12-29 [1] CRAN (R 4.6.0)
#>  mcmc                0.9-8      2023-11-16 [1] CRAN (R 4.6.1)
#>  MCMCpack          * 1.7-1      2024-08-27 [1] CRAN (R 4.6.1)
#>  methods           * 4.6.1      2026-06-24 [2] local
#>  mgcv              * 1.9-4      2025-11-07 [1] CRAN (R 4.6.0)
#>  mi                * 1.3.1      2026-07-28 [1] CRAN (R 4.6.1)
#>  mice              * 3.19.0     2025-12-10 [1] CRAN (R 4.6.1)
#>  miceadds          * 3.20-10    2026-05-28 [1] CRAN (R 4.6.1)
#>  miceFast          * 0.9.1      2026-02-26 [1] CRAN (R 4.6.1)
#>  minqa               1.2.8      2024-08-17 [1] CRAN (R 4.6.1)
#>  miscTools         * 0.6-30     2026-01-20 [1] CRAN (R 4.6.1)
#>  missForest        * 1.6.1      2025-10-26 [1] CRAN (R 4.6.1)
#>  mitml             * 0.4-5      2023-03-08 [1] CRAN (R 4.6.1)
#>  mitools             2.7        2026-08-31 [1] CRAN (R 4.6.1)
#>  multcomp            1.4-32     2026-08-21 [1] CRAN (R 4.6.1)
#>  mvtnorm           * 1.4-2      2026-07-12 [1] CRAN (R 4.6.1)
#>  nlme              * 3.1-171    2026-09-01 [1] CRAN (R 4.6.1)
#>  nloptr              2.2.1      2025-03-17 [1] CRAN (R 4.6.1)
#>  nnet                7.3-21     2026-08-03 [1] CRAN (R 4.6.1)
#>  numDeriv            2016.8-1.1 2019-06-06 [1] CRAN (R 4.6.1)
#>  opdisDownsampling   1.7        2026-09-04 [1] CRAN (R 4.6.1)
#>  otel                0.2.0      2025-08-29 [1] CRAN (R 4.6.1)
#>  pan                 2.0        2026-06-30 [1] CRAN (R 4.6.1)
#>  parallel          * 4.6.1      2026-06-24 [2] local
#>  parallelly        * 1.48.0     2026-06-29 [1] CRAN (R 4.6.1)
#>  pbmcapply         * 1.5.1      2022-04-28 [1] CRAN (R 4.6.1)
#>  performance       * 0.18.2     2026-09-10 [1] CRAN (R 4.6.1)
#>  pillar              1.11.1     2025-09-17 [1] CRAN (R 4.6.1)
#>  pkgbuild            1.4.8      2025-05-26 [1] CRAN (R 4.6.1)
#>  pkgconfig           2.0.3      2019-09-22 [1] CRAN (R 4.6.1)
#>  plyr                1.8.9      2023-10-02 [1] CRAN (R 4.6.1)
#>  png                 0.1-9      2026-03-15 [1] CRAN (R 4.6.0)
#>  polspline           1.1.25     2024-05-10 [1] CRAN (R 4.6.1)
#>  posterior         * 1.7.0      2026-04-01 [1] CRAN (R 4.6.1)
#>  pracma              2.4.6      2025-10-22 [1] CRAN (R 4.6.1)
#>  prettyunits         1.2.0      2023-09-24 [1] CRAN (R 4.6.1)
#>  ProfileLikelihood * 1.3        2023-08-25 [1] CRAN (R 4.6.1)
#>  progress          * 1.2.3      2023-12-06 [1] CRAN (R 4.6.1)
#>  proxy               0.4-29     2025-12-29 [1] CRAN (R 4.6.1)
#>  purrr             * 1.2.2      2026-04-10 [1] CRAN (R 4.6.1)
#>  qqconf              1.3.2      2023-04-14 [1] CRAN (R 4.6.1)
#>  qqplotr           * 0.0.7      2025-09-05 [1] CRAN (R 4.6.1)
#>  quantreg          * 6.1        2025-03-10 [1] CRAN (R 4.6.1)
#>  QuickJSR            1.11.0     2026-08-21 [1] CRAN (R 4.6.1)
#>  R6                  2.6.1      2025-02-15 [1] CRAN (R 4.6.1)
#>  randomForest      * 4.7-1.2    2024-09-22 [1] CRAN (R 4.6.1)
#>  ranger              0.18.0     2026-01-16 [1] CRAN (R 4.6.1)
#>  rbibutils           2.4.1      2026-01-21 [1] CRAN (R 4.6.1)
#>  RColorBrewer        1.1-3      2022-04-03 [1] CRAN (R 4.6.1)
#>  Rcpp              * 1.1.2      2026-07-05 [1] CRAN (R 4.6.1)
#>  RcppParallel        6.2.1      2026-08-27 [1] CRAN (R 4.6.1)
#>  Rdpack              2.6.6      2026-02-08 [1] CRAN (R 4.6.1)
#>  readr             * 2.2.0      2026-02-19 [1] CRAN (R 4.6.1)
#>  reformulas          0.4.4      2026-02-02 [1] CRAN (R 4.6.1)
#>  rematch2            2.1.2      2020-05-01 [1] CRAN (R 4.6.1)
#>  reshape2          * 1.4.5      2025-11-12 [1] CRAN (R 4.6.1)
#>  reticulate        * 1.47.0     2026-09-03 [1] CRAN (R 4.6.1)
#>  rlang               1.3.0      2026-07-05 [1] CRAN (R 4.6.1)
#>  rmarkdown         * 2.32       2026-09-01 [1] CRAN (R 4.6.1)
#>  rms               * 8.1-1      2026-02-18 [1] CRAN (R 4.6.1)
#>  rngtools            1.5.2      2021-09-20 [1] CRAN (R 4.6.1)
#>  robustbase          0.99-7     2026-02-05 [1] CRAN (R 4.6.1)
#>  rpart               4.1.27     2026-03-27 [1] CRAN (R 4.6.0)
#>  rprojroot           2.1.1      2025-08-26 [1] CRAN (R 4.6.1)
#>  rstan             * 2.32.7     2025-03-10 [1] CRAN (R 4.6.1)
#>  rstantools          2.7.1      2026-09-04 [1] Github (stan-dev/rstantools@4d84659)
#>  rstudioapi          0.19.0     2026-06-11 [1] CRAN (R 4.6.1)
#>  S7                  0.2.2      2026-04-22 [1] CRAN (R 4.6.1)
#>  sandwich            3.1-3      2026-08-03 [1] CRAN (R 4.6.1)
#>  scales              1.4.0      2025-04-24 [1] CRAN (R 4.6.1)
#>  sessioninfo         1.2.4      2026-06-04 [1] CRAN (R 4.6.1)
#>  shape               1.4.6.1    2024-02-23 [1] CRAN (R 4.6.1)
#>  showtext          * 0.9-8      2026-03-21 [1] CRAN (R 4.6.1)
#>  showtextdb        * 3.0        2020-06-04 [1] CRAN (R 4.6.1)
#>  sp                  2.2-3      2026-07-19 [1] CRAN (R 4.6.1)
#>  SparseM           * 1.84-2     2024-07-17 [1] CRAN (R 4.6.1)
#>  splines           * 4.6.1      2026-06-24 [2] local
#>  StanHeaders       * 2.39.1     2026-09-02 [1] CRAN (R 4.6.1)
#>  Statamarkdown     * 1.0.0      2026-08-21 [1] CRAN (R 4.6.1)
#>  statmod           * 1.5.2      2026-05-17 [1] CRAN (R 4.6.1)
#>  stats             * 4.6.1      2026-06-24 [2] local
#>  stats4            * 4.6.1      2026-06-24 [2] local
#>  stringi             1.8.9      2026-08-04 [1] CRAN (R 4.6.1)
#>  stringr           * 1.6.0      2025-11-04 [1] CRAN (R 4.6.1)
#>  survival          * 3.8-12     2026-09-09 [1] CRAN (R 4.6.1)
#>  svglite           * 2.2.2      2025-10-21 [1] CRAN (R 4.6.1)
#>  svgPanZoom          0.3.4      2020-02-15 [1] CRAN (R 4.6.1)
#>  svUnit              1.0.8      2025-08-26 [1] CRAN (R 4.6.1)
#>  sysfonts          * 0.8.9      2024-03-02 [1] CRAN (R 4.6.1)
#>  systemfonts         1.3.2      2026-03-05 [1] CRAN (R 4.6.1)
#>  tensorA             0.36.2.1   2023-12-13 [1] CRAN (R 4.6.0)
#>  texPreview        * 2.1.0      2024-01-24 [1] CRAN (R 4.6.0)
#>  textshaping         1.0.5      2026-03-06 [1] CRAN (R 4.6.0)
#>  TH.data             1.1-5      2025-11-17 [1] CRAN (R 4.6.0)
#>  tibble            * 3.3.1      2026-01-11 [1] CRAN (R 4.6.1)
#>  tidybayes         * 3.0.7      2024-09-15 [1] CRAN (R 4.6.1)
#>  tidyr             * 1.3.2      2025-12-19 [1] CRAN (R 4.6.1)
#>  tidyselect          1.2.1      2024-03-11 [1] CRAN (R 4.6.1)
#>  tidyverse         * 2.0.0      2023-02-22 [1] CRAN (R 4.6.1)
#>  timechange          0.4.0      2026-01-29 [1] CRAN (R 4.6.1)
#>  tinytex           * 0.60       2026-06-16 [1] CRAN (R 4.6.1)
#>  tools               4.6.1      2026-06-24 [2] local
#>  twosamples          2.0.1      2023-06-23 [1] CRAN (R 4.6.1)
#>  tzdb                0.5.0      2025-03-15 [1] CRAN (R 4.6.1)
#>  utils             * 4.6.1      2026-06-24 [2] local
#>  V8                  8.2.0      2026-04-21 [1] CRAN (R 4.6.0)
#>  vcd                 1.4-14     2026-07-29 [1] CRAN (R 4.6.1)
#>  vctrs               0.7.3      2026-04-11 [1] CRAN (R 4.6.1)
#>  VIM               * 7.3.1      2026-09-11 [1] CRAN (R 4.6.1)
#>  viridisLite         0.4.3      2026-02-04 [1] CRAN (R 4.6.1)
#>  wesanderson       * 0.3.7      2023-10-31 [1] CRAN (R 4.6.1)
#>  whisker             0.4.1      2022-12-05 [1] CRAN (R 4.6.1)
#>  withr               3.0.3      2026-06-19 [1] CRAN (R 4.6.1)
#>  xfun              * 0.60       2026-07-09 [1] CRAN (R 4.6.1)
#>  xgboost             3.2.1.1    2026-03-18 [1] CRAN (R 4.6.1)
#>  xml2                1.6.0      2026-06-22 [1] CRAN (R 4.6.1)
#>  xtable            * 1.8-8      2026-02-22 [1] CRAN (R 4.6.1)
#>  yaml                2.3.12     2025-12-10 [1] CRAN (R 4.6.1)
#>  yardstick         * 1.4.0      2026-04-07 [1] CRAN (R 4.6.1)
#>  zoo                 1.9-0      2026-07-31 [1] CRAN (R 4.6.1)
#> 
#>  [1] /opt/homebrew/lib/R/4.6/site-library
#>  [2] /opt/homebrew/Cellar/r/4.6.1/lib/R/library
#>  * ── Packages attached to the search path.
#> 
#> ─ External software ────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting        value
#>  cairo          1.18.4
#>  cairoFT        2.14.3/2.18.1
#>  pango
#>  png
#>  jpeg
#>  tiff
#>  tcl            9.0
#>  curl           8.7.1
#>  zlib           1.2.12
#>  bzlib          1.0.8, 13-Jul-2019
#>  xz             5.8.3
#>  deflate
#>  zstd           1.5.7
#>  PCRE           10.48 2026-08-31
#>  ICU            78.1
#>  TRE            TRE 0.8.0 R_fixes (BSD)
#>  iconv          Apple or GNU libiconv 1.11 /usr/lib/libiconv.2.dylib
#>  readline       8.3
#>  BLAS           /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib
#>  lapack         /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib
#>  lapack_version 3.12.1
#> 
#> ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> $RNGkind
#> [1] "Mersenne-Twister" "Inversion"        "Rejection"

Stata Environment

set linesize 255
display %tcMonth_dd,CCYY_hh:MM_am now()
about
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#>  September 13,2026 5:39 am
#> 
#> 
#> StataNow/MP 19.5 for Mac (Apple Silicon)
#> Revision 12 Aug 2026
#> Copyright 1985-2025 StataCorp LLC
#> 
#> Total physical memory: 48.01 GB
#> 
#> Stata license: Single-user 2-core , expiring  6 Feb 2027
#> Serial number: 501909358563
#>   Licensed to: Zad Rafi
#>

References

Back to top

References

1. Cole SR, Chu H, Greenland S. (2013). “Maximum likelihood, profile likelihood, and penalized likelihood: A primer.” American Journal of Epidemiology. 179:252–260. doi: 10.1093/aje/kwt245. https://doi.org/10.1093/aje/kwt245.
2. Cox DR. (2006). “Principles of Statistical Inference.” Cambridge University Press.

Citation

BibTeX citation:
@online{rafi2026,
  author = {Rafi, Zad and Rafi, Zad},
  title = {Constructing {Likelihood} {Functions} in {R,} {Stata,}
    {Python,} {Julia,} {MATLAB,} and {SQL}},
  date = {2026-09-16},
  url = {https://lesslikely.com/statistics/likelihood-polyglot.html},
  langid = {en-US}
}
]]> statistics https://lesslikely.com/statistics/likelihood-polyglot.html Wed, 16 Sep 2026 04:00:00 GMT How to Be a Good Collaborator to a Statistician and a Data Engineer Zad Rafi https://lesslikely.com/statistics/good-collaborator.html

Most failed projects are not failures due to technical matters, but failures of a good collaboration. They fail earlier, in the part nobody writes up: a question that was never pinned down, a column whose meaning changed in 2023, a “quick pull” that quietly became the number in the board deck. The statistician gets blamed for the estimate and the data engineer gets blamed for the pipeline, but the decisive errors were usually committed by well-meaning people who thought they were being helpful by not bothering anyone.

The influential statistician and geneticist, Ronald A. Fisher put it elegantly, in his 1938 presidential address to the Indian Statistical Congress:

consulting the statistician after the experiment is finished is often merely asking him to perform a post mortem examination — he can perhaps say what the experiment died of.1.

This is a piece about the habits that keep the patient alive. It is deliberately two-sided. Collaborators owe analysts specific things; analysts owe collaborators specific things back, and “the data wasn’t clean” is not a professional deliverable.

Why This Is Harder Than It Looks

The intuition that a well-specified question has one obvious analysis is wrong, and it is wrong by a measurable amount.

In the best-known demonstration, 29 teams comprising 61 analysts were given the same dataset and the same question — whether soccer referees give more red cards to dark-skin-toned players. The 29 analyses used 21 unique combinations of covariates. Estimated effects ranged from an odds ratio of 0.89 to 2.93, median 1.31; 20 teams (69%) reported a statistically significant positive effect and 9 (31%) did not. Neither analysts’ prior beliefs nor their expertise nor peer ratings of analysis quality explained the spread2.

The same pattern shows up in neuroimaging. Seventy teams analysed one fMRI dataset against nine pre-specified hypotheses. No two teams chose identical workflows. The resulting variation in hypothesis-test outcomes was substantial even between teams whose intermediate statistical maps were highly correlated3.

The lesson is not that analysis is arbitrary. It is that the analysis is underdetermined by the question, and the gap gets filled by defensible-looking judgment calls — the same garden of forking paths that makes untested model assumptions so consequential. Every one of those calls is a place where your context — what the variable actually measures, which exclusions are clinically or commercially sensible, what decision the number feeds — either arrives in time or doesn’t.

NoteThe Load-Bearing Claim

The variation in results across competent analysts is large enough that your contribution as a collaborator — supplying context that narrows the defensible choice set — is not a courtesy. It is part of the estimate.

What You Owe the Statistician

In Research Settings

Bring them in at design, not at submission. The single highest-leverage change is temporal. Kass and colleagues put “statistical analysis is more than a set of computations” and “plan ahead, really ahead” near the top of their ten rules for effective statistical practice, precisely because analysis choices are constrained by design choices that have already been locked in by the time most people call4. A statistician consulted at design can change the sample size. A statistician consulted at submission can only change the adjectives.

State the estimand before the method. Not “can you run a regression,” but: what quantity, in what population, under what contrast, with what handling of intercurrent events. If you cannot say it in a sentence, that is the finding, and it is worth knowing before you collect data.

Disclose everything you already looked at. Selective reporting is not a moral failing you have to confess; it is a fact about the data-generating process that changes what the numbers mean. An analyst who learns on submission day that you already ran eleven subgroup comparisons has been handed a different problem than the one they thought they were solving.

Settle authorship and credit at the start. The ASA’s Ethical Guidelines for Statistical Practice — eight principles across 72 elements, approved in 2022 — put responsibilities to stakeholders and integrity of data and methods on the same footing5. Statisticians are expected to refuse to produce misleading analyses. That obligation is much easier to honour when they are a named collaborator than when they are an unacknowledged service provider being paid by the hour.

Accept that the honest answer may be “this design can’t answer that.” The most valuable thing a statistician does is sometimes to tell you your study is not powered to detect what you care about. Treating this as obstruction rather than information is how you end up with an underpowered study and a strained relationship.

In Industry and Analytics Settings

Bring the decision, not the query. “Pull last quarter’s same-store sales by region” is a specification. “We’re deciding whether to close three underperforming stores and I need to know whether the recent dip is real” is a problem. The second lets the analyst tell you the first request would have misled you — because, say, the comp-store definition excludes the exact stores you’re evaluating.

Define the metric once, in writing, and stop redefining it in meetings. Most “the numbers don’t match” escalations are two correct calculations of two different things. A metric definition should name its grain, its filters, its date logic, its denominator, and its owner.

Say what happens if the number is wrong. An estimate for a monthly newsletter and an estimate feeding a pricing change deserve different levels of scrutiny, and analysts cannot calibrate effort they cannot see. Tell them the blast radius.

Give real deadlines and real slack. “EOD” for a request that requires joining three source systems is not a deadline; it is a request for an unvalidated guess with your name on it.

What You Owe the Data Engineer

Data engineers are the ones who find out that your business logic is inconsistent, usually at 2 a.m. Treat their constraints as physics, not preferences.

Send the reproduction, not the vibe. “The dashboard looks off” costs an engineer hours. “Row for store 412, 2026-07-14, shows fuel gallons of 0 but the POS extract shows 8,431 — here’s the query” costs minutes.

A defect report has four parts, and “something is wrong” is none of them. Telling an engineer or statistician that a number is wrong hands them a search problem with no search criteria. A usable report specifies:

  1. What is wrong — the exact rows, cells, or figures, with the query or filter that reproduces them.
  2. Why it is wrong — the business or domain fact that contradicts it: “these stores were acquired in March, so they can’t have January comp sales.”
  3. What correct looks like — the expected value or behaviour, with the independent source it comes from (the POS extract, the vendor invoice, the trial protocol).
  4. How to build the correct logic — the rule in implementable terms: “exclude stores from the comp base until they have 13 full months of history, keyed on acquisition_date, not open_date.”

The fourth part is the one collaborators most often skip, because it feels like doing the engineer’s job. It is the opposite: you are the only person who knows the rule. The engineer can translate a rule into SQL in an afternoon; they cannot conjure the rule itself, and if forced to guess, you have just delegated your business logic to the person with the least context for it — which is how the definition-drift problem below gets minted in the first place.

Announce upstream changes before they ship. A renamed field, a new store format, a changed promotion code, a vendor migration — every one of these breaks something downstream. The cost of a two-line heads-up is roughly zero. The cost of discovering it via a silently wrong report is not.

Understand that “just add a column” is rarely just adding a column. It may mean a backfill, a grain change, a contract change with every downstream consumer, and a reconciliation against numbers people have already quoted externally.

Stop building shadow pipelines. The spreadsheet you maintain because the official table was inconvenient will eventually be cited in a meeting as a source of truth, and someone will have to reconcile it. If the official table is wrong, file that as a defect rather than routing around it.

Treat data quality as a shared problem, not a service-desk ticket. In dbt Labs’ 2025 practitioner survey (459 respondents, collected October–December 2024 — a self-selected industry sample, not a probability sample), poor data quality was the most commonly cited challenge, named by more than 56% of respondents6. Data quality originates in operational systems and business processes, which means it is mostly not within the data team’s control, which means it will not be fixed by escalating to the data team.

What Statisticians and Data Engineers Owe Back

The obligations run both directions, and this is where the genre usually goes soft.

Learn the domain well enough to be wrong intelligently. Vance and Smith’s ASCCR framework — Attitude, Structure, Content, Communication, Relationship — is drawn from directing statistical collaboration labs across more than 900 direct projects and supervision of 4,000 more, and it puts Attitude and Relationship on equal footing with Content7. The framework exists because technical correctness delivered incomprehensibly has an effect size of zero.

Report uncertainty as uncertainty. Give the interval and the assumptions it rests on. Treating a P-value as a binary verdict rather than as continuous evidence throws away most of what the analysis produced — the case for S-values and the full compatibility/consonance function rather than a single dichotomized interval.

Say what would change your mind. Before running the analysis, state which results would be surprising and what you would do about them. This is the analyst-side equivalent of pre-registration and it is free.

Make it reproducible or don’t ship it. Given the demonstrated spread across analysts2, 3, an analysis whose choices aren’t recorded is not a result — it’s an anecdote with decimal places. Code in version control, the query saved next to the number, the exclusions written down.

Push back early and in writing. An analyst who accepts an impossible framing in the meeting and privately resents it for six weeks has chosen the worst available option.

Explain the constraint, not just the refusal. “That’s a three-week change because the fact table’s grain is transaction-line and you’re asking for basket-level” is a sentence a business partner can act on. “That’s not possible” is not.

Everything Runs on Assumptions — Ledger Them, Don’t Launder Them

Every model is a bundle of assumptions: linearity, exchangeability, missingness mechanisms, the proposition that the measurement instrument measures the thing. But so is everything else in the collaboration. You assume the chair holds, the laptop boots, the warehouse job ran last night, the engineer understood the ticket, the stakeholder meant what the ticket says. There is no assumption-free work — there is only work whose assumptions are written down and work whose assumptions are discovered later, at a worse time, by someone angrier.

The failure mode is not making assumptions. It is laundering them — letting them pass silently into the result so the final number looks cleaner than the process that produced it. The analyst-variability results above2, 3 are precisely this: dozens of defensible assumption sets, each invisible in its own write-up, visible only when laid side by side.

So the practice, for both sides of the table, is a two-column discipline. For each assumption, state:

  1. The assumption, in one falsifiable sentence.
  2. What is different if it’s wrong — which conclusions survive, which die, and what you’d check first.

A short worked example:

Assumption If it’s wrong
The nightly load completed before the snapshot Yesterday’s figures are partial; totals are biased downward; check the job log before anything else
acquisition_date marks when a store enters the comp base Comp growth mixes organic and acquired stores; the trend is uninterpretable; re-key the base
Missingness in loyalty ID is unrelated to basket size Basket-level estimates are biased toward small transactions; bound the effect or model the missingness
The stakeholder’s “sales” means net of returns Every reconciliation against finance fails by the return rate; re-confirm the definition

Two things happen when you do this. First, most of the second column turns out to be checkable in minutes — the ledger converts vague dread into a short verification queue. Second, disagreements relocate to where they belong: not “your number is wrong” but “I don’t accept row three,” which is a conversation two professionals can actually finish. An assumption stated with its failure consequence is a hypothesis; an assumption left implicit is a landmine with your name on the deed.

An Intake Template

Put this at the top of every request. If it cannot be filled in, the work isn’t ready to start.

Field Why it matters
Decision What action changes based on the answer
Question One sentence, in domain language
Estimand / metric Quantity, population, contrast, grain, denominator
Population & exclusions Who’s in, who’s out, and why
Time window Including how “current” is defined
Known data issues Migrations, backfills, definition changes
Prior looks What has already been run or explored
Blast radius External? Regulatory? Internal only?
Deadline & slack Real date, and what happens if it slips
Owner Who signs off on the definition
Assumption ledger Each assumption, and what changes if it’s wrong

Two Checklists

Before You Ask

Before You Deliver

The Failure Modes Worth Naming

The post-mortem request. Design is locked, data are collected, and the ask is to find something. See Fisher, above.

The definition drift. Two teams, two correct numbers, one metric name. Costs more meeting-hours annually than any modelling error.

The heroic pull. An engineer bypasses the pipeline under deadline pressure; the one-off becomes load-bearing; nobody remembers the filters six months later.

The silent constraint. The analyst knows the design cannot support the claim, says so once, softly, in a meeting, and is not heard. Write it down. Writing it down is the whole technique.

None of this is sophisticated. That is rather the point: the returns to being a good collaborator come almost entirely from unglamorous, cheap, early habits, and almost not at all from anyone being cleverer.

TipOne-Line Summary

Bring the decision, define the metric, disclose the prior looks, warn about upstream changes — and, on the other side, explain the constraint, report the uncertainty, and make it reproducible.

References

1. Fisher RA. (1938). “Presidential address to the first indian statistical congress.” Sankhyā: The Indian Journal of Statistics. 4:14–17.
2. Silberzahn R, Uhlmann EL, Martin DP, Anselmi P, Aust F, Awtrey E, et al. (2018). “Many analysts, one data set: Making transparent how variations in analytic choices affect results.” Advances in Methods and Practices in Psychological Science. 1:337–356. doi: 10.1177/2515245917747646.
3. Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M, et al. (2020). “Variability in the analysis of a single neuroimaging dataset by many teams.” Nature. 582:84–88. doi: 10.1038/s41586-020-2314-9.
4. Kass RE, Caffo BS, Davidian M, Meng X-L, Yu B, Reid N. (2016). “Ten simple rules for effective statistical practice.” PLOS Computational Biology. 12:e1004961. doi: 10.1371/journal.pcbi.1004961.
5. American Statistical Association. (2022). “Ethical guidelines for statistical practice.” https://www.amstat.org/docs/default-source/amstat-documents/EthicalGuidelines.pdf.
6. dbt Labs. (2025). “The 2025 state of analytics engineering report.” https://www.getdbt.com/resources/state-of-analytics-engineering-2025.
7. Vance EA, Smith HS. (2019). “The ASCCR frame for learning essential collaboration skills.” Journal of Statistics Education. 27:265–274. doi: 10.1080/10691898.2019.1687370.
Back to top

Citation

BibTeX citation:
@online{rafi2026,
  author = {Rafi, Zad and Rafi, Zad},
  title = {How to {Be} a {Good} {Collaborator} to a {Statistician} and a
    {Data} {Engineer}},
  date = {2026-08-11},
  url = {https://lesslikely.com/statistics/good-collaborator.html},
  langid = {en-US}
}
]]> statistics https://lesslikely.com/statistics/good-collaborator.html Tue, 11 Aug 2026 04:00:00 GMT The Octagon Is a One-Trial Experiment Zad Rafi https://lesslikely.com/statistics/mma-single-event.html Jon Jones was, by almost any measure, the most complete mixed martial artist ever assembled. In 2023 he returned from a three-year layoff to fight Ciryl Gane for the heavyweight championship. Jones won by first-round submission. It was not particularly close.

Now imagine you had never seen Jones fight. You watched exactly one round of that contest and were asked: what is the probability Jones wins this match?

If you’re a frequentist, you have a problem. Frequency-based probability is formally defined as a limiting ratio — the number of times an event occurs divided by the number of opportunities, as the number of opportunities goes to infinity. A single fight is one trial. You cannot take a limit over one data point. So strictly speaking, the frequentist cannot assign a probability to this specific contest.

This is not a minor inconvenience. It is one of the oldest and most discussed objections in the philosophy of probability: the single-case problem, treated at length by Hájek (2009) and Gillies (2000).

But here is the thing: most people watching that fight thought Jones was a heavy favorite, and they were right. And they were right in a way that a frequentist framework, properly applied, can actually explain — just not in the way the objection implies.

The standard critique goes like this: frequentism requires repeatable experiments. A unique event — this fight, this night, with this version of these two people — can’t be repeated. Therefore frequentism is useless for one-off events. Bayesians, by contrast, treat probability as a degree of belief updated on evidence, which works fine for singular events.

The critique is not wrong as far as it goes. But it conflates two things: the probability of a fight type and the probability of this specific fight.

When a model estimates that Jones wins a matchup like his bout with Gane with, say, a 70% probability, it is not claiming something about a unique unrepeatable event. It is claiming something about a class of contests — fights between a wrestler of Jones’s profile and a striker of Gane’s profile, under roughly these conditions. The probability is a statement about the reference class, not the individual.

This is exactly von Mises’s original framing: probability is always about collectives, not single instances. What we need is the right collective — and choosing that collective is the hard part.

The empirical floor: what the betting market knows

Betting markets in MMA aggregate the predictions of many informed people, and their track record gives us a useful empirical anchor.

The numbers in this section come from market data, not controlled experiments. They describe the aggregate accuracy of odds-setters and bettors, not the underlying probability of any single fight.

Across all UFC fights, favorites win somewhere in the range of 60 to 65 percent of the time — meaning underdogs pull off upsets in more than a third of all fights. That 35-to-40 percent upset rate is notably higher than in most major team sports.

Heavy favorites do better: UFC odds priced between -400 and -900 (implying an 80-to-90 percent win probability) have returned an 88 to 93 percent win rate since 2013. At the other end, fights where odds land between +100 and -122 — roughly a coin flip — have resolved in favor of the nominal favorite only 51 percent of the time.

The market, in other words, does predict fight outcomes with meaningful accuracy when there is a genuine skill gap. It fails — or at least flattens — when the gap is small.

Skill is real; one fight is still noisy

Here is the central claim, stated plainly: a fighter being more skilled than their opponent does not mean they win every fight. It means their probability of winning a single fight is greater than 0.5, perhaps substantially so. But a 70% win probability is also a 30% loss probability.

This is not a trivial point. It is, in miniature, the same confusion that afflicts applied researchers who treat a statistically meaningful estimate as proof that the effect always occurs. The estimate is about the distribution. The distribution is not the single draw.

When Georges St-Pierre was 170 pounds of organized violence in his prime, his aggregate probability of winning any given welterweight fight was very high. But any single fight carried:

  • Opponent stylistic advantages that a model underweights
  • Chance of injury at any moment
  • Variance in execution (did the game plan hold? did the cuts go as expected?)
  • Pure stochastic noise — scrambles in grappling are notoriously non-deterministic

A single punch, a well-timed submission, or a referee stoppage can override any statistical edge a fighter might carry on paper. A scramble on the mat can convert a dominant position into a finished fight in one second.

If you ran that version of GSP against a solid but beatable opponent ten times, you’d expect him to win eight or nine. In one fight, you’d expect him to win — but “expect” means something probabilistic, not certain.

The reference-class problem is the real fight

Choosing the right reference class is where frequentist analysis actually gets difficult in MMA, and where most naive applications go wrong.

Holmes, McHale, and Żychaluk (2023) proposed a Markov chain simulation approach that explicitly estimates fighters’ skills in key aspects of the sport — striking, grappling, clinch work, etc. — and then simulates the fight rather than simply predicting a binary outcome. This matters because the outcome of a fight is not a single draw from a bivariate distribution. It is the aggregate result of hundreds of micro-decisions, each of which has its own stochastic component.

When you model it that way — as they did, comparing favorably to bookmaker odds — you are treating the fight as a sequence of probabilistic events from a well-defined class. The frequentist framework is doing real work. It is just applied at the level of techniques and positions rather than whole fights.

That is a defensible and scientifically useful construction. It is also, notably, the same move statisticians make in clinical medicine: you cannot run a randomized trial on a specific patient, but you can estimate their probability of response from a well-defined reference population and then apply that estimate to guide the decision.

What this means for how you talk about fight outcomes

The mistake most analysts — and most fans — make is binary: the favorite “should” win, the underdog “shouldn’t,” and an upset is a surprise requiring explanation.

The better frame: a fight between a 70% favorite and a 30% underdog is not a fight with a predetermined outcome. It is a draw from a distribution with that shape. Three out of ten times, the underdog wins. That is not an upset requiring special explanation. That is the distribution working correctly.

An upset does warrant explanation if it happens at a rate higher than the odds imply — i.e., if the market is miscalibrated. The Fightful analysis of heavy favorites suggests the market is reasonably calibrated at the extremes, noisier in the middle.

The key methodological point: a single fight is a sample of size one from a fighter’s underlying skill distribution. Skilled fighters have higher-mean distributions with less variance; fighters with fight-ending power but defensive liabilities have wide distributions. The outcome of one draw tells you something, but not much.

This is why it is statistically incoherent to say, after a single upset, that the underdog was “always better.” It is equally incoherent to say, after a dominant win, that the favorite “always wins this.” One draw is one draw.

The Bayesian objection, and why it doesn’t fully resolve things

Bayesians handle this better in some respects. They assign a prior belief about the outcome, update on evidence, and end up with a posterior. For singular events, this is the most coherent framework — a posterior probability genuinely represents a degree of belief about this specific contest, not a reference class.

But the Bayesian approach faces its own problem in combat sports: you need a prior, and priors in MMA are hard to specify. Two fighters may never have faced similar opponents, may have competed in different promotions, and may have trained specifically for this contest in ways the market cannot observe.

In practice, the most useful approach is not doctrinal but instrumental: build the best reference class you can using comparable fights and fighters, fit a model, and interpret the output as a probability statement about the type of contest this represents, while acknowledging that any single fight is an unreliable single observation of the underlying process.

That is frequentism in practice — not a claim that probabilities don’t apply to single fights, but a demand that you ground the probability in something empirically defined.

The upshot

MMA is genuinely less predictable than most sports. The upset rate is substantially higher than in major team sports, and the mechanisms are clear: it is a sport defined by rapid phase transitions (standup, clinch, ground), each carrying its own variance.

That unpredictability is not a bug in the analysis. It is a feature of the process. A skilled fighter’s edge is real and measurable in the aggregate — the betting market shows this plainly at heavy price differentials — but it is not deterministic in a single contest.

The fighter who “should” win is the one with the higher probability under the best available model. Whether they win this time is a separate question, and anyone who collapses the two has made the same mistake researchers make when they treat a significant p-value as proof that the effect always occurs.

One fight is one trial. The distribution is real. The single draw tells you much less than you think.

See also: - Semantic and Cognitive Tools to Aid Statistical Science — the case for replacing “significant” with “compatible” and describing what the data are more and less consistent with - [TODO: link to another relevant post on inference under uncertainty]

References

Back to top ]]> statistics https://lesslikely.com/statistics/mma-single-event.html Tue, 23 Jun 2026 04:00:00 GMT Multi-Language Code Examples Zad Rafi https://lesslikely.com/statistics/code-examples.html

Introduction

This post demonstrates how to include Stata, Julia, and pseudocode in your statistical posts on Less Likely.

Stata Examples

Stata code can be included for syntax highlighting:

* ============================================
* Example: Linear Regression Analysis
* ============================================

* Load example dataset
sysuse auto, clear

* Descriptive statistics
summarize price mpg weight foreign

* Create a scatter plot
scatter price mpg, title("Price vs. MPG")

* Run linear regression
regress price mpg weight foreign

* Store estimates
estimates store model1

* Display results with confidence intervals
regress price mpg weight foreign, level(95)

* Test joint significance
test mpg weight

* Predict fitted values
predict price_hat

* Calculate residuals
predict residuals, residuals

* Plot residuals
scatter residuals price_hat, ///
  yline(0) ///
  title("Residual Plot")

Advanced Stata: Panel Data Analysis

* ============================================
* Panel Data Example
* ============================================

* Load panel dataset
webuse nlswork, clear

* Declare panel structure
xtset idcode year

* Summary statistics by panel
xtsum ln_wage age

* Fixed effects regression
xtreg ln_wage age tenure, fe

* Random effects regression
xtreg ln_wage age tenure, re

* Hausman test
hausman fe re

Julia Examples

Julia code for statistical computing:

# ============================================
# Example: Linear Regression in Julia
# ============================================

using DataFrames, Statistics, GLM, Plots

# Create sample data
n = 100
x = randn(n)
y = 2.0 .+ 3.0 .* x .+ randn(n) .* 0.5

# Create DataFrame
df = DataFrame(x = x, y = y)

# Fit linear model
model = lm(@formula(y ~ x), df)

# Display results
println(model)

# Plot data and fitted line
scatter(x, y, label="Data", alpha=0.6)
plot!(x, predict(model), label="Fitted Line", linewidth=2)
xlabel!("X")
ylabel!("Y")
title!("Linear Regression")

Advanced Julia: Bootstrap Confidence Intervals

# ============================================
# Bootstrap Example
# ============================================

using Distributions, StatsBase

# Define a function to bootstrap
function bootstrap_mean(data, n_bootstrap=1000)
    n = length(data)
    bootstrap_means = zeros(n_bootstrap)

    for i in 1:n_bootstrap
        # Resample with replacement
        sample = rand(data, n)
        bootstrap_means[i] = mean(sample)
    end

    return bootstrap_means
end

# Generate sample data
data = randn(100) .+ 5.0

# Perform bootstrap
bootstrap_samples = bootstrap_mean(data)

# Calculate confidence interval
ci_lower = quantile(bootstrap_samples, 0.025)
ci_upper = quantile(bootstrap_samples, 0.975)

println("95% Bootstrap CI: [", ci_lower, ", ", ci_upper, "]")

# Plot bootstrap distribution
histogram(bootstrap_samples,
         bins=30,
         label="Bootstrap Distribution",
         xlabel="Sample Mean",
         ylabel="Frequency")
vline!([ci_lower, ci_upper],
       label="95% CI",
       linewidth=2,
       color=:red)

Pseudocode Examples

Example 1: Algorithm Block

Algorithm 1 Bootstrap Confidence Interval

Input: Data , statistic , confidence level

Output: Confidence interval

  1. for to do
    • Draw bootstrap sample by sampling observations from with replacement
    • Calculate
  2. end for
  3. Sort in ascending order
  4. Set
  5. Set
  6. return

Example 2: Simple Pseudocode Block

ALGORITHM: Linear Regression via Gradient Descent INPUT: X (n × p design matrix), y (n × 1 response vector), learning rate α OUTPUT: β (p × 1 coefficient vector) 1. Initialize β = 0, iteration = 0, maxiter = 1000 2. REPEAT until convergence OR iteration > maxiter: a. Compute predictions: ŷ = Xβ b. Compute residuals: r = y - ŷ c. Compute gradient: ∇L = -2X'r/n d. Update parameters: β = β - α∇L e. iteration = iteration + 1 3. RETURN β

Example 3: Detailed Algorithm with Comments

Algorithm 2 Maximum Likelihood Estimation

Input: Data , log-likelihood function , initial value

Output:

  1. Set ,
  2. while not converged do
    • Compute score function:
    • Compute Hessian:
    • Update:
    • Check convergence: if then break
    • if then break
  3. end while
  4. Compute standard errors:
  5. return

Comparing Approaches

Here’s how the same analysis looks across languages:

Task: Calculate Mean and 95% CI

R:

data <- rnorm(100, mean = 5, sd = 2)
mean_val <- mean(data)
ci <- t.test(data)$conf.int

Julia:

using Statistics, HypothesisTests
data = randn(100) .* 2 .+ 5
mean_val = mean(data)
ci = confint(OneSampleTTest(data))

Stata:

clear
set obs 100
generate data = rnormal(5, 2)
mean data

Python:

import numpy as np
from scipy import stats
data = np.random.normal(5, 2, 100)
mean_val = np.mean(data)
ci = stats.t.interval(0.95, len(data)-1,
                      loc=mean_val,
                      scale=stats.sem(data))

Best Practices

  1. Use syntax highlighting for display-only code - Most readers just need to see the code
  2. Execute code sparingly - Only execute when output is essential
  3. Document your setup - Note required packages and versions
  4. Test before publishing - Ensure code runs without errors
  5. Provide context - Explain what the code does and why

Resources

Conclusion

With support for multiple languages, you can now share statistical methods and algorithms in the language that best suits your analysis and audience.

Back to top ]]> statistics programming tutorial https://lesslikely.com/statistics/code-examples.html Tue, 06 Jan 2026 05:00:00 GMT A Critical Look At The REDUCE-IT Trial Zad Rafi https://lesslikely.com/medicine/reduce-it.html It’s finally here. The results of the REDUCE-IT trial were presented today at the American Heart Association conference and the paper is also up on The New England Journal of Medicine. I’ve been excited to see the results of this trial ever since the announcement in September (which I wrote about here) so let’s get right into it.

I’ll just copy and paste what I’d written before in September regarding the inclusion/criteria and statistical analyses so that I can get right into the results. You can skip this section and go to the results if you’d like or you can reread it to get reacquainted with the trial design.

September 2018:

That’s why I thought it would be a good time to review the preregistration protocol of the REDUCE-IT trial on ClinicalTrials.gov and a detailed paper that describes the rationale behind the study design. Let’s get into it.

So very clearly, the authors state what the study here is doing,

“AMR101 (icosapent ethyl [ethyl-EPA]) is a highly purified ethyl ester of eicosapentaenoic acid (EPA) being developed by Amarin Pharma Inc. for the treatment of hypertriglyceridemia.

The purpose of this study is to evaluate whether this drug, combined with a statin therapy, will be superior to the statin therapy alone, when used as a prevention in reducing long-term cardiovascular events in high-risk patients with mixed dyslipidemia.”

Comparing statins + highly purified, high doses of EPA to statins alone. Why are they using such high doses? The authors explain here,

“It is worth noting that the promising results from JELIS occurred with a high‐purity EPA preparation dosed at 1.8 g/d in a Japanese population, for whom the baseline EPA levels are higher than in western populations due to greater dietary intake of marine omega‐3 fatty acids.

Icosapent ethyl 12‐week dosing at 4 g/d in a high‐risk population similar to that within the Reduction of Cardiovascular Events with Icosapent Ethyl–Intervention Trial (REDUCE‐IT) who had persistent elevations of TG after treatment with statins resulted in significant reductions in TG and atherogenic lipoproteins, as well as comparable plasma EPA levels as the 1.8 g/d dosing group in JELIS._

T herefore, a dose of 4 g/d was selected as the dose for further study. In this context, REDUCE‐IT was designed to determine if treatment with icosapent ethyl 4 g/d vs placebo would reduce ischemic events in patients at increased CV risk already being treated with statins.”

Also “worth noting” that the JELIS trial was included in both meta-analyses mentioned at the beginning of the blog post.

The primary outcome is a composite endpoint that consists of CV death, MI, stroke, coronary revascularization, and hospitalization for unstable angina. Whenever I hear composite endpoint, I often become skeptical because I’ve seen composite endpoints abused or misused in several ways, resulting in very misleading results. There was a very nice post on composite endpoints by Hilda Bastian here.

The study (REDUCE-IT) enrolled 8,000 participants and began in 2011 and ended in 2018.

Why 8,000 participants?

The authors explain here,

“The sample‐size calculation was based on a hazard ratio assumption of 0.85. Therefore, 1612 events would be required to have approximately 90% power with a 1‐sided α‐level of 2.5% and with 2 interim analyses. This results in a total target sample size of 7990 patients.

Approximately 70% of randomized patients were to be in CV risk stratum 1 (established CVD) and approximately 30% of randomized patients were to be in CV risk stratum 2 (high‐risk primary prevention defined by diabetes mellitus and other risk factors). Randomization was stratified by CV risk strata, ezetimibe use, and by geographical region.”

The design power calculations seem like they were based on very optimistic expectations of the effect size, and apparently, justifiably so!

A bit unfortunate that there is no mention anywhere of what statistical analyses they planned on doing exactly.

In order for participants to be eligible for the trial:

  • they had to be over the age of 45

  • have high levels of triglycerides (hypertriglyceridemia) originally >150 mg/dL but then changed to >200 mg/dL via a protocol change

  • been on statins for at least a month

  • they had to have a history of heart disease

  • There’s a bunch of exclusion criteria too that I don’t really feel the need to go over, but you can see them listed out on the registry.

Also, worth mentioning of course that the trial is funded by Amarin, AKA the manufacturer of Vascepa.

November 2018

Results:

Quick Characteristics of Participants

  • Median Age: 64 years old

  • Females: 28.8%

  • Median LDL: 75.0 mg/dL

  • Median HDL: 40 mg/dL

  • Median Triglycerides: 216.0 mg/dL

Changes in Lipids:

The median change in triglycerides from baseline to 1 year was a decrease of 18.3% (−39.0 mg per deciliter) for the Vascepa group and an increase of 2.2% (4.5 mg per deciliter).

A between-group comparison of change scores shows that the median group reduction was 19.7% higher (44.5 mg/dL, p<0.001) in the Vascepa group than it was in the placebo group.

LDL increased in both groups from baseline to year 1, however, the Vascepa group had a 6.6% (5.0 mg per deciliter, p<0.001) lower increase than the placebo group.

Differences in Clinical Endpoints (what we really care about):

Survival curve showing the primary endpoints for the REDUCE-IT trial

A total of 1606 participants experienced one of the primary end-point events.

Approximately 17.2% of the participants in the Vascepa group experienced a primary end-point whereas 22.2% of the participants in the control group experienced a primary end-point.

Thus, the between-group difference is 4.8% (95% CI, 3.1 to 6.5). The results are: (hazard ratio, 0.75; 95% confidence interval [CI], 0.68 to 0.83; P<0.001).

So very clearly we can see that there was a statistically significant difference and the confidence interval leans towards an effect.

The authors calculated a number needed to treat for this between-group difference (NNT: 21 (95% CI, 15 to 33)) and its confidence interval which slightly makes me laugh. I’ve written about the NNT before and its flaws and the reality is that it doesn’t do what it claims to do and has very bizarre statistical properties.

Survival curve showing the secondary endpoints for the REDUCE-IT trial

With regards to secondary efficacy end-point events, 11.2% of the participants in the Vascepa group experienced an event compared to 14.8% of the participants in the placebo group.

The absolute between-group difference was 3.6% (95% CI, 2.1 to 5.0).

Again, very clearly a difference.

A forest plot showing prespecified subgroup analyses for the REDUCE-IT trial

They also ran prespecified subgroup analyses to see whether the effects would be influenced. What was interesting to me was when they claimed this:

“Baseline triglyceride levels (≥150 vs. <150 mg per deciliter or ≥200 or <200 mg per deciliter) had no influence on the primary or key secondary efficacy end points (Figures 2 and Figure 3).”

This is interesting to me because there were so few patients that were enrolled into the trial with triglyceride levels that were beneath 150 or hovering around 150. This is very clearly reflected by the wide confidence intervals in that subgroup which can be seen in the figure above. This is because the protocol changed early in the trial to restrict participant inclusion to people with higher triglycerides,

“The first protocol amendment in May 2013 changed the lower limit of the acceptable triglyceride level from 150 mg per deciliter to 200 mg per deciliter (2.26 mmol per liter), with no allowance for variability.”

Whereas originally, it was,

“Eligible patients had a fasting triglyceride level of 150 to 499 mg per deciliter (1.69 to 5.63 mmol per liter)… because of the intraindividual variability of triglyceride levels, the initial protocol allowed for a 10% lower triglyceride level from the target lower limit, which permitted patients to be enrolled if they had a triglyceride level of at least 135 mg per deciliter (1.52 mmol per liter).”

Seems like more data is needed to say anything about this with confidence.

Hierarchical Tests

For their prespecified hierarchical tests, this is what they found:

  • Rates of Composite Ischemic Endpoints + Rates of Cardiovascular Death: (Vascepa: 4.3% vs. Placebo: 5.2%; hazard ratio, 0.80; 95% CI, 0.66 to 0.98; P=0.03)

  • Rate of Death From Any Cause: Vascepa: 6.7% vs. Placebo: 7.6% (hazard ratio, 0.87; 95% CI, 0.74 to 1.02).

  • Doesn’t seem to be statistically significant, but the coverage of the intervals leans towards an effect.

  • Rates of Adjudicated Sudden Cardiac Death: Vascepa: 1.5% vs. Placebo: 2.1% (hazard ratio, 0.69; 95% CI, 0.50 to 0.96)

  • Rates of Cardiac Arrest: Vascepa: 0.5% vs. Placebo: 1.0% (hazard ratio, 0.52; 95% CI, 0.31 to 0.86)

A forest plot showing hierarchial analyses for the REDUCE-IT trial

So most seem to favor the Vascepa group.

Safety

What about its safety? A look at the supplementary appendix gives us this:

Table showing the number of serious adverse events per group in the REDUCE-IT trial

The analyses failed to show any statistically significant differences for serious treatment-emergent adverse events.

Table showing the number of adverse events per group in the REDUCE-IT trial

However, among other treatment-emergent adverse events, Peripheral Edema, Constipation, and Atrial Fibrillation were significantly more frequent in the Vascepa group.

There you have it.

Conclusions

The results of this large trial clearly suggest that Vascepa had a notable effect on triglycerides and clinical endpoints. These data along with the results from the JELIS trial should be considered as supporting evidence for the hypothesis that high doses of EPA (not your over-the-counter normal fish oil) may have an effect on cardiovascular disease events (even though JELIS was an open-label study).

Hopefully, future meta-analyses make high doses of EPA an inclusion criteria to avoid pooling null results from trials using smaller dosages, where it may be more difficult to see an effect. Currently, there are a few other ongoing trials that are also using high doses of EPA:

“Ongoing trials of moderate-to-high doses of pure EPA ethyl ester will provide further information on the effects of these agents. These trials include the Randomized Trial for Evaluation in Secondary Prevention Efficacy of Combination Therapy–Statin and EPA (RESPECT-EPA; UMIN Clinical Trials Registry number, UMIN000012069), a secondary prevention outcomes trial involving statin-treated patients in Japan, and the Effect of Vascepa on Improving Coronary Atherosclerosis in People with High Triglycerides Taking Statin Therapy (EVAPORATE; ClinicalTrials.gov number, NCT02926027), which is examining changes in coronary plaque over 9 to 18 months.”

Even though REDUCE-IT has been successful, it is only one trial, and it’s always worth remembering what the history of fish is, even though this trial didn’t look at fish oil but rather a pharmaceutical formulation of EPA.

A Brief Timeline of Fish Oil

  • 1970s | A group of explorers noticed that people who consumed fish also had lower incidences of heart disease.

  • 1980s-1990s | Early observational studies also saw a connection between fish consumption and heart disease risk

  • 1990s | A few randomized trials were conducted to see the effects of fish oil supplementation and CVD outcomes

  • 2002 | The American Heart Association put out a statement suggesting that fish oil supplementation was beneficial for fatal outcomes

  • 2002 - 2016 | Several randomized trials and systematic reviews found inconsistent evidence for the effects of fish oil supplementation and CVD outcomes

  • 2016 | The American Heart Association creates a scientific advisory group to look at all the evidence and concludes that despite the controversy, there was a small benefit from fish oil and it could be worth it for people who had a history of CVD events.

  • 2018 | Large meta-analysis published in JAMA with 77,000 + participants found no benefit from fish oil supplementation on real clinical endpoints.

  • Mid - 2018 | Most comprehensive meta-analysis and systematic review with 112,000 participants by Cochrane finds very small benefits to several CVD-related outcomes.

  • Late 2018 | Announcement by the manufacturer of Vascepa exclaiming that high-doses of fish oil in the REDUCE-IT trial had a large effect (25% reduction) on CVD outcomes.

Edit: It came to my attention after writing this blog post that the placebo formulation included large amounts of mineral oil. Thus, the control arm was taking mineral oil + statins, while the intervention group was taking Vascepa + statins. This is problematic because mineral oil is a known laxative and could possibly interfere with the absorption of statins. Some data to suggest this is the change in LDL within the control arm. LDL increased after the trial, which doesn’t make much sense given that they were taking statins.

We have sufficient evidence from the literature to suggest that statins should lower LDL. It is also worth pointing out that side effects like diarrhea were far more prevalent within the control group in a 7,000+ clinical trial. Thus, I suspect that the control group may have been given enough mineral oil that it possibly interfered with the absorption of statins within the gut, thus reducing their efficacy, and making the difference between the Vascepa group and control group even more substantial.

Amarin, the manufacturer of Vascepa, has responded to some of these criticisms here.

Back to top ]]> medicine https://lesslikely.com/medicine/reduce-it.html Tue, 06 Jan 2026 05:00:00 GMT Seductive Surrogates Can Be Deadly Zad Rafi https://lesslikely.com/medicine/seductive-surrogates.html In clinical trials, it’s not always possible to measure hard endpoints like cardiovascular disease events and cancer remission rates. Studies that use clinical outcomes often dichotomize these variables, and as a result, they need to have a large number of participants and be long in duration to detect differences between groups.

Again, this type of research is expensive and not always feasible. In many scenarios, a more practical alternative is to focus on intermediate markers. Intermediate markers are biomarkers associated with a clinical outcome.1

For example, C-reactive protein (CRP) is a molecule that is strongly associated with inflammation, and you can often find that CRP levels increase when inflammation increases.2–4 Because inflammation is associated with coronary heart disease (CHD)5–8, an investigator may choose to focus on decreasing levels of inflammation in a clinical trial (measured by CRP), rather than focus on how many CHD-related deaths the intervention prevents.

If the changes in an intermediate marker can robustly predict changes in a hard endpoint, and if it’s part of the primary pathway of the clinical outcome, then your biomarker can be considered a surrogate marker for that clinical outcome.1 (I’m not sure if CRP is a good surrogate outcome for CHD events, just used it as an example).

Low-density lipoprotein (LDL) is considered an excellent surrogate marker for CHD because reducing LDL levels also seems to reduce the number of CHD events.9,10 Unfortunately, this can all go wrong if the intermediate marker is associated with a clinical outcome, but is not involved in the causal pathway of the outcome and is confounded by other phenomena.

Torcetrapib

A great example of this is the story of high-density lipoprotein (HDL) and myocardial infarction (heart attacks). Several studies had found associations between low levels of HDL, often considered “good cholesterol,” and heart attacks.11–17

So, it shouldn’t come as a surprise that a drug (torcetrapib) was produced by Pfizer that attempted to increase the amount of HDL with the hopes that it would reduce the number of CVD events.

A group of researchers administered the drug to thousands of patients.18 The drug was successful in changing the lipids of the participants to more favorable numbers. Patients who received the drug had a 24.9% decrease in LDL and a 72.1% increase in HDL. Seems pretty great. However, the number of deaths increased by 58% and the number of heart attacks increased by 21%.

A systematic review and meta-analysis19 published in the BMJ a few years later concluded the following after pooling studies that focused on interventions that primarily increased HDL and interventions that primarily decreased LDL,

“Available data suggest that simply increasing the amount of circulating high-density lipoprotein cholesterol does not reduce the risk of coronary heart disease events, coronary heart disease deaths, or total deaths. The results support reduction in low-density lipoprotein cholesterol as the primary goal for lipid modifying interventions.”

Svensson20 provides us with a very lovely table showing other scenarios for which a drug had a favorable effect on a surrogate marker but had a negative impact on the clinical outcome.

Surrogate endpoints and how they've conflicted with clinical endpoints.

What does this tell us? That focusing only on improving surrogate markers is not a very good idea if there is not much concordance between trials that focus on surrogates and trials that focus on hard endpoints. Until we’ve established a robust causal link between a surrogate marker and a hard endpoint, we shouldn’t be seduced by surrogates.

Back to top

References

1.
Hulley SB. Designing Clinical Research. Lippincott Williams & Wilkins; 2007.
2.
Bray C, Bell LN, Liang H, et al. Erythrocyte Sedimentation Rate and C-Reactive Protein Measurements and Their Relevance in Clinical Medicine. WMJ : official publication of the State Medical Society of Wisconsin. 2016;115(6):317-321.
3.
Chew KS. What’s new in Emergencies Trauma and Shock? C-Reactive protein as a potential clinical biomarker for influenza infection: More questions than answers. Journal of Emergencies, Trauma, and Shock. 2012;5(2):115. doi:10.4103/0974-2700.96477
4.
Pepys MB, Hirschfield GM. C-reactive protein: A critical update. Journal of Clinical Investigation. 2003;111(12):1805-1812. doi:10.1172/JCI18921
5.
Frangogiannis NG. Regulation of the inflammatory response in cardiac repair. Circulation Research. 2012;110(1):159-173. doi:10.1161/CIRCRESAHA.111.243162
6.
Hansson GK. Inflammation, atherosclerosis, and coronary artery disease. New England Journal of Medicine. 2005;352(16):1685-1695. doi:10.1056/NEJMra043430
7.
Libby P. Inflammation and cardiovascular disease mechanisms. American Journal of Clinical Nutrition. 2006;83(2):456S-460S. doi:10.1093/ajcn/83.2.456S
8.
Ruparelia N, Chai JT, Fisher EA, Choudhury RP. Inflammatory processes in cardiovascular disease: A route to targeted therapies. Nature Reviews Cardiology. 2017;14(3):133-144. doi:10.1038/nrcardio.2016.185
9.
Cholesterol Treatment Trialists’ (CTT) Collaboration, Fulcher J, O’Connell R, et al. Efficacy and safety of LDL-Lowering therapy among men and women: Meta-analysis of individual data from 174,000 participants in 27 randomised trials. Lancet (London, England). 2015;385(9976):1397-1405. doi:10.1016/S0140-6736(14)61368-4
10.
Taylor F, Huffman MD, Macedo AF, et al. Statins for the primary prevention of cardiovascular disease. Cochrane Database Syst Rev. 2013;(1):CD004816. doi:10.1002/14651858.CD004816.pub5
11.
Assmann G, Schulte H, Eckardstein A von, Huang Y. High-density lipoprotein cholesterol as a predictor of coronary heart disease risk. The PROCAM experience and pathophysiological implications for reverse cholesterol transport. Atherosclerosis. 1996;124 Suppl:S11-20.
12.
Curb JD, Abbott RD, Rodriguez BL, et al. A prospective study of HDL-C and cholesteryl ester transfer protein gene mutations and the risk of coronary heart disease in the elderly. Journal of Lipid Research. 2004;45(5):948-953. doi:10.1194/jlr.M300520-JLR200
13.
Gordon T, Castelli WP, Hjortland MC, Kannel WB, Dawber TR. High density lipoprotein as a protective factor against coronary heart disease. The Framingham Study. American Journal of Medicine. 1977;62(5):707-714.
14.
Gordon DJ, Probstfield JL, Garrison RJ, et al. High-density lipoprotein cholesterol and cardiovascular disease. Four prospective American studies. Circulation. 1989;79(1):8-15.
15.
Rahilly-Tierney CR, Spiro A 3rd, Vokonas P, Gaziano JM. Relation between high-density lipoprotein cholesterol and survival to age 85 years in men (from the VA normative aging study). American Journal of Cardiology. 2011;107(8):1173-1177. doi:10.1016/j.amjcard.2010.12.015
16.
Sharrett AR, Ballantyne CM, Coady SA, et al. Coronary heart disease prediction from lipoprotein cholesterol levels, triglycerides, lipoprotein(a), apolipoproteins A-I and B, and HDL density subfractions: The Atherosclerosis Risk in Communities (ARIC) Study. Circulation. 2001;104(10):1108-1113.
17.
Turner RC, Millns H, Neil HA, et al. Risk factors for coronary artery disease in non-insulin dependent diabetes mellitus: United Kingdom Prospective Diabetes Study (UKPDS: 23). BMJ (Clinical research ed). 1998;316(7134):823-828.
18.
Barter PJ, Caulfield M, Eriksson M, et al. Effects of torcetrapib in patients at high risk for coronary events. New England Journal of Medicine. 2007;357(21):2109-2122. doi:10.1056/NEJMoa0706628
19.
Briel M, Ferreira-Gonzalez I, You JJ, et al. Association between change in high density lipoprotein cholesterol and cardiovascular disease morbidity and mortality: Systematic review and meta-regression analysis. BMJ (Clinical research ed). 2009;338:b92. doi:10.1136/bmj.b92
20.
Svensson S, Menkes DB, Lexchin J. Surrogate outcomes in clinical trials: A cautionary tale. JAMA Intern Med. 2013;173(8):611-612. doi:10.1001/jamainternmed.2013.3037
]]> medicine https://lesslikely.com/medicine/seductive-surrogates.html Tue, 06 Jan 2026 05:00:00 GMT How Did We Figure Out Smoking Causes Lung Cancer? Zad Rafi https://lesslikely.com/medicine/smoking-and-lung-cancer.html Proving a cause and effect relationship isn’t easy. Causality is complex, and there are thousands of philosophical and mathematical texts on it that are beyond my understanding. However, I do understand enough to discuss how we arrive at cause and effect relationships in the sciences.

One of the first things drilled into students in a research methods course is that correlation does not equal causation. It’s also taught in a cult-like way that one of the best ways to prove causality in the health and social sciences is with a perfectly designed randomized controlled trial (RCT). Why?

A few reasons:

These are difficult to do with observational/epidemiological studies. There are always some spurious links that have nothing to do with the phenomena of interest as seen below:

Spurious associations such as the number of letters in the winning word of the spelling bee and the number of people killed by venomous spiders

So perfect RCTs are believed to be one of the best ways to prove causality, and at some point in history, we determined that smoking causes lung cancer in humans. Does that mean we recruited participants, randomized them to a group, and made them smoke?

Nope. Although several unethical experiments on humans have been carried out in the past, this was not one of them. Institutional Review Boards wouldn’t allow an investigation like that, where participants could potentially be harmed to such a large degree.

Wait, if there are no RCTs testing the effects of cigarettes, then how did we determine that smoking causes lung cancer? In the mid-1900s, two researchers (Richard Doll and Austin Bradford Hill) produced data showing that smoking frequency was strongly correlated with lung cancer incidence. The data were compelling. Look at the numbers below.

Data from the British Doctor's study

But again, these were associations from observational studies (case controls), not experiments. What if people who had lung cancer also had genes that made them more likely to smoke? What if they happened to be drawn to smoking? You couldn’t dismiss these possibilities with correlations.

This is what legendary statistician, empiricist, and potato scientist, Ronald A. Fisher, exclaimed at the time.

Picture of Ronald A. Fisher

Fisher was often a critic of these findings and of the surgeon general’s report, that smoking causes lung cancer. He claimed that these conclusions were purely correlational and it’s also what tobacco companies exclaimed to defend their highly profitable product. And they weren’t wrong.

Alright, we’re in a dilemma. Perfectly designed randomized trials are the one of best ways to establish causality, but you can’t conduct randomized experiments to see whether smoking causes cancer, because it’s unethical, so you’re limited to epidemiological/observational studies.

So researchers have to work with what they have. And that’s what they did in the mid-1900s. There were already suspicions at the time, that consumption of tobacco products caused harmful gene mutations from studies carried out in animal models. Specific compounds found in tobacco products had also been established as carcinogens decades before the surgeon general’s report, and researchers were already open to the idea that smoking wasn’t the greatest habit. Unfortunately, rigorous human data were missing.

The results from the British Doctor’s Study (the study conducted by Richard Doll and Austin Bradford Hill) was the nail in the coffin because it provided evidence (although correlational) that smoking frequency increased the incidence rates of lung cancer.

One of the researchers on that original paper (Austin Bradford Hill) went on to establish the Bradford Hill Criteria, a set of viewpoints that Hill had discussed in a speech that he believed helped establish the link between smoking and lung cancer.

Interestingly enough, he was also a pioneer of the randomized controlled trial!

Picture of Austin Bradford Hill

In Hill’s speech, he exclaimed that one could potentially arrive at causality from nine viewpoints:

  1. The strength of the association

  2. Consistency

  3. Specificity

  4. Temporality

  5. Dose dependency

  6. Biological mechanism

  7. Coherence

  8. Experiment

  9. Analogy

The relationship between smoking and lung cancer fit many of these scenarios. There was a biological mechanism (DNA damage from carcinogens in tobacco products), the effect sizes of the associations were large, there was a dose-dependent relationship (groups that smoked more had a higher incidence of lung cancer), there was consistency amongst the data observed, and there was coherence ( multiple lines of evidence to support a relationship: animal models, cellular biology, molecular, and epidemiological, etc.). Robust research needs multiple lines of evidence.

Even without experiments, the evidence from multiple lines was overwhelming.

It’s still worth mentioning that epidemiology has its limits. Temporality is often a problem because it’s hard to figure out which variable caused the other, and not all confounding variables can be adjusted for (over adjustment bias can be a problem and there will always be residual confounding), but if there are multiple lines of evidence to suggest a relationship, then there could be one.

Back to top ]]> medicine https://lesslikely.com/medicine/smoking-and-lung-cancer.html Tue, 06 Jan 2026 05:00:00 GMT The Rise, Fall, and Rise Again of Fish Oil? Zad Rafi https://lesslikely.com/medicine/vascepa-reduce-it.html With the exception of this past week, it hasn’t been a very good year for fish oil. In early 2018, a large meta-analysis and systematic review of several randomized trials was published in JAMA.

This large pooled analysis (77,000 + participants) assessed the impact of supplementing fish oil on cardiovascular disease events and found that there was no statistically significant effect (p=.10). The relative risks (RR) and coverage of the confidence intervals (CI) also suggested a very small effect (RR: 0.97, 95% CI: 0.93 to 1.01) for any major vascular event.

A few months later Cochrane came out with their own systematic review and meta-analysis which was more comprehensive (79 studies, 112,000 participants) than the JAMA meta-analysis but also came to very similar conclusions of there being a small effect for several outcomes.

Cochrane Primary Outcomes Results:

  • All-Cause Mortality: (RR: 0.98, 95% CI: 0.93 to 1.03)

  • Cardiovascular Mortality: (RR: 0.95, 95% CI: 0.87 to 1.03)

  • Combined Cardiovascular Events: (RR: 0.99, 95% CI: 0.94 to 1.04)

  • Coronary Heart Mortality: (RR: 0.93, 95% CI: 0.79 to 1.09)

  • Coronary Heart Disease Events: (RR: 0.93, 95% CI: 0.88 to 0.97)

  • Stroke: (RR: 1.06, 95% CI: 0.96 to 1.16)

  • Arrhythmia: (RR: 0.97, 95% CI: 0.90 to 1.05)

At this time, it wasn’t looking particularly good for fish oil. Science-Based Medicine wrote an article titled, “No Benefit from Fish Oil,” which largely suggested that fish oil had little utility based on the 2018 meta-analyses.

While these two recent meta-analyses of fish oil were very comprehensive, they weren’t able to include the results of two trials (STRENGTH & REDUCE-IT) that were using extremely high doses of fish oil (3-4 g/d) because those trials were still ongoing, and the data were not available.

A few days ago, Amarin (the manufacturer of the patented, high-dose, purified EPA product, Vascepa) made an announcement that the results from the REDUCE-IT trial suggested a fairly notable reduction in CVD events (25% reduction in the relative risk, apparently, RR: 0.75, p<0.001).

Several reports of this announcement declared it as a major event in the field of cardiology, comparable to the significance of statins. Of course, many folks (including myself) are still skeptical and will remain so until the results are published.

For those who are unaware, fish oil has had one of the most interesting histories of all nutritional supplements that are available today. I’ve used this timeline before in a previous blog post but have updated it to include the newer events/studies.

A Brief Timeline of Fish Oil

  • 1970s | A group of explorers noticed that people who consumed fish also had lower incidences of heart disease.

  • 1980s-1990s | Early observational studies also saw a connection between fish consumption and heart disease risk

  • 1990s | A few randomized trials were conducted to see the effects of fish oil supplementation and CVD outcomes

  • 2002 | The American Heart Association put out a statement suggesting that fish oil supplementation was beneficial for fatal outcomes

  • 2002 - 2016 | Several randomized trials and systematic reviews found inconsistent evidence for the effects of fish oil supplementation and CVD outcomes

  • 2016 | The American Heart Association creates a scientific advisory group to look at all the evidence and concludes that despite the controversy, there was a small benefit from fish oil and it could be worth it for people who had a history of CVD events.

  • 2018 | Large meta-analysis published in JAMA with 77,000 + participants found no benefit from fish oil supplementation on real clinical endpoints.

  • Mid - 2018 | Most comprehensive meta-analysis and systematic review with 112,000 participants by Cochrane finds very small benefits to several CVD-related outcomes.

  • Late 2018 | Announcement by the manufacturer of Vascepa exclaiming that high-doses of fish oil in the REDUCE-IT trial had a large effect (25% reduction) on CVD outcomes.

Fish oil was accompanied by a lot of hype in the early days and little evidence, but eventually, like most things, it took a beating once several research groups started prodding the existing evidence.

The results of this trial (when they are presented in November at the AHA conference) and when they are published in a peer-reviewed journal, may potentially change the minds of skeptics everywhere.

That’s why I thought it would be a good time to review the preregistration protocol of the REDUCE-IT trial on ClinicalTrials.gov and a detailed paper that describes the rationale behind the study design. Let’s get into it.

So very clearly, the authors state what the study here is doing,

“AMR101 (icosapent ethyl [ethyl-EPA]) is a highly purified ethyl ester of eicosapentaenoic acid (EPA) being developed by Amarin Pharma Inc. for the treatment of hypertriglyceridemia.

The purpose of this study is to evaluate whether this drug, combined with a statin therapy, will be superior to the statin therapy alone, when used as a prevention in reducing long-term cardiovascular events in high-risk patients with mixed dyslipidemia.”

Comparing statins + highly purified, high doses of EPA to statins alone. Why are they using such high doses? The authors explain here,

“It is worth noting that the promising results from JELIS occurred with a high‐purity EPA preparation dosed at 1.8 g/d in a Japanese population, for whom the baseline EPA levels are higher than in western populations due to greater dietary intake of marine omega‐3 fatty acids.

Icosapent ethyl 12‐week dosing at 4 g/d in a high‐risk population similar to that within the Reduction of Cardiovascular Events with Icosapent Ethyl–Intervention Trial (REDUCE‐IT) who had persistent elevations of TG after treatment with statins resulted in significant reductions in TG and atherogenic lipoproteins, as well as comparable plasma EPA levels as the 1.8 g/d dosing group in JELIS.

Therefore, a dose of 4 g/d was selected as the dose for further study. In this context, REDUCE‐IT was designed to determine if treatment with icosapent ethyl 4 g/d vs placebo would reduce ischemic events in patients at increased CV risk already being treated with statins.”

Also “worth noting” that the JELIS trial was included in both meta-analyses mentioned at the beginning of the blog post.

The primary outcome is a composite endpoint that consists of CV death, MI, stroke, coronary revascularization, and hospitalization for unstable angina. Whenever I hear composite endpoint, I often become skeptical because I’ve seen composite endpoints abused or misused in several ways, resulting in very misleading results. There was a very nice post on composite endpoints by Hilda Bastian here.

The study (REDUCE-IT) enrolled 8,000 participants and began in 2011 and ended in 2018.

Why 8,000 participants?

The authors explain here,

“The sample‐size calculation was based on a hazard ratio assumption of 0.85. Therefore, 1612 events would be required to have approximately 90% power with a 1‐sided α‐level of 2.5% and with 2 interim analyses. This results in a total target sample size of 7990 patients.

Approximately 70% of randomized patients were to be in CV risk stratum 1 (established CVD) and approximately 30% of randomized patients were to be in CV risk stratum 2 (high‐risk primary prevention defined by diabetes mellitus and other risk factors). Randomization was stratified by CV risk strata, ezetimibe use, and by geographical region.”

The design power calculations seem like they were based on very optimistic expectations of the effect size, and apparently, justifiably so!

A bit unfortunate that there is no mention anywhere of what statistical analyses they planned on doing exactly.

In order for participants to be eligible for the trial:

  • they had to be over the age of 45

  • have high levels of triglycerides (hypertriglyceridemia)

  • been on statins for at least a month

  • they had to have a history of heart disease

There’s a bunch of exclusion criteria too that I don’t really feel the need to go over, but you can see them listed out on the registry.

Also, worth mentioning of course that the trial is funded by Amarin, AKA the manufacturer of Vascepa.

Even though my title mentions “fish oil” it really is important to point out that this is NOT the average fish oil supplement that you’ll get over the counter. It’s a highly purified substance, that’s currently being prescribed by doctors in high dosages for hypertriglyceridemia. It is a drug.

Vascepa versus normal fish oil

An extremely expensive one ($250+ for a 30 day supply… ) that most insurance companies will not cover unless your blood tests show an indication of high triglycerides.

So, while the substance technically is fish oil, a very purified, high-dose version of it, the results of the REDUCE-IT trial are not evidence of your average fish oil supplement having these same effects.

I look forward to hearing about the results of this study in November.

Edit: I’ve covered that study in this blog post.

Back to top ]]> medicine https://lesslikely.com/medicine/vascepa-reduce-it.html Tue, 06 Jan 2026 05:00:00 GMT Revisiting Eggs and Dietary Cholesterol Zad Rafi https://lesslikely.com/nutrition/revisiting-eggs.html Buckle up, folks. This is going to be a long one. In this post, I want to discuss three main things:

Let’s get right into it.

A Summary of the JAMA Study

Last week, a nutritional epidemiological study (from now on nutri epi) published in JAMA posited that egg consumption was associated with an increased risk of cardiovascular disease incidence.1 The results gained significant coverage from the media.

With that coverage came the usual criticism of “correlation not being causation,” and how nutri epi studies produce findings that are often contradicted by randomized trials, etc. Unlike many of these surface-level criticisms, I want to dive a little deeper into the findings of this study and the utility and limitations of studies like this. But first, a quick summary of the study.

Who’s Being Studied?

This study included data from six prospective cohorts and had a total sample size of 29,615 adults. That’s a lot of people. Those who follow nutrition studies should be familiar with many of the cohorts included such as:

  • the Atherosclerosis Risk in Communities (ARIC) Study
  • the Coronary Artery Risk Development in Young Adults (CARDIA) Study
  • the Framingham Heart Study (FHS)
  • the Framingham Offspring Study (FOS)
  • the Jackson Heart Study (JHS)
  • the Multi-Ethnic Study of Atherosclerosis (MESA)

Data collection started in early 1985 and halted in the summer of 2016 (nearly 31 years of data). The project was involved in answering the question of whether dietary cholesterol was associated with cardiovascular disease incidence.

What and How Were the Data Collected?

So how were dietary data collected? The authors give a very brief description in the methods,

Using a standardized protocol and data dictionary, diet data were harmonized cohort by cohort. Briefly, consumption frequencies were converted into estimated number per day using the middle value (eg, 3-4 times per week = 0.5 times per day).

However, a look at the supplementary files gives us more insight into how the data were collected:

A table describing the data collection methods for the six cohorts

So they mainly used various food frequency questionnaires which differed depending on the cohort.

The primary outcomes were…

The primary outcomes were incident CVD and all-cause mortality. Incident CVD was a composite end point of fatal and nonfatal coronary heart disease (0), stroke, heart failure, and CVD death from other causes.

Here are some of the covariates they collected data for

Standardized questionnaires and laboratory protocols were used to collect information on the following variables: age, sex, race/ethnicity, education, lifestyle factors (including smoking, alcohol intake, and physical activity), body mass index (BMI, calculated as weight in kilograms divided by height in meters squared), blood pressure, lipid profile, medication use, and medical conditions.

and what they adjusted for in their three statistical models. Let’s focus mainly on model 2 since they use that as a reference.

Models were sequentially adjusted for age, sex, race/ethnicity (white, black, Hispanic, Chinese), education (<high school, high school, some college or more) (model 1); plus total energy, smoking status (current, former, never), smoking pack-years (0, 0.1-4.9, 5.0-9.9, 10-19.9, 20-29.9, 30-39.9, ≥40), cohort-specific physical activity z score, alcohol intake (gram), use of hormone therapy (y/n) (model 2);

What Were the Results?

The authors graphed the hazard ratios by the amount of dietary cholesterol consumed per day (mg) for CVD incidence…

Graphed models of the associations of dietary cholesterol intake and CVD incidence

So what were the main results that the authors reported?

Based on model 2, each additional 300 mg of dietary cholesterol consumed per day was significantly associated with higher risk of incident CVD (adjusted HR, 1.17 [95% CI, 1.09-1.26]; adjusted ARD, 3.24% [95% CI, 1.39%-5.08%]; Figure 2A) and all-cause mortality (adjusted HR, 1.18 [95% CI, 1.10-1.26]; adjusted ARD, 4.43% [95% CI, 2.51%-6.36%]; Figure 2B).

So a 17% increase in the risk of incident CVD where a 9% increase all the way up to a 26% increase were reasonably compatible with the data and its assumptions. The results are practically the same for all-cause mortality.

The authors also adjusted for individual foods or combinations of foods to see how the effect deviated with model 2 being used as the reference.

Forest plot of dietary cholesterol and CVD incidence

  • For incident CVD (shown above), the effect estimate increased from 17% to 20% when adjusted for eggs alone, and decreased to 13% when adjusted for eggs, unprocessed red meat, and processed meat.

  • For all-cause mortality (graph not shown), the effect estimate dropped from a 17% increase to a 13% increase when adjusted for eggs alone and dropped to a 5% increase when adjusted for eggs, unprocessed red meat, and processed meat.

Here’s what the authors concluded:

The current study found that the significant associations of dietary cholesterol consumption with incident CVD and all-cause mortality were independent of fat amount and quality of the diet. Further, the significant association between egg consumption and incident CVD was fully accounted for by the cholesterol content in eggs…

Mechanistically, eggs and processed or unprocessed red meat are rich in other nutrients such as choline, iron, carnitine, and added sodium (for processed meat) that have been implicated in CVD risk via different pathways.

And then they discuss the limitations of their study:

First, appropriate interpretation of the study findings requires consideration of measurement error for self-reported diet data. Further, this study relied on single measurement of egg and dietary cholesterol consumption. Exposure misclassification may be of concern, but results were similar when censoring participants at different time points…

…Fifth, generalizing our results to non-US populations requires caution due to different nutrition and food environments and chronic disease epidemiology. Sixth, the study findings are observational and cannot establish causality.

Interestingly enough, right after saying the results cannot establish causality, they write this,

Among US adults, higher consumption of dietary cholesterol or eggs was significantly associated with higher risk of incident CVD and all-cause mortality in a dose-response manner. These results should be considered in the development of dietary guidelines and updates.

Well, I have a lot to say, but let’s first take a look at how the media interpreted and reported these results. For that, we shall consult the Google oracle.

How Did the Media Respond?

The media response to the publication of the JAMA egg study

This is one of my favorite ones.

Fear mongering headlines about eggs

Okay, now that we’ve done that, we can discuss the measurements and analyses.

How Reliable Are the Measurements and Analyses?

As the authors mentioned in the methods, and supplementary files, most of the dietary data were collected via a food frequency questionnaire (FFQ). This is important for a couple of reasons, and I’m going to dive a little deeper than the usual “FFQ data are useless”.

FFQ data are known to suffer from serious measurement error, and there is a giant body of literature on this. For example, such data can be prone to classical measurement error (), systematic error, heteroscedastic error, and differential error.

Methods like regression calibration, moment reconstruction, multiple imputation, and graphical methods can be used to help address many of these errors, along with sensitivity analyses. For a more detailed read of this, see Keogh & White, 2014.2

Unfortunately, the authors of the JAMA study hardly bothered with this, and merely pointed out in their limitations section that their study was limited by measurement error.

This study has several limitations. First, appropriate interpretation of the study findings requires consideration of measurement error for self-reported diet data. Further, this study relied on single measurement of egg and dietary cholesterol consumption. Exposure misclassification may be of concern, but results were similar when censoring participants at different time points…

This isn’t something to take this lightly given how serious of a problem it is. There have been extensive criticisms published of FFQs with one of the most prominent of reviews stating the following six arguments as to why memory-based questionnaires should not be used.3

  1. The use of memory-based methods is founded upon two inter-related logical fallacies: a category error and reification
  2. Human memory and recall are not valid instruments for scientific data collection
  3. The measurement errors of memory-based dietary assessment methods are neither quantifiable nor falsifiable; this renders these methods and data pseudoscientific
  4. The post hoc process of pseudoquantification is impermissible and invalid
  5. Memory-based dietary data were repeatedly demonstrated to be physiologically implausible (i.e., meaningless numbers)
  6. The failure to cite or acknowledge contrary evidence and empirical refutations contributed to a fictional discourse on diet-disease relations

The authors don’t even believe that the measurement error from memory-based questionnaires can even be addressed! They conclude the following:

Our conclusion is that nutrition epidemiology is a degenerating research paradigm in which the use of expedient but nonfalsifiable anecdotal evidence impeded scientific progress and engendered a fictional discourse on diet-health relations. The continued funding and use of memory-based methods such as Food Frequency Questionnaires and 24-hour dietary interviews is anathema to evidence-based public policy.

Thus, our recommendation is simply that the use of memory-based methods must stop and all previous articles presenting memory-based methods data and conclusions must be corrected to include contrary evidence and address the consequences of a half century of pseudoscientific claims.

Ouch. Those are quite polarizing statements. Some researchers have argued that Archer et al have overstated their case against FFQs, but agree that FFQs are far from perfect.4 This discussion isn’t new though. FFQs have been criticized for so long that they’ve spurred parody videos like the one below:

Those are some of the issues with the measurements, now onto the analyses (I’ll be using many of the arguments from my nutri epi blog post). Unfortunately, the analytical plans of studies like this are rarely registered in advance, especially because data collection for many of these studies began decades ago, and most of the analyses are often viewed as being “exploratory.” This means that there is a great deal of analytical flexibility in what covariates to include in a model.

Some of the biggest critics of this have been statistician Stan Young and meta-researcher John Ioannidis. Young wrote the following in the journal Significance.5

For example, consider the use of linear regression to adjust the risk levels of two treatments to the same background level of risk. There can be many covariates, and each set of covariates can be in or out of the model. With ten covariates, there are over 1000 possible models. Consider a maze as a metaphor for modelling (Figure 3).

Analytical flexibility depicted by attempting to get through a maze
The maze of forking paths.

The red line traces the correct path out of the maze. The path through the maze looks simple, once it is known. Returning to a linear regression model, terms can be put into and taken out of a regression model. Once you get a p‐value smaller than 0.05, the model can be frozen and the model selection justified after the fact. It is easy to justify each turn.

Patel, Burford, and Ioannidis call this flexibility and its impact on the effect size, the “vibration of effects.”6 In their study (which I’ve mentioned several times in previous blog posts), they downloaded several variables from the NHANES dataset that were linked to all-cause mortality and were able to substantially change the effect size by including certain covariates and excluding others,

Ioannidis and colleagues downloaded 13 variables from the NHANES dataset that were linked to all-cause mortality, and that had a substantial number of participants associated with each variable (at least 1000 participants and 100 deaths). From those 13 variables, they were able to produce 8,192 different statistical models that all resulted in different hazard ratios.

Their recommendations were to report all possible statistical models from all possible combinations of covariates and to report the median of all these models, rather than selectively reporting a few models. And to note whether there was something called a “Janus effect”, where the effect size would go in both directions.

In the last pattern, as exemplified by α-tocopherol, the estimated HRs can be both greater and less than the null value (HR > 1 and HR ≤ 1) depending on what adjustments were made. We call this the Janus effect after the two-headed representation of the ancient Roman god.

For α-tocopherol, most of the HR and p-values were concentrated around 1 and non-significance, respectively… The Janus effect is common: 131 (31%) of the 417 variables had their 99th percentile HR>1 and their 1st percentile HR<1.

However, statistician/epidemiologist, Sander Greenland has been a critic of meta-research and epidemiological studies that lack biochemical sophistication and lump several compounds together as if they were the same and behaved the same in the body.

“For nutrition, the lack of biochemistry sophistication among the trial designers leads to a lot of dubious and noncomparable studies, while the meta-analysts and reviewers do a lot of distortive lumping, e.g., talking about “vitamin E” as if that were a single entity.

A recent review by prominent authors didn’t even notice that almost all trials used the racemic synthetic mixture, dl-alpha-tocopheryl, (which they misidentify with “alpha tocopherol”) with vastly diferent and unjustifed dosages, and which hardly resembles the eight or so natural d-tocopherols or d-tocotrienols that account for dietary intake.”6

Ioannidis and Young are also prominent critics of the fact that nutri epi studies and randomized trials have poor concordance. For example, in Young’s 2011 paper published in Significance (which Ioannidis cites often), he (Young) states the following,

We ourselves carried out an informal but comprehensive accounting of 12 randomised clinical trials that tested observational claims – see Table 1. The 12 clinical trials tested 52 observational claims. They all confirmed no claims in the direction of the observational claims. We repeat that figure: 0 out of 52. To put it another way, 100% of the observational claims failed to replicate.

In fact, five claims (9.6%) are statistically significant in the clinical trials in the opposite direction to the observational claim. To us, a false discovery rate of over 80% is potent evidence that the observational study process is not in control. The problem, which has been recognised at least since 1988, is systemic.

This seems pretty bad. I can agree with Young and Ioannidis that nutri epi studies can often produce much noise given the analytical flexibility and measurement error. However, I’m not particularly convinced by the poor concordance data for two reasons.

  1. Most RCTs have designs that are likely to be underpowered for an effect size of interest. Contrast this with nutri epi studies which are often enormous and span decades. So fixating on statistical significance with this in mind makes little sense to me.

  2. Statistical significance has far more utility in studies that have some random mechanism, whether that be random assignment or random sampling. Otherwise, they make little sense to fixate on, especially in nutri epi studies. Greenland, 1990 summarizes this well below.7

Randomization provides the key link between inferential statistics and causal parameters. Inferential statistics, such as P values, confidence intervals, and likelihood ratios, have very limited meaning in causal analysis when the mechanism of exposure assignment is largely unknown or is known to be nonrandom. It is my impression that such statistics are often given a weight of authority appropriate only in randomized studies

I’m reasonably confident that Greenland has changed his mind since then on the appropriate weight of authority such statistics should be given, even in randomized studies. Especially given the recent Nature commentary he published on the problems with statistical significance and dichotomous thinking.8

In the same 1990 paper, Greenland also points out the problems with several probabilistic arguments that are used to justify classical statistics in areas where the exposure is unknown or unlikely to be random. He proposes some possible solutions:

In causal analysis of observational data, valid use of inferential statistics as measures of compatibility, conflict, or support depends crucially on randomization assumptions about which we are at best agnostic and more usually doubtful. Among the possible remedies are:

  • Restrain our interpretation of classical statistics by explicating and criticizing any randomization assumptions that are necessary for probabilistic interpretations;
  • train our students and retrain ourselves to focus on nonprobabilistic interpretations of inferential statistics;
  • deemphasize inferential statistics in favor of pure data descriptors, such as graphs and tables;
  • expand our analytic repertoire to include more elaborate techniques that depend on assumptions in the “agnostic” rather than the “doubtful” realm, and subject the results of these techniques to influence and sensitivity analysis.

These are neither mutually exclusive nor exhaustive possibilities, but I think any one of them would constitute an improvement over much of what we have done in the past.

Now, with all of these nuances in mind, how convincing is this observational study that has produced a hazard ratio of 1.17 with a 95% CI ranging from 1.09 - 1.26? I don’t think it’s that convincing. I’m not saying that dietary cholesterol or eggs don’t impact CVD incidence and serum cholesterol, but just that I don’t find this study’s data to be very convincing.

Perry Wilson also has an interesting argument where he utilized graphical methods to assess the plausibility of the results,

Does Dietary Cholesterol Impact Serum Cholesterol?

This has been a very controversial subject ever since dietary cholesterol was first implicated in cardiovascular disease in the 1900s with one of the most famous epidemiological studies being the Seven Countries Study.9 I’m not going to touch on that study since I wouldn’t be doing it justice, but there’s plenty of reading material out there.

But here’s a very brief timeline of dietary cholesterol:

Brief Timeline of Dietary Cholesterol

  • 1950s | Dietary cholesterol is implicated in heart disease.10
  • Late 1900s | Seven Countries Study is published.9
  • Late 1900s | Metabolic ward experiments find that dietary cholesterol impacts serum cholesterol.11
  • Late 1900s | Other trials fail to find an effect.12
  • Late 1900s - Early 2000s | Government agencies and medical organization recommend limiting dietary cholesterol.13
  • 2013 | American Heart Association states that there’s insufficient evidence whether removing cholesterol is beneficial.14
  • 2015 | Dietary guidelines drop arbitrary recommendations on limiting dietary cholesterol, but still recommend little intake.

Wow, that can be pretty confusing and frustrating… Wait till you see the next study I talk about.

What Do Reviews of Randomized Trials Say?

A few months ago, a meta-regression of only randomized controlled trials that were at least two weeks long (and a million other inclusion/exclusion criteria) was published in The American Journal of Clinical Nutrition.15 This study was interested in assessing the relationship between dietary cholesterol and serum cholesterol. Here are some characteristics of the study:

  • Included 55 randomized trials (2,652 subjects), with 44 being crossover trials.
  • Used Bayesian methods with uninformed priors.
  • Used the Mensink equation to subtract the impact of fatty acids on cholesterol to isolate the impact of dietary cholesterol better.
  • Explored both linear and nonlinear models.

So what were the results?

The nonlinear models better fit the data than the linear model and at 100 mg/d of dietary cholesterol change, there was an associated 4.58 mg/dL change in LDL cholesterol. The linear model estimated this change to be 1.90 mg/dL, which is similar to estimates found by the meta-analysis of metabolic ward studies.11 At 200-400 mg/d of dietary cholesterol change, the nonlinear and linear models had substantial differences in their estimations (see the table below and then the graph).

Table of changes in LDL associated with dietary cholesterol

Linear, MM, and Hill models graphed

Wow. So dietary cholesterol does indeed impact serum cholesterol (LDL)! Here’s what the authors write in their conclusion…

The meta-regression analyses presented herein indicate that, after providing a theoretical control for intakes of SFAs, MUFAs, PUFAs, and when possible, TFAs, there is a positive, dose-related trend between changes in cholesterol intake and changes in circulating LDL cholesterol, a finding that is consistent with the published literature.

The nonlinear dose-response shapes (MM and Hill) best fit the data. For a 100-mg/d increase in dietary cholesterol intake, circulating LDL-cholesterol concentrations are predicted to increase by 1.90 mg/dL (linear model), 4.46 mg/dL (MM model), or 4.58 mg/dL (Hill model).

Then they discuss the practical significance of this.

The results from these meta-analyses have potential implications for dietary recommendations. For comparison, the equation given by Mensink et al. predicts a change of 1.23 mg/dL for each 1% increase in SFAs in exchange for carbohydrate.

Therefore, based on the nonlinear models, increasing dietary cholesterol by 100 mg/d would be predicted to have an effect comparable to that of increasing dietary SFAs by 3.7%, and increasing dietary cholesterol by 200 mg/d is predicted to be comparable to increasing dietary SFAs by 5.5%.

A large egg contains ∼185 mg cholesterol and would therefore be expected to increase the LDL-cholesterol concentration by 6.5 mg/dL based on the nonlinear models. However, the majority of dietary cholesterol is not attributable to egg intake in the United States, except for individuals in the highest quartile of TC intake.

According to a recent publication reporting dietary sources of cholesterol in US adults aged ≥20 y based on NHANES 2013–2014 data, mean dietary cholesterol intake was 293 mg/d and the primary dietary cholesterol source in the overall population was meat (defined as poultry, mixed dishes, red meat, processed meat, and seafood), which accounted for 42% of dietary cholesterol.

Wait, so Should We Be Worried About Eggs?

I’m going to argue that we probably shouldn’t. The reason is that just because a food contains dietary cholesterol doesn’t mean it will be absorbed efficiently in the gastrointestinal tract. Foods are not homogenous and do not contain the same nutrients, anti-nutrients, and minerals. Consuming eggs in a sitting is not the same as consuming pure dietary cholesterol nor is it the same as consuming other foods high in cholesterol (poultry, red meat, and processed meat).

This also seems to be supported by a few randomized crossover trials that have shown that the cholesterol in eggs doesn’t seem to be well absorbed.16 Several other trials have also failed to show an effect on serum cholesterol.12,17,18 Is it possible that these trials were just underpowered in the first place? Very possible.

This all needs to be taken into account with the fact that in addition to the randomized trials included in the meta-regression mentioned above,15 there exists a plethora of metabolic ward studies suggesting an effect of dietary cholesterol on serum cholesterol.11 However, many of the metabolic ward trials included in the prominent meta-analysis by Clarke et al11 were also nonrandomized,

Solid food diets were assessed in 72 of these reports among 129 groups of subjects in 395 experiments with various designs (109 randomised crossover, 57 randomised or matched parallel, 77 non-randomised Latin square, and 152 non-randomised sequential).

So while there is plenty of evidence suggesting that a diet high in dietary cholesterol impacts serum cholesterol, the evidence to suggest that eggs in particular have this effect seems far less concrete. There also may be a number of nutrients found in eggs that inhibit cholesterol absorption.19,20 Here are some excerpts from Kuang et al20

Yang et al. reported that dietary PUFAs, PC, and SM significantly inhibit the uptake of cholesterol in Caco-2 monolayer, which may have potential therapeutic effect on reducing cholesterol absorption as functional food ingredients [44, 45]….

Ezetimibe reduces the absorption of cholesterol in the small intestine by inhibiting the activity of NPC1L1 protein [52].

Therefore, consuming eggs may inhibit the intestinal absorption of cholesterol via the phospholipids in eggs. A fundamental question is how this biological effect is regulated.

How convincing is this evidence? I’m still not sure. Much of it seems to be very preliminary, but I’m leaning towards the idea that most people probably don’t have to worry about eggs, especially since Americans don’t even get most of their dietary cholesterol from eggs.15 I don’t find the existing data for eggs (including those from this study) to be convincing. More better research is needed.

We need less research, better research, and research done for the right reasons.21 - Doug Altman

See my other articles on nutritional epidemiology:

Back to top

References

1.
Zhong VW, Horn LV, Cornelis MC, et al. Associations of Dietary Cholesterol or Egg Consumption With Incident Cardiovascular Disease and Mortality. JAMA. 2019;321(11):1081-1095. doi:10.1001/jama.2019.1572
2.
Keogh RH, White IR. A toolkit for measurement error correction, with a focus on nutritional epidemiology. Statistics in Medicine. 2014;33(12):2137-2155. doi:10.1002/sim.6095
3.
Archer E, Marlow ML, Lavie CJ. Controversy and debate: Memory-Based Methods Paper 1: The fatal flaws of food frequency questionnaires and other memory-based dietary assessment methods. Journal of Clinical Epidemiology. 2018;104:113-124. doi:10.1016/j.jclinepi.2018.08.003
4.
Subar AF, Freedman LS, Tooze JA, et al. Addressing Current Criticism Regarding the Value of Self-Report Dietary Data12. The Journal of Nutrition. 2015;145(12):2639-2645. doi:10.3945/jn.115.219634
5.
Young SS, Karr A. Deming, data and observational studies. Significance. 2011;8(3):116-120. doi:10.1111/j.1740-9713.2011.00506.x
6.
Patel CJ, Burford B, Ioannidis JPA. Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. J Clin Epidemiol. 2015;68(9):1046-1058. doi:10.1016/j.jclinepi.2015.05.029
7.
Greenland S. Randomization, statistics, and causal inference. Epidemiology. 1990;1(6):421-429. doi:10.1097/00001648-199011000-00003
8.
Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature. 2019;567(7748):305. doi:10.1038/d41586-019-00857-9
9.
Feinleib M. Seven Countries: A Multivariate Analysis of Death and Coronary Heart Disease. JAMA. 1981;245(5):511-512. doi:10.1001/jama.1981.03310300063026
10.
Duff GL, McMILLAN GC. Pathology of atherosclerosis. The American Journal of Medicine. 1951;11(1):92-108.
11.
Clarke R, Frost C, Collins R, Appleby P, Peto R. Dietary lipids and blood cholesterol: Quantitative meta-analysis of metabolic ward studies. BMJ (Clinical research ed). 1997;314(7074):112-117.
12.
Bowman MP, Van Doren J, Taper LJ, Thye FW, Ritchey SJ. Effect of dietary fat and cholesterol on plasma lipids and lipoprotein fractions in normolipidemic men. The Journal of Nutrition. 1988;118(5):555-560. doi:10.1093/jn/118.5.555
13.
Brownawell AM, Falk MC. Cholesterol: Where science and public health policy intersect. Nutrition Reviews. 2010;68(6):355-364. doi:10.1111/j.1753-4887.2010.00294.x
14.
Eckel RH, Jakicic JM, Ard JD, et al. 2013 AHA/ACC guideline on lifestyle management to reduce cardiovascular risk: A report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Journal of the American College of Cardiology. 2014;63(25 Pt B):2960-2984. doi:10.1016/j.jacc.2013.11.003
15.
Vincent MJ, Allen B, Palacios OM, Haber LT, Maki KC. Meta-regression analysis of the effects of dietary cholesterol intake on LDL and HDL cholesterol. Am J Clin Nutr. 2019;109(1):7-16. doi:10.1093/ajcn/nqy273
16.
Kim JE, Campbell WW. Dietary Cholesterol Contained in Whole Eggs Is Not Well Absorbed and Does Not Acutely Affect Plasma Total Cholesterol Concentration in Men and Women: Results from 2 Randomized Controlled Crossover Studies. Nutrients. 2018;10(9). doi:10.3390/nu10091272
17.
Chenoweth W, Ullmann M, Simpson R, Leveille G. Influence of dietary cholesterol and fat on serum lipids in men. The Journal of Nutrition. 1981;111(12):2069-2080. doi:10.1093/jn/111.12.2069
18.
Vorster HH, Benadé AJ, Barnard HC, et al. Egg intake does not change plasma lipoprotein and coagulation profiles. The American Journal of Clinical Nutrition. 1992;55(2):400-410. doi:10.1093/ajcn/55.2.400
19.
Jesch ED, Carr TP. Food Ingredients That Inhibit Cholesterol Absorption. Preventive Nutrition and Food Science. 2017;22(2):67-80. doi:10.3746/pnf.2017.22.2.67
20.
Kuang H, Yang F, Zhang Y, Wang T, Chen G. The Impact of Egg Nutrient Composition and Its Consumption on Cholesterol Homeostasis. Cholesterol. 2018. doi:10.1155/2018/6303810
21.
Altman DG. The scandal of poor medical research. BMJ. 1994;308(6924):283-284. doi:10.1136/bmj.308.6924.283
]]> nutrition https://lesslikely.com/nutrition/revisiting-eggs.html Tue, 06 Jan 2026 05:00:00 GMT Confidence, Posteriors, and the Bootstrap Zad Rafi Sir Panda https://lesslikely.com/statistics/bootstrap.html

The Power of the Bootstrap

The bootstrap1, 2 is an incredibly powerful method of approximation. Indeed, frequentists are even able to approximate posterior distributions using it, In an excerpt from the paper below, Efron describes how these procedures are similar and gives examples using the parametric bootstrap.

The point of this brief note is to say that in some situations the bootstrap, in particular the parametric bootstrap, offers an easier path toward the calculation of Bayes posterior distributions. An important but somewhat under-appreciated article by Newton and Raftery (1994) made the same point using nonparametric bootstrapping. By “going parametric,” we can illustrate more explicitly the bootstrap/MCMC connection. The arguments here will be made mainly in terms of a simple example, with no attempt at the mathematical justifications seen in Newton and Raftery.

It is not really surprising that the bootstrap and MCMC share some overlapping territory. Both are general-purpose computer-based algorithmic methods for assessing statistical accuracy, and both enable the statistician to deal effectively with nuisance parameters, obtaining inferences for the interesting part of the problem. On the less salubrious side, both share the tendency of general-purpose algorithms toward overuse.

Of course the two methodologies operate in competing inferential realms: frequentist for the bootstrap, Bayesian for MCMC. Here, as in Newton and Raftery, we leap over that divide, bringing bootstrap calculations to bear on the Bayesian world. The working assumption is that we have a Bayesian prior in mind and the only question is how to compute its posterior distribution. Arguments about the merits of Bayesian versus frequentist analysis will not be taken up here, except for our main point that the two camps share some common ground.

However, they can not only approximate posterior distributions, but confidence distributions too. Confidence distributions are an incredibly dense and technical topic and is practically an umbrella term to refer to multiple concepts, for example, they can refer to confidence curves, confidence densities and deviance functions. For a definitive source to learn more about what they are, see the following3, 4

In a paper prepared for Efron’s Fisher lecture, he writes that confidence distributions may provide the key to syncretism between frequentist and bayesian motivations:

Fiducial distributions, or confidence densities, of- fer a way to finesse this difficulty. A good argument can be made that the confidence density is the posterior density for the parameter of interest, after all of the nuisance parameters have been integrated out in an objective way. If this argument turns out to be valid, then our progress in constructing approximate confidence intervals, and approximate confidence densities, could lead to an easier use of Bayesian thinking in practical problems. This is all quite speculative, but here is a safe prediction for the 21st century: statisticians will be asked to solve bigger and more complicated prob- lems. I believe that there is a good chance that objective Bayes methods will be developed for such problems, and that something like fiducial infer- ence will play an important role in this develop- ment. Maybe Fisher’s biggest blunder will become a big hit in the 21st century!

Indeed, some authors have shown that the bootstrap distribution is equal to the confidence distribution because it meets the definition of a confidence distribution.5–7. Some have even referred to posteriors as a quick and dirty approximation to confidence distributions.8

Properties of Confidence Distributions

Xie and Singh, as summarized by Fraser define these properties as:

The first is the classical definition CL and defines the confidence distribution function as the distribution function version of the confidence quantile function ̃, where is a level confidence interval.

The second definition (2.1) identifies a confidence distribution function as a distribution function in θ for each given y and as a pivot with a uniform distribution for eachθ. But this also closely identifies with what Fisher (1930) offered: fiducial but with Fisher’s promotion replaced by a claim that the argument is pure frequentist. Of course it is pure frequentist just as Fisher (1930) was pure frequentist, except for Fisher’s accompanying claim to having purer probabilities, which was then a direct confrontation to the Bayes aficionados of the time.

The third definition (2.2) essentially gives just the quantile equivalent say ) of the distribution function where u is Uniform (0,1). Aren’t confidence quantile and confidence upper bound just different labelling for the same object? Confidence distributions can have many properties: the distribution function should of course be , but also it should inherit continuity when present in the model, should use all available information, and should generally be sensible. These properties aren’t really addressed in the authors’ proposal. A promotion of confidence distributions should acknowledge these inherent issues and also mention marginal and conditional conflicts as discussed in the literature.

The bootstrap distribution and the asymptotic consonance distribution would be defined as:

Efron writes,

As the authors point out, all of this has something to do with the bootstrap. Let represent B bootstrap replications ofθˆ, an estimator of parameter θ(possibly in the presence of nuisance parameters). The -th empirical quantile of the θˆ∗’s is then theupper endpoint of a first-order accurate-levelαconfidence interval. In this sense, the bootstrap distribution is an approximate confidence distribution. The BCa density, Efron & Tibshirani (1998), improves the confidence accuracy by reweighting the values. Let Gˆbe this empirical cdf, and and a be the bias correction and acceleration constants in my 1987 paper. Rather than equal weights 1/B, the BCa density puts weight proportional to . The reweighted bootstrap distribution then becomes a second-order accurate confidence distribution. Efron (2012) discusses this construction in the context of objective Bayes inference.

Fraser, however, argues that the connection is nothing special

The bootstrap.The bootstrap as in Section 2.3 provides an approximation to the distributions described by a model, and in doing this the bootstrap can also eliminate the influence of nuisance parameters. It can be applied to statistics or to pivots, with faster effect using suitable pivots. It can be used with least squares, or with maximum like lihood statistics, or with statistical quantities, or anywhere where distributions are wanted. Ofcourse confidence calculations are just one such use but there are many others including of course testing. So there is no particular attachment of the bootstrap to confidence distribution functions other than providing an approximate means of calculation for such.

Certain bootstrap methods such as the BCa method and t-bootstrap method also yield second order accuracy of consonance distributions.

Here, I demonstrate how to use these particular bootstrap methods to arrive at consonance curves and densities.

We’ll use the Iris dataset and construct a function that’ll yield a correlation coefficient and we will try to estimate the confidence intervals for them.

The Nonparametric Bootstrap

iris <- datasets::iris
foo <- function(data, indices) {
  dt <- data[indices, ]
  c(cor(dt[, 1], dt[, 2], method = "p"))
}

We can now use the curve_boot() method to construct a function. The default method used for this function is the “Bca” method provided by the bcaboot package (also written by Efron).

I will suppress the output of the function because it is unnecessarily long. But we’ve placed all the estimates into a list object called y.

The first item in the list will be the consonance distribution constructed by typical means, while the third item will be the bootstrap approximation to the consonance distribution.

ggplot2::qplot(boot::boot(iris, foo, 1000)$t, geom = "histogram") +
  theme_light()

ggcurve(data = y[[2]], type = "cd", nullvalue = TRUE, fill= "red") +
  theme_light()

We can also print out a table for TeX documents

ztable(gg <- curve_table(data = y[[1]]))
Lower Limit Upper Limit Interval Width Interval Level (%) CDF P-value S-value (bits)
2500 -0.14 -0.10 0.05 25.0 0.62 0.75 0.42
5000 -0.17 -0.07 0.10 50.0 0.75 0.50 1.00
7500 -0.20 -0.03 0.17 75.0 0.88 0.25 2.00
8000 -0.21 -0.02 0.19 80.0 0.90 0.20 2.32
8500 -0.22 -0.01 0.21 85.0 0.92 0.15 2.74
9000 -0.24 0.00 0.24 90.0 0.95 0.10 3.32
9500 -0.26 0.03 0.29 95.0 0.98 0.05 4.32
9750 -0.28 0.05 0.33 97.5 0.99 0.03 5.32
9900 -0.30 0.07 0.38 99.0 1.00 0.01 6.64

More bootstrap replications will lead to a smoother function. But for now, we can compare these two functions to see how similar they are.

result <- curve_from_se(estimate = 2.5, se = 0.8, measure = "mean")
ggcurve(result[[1]], type = "c", nullvalue = TRUE)


# Odds ratio with SE (on log scale internally)
result <- curve_from_se(estimate = 1.5, se = 0.2, measure = "ratio")
ggcurve(result[[1]], type = "c", nullvalue = TRUE)

If we wanted to look at the bootstrap standard errors, we could do so by loading the fifth item in the list

where in the top row, theta is the point estimate, and sdboot is the bootstrap estimate of the standard error, sdjack is the jacknife estimate of the standard error. z0 is the bias correction value and a is the acceleration constant.

The values in the second row are essentially the internal standard errors of the estimates in the top row.

One can also construct the confidence density, here I provide an example from the pvaluefunction R package The consonance curve and density are nearly identical. With more bootstrap replications, they are very likely to converge. [aadland2015ajcn]

library("pvaluefunctions")
result <- bcaboot::bcajack(
  x = iris, B = 10000, func = foo,
  alpha = c((1:99) / 100), verbose = FALSE
)

bca <- data.frame(result[["lims"]][, 1])

ggplot2::qplot(bca$result...lims......1., geom = "density") +
  theme_less()

The Parametric Bootstrap

For the examples above, we mainly used nonparametric bootstrap methods. Here I show an example using the parametric Bca bootstrap and the results it yields.

data(diabetes, package = "bcaboot")
X <- diabetes$x
y <- scale(diabetes$y, center = TRUE, scale = FALSE)
lm.model <- lm(y ~ X - 1)
mu.hat <- lm.model$fitted.values
sigma.hat <- stats::sd(lm.model$residuals)
t0 <- summary(lm.model)$adj.r.squared
y.star <- sapply(mu.hat, rnorm, n = 1000, sd = sigma.hat)
tt <- apply(y.star, 1, function(y) summary(lm(y ~ X - 1))$adj.r.squared)
b.star <- y.star %*% X

Now, we’ll use the same function, but set the method to bcapar for the parametric method.

bcadf <- (bcaboot::bcapar(
  t0 = t0, alpha = c((1:99) / 100),
  tt = tt, bb = b.star, cd = 1
)[["lims"]][, 1])

Now we can look at our outputs.

ggplot2::qplot((unlist(bcadf)), geom = "density") +
  theme_less()

That concludes our demonstration of the bootstrap method to approximate consonance functions.

See also: What Makes a Sensitivity Analysis? — the hub piece tying this material to the broader cluster on assumptions, robustness, and what happens when models bend.

References

Back to top

References

1. Efron B. (1979). “Bootstrap methods: Another look at the jackknife.” The Annals of Statistics. 7:1–26. doi: 10.1214/aos/1176344552. https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods-Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full.
2. Cox DR, Efron B. (2017). “Statistical thinking for 21st century scientists.” Science Advances. 3:e1700768. doi: 10.1126/sciadv.1700768.
3. Cole SR, Edwards JK, Greenland S. (2020). “Surprise!” American Journal of Epidemiology. doi: 10/gg63md.
4. Brown HK, Hussain-Shamsy N, Lunsky Y, Dennis C-LE, Vigod SN. (2017). “The association between antenatal exposure to selective serotonin reuptake inhibitors and autism: A systematic review and meta-analysis.” The Journal of Clinical Psychiatry. 78:e48–e58. doi: 10.4088/JCP.15r10194.
5. Efron B. (1998). “R. A. Fisher in the 21st century (Invited paper presented at the 1996 R. A. Fisher Lecture).” Statistical Science. 13:95–122. doi: 10/cxg354.
6. Efron B, Narasimhan B. (2018). “The automatic construction of bootstrap confidence intervals.” :17.
7. Xie M, Singh K. (2013). “Confidence Distribution, the Frequentist Distribution Estimator of a Parameter: A Review.” International Statistical Review. 81:3–39. doi: 10.1111/insr.12000.
8. Fraser DaS. (2011). “Is Bayes Posterior just Quick and Dirty Confidence?” Statistical Science. 26:299–316. doi: 10/df4n7n.
]]> statistics https://lesslikely.com/statistics/bootstrap.html Sun, 12 Jan 2025 05:00:00 GMT Computing Confidence Interval Functions with cifunction Zad Rafi Zad R https://lesslikely.com/statistics/cifunction.html

1 Overview

Traditional confidence intervals present a single level of uncertainty (typically 95%), but this dichotomous approach discards valuable information about the data. The confidence interval function (CI-function) provides a more complete picture by displaying all possible confidence intervals simultaneously.

NoteAlternative Names

The CI-function has been described in the literature under various names:

  • Confidence curve1
  • P-value function2, 3
  • Consonance interval4
  • Confidence density5

2 Installation

Install cifunction from SSC (Statistical Software Components):

3 Syntax

{stata, echo=TRUE, collectcode=TRUE, statapath="/Applications/StataNow/StataMP.app/Contents/MacOS/StataMP"}cifunction #b, se(#) [options]

3.1 Required Arguments

Required parameters for cifunction
Argument Description
#b Point estimate (coefficient or exponentiated value like OR, HR, IRR)
se(#) Standard error of the estimate

3.2 Optional Arguments

Optional parameters for cifunction
Option Description
df(#) Degrees of freedom (for t-distributed estimates)
eform Indicates coefficient is exponentiated (OR, RR, IRR)
figure[(options)] Produces CI-function plot with optional twoway options
saving(filename, replace) Saves results to file

4 Understanding CI-Functions

A CI-function graphically represents the relationship between confidence levels and confidence intervals:

  • Peak of curve: Indicates the point estimate
  • Concentration around peak: Reflects precision of estimate
  • Narrow curve: High precision (large sample size)
  • Broad curve: Low precision (small sample size)

4.1 Advantages Over Single-Level CIs

  1. Complete information: Shows compatibility of all values with data
  2. Precision visualization: Curve width directly shows estimate precision
  3. Multiple comparisons: Reveals whether data provide reassurance about null effects
  4. Nuanced interpretation: Moves beyond binary “significant/non-significant” thinking
TipInterpretation Guide

Values toward the peak are more compatible with the observed data (assuming the statistical model is correct). The curve explicitly shows the gradual nature of statistical evidence rather than suggesting arbitrary cutoffs.

5 Practical Examples

5.1 Example 1: Basic Risk Ratio

Reproducing Figure 1 from Sullivan and Foster (1990):

* Get standard error for RR=2.0 with p=0.05
getregstats 2.0, p(0.05) mod(rr)

* Create CI-function with log-scaled x-axis
cifunction 2.0, se(.707306) eform ///
    fig(xscale(log) xlab(0.5 1 2 5 10) ///
    xtitle("Risk Ratio (Log Scale)"))
## file ()print() not found
## r(601);
## 
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |          2    .707306    -1.96   0.050            1           4
## ------------------------------------------------------------------------------

5.2 Example 2: Age-Stratified Rate Ratios

Using Doll-Hill smoking data, stratified by age group:

* Load data and fit models
webuse dollhill3, clear

* Fit Poisson models for each age category
forvalues i = 1/4 {
    poisson deaths smokes if agecat==`i', exposure(pyears) irr
}

* Plot all four age-stratified CI-functions
cifunction 5.736638 2.138812 1.46824 1.35606, ///
    se(4.181258 .6520701 .295728 .2748845) eform ///
    fig(xscale(log) xlabel(.3 .5 1 2 5 10 20 50 100) ///
    legend(label(1 "35-44") label(2 "45-54") ///
           label(3 "55-64") label(4 "65-74") ///
           title("Age Group", size(small))) ///
    xtitle("Rate Ratio (Log Scale)"))
## file ()print() not found
## r(601);
## 
## 
## (Doll and Hill (1966))
## 
## 
## Iteration 0:  Log likelihood = -4.0705386  
## Iteration 1:  Log likelihood = -3.9613823  
## Iteration 2:  Log likelihood = -3.9612634  
## Iteration 3:  Log likelihood = -3.9612634  
## 
## Poisson regression                                      Number of obs =      2
##                                                         LR chi2(1)    =   9.73
##                                                         Prob > chi2   = 0.0018
## Log likelihood = -3.9612634                             Pseudo R2     = 0.5511
## 
## ------------------------------------------------------------------------------
##       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
## -------------+----------------------------------------------------------------
##       smokes |   5.736638   4.181258     2.40   0.017     1.374811    23.93712
##        _cons |   .0001064   .0000753   -12.94   0.000     .0000266    .0004256
##   ln(pyears) |          1  (exposure)
## ------------------------------------------------------------------------------
## Note: _cons estimates baseline incidence rate.
## 
## Iteration 0:  Log likelihood = -5.4793565  
## Iteration 1:  Log likelihood = -5.4104407  
## Iteration 2:  Log likelihood =   -5.41027  
## Iteration 3:  Log likelihood =   -5.41027  
## 
## Poisson regression                                      Number of obs =      2
##                                                         LR chi2(1)    =   7.59
##                                                         Prob > chi2   = 0.0059
## Log likelihood = -5.41027                               Pseudo R2     = 0.4123
## 
## ------------------------------------------------------------------------------
##       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
## -------------+----------------------------------------------------------------
##       smokes |   2.138812   .6520701     2.49   0.013     1.176691    3.887609
##        _cons |   .0011243   .0003246   -23.52   0.000     .0006385    .0019798
##   ln(pyears) |          1  (exposure)
## ------------------------------------------------------------------------------
## Note: _cons estimates baseline incidence rate.
## 
## Iteration 0:  Log likelihood = -6.2456963  
## Iteration 1:  Log likelihood = -6.1713863  
## Iteration 2:  Log likelihood =  -6.171298  
## Iteration 3:  Log likelihood =  -6.171298  
## 
## Poisson regression                                      Number of obs =      2
##                                                         LR chi2(1)    =   4.01
##                                                         Prob > chi2   = 0.0453
## Log likelihood = -6.171298                              Pseudo R2     = 0.2450
## 
## ------------------------------------------------------------------------------
##       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
## -------------+----------------------------------------------------------------
##       smokes |    1.46824    .295728     1.91   0.057     .9893522     2.17893
##        _cons |   .0049037   .0009267   -28.14   0.000     .0033858    .0071021
##   ln(pyears) |          1  (exposure)
## ------------------------------------------------------------------------------
## Note: _cons estimates baseline incidence rate.
## 
## Iteration 0:  Log likelihood = -6.1640322  
## Iteration 1:  Log likelihood = -6.1203079  
## Iteration 2:  Log likelihood = -6.1202768  
## Iteration 3:  Log likelihood = -6.1202768  
## 
## Poisson regression                                      Number of obs =      2
##                                                         LR chi2(1)    =   2.43
##                                                         Prob > chi2   = 0.1189
## Log likelihood = -6.1202768                             Pseudo R2     = 0.1658
## 
## ------------------------------------------------------------------------------
##       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
## -------------+----------------------------------------------------------------
##       smokes |    1.35606   .2748845     1.50   0.133     .9114509    2.017551
##        _cons |   .0108317    .002047   -23.95   0.000     .0074789    .0156877
##   ln(pyears) |          1  (exposure)
## ------------------------------------------------------------------------------
## Note: _cons estimates baseline incidence rate.

This example demonstrates how CI-functions can reveal patterns across subgroups—in this case, showing how the smoking-mortality relationship varies by age.

5.3 Example 3: Crude vs Adjusted Estimates

Comparing crude and pooled odds ratios from Mann et al. (1976):

* Case-control analysis
cci 10 36 5 40, woolf
cci 18 78 7 86, woolf

* Get standard errors
getregstats 2.835165, ucl(7.151794) mod(or)
getregstats 2.222222, ucl(7.118025) mod(or)

* Compare crude vs pooled estimates
cifunction 2.222222 2.835165, se(1.319891 1.33843) eform ///
    fig(xscale(log) xlabel(.2 .5 1 2 5 10 20) ///
    legend(label(1 Crude) label(2 Pooled)) ///
    xtitle("Odds Ratio (Log Scale)"))
## file ()print() not found
## r(601);
## 
## 
## 
##                                                          Proportion
##                  |   Exposed   Unexposed  |      Total      exposed
## -----------------+------------------------+------------------------
##            Cases |        10          36  |         46       0.2174
##         Controls |         5          40  |         45       0.1111
## -----------------+------------------------+------------------------
##            Total |        15          76  |         91       0.1648
##                  |                        |
##                  |      Point estimate    |    [95% conf. interval]
##                  |------------------------+------------------------
##       Odds ratio |         2.222222       |      .69377    7.118025 (Woolf)
##  Attr. frac. ex. |              .55       |      -.4414    .8595116 (Woolf)
##  Attr. frac. pop |         .1195652       |
##                  +-------------------------------------------------
##                                chi2(1) =     1.87  Pr>chi2 = 0.1719
## 
## 
##                                                          Proportion
##                  |   Exposed   Unexposed  |      Total      exposed
## -----------------+------------------------+------------------------
##            Cases |        18          78  |         96       0.1875
##         Controls |         7          86  |         93       0.0753
## -----------------+------------------------+------------------------
##            Total |        25         164  |        189       0.1323
##                  |                        |
##                  |      Point estimate    |    [95% conf. interval]
##                  |------------------------+------------------------
##       Odds ratio |         2.835165       |    1.123936    7.151794 (Woolf)
##  Attr. frac. ex. |         .6472868       |    .1102698    .8601749 (Woolf)
##  Attr. frac. pop |         .1213663       |
##                  +-------------------------------------------------
##                                chi2(1) =     5.18  Pr>chi2 = 0.0228
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |   2.835165   1.338429     2.21   0.027     1.123936    7.151794
## ------------------------------------------------------------------------------
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |   2.222222   1.319891     1.34   0.179     .6937698    7.118025
## ------------------------------------------------------------------------------

5.4 Example 4: Multi-Study Comparison

Comparing results across four geographic regions (Gufferman et al. 1984):

* Get standard errors from confidence limits
getregstats 0.7, l(.2) mod(rr)
getregstats 1.2, l(.8) mod(rr)
getregstats 1.3, l(.5) mod(rr)
getregstats 3.6, l(1.3) mod(rr)

* Plot all studies
cifunction .7 1.2 1.3 3.6, ///
    se(.4474236 .2482485 .6337692 1.870876) eform ///
    fig(xscale(log) xlabel(.05 .1 .2 .5 1 2 5 10 30) ///
    legend(label(1 "Finland") label(2 "NO/LA") ///
           label(3 "Denmark") label(4 "Upstate NY")) ///
    xtitle("Relative Risk (Log Scale)"))
## file ()print() not found
## r(601);
## 
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |         .7   .4474236    -0.56   0.577           .2        2.45
## ------------------------------------------------------------------------------
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |        1.2   .2482485     0.88   0.378           .8         1.8
## ------------------------------------------------------------------------------
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |        1.3   .6337692     0.54   0.590           .5        3.38
## ------------------------------------------------------------------------------
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |        3.6   1.870876     2.46   0.014          1.3    9.969231
## ------------------------------------------------------------------------------

This visualization makes it immediately apparent which studies had greater precision and how their point estimates compare.

5.5 Example 5: Meta-Analysis Context

From Rothman et al. (2008), Figure 10.5:

* Get standard errors
getregstats 0.87, l(0.70) model(or)

* Compare individual study vs meta-analysis
cifunction 0.87 0.88, se(.0965065 .0658232) eform ///
    fig(xlab(.5(.1)1.3) xtitle("Odds Ratio") ///
    legend(label(1 "Eisenberger et al (1998)") ///
           label(2 "Summary of 10 studies") ///
           ring(0) position(11))) ///
    sav(figdata, replace)
## file ()print() not found
## r(601);
## 
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |        .87   .0965065    -1.26   0.209           .7    1.081286
## ------------------------------------------------------------------------------

6 Output Variables

When using saving(), cifunction stores these variables:

Variables saved in output dataset
Variable Description
cilev Confidence levels (0 to 99.99)
plev P-values (0.0001 to 1.0)
sval S-values (Shannon information, see6)
lcl# Lower confidence limits for estimate #
ucl# Upper confidence limits for estimate #
NoteS-Values

S-values represent the information content against a null hypothesis, measured in bits. An S-value of 4.32 corresponds to p=0.05 (roughly 4.3 bits of information against the null). See6 for details.

7 Best Practices

  1. Always use log scale for ratio measures (OR, RR, HR, IRR)
  2. Label curves clearly when plotting multiple estimates
  3. Save output data for further analysis or custom plotting
  4. Consider degrees of freedom when using regression-based estimates
  5. Interpret curves holistically rather than focusing on single cutoffs

8 Theoretical Foundation

CI-functions rest on the principle that statistical evidence is continuous rather than dichotomous. As2 argues:

“The P-value function provides a convenient way to present the information in the data concerning the magnitude of an effect… [It] avoids the arbitrariness of any single significance level.”

The approach aligns with modern calls for moving beyond p-value thresholds7 by presenting the full gradient of statistical evidence.

9 Comparison with Other Approaches

Comparison of uncertainty visualization methods
Approach Information Provided Computational Complexity
Single CI One interval at one level Low
Multiple CIs Discrete intervals Low
CI-function Continuous curve of all intervals Low
Bootstrap distribution Empirical sampling distribution High
Likelihood profile Likelihood across parameter space Medium-High

10 Limitations

  • Assumes the statistical model is correctly specified
  • Does not account for unmeasured confounding or bias
  • Cannot compensate for poor study design or execution
  • Interpretation requires understanding of continuous evidence
WarningModel Assumptions

CI-functions, like all frequentist intervals, are valid only to the extent that the underlying statistical model and assumptions are appropriate for the data.

11 Citation

cifunction is free research software. Please cite it as:

Linden, A. (2019). CIFUNCTION: Stata module for computing and graphically displaying all possible confidence intervals around a point estimate. Statistical Software Components, Boston College Department of Economics.

12 Author

Ariel Linden President, Linden Consulting Group, LLC Email: alinden@lindenconsulting.org

13 See Also

  • getregstats (SSC): Helper for obtaining standard errors
  • niceloglabels (SSC): Improved logarithmic axis labels
  • concurve (R package): Similar functionality in R

14 References

1. Birnbaum A. (1961). “Confidence curves: An omnibus technique for estimation and testing statistical hypotheses.” Journal of the American Statistical Association. 56:246–249.
2. Poole C. (1987). “Beyond the confidence interval.” American Journal of Public Health. 77:195–199.
3. Miettinen OS. (1985). “Theoretical epidemiology: Principles of occurrence research in medicine.” New York: John Wiley & Sons.
4. Folks JL. (1981). “Ideas of statistics.” New York: John Wiley & Sons.
5. Xie M, Singh K. (2013). “Confidence Distribution, the Frequentist Distribution Estimator of a Parameter: A Review.” International Statistical Review. 81:3–39. doi: 10.1111/insr.12000.
6. Greenland S. (2019). “Valid p-values behave exactly as they should: Some misleading criticisms of p-values and their resolution with s-values.” The American Statistician. 73:106–114. doi: 10.1080/00031305.2018.1529625.
7. Wasserstein RL, Schirm AL, Lazar NA. (2019). “Moving to a world beyond "p < 0.05".” The American Statistician. 73:1–19. doi: 10.1080/00031305.2019.1583913.
Back to top

15 Appendix: Technical Details

15.1 Mathematical Formulation

For a normally distributed estimator with standard error , the confidence limits at level are:

The CI-function plots these limits across all values of .

For t-distributed estimators (when degrees of freedom are specified):

15.2 Exponentiated Coefficients

When eform is specified, computations occur on the log scale:

  1. Transform: with
  2. Compute CI limits on log scale
  3. Back-transform:

This ensures symmetric intervals on the log scale and proper geometric interpretation.

Last updated: 2026-09-13 Source code: Available on request License: Free for academic and non-commercial use

Citation

BibTeX citation:
@article{rafi2024,
  author = {Rafi, Zad and R, Zad},
  title = {Computing {Confidence} {Interval} {Functions} with
    Cifunction},
  journal = {Less Likely},
  date = {2024-03-23},
  url = {https://lesslikely.com/statistics/cifunction},
  langid = {en-US},
  abstract = {The cifunction package for Stata computes and graphically
    displays all possible confidence intervals around a point estimate,
    creating what’s known as a confidence interval function, confidence
    curve, p-value function, or consonance interval. This approach
    conveys more information than traditional single-level confidence
    intervals by showing the full range of values compatible with the
    data at all confidence levels.}
}
]]> statistics stata confidence intervals data visualization statistical methods https://lesslikely.com/statistics/cifunction.html Sat, 23 Mar 2024 04:00:00 GMT Your Models Are Neither Useful Nor Approximate Zad Rafi Sir Panda https://lesslikely.com/statistics/assumptions.html

If the assumptions of a model are not derived from theory, and if predictions are not tested against reality, then deductions from the model must be quite shaky. However, without the model, the data cannot be used to answer the research question. Indeed, the research hypothesis may not really be translatable into an empirical claim except as a statement about nominal significance levels of coefficients in a model. - David Freedman, 1991

Statistical Models as Approximations

Scientific inference is a highly complex process often accompanied by numerical outputs from statistical models. To some, such outputs are not convincing as they believe that anyone can lie with numbers, but the reality is that it is a lot easier to lie without them, making them crucial because if deployed correctly, they may be close approximations to reality. The presence of a statistical model can allow others to view an answer to a complex question from a numerical standpoint that many may be able to verify.

Unfortunately, there are rampant issues in the building, reporting, and interpretation of such statistical models, and such issues can lead to devastating policies. We have elsewhere reviewed proposals to reform terminology and improve interpretations of conventional statistics by emphasizing logical and information concepts over probability concepts (1–3), as the latter has continued to confuse even well-credentialed statisticians familiar with the misinterpretations.

What Goes into a Model?

We here focus on how common descriptions of statistical outputs can be technically accurate yet still mislead when there is considerable uncertainty about what are known as background or auxiliary assumptions. Such uncertainty is typical in many contexts, where assumptions may include both mathematical and methodological ones:

For example, in the popular regression book by Gelman, the following assumptions are given for regression models in decreasing order of importance:

  • validity
  • additivity and linearity
  • independence of errors
  • equal variance of errors
  • normality of errors

This is typical to see in many regression modeling books, but important methodological assumptions underlying the model are rarely discussed such as:

  • patient outcomes are independent
  • interactions and trends follow the regression model used for analysis (e.g., linear or logistic)
  • no uncontrolled confounding
  • subject-selection bias
  • measurement error
  • sparse-data bias
  • no selective reporting of results* based on their P-values, interval estimates, or any other output – or if there is such selection, any bias it could produce has been adjusted for (or blocked) in the computation and presentation of the final results.

P-value assumptions

Conditional versus unconditional interpretations of P-values, S-values, and compatibility intervals (CIs). (A) Conditional interpretation, in which background model assumptions, such as no systematic error, are assumed to be correct; thus,the information provided by the P-value and S-value is targeted towards the test hypothesis. (B)Unconditional interpretation, in which no aspect of the statistical model is assumed to be correct; thus,the information provided by the P-value and S-value is targeted toward the entire test model.

In addition to these general statistical and methodological assumptions, many assume that the statistical model being fit is answering a proper scientific question that has been translated into a statistical one, with assumptions that match or approximate reality. When there is a disconnect between the scientific question and the statistical one being answered by the model, the analyst is committing a type-III error, and will produce statistical outputs with of little use, unless the assumption violations of the chosen and reported are interrogated/varied and considered in the overall analysis.

Many are not aware of what a type-III error is due to the focus on type-I and type-II errors. I paste the following definition from the APA dictionary, which I found myself agreeing with:

Type-III Error: the error that occurs when there is a discrepancy between the research focus and the hypothesis actually tested. For example, a Type III error would have happened if a researcher collected data on individual differences within a sample and determined the causes of variation but the question of interest concerned differences between populations. In other words, a Type III error involves providing the right answer for the wrong question. Also called Type 0 error.

Violation of Assumptions

Misleading interpretations arise because the descriptions of such statistical outputs are conditional on such assumptions (i.e., they treat them as given/being true), and so do not factor uncertainty regarding the assumptions into their assessments. While various risk-assessment methods can incorporate additional sources in quantified uncertainties, such methods demand considerably more skilled user input than do conventional regression methods (5–9) and still do not account for unquantified sources of uncertainty. We thus present a more direct and nontechnical approach based on logic and associated language to honor assumption uncertainty, called deconditioning.

For example, suppose that a model assumes independence, a common assumption in many statistical models, but this violated due to the presence of a correlation, this will lead to P-values that are invalid, demonstrable by their failure to achieve asymptotic uniformity.

For example, a model that is approximately correct will generate P-values that are asymptotically uniform (a characteristic used to assess the validity of P-values)

#' @title Simulation of valid P-values where test hypothesis is true
#' @param X The first variable we are simulating
#' @param Y The second variable we are simulating
#' @param n.sim # The number of simulations
#' @param t The object storing the t-test results
#' @param t.sim # Empty numeric vector to contain values
#' @param n.samp # Sample size in each group
#' @NOTE The null hypothesis does not have to be 0, it can be any value.

n.sim <- 10000
t.sim <- numeric(n.sim)
n.samp <- 1000

for (i in 1:n.sim) {
  X <- rnorm(n.samp, mean = 0, sd = 1)
  Y <- rnorm(n.samp, mean = 0, sd = 1)
  df <- data.frame(X, Y)
  t <- t.test(X, Y,
    mu = 0, paired = FALSE,
    var.equal = TRUE, data = df)
  t.sim[i] <- t[[3]]}

#' @title Simulation of invalid P-values where test hypothesis is false
#' @param n_simulations # The number of simulations
#' @param p_values The object storing the t-test results
#' @param p_value_data # Empty numeric vector to contain values
#' @param n_subjects # Sample size in each group
#' @param correlation # correlation to violate independence

library(MASS)
n_simulations <- 1000
n_subjects <- 50
correlation <- 0.5

generate_correlated_data <- function(n_subjects, correlation) {
    Sigma <- matrix(correlation, n_subjects, n_subjects) +
        diag(1 - correlation, n_subjects)
    data <- mvrnorm(n = 2, mu = rep(0, n_subjects),
        Sigma = Sigma)
    return(data)
}
run_simulation <- function(n_subjects, correlation) {
    data <- generate_correlated_data(n_subjects, correlation)
    t_test_result <- t.test(data[1, ], data[2, ])
    t_test_p_value <- t_test_result$p.value
    return(t_test_p_value)
}
p_values <- replicate(n_simulations, run_simulation(n_subjects,
    correlation))
p_value_data <- data.frame(PValue = p_values)

Deconditioning shifts the logical status of assumptions in the description of results from what is assumed as given (conditioned on) to an integral part of what is being checked by common statistics. This shift makes clear that the results might be explained by violations of background assumptions rather than, or in addition to, failure of a targeted hypothesis. Such deconditioning does not change calculations; hence it does not change any P-value, interval estimate, or posterior distribution. Instead, it provides a different interpretation of these statistics, one arising naturally in thorough contextual discussions, but which deserves more emphasis in basic statistical practice, including teaching and presentations.

In philosophical accounts of theory testing, it is well recognized that an apparent violations of a hypothesis may instead be due to violation of background or auxiliary assumptions (10–13). Among these assumptions are researcher competence and integrity in the conduct and reporting of statistical analyses which have come under scrutiny in the “replication crisis”) as well as common statistical assumptions such as linearity, outcome independence, and effect homogeneity, and methodological assumptions such as no uncontrolled confounding or selection bias. Perhaps because of this generality, we have found that our recommendation to decondition inferences (present inferences using an unconditional interpretation) (1,3) is often difficult for statistically sophisticated readers to comprehend or accept, and is even resisted, misunderstood, and misrepresented by some reviewers

See also: What Makes a Sensitivity Analysis? — the hub piece tying this material to the broader cluster on assumptions, robustness, and what happens when models bend.

Back to top

Citation

BibTeX citation:
@misc{rafi2024,
  author = {Rafi, Zad and Panda, Sir},
  title = {Your {Models} {Are} {Neither} {Useful} {Nor} {Approximate}},
  date = {2024-01-01},
  url = {https://lesslikely.com/statistics/assumptions},
  langid = {en-US},
  abstract = {A discussion about models and the assumptions that
    underlie them.}
}
]]> statistics https://lesslikely.com/statistics/assumptions.html Mon, 01 Jan 2024 05:00:00 GMT Using Stata: Producing Consonance Functions Zad Rafi Sir Panda https://lesslikely.com/statistics/stata.html

Although concurve was originally designed to be used in R, it is possible to achieve very similar results in Stata. We can use some datasets that are built into Stata to show how to achieve this. I’ll use the Statamarkdown R package so that I can obtain Stata outputs using RMarkdown via my Stata 16 package.

First, let’s load the auto2 dataset which contains data about cars and their characteristics.

sysuse auto2
#> (1978 automobile data)

Browse the data set in your data browser to get more familiar with some of the variables. Let’s say we’re interested in the relationship between miles per gallon and price. We could fit a very simple linear model to assess that relationship.

First, let’s visualize the data with a scatter plot.

sysuse auto2
scatter price mpg, mcolor(dkorange) scale(0.70)
graph export "scatter.svg", replace
#> (1978 automobile data)
#> 
#> 
#> file scatter.svg saved as SVG format

scatter

scatter

That’s what our data looks like. Clearly there seems to be an inverse relationship between miles per gallon and price.

Now we could fit a very simple linear model with miles per gallon being the predictor and price being the outcome and get some estimates of the relationship.

sysuse auto2
regress price mpg
#> (1978 automobile data)
#> 
#> 
#>       Source |       SS           df       MS      Number of obs   =        74
#> -------------+----------------------------------   F(1, 72)        =     20.26
#>        Model |   139449474         1   139449474   Prob > F        =    0.0000
#>     Residual |   495615923        72  6883554.48   R-squared       =    0.2196
#> -------------+----------------------------------   Adj R-squared   =    0.2087
#>        Total |   635065396        73  8699525.97   Root MSE        =    2623.7
#> 
#> ------------------------------------------------------------------------------
#>        price | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>          mpg |  -238.8943   53.07669    -4.50   0.000    -344.7008   -133.0879
#>        _cons |   11253.06   1170.813     9.61   0.000     8919.088    13587.03
#> ------------------------------------------------------------------------------

That’s what our output looks like.

Our output also gives us 95% consonance (confidence) intervals by default. But suppose we wished to fit a fractional polynomial model and graph it and get the confidence bands, here’s what we would do.

sysuse auto2
mfp: glm price mpg
twoway (fpfitci price mpg, estcmd(glm) fcolor(dkorange%20) alcolor(%40))  || scatter price mpg, mcolor(dkorange) scale(0.75)
graph export "mfp.svg", replace
#> (1978 automobile data)
#> 
#> 
#> Deviance for model with all terms untransformed = 1373.079, 74 observations
#> 
#> Variable     Model (vs.)   Deviance  Dev diff.   P      Powers   (vs.)
#> ----------------------------------------------------------------------
#> mpg          Lin.   FP2    1373.079    19.565  0.000+   1         -2 -2
#>              FP1           1356.927     3.413  0.182    -2        
#>              Final         1356.927                     -2
#> 
#> 
#> Transformations of covariates:
#> 
#> -> gen double Impg__1 = X^-2-.2204707671 if e(sample) 
#>    (where: X = mpg/10)
#> 
#> Final multivariable fractional polynomial model for price
#> --------------------------------------------------------------------
#>     Variable |    -----Initial-----          -----Final-----
#>              |   df     Select   Alpha    Status    df    Powers
#> -------------+------------------------------------------------------
#>          mpg |    4     1.0000   0.0500     in      2     -2
#> --------------------------------------------------------------------
#> 
#> Generalized linear models                         Number of obs   =         74
#> Optimization     : ML                             Residual df     =         72
#>                                                   Scale parameter =    5533697
#> Deviance         =  398426217.4                   (1/df) Deviance =    5533697
#> Pearson          =  398426217.4                   (1/df) Pearson  =    5533697
#> 
#> Variance function: V(u) = 1                       [Gaussian]
#> Link function    : g(u) = u                       [Identity]
#> 
#>                                                   AIC             =    18.3909
#> Log likelihood   = -678.4632599                   BIC             =   3.98e+08
#> 
#> ------------------------------------------------------------------------------
#>              |                 OIM
#>        price | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>      Impg__1 |   13163.85   2013.016     6.54   0.000      9218.41    17109.29
#>        _cons |   5538.395   289.7737    19.11   0.000     4970.449    6106.341
#> ------------------------------------------------------------------------------
#> Deviance = 1356.927.
#> 
#> 
#> file mfp.svg saved as SVG format

That’s what our model looks graphed.

fractional polynomial model

fractional polynomial model

Now suppose we got a single estimate (point or interval) for a parameter, and we wanted all the intervals for it at every level.

Here’s the code that we’ll be using to achieve that in Stata.

#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> r(4);

That’s a lot and may seem intimidating at first, but I’ll explain it line by line.

postfile topost level pvalue svalue lointerval upinterval using my_new_data, replace
#> no; dataset in memory has changed since last saved
#> r(4);

“postfile” is the command that will be responsible for pasting the data from our overall loop into a new dataset. Here, we are telling Stata that the internal Stata memory used to hold these results (the post) will be named “topost” and that it will have five variables, “level”, “pvalue”, “svalue”, “lointerval”, and “upinterval.”

  • “level” will contain the consonance level that corresponds to the limits of the interval, with “lointerval” being the lower bound of the interval and “upinterval” being the upper bound.

  • “pvalue”is computed by taking 1 - “level”, which is alpha.

  • “svalue”is computed by taking the of the computed P-value, and this column will be used to plot the surprisal function.

  • “my_new_data” is the filename that we’ve assigned to our new dataset.

  • “replace” indicates that if there is an existing filename that already exists, we’re willing to relace it.

Here are the next few major lines

#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> post topost not found
#> r(111);
#> 
#> r(111);

The command “forvalues” is responsible for taking a set of numbers that we provide it, and running the contents within the braces through those numbers. So here, we’ve set the local macro “i” to contain numbers between 10 and 99.99 for our consonance levels. Why 10? Stata cannot compute consonance intervals lower than 10%.

Our next line contains the actual contents of what we want to do. Here, it says that we will run a simple linear regression where mpg is the predictor and where price is the outcome, and that the outputs for each loop will be suppressed, hence the “quiet.”

Then, we have the command “level” with the local macro “i” inside of it. As you may already know, “level” dictates the consonance level that Stata provides us. By default, this is set to 95%, but here, we’ve set it “i”, which we established via “forvalues” as being set to numbers between 10 and 99.

The next line two lines

#> no; dataset in memory has changed since last saved
#> r(4);

indicate that we will take variables of a certain class r(), (this class contains the interval bounds we need) and place them within a matrix called E. Then we will list the contents of this matrix.

post topost (`i') (1-`i'/100) ( ln(1-`i'/100)/ln(2) * -1) (E[5,1]) (E[6,1])
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> post topost not found
#> r(111);
#> 
#> r(111);

From the contents of this matrix list, we will take the estimates from the fifth and sixth rows (look at the last two paranthesis of this line of code above and then the image below) in the first column which contain our consonance limits, with the fifth row containing the lower bound of the interval and the sixth containing the upper bound.

Trace plot of imputed datasets.

Trace plot of imputed datasets.

We will place the contents from the fifth row into the second variable we set originally for our new dataset, which was “lointerval.” The contents of the sixth row will be placed into “upinterval.”

All potential values of “i” (10-99) will be placed into the first variable that we set, “level”. From this first variable, we can compute the second variable we set up, which was “Pvalue” and we’ve done that here by subtracting “level” from 1 and then dividing the whole equation by 100, so that our P-value can be on the proper scale. Our third variable, which is the longest, computes the “Svalue” by using the previous variable, the “Pvalue” and taking the of it.

The relationships between the variables on this line and the variables we set up in the very first line are dictated by the order of the commands we have set, and therefore they correspond to the same order.

“post topost” is writing the results from each loop as new observations in this data structure.

With that, our loop has concluded, and we can now tell Stata that “post” is no longer needed

postclose topost
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> post topost not found
#> r(111);
#> 
#> r(111);

We then tell Stata to clear its memory to make room for the new dataset we just created and we can list the contents of this new dataset.

use my_new_data, clear
list
#> no; dataset in memory has changed since last saved
#> r(4);

Now we have an actual dataset with all the consonance intervals at all the levels we wanted, ranging from 10% all the way up to 99%.

In order to get a function, we’ll need to be able to graph these results, and that can be tricky since for each observation we have one y value (the consonance level), and two x values, the lower bound of the interval and the upper bound of the interval.

So a typical scatterplot will not work, since Stata will only accept one x value. To bypass this, we’ll have to use a paired-coordinate scatterplot which will allow us to plot two different y variables and two different x variables.

Of course, we don’t need two y variables, so we can set both options to the variable “level”, and then we can set our first x variable to “lointerval” and the second x variable to “upinterval.”

This can all be done with the following commands, which will also allow us to set the title and subtitle of the graph, along with the titles of the axes.

twoway (pcscatter level lointerval level upinterval), ///
ytitle(Consonance Level (%)) xtitle(Consonance Limits) ///
title(Consonance Curve) ///
subtitle(A function comprised of several consonance intervals at various levels.)
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> variable level not found
#> r(111);
#> 
#> r(111);

However, I would recommend using the menu to customize the plots as much as possible. Simply go to the Graphics menu and select Twoway Graphs. Then create a new plot definition, and select the Advanced plots and choose a paired coordinate scatterplot and fill in the y variables, both of which will be “levels” and the x variables, which will be “lointerval” and “upinterval”.

So now, here’s what our confidence/consonance function looks like.

clear
sysuse auto2
postfile topost level pvalue svalue lointerval upinterval using my_new_data, replace

forvalues i = 10/99.9 {
      quietly regress price weight, level(`i')
      matrix E = r(table)
      quietly matrix list E
      post topost (`i') (1-`i'/100) ( ln(1-`i'/100)/ln(2) * -1) (E[5,1]) (E[6,1])
    }

postclose topost
use my_new_data, clear

twoway (pcscatter pvalue lointerval pvalue upinterval, mcolor(maroon)), ytitle(Consonance Level (%)) xtitle(Consonance Limits) scale(0.75) ///
title(Consonance Curve) subtitle(A function comprised of several consonance intervals at various levels.)
graph export "confidence.svg", replace
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> (1978 automobile data)
#> 
#> 
#> 
#> 
#> 
#> 
#> file confidence.svg saved as SVG format

Pretty neat, eh? And below is what our surprisal function looks like, which is simply the (p) transformation of the observed P-value. For a more comprehensive discussion on surprisals, see this page and check out some of the references at the bottom.

clear
sysuse auto2
postfile topost level pvalue svalue lointerval upinterval using my_new_data, replace

forvalues i = 10/99.9 {
      quietly regress price weight, level(`i')
      matrix E = r(table)
      quietly matrix list E
      post topost (`i') (1-`i'/100) ( ln(1-`i'/100)/ln(2) * -1) (E[5,1]) (E[6,1])
    }

postclose topost
use my_new_data, clear

twoway (pcscatter svalue lointerval svalue upinterval, mcolor(maroon)), ytitle(Consonance Level (%)) xtitle(Consonance Limits)  scale( 0.75) ///
title(Surprisal Curve) subtitle(A function comprised of several consonance intervals at various levels.)
graph export "surprisal.svg", replace
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> (1978 automobile data)
#> 
#> 
#> 
#> 
#> 
#> 
#> file surprisal.svg saved as SVG format

Surprisal Function

Surprisal Function

It’s clear that in both plots, we’re missing values of intervals with a confidence/consonance level of less than 10%, but unfortunately, this is the best Stata can do, and what we’ll have to work with. It may not look as pretty as an output from R, but it’s far more useful than blankly staring at a 95% interval and thinking that it is the only piece of information we have regarding compatibility of different effect estimates.

The code that I have pasted above can be used for most commands in Stata that have an option to calculate a consonance level. Thus, if there’s an option for “level”, then the commands above will work to produce a data set of several consonance intervals. Though I am seriously hoping that a Stata expert will see this post and point out how I am wrong.

Now, suppose we wished to fit a generalized linear model, here’s what our code would look like.

#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> (1978 automobile data)
#> 
#> 
#> 
#> 
#> 
#> 
#>      +---------------------------------------------------+
#>      | level   pvalue     svalue   lointer~l   upinter~l |
#>      |---------------------------------------------------|
#>   1. |    10       .9   .1520031    -245.564   -232.2247 |
#>   2. |    11      .89   .1681228   -246.2351   -231.5536 |
#>   3. |    12      .88   .1844246   -246.9073   -230.8814 |
#>   4. |    13      .87   .2009127   -247.5808   -230.2079 |
#>   5. |    14      .86   .2175914   -248.2557    -229.533 |
#>      |---------------------------------------------------|
#>   6. |    15      .85   .2344653   -248.9321   -228.8566 |
#>   7. |    16      .84   .2515388   -249.6102   -228.1785 |
#>   8. |    17      .83   .2688168     -250.29   -227.4987 |
#>   9. |    18      .82   .2863042   -250.9717    -226.817 |
#>  10. |    19      .81   .3040062   -251.6554   -226.1333 |
#>      |---------------------------------------------------|
#>  11. |    20       .8   .3219281   -252.3412   -225.4475 |
#>  12. |    21      .79   .3400754   -253.0292   -224.7595 |
#>  13. |    22      .78    .358454   -253.7197    -224.069 |
#>  14. |    23      .77   .3770697   -254.4126   -223.3761 |
#>  15. |    24      .76   .3959287   -255.1083   -222.6804 |
#>      |---------------------------------------------------|
#>  16. |    25      .75   .4150375   -255.8067    -221.982 |
#>  17. |    26      .74   .4344028    -256.508   -221.2807 |
#>  18. |    27      .73   .4540316   -257.2125   -220.5762 |
#>  19. |    28      .72   .4739312   -257.9202   -219.8685 |
#>  20. |    29      .71   .4941091   -258.6312   -219.1575 |
#>      |---------------------------------------------------|
#>  21. |    30       .7   .5145732   -259.3459   -218.4428 |
#>  22. |    31      .69   .5353317   -260.0642   -217.7244 |
#>  23. |    32      .68   .5563933   -260.7865   -217.0022 |
#>  24. |    33      .67    .577767   -261.5129   -216.2758 |
#>  25. |    34      .66   .5994621   -262.2435   -215.5452 |
#>      |---------------------------------------------------|
#>  26. |    35      .65   .6214884   -262.9785   -214.8102 |
#>  27. |    36      .64   .6438562   -263.7183   -214.0704 |
#>  28. |    37      .63   .6665763   -264.4628   -213.3259 |
#>  29. |    38      .62   .6896599   -265.2124   -212.5762 |
#>  30. |    39      .61   .7131189   -265.9673   -211.8213 |
#>      |---------------------------------------------------|
#>  31. |    40       .6   .7369656   -266.7278   -211.0609 |
#>  32. |    41      .59   .7612131    -267.494   -210.2947 |
#>  33. |    42      .58   .7858752   -268.2662   -209.5225 |
#>  34. |    43      .57   .8109662   -269.0446    -208.744 |
#>  35. |    44      .56   .8365012   -269.8297    -207.959 |
#>      |---------------------------------------------------|
#>  36. |    45      .55   .8624965   -270.6215   -207.1672 |
#>  37. |    46      .54   .8889687   -271.4204   -206.3683 |
#>  38. |    47      .53   .9159358   -272.2268   -205.5619 |
#>  39. |    48      .52   .9434165    -273.041   -204.7477 |
#>  40. |    49      .51   .9714308   -273.8633   -203.9254 |
#>      |---------------------------------------------------|
#>  41. |    50       .5          1    -274.694   -203.0947 |
#>  42. |    51      .49   1.029146   -275.5337    -202.255 |
#>  43. |    52      .48   1.058894   -276.3825   -201.4061 |
#>  44. |    53      .47   1.089267   -277.2411   -200.5475 |
#>  45. |    54      .46   1.120294   -278.1099   -199.6788 |
#>      |---------------------------------------------------|
#>  46. |    55      .45   1.152003   -278.9893   -198.7994 |
#>  47. |    56      .44   1.184425   -279.8798   -197.9089 |
#>  48. |    57      .43   1.217591    -280.782   -197.0067 |
#>  49. |    58      .42   1.251539   -281.6965   -196.0922 |
#>  50. |    59      .41   1.286304   -282.6239   -195.1648 |
#>      |---------------------------------------------------|
#>  51. |    60       .4   1.321928   -283.5648   -194.2239 |
#>  52. |    61      .39   1.358454     -284.52   -193.2687 |
#>  53. |    62      .38   1.395929   -285.4902   -192.2985 |
#>  54. |    63      .37   1.434403   -286.4762   -191.3125 |
#>  55. |    64      .36   1.473931   -287.4789   -190.3098 |
#>      |---------------------------------------------------|
#>  56. |    65      .35   1.514573   -288.4992   -189.2894 |
#>  57. |    66      .34   1.556393   -289.5383   -188.2504 |
#>  58. |    67      .33   1.599462   -290.5971   -187.1916 |
#>  59. |    68      .32   1.643856   -291.6769   -186.1118 |
#>  60. |    69      .31    1.68966    -292.779   -185.0097 |
#>      |---------------------------------------------------|
#>  61. |    70       .3   1.736966   -293.9048   -183.8839 |
#>  62. |    71      .29   1.785875   -295.0559   -182.7328 |
#>  63. |    72      .28   1.836501   -296.2341   -181.5546 |
#>  64. |    73      .27   1.888969   -297.4413   -180.3474 |
#>  65. |    74      .26   1.943416   -298.6794   -179.1092 |
#>      |---------------------------------------------------|
#>  66. |    75      .25          2   -299.9511   -177.8376 |
#>  67. |    76      .24   2.058894   -301.2588   -176.5299 |
#>  68. |    77      .23   2.120294   -302.6054   -175.1833 |
#>  69. |    78      .22   2.184425   -303.9944   -173.7943 |
#>  70. |    79      .21   2.251539   -305.4294   -172.3592 |
#>      |---------------------------------------------------|
#>  71. |    80       .2   2.321928   -306.9149   -170.8738 |
#>  72. |    81      .19   2.395929   -308.4555   -169.3331 |
#>  73. |    82      .18   2.473931   -310.0572   -167.7315 |
#>  74. |    83      .17   2.556393   -311.7264   -166.0623 |
#>  75. |    84      .16   2.643856   -313.4709   -164.3178 |
#>      |---------------------------------------------------|
#>  76. |    85      .15   2.736966   -315.2999   -162.4888 |
#>  77. |    86      .14   2.836501   -317.2245   -160.5642 |
#>  78. |    87      .13   2.943416   -319.2578   -158.5308 |
#>  79. |    88      .12   3.058894   -321.4166   -156.3721 |
#>  80. |    89      .11   3.184425   -323.7211   -154.0676 |
#>      |---------------------------------------------------|
#>  81. |    90       .1   3.321928   -326.1977    -151.591 |
#>  82. |    91      .09   3.473931   -328.8804   -148.9082 |
#>  83. |    92      .08   3.643856    -331.815   -145.9737 |
#>  84. |    93      .07   3.836501   -335.0646   -142.7241 |
#>  85. |    94      .06   4.058894   -338.7206   -139.0681 |
#>      |---------------------------------------------------|
#>  86. |    95      .05   4.321928   -342.9227    -134.866 |
#>  87. |    96      .04   4.643856   -347.9005   -129.8882 |
#>  88. |    97      .03   5.058894   -354.0756   -123.7131 |
#>  89. |    98      .02   5.643856   -362.3692   -115.4195 |
#>  90. |    99      .01   6.643856   -375.6108   -102.1779 |
#>      +---------------------------------------------------+
#> 
#> 
#> command ytitle is unrecognized
#> r(199);
#> 
#> r(199);

We simply replace the first line within the loop with our intended command, just as I’ve replaced

regress price mpg
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#>       Source |       SS           df       MS      Number of obs   =        74
#> -------------+----------------------------------   F(1, 72)        =     20.26
#>        Model |   139449474         1   139449474   Prob > F        =    0.0000
#>     Residual |   495615923        72  6883554.48   R-squared       =    0.2196
#> -------------+----------------------------------   Adj R-squared   =    0.2087
#>        Total |   635065396        73  8699525.97   Root MSE        =    2623.7
#> 
#> ------------------------------------------------------------------------------
#>        price | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>          mpg |  -238.8943   53.07669    -4.50   0.000    -344.7008   -133.0879
#>        _cons |   11253.06   1170.813     9.61   0.000     8919.088    13587.03
#> ------------------------------------------------------------------------------

with

glm price mpg
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> Iteration 0:  Log likelihood = -686.53958  
#> 
#> Generalized linear models                         Number of obs   =         74
#> Optimization     : ML                             Residual df     =         72
#>                                                   Scale parameter =    6883554
#> Deviance         =  495615922.6                   (1/df) Deviance =    6883554
#> Pearson          =  495615922.6                   (1/df) Pearson  =    6883554
#> 
#> Variance function: V(u) = 1                       [Gaussian]
#> Link function    : g(u) = u                       [Identity]
#> 
#>                                                   AIC             =   18.60918
#> Log likelihood   = -686.5395809                   BIC             =   4.96e+08
#> 
#> ------------------------------------------------------------------------------
#>              |                 OIM
#>        price | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>          mpg |  -238.8943   53.07669    -4.50   0.000    -342.9227    -134.866
#>        _cons |   11253.06   1170.813     9.61   0.000      8958.31    13547.81
#> ------------------------------------------------------------------------------

If we wanted fit something more complex, like a multilevel mixed model that used restricted maximum likelihood, here’s what our code would look like:

#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> (1978 automobile data)
#> 
#> 
#> variable outcome not found
#> r(111);
#> 
#> r(111);

Basically, our code doesn’t really change that much and with only a few lines of it, we are able to produce graphical tools that can better help us interpret the wide range of effect sizes that are compatible with the model and its assumptions.

Using the cifunction Command for Confidence Interval Functions

An alternative approach to producing confidence interval functions in Stata is to use the cifunction command, which computes and graphically displays all possible confidence intervals around a point estimate.

The cifunction command is an immediate command that allows you to specify a point estimate and standard error, and it will produce a confidence interval function (also called a confidence curve, P-value function, or consonance interval).

Syntax

cifunction #b , se(#) [ df(#) eform figure[(twoway_options)] saving(filename, replace) ]

Where #b can be specified as a coefficient or exponentiated value (e.g., OR, HR, IRR).

Options

  • se(#): specifies the standard error of the estimate (required)
  • df(#): specify the degrees of freedom if #b is t-distributed
  • eform: indicate that the coefficient is exponentiated (e.g., OR, RR, IRR)
  • figure[(twoway_options)]: produces a confidence interval function plot
  • saving(filename, replace): save results to filename

Example 1: Simple Risk Ratio

Reproduce a confidence interval function for a risk ratio of 2.0 with p-value of 0.05. First, we use getregstats to get the standard error:

getregstats 2.0, p(0.05) mod(rr)
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> 
#> ------------------------------------------------------------------------------
#>              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
#> -------------+----------------------------------------------------------------
#>    Estimates |          2    .707306    -1.96   0.050            1           4
#> ------------------------------------------------------------------------------

Then use cifunction specifying the point estimate and standard error. We modify the figure to improve presentation:

cifunction 2.0, se(.707306) eform fig(xscale(log) xlab(0.5 1 2 5 10) xtitle("Risk Ratio (Log Scale)"))
#> no; dataset in memory has changed since last saved
#> r(4);

Example 2: Multiple Rate Ratios by Age Group

For multiple estimates across age groups from the dollhill3 dataset:

webuse dollhill3, clear
poisson deaths smokes if agecat==1, exposure(pyears) irr
poisson deaths smokes if agecat==2, exposure(pyears) irr
poisson deaths smokes if agecat==3, exposure(pyears) irr
poisson deaths smokes if agecat==4, exposure(pyears) irr
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> (Doll and Hill (1966))
#> 
#> 
#> Iteration 0:  Log likelihood = -4.0705386  
#> Iteration 1:  Log likelihood = -3.9613823  
#> Iteration 2:  Log likelihood = -3.9612634  
#> Iteration 3:  Log likelihood = -3.9612634  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   9.73
#>                                                         Prob > chi2   = 0.0018
#> Log likelihood = -3.9612634                             Pseudo R2     = 0.5511
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |   5.736638   4.181258     2.40   0.017     1.374811    23.93712
#>        _cons |   .0001064   .0000753   -12.94   0.000     .0000266    .0004256
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.
#> 
#> 
#> Iteration 0:  Log likelihood = -5.4793565  
#> Iteration 1:  Log likelihood = -5.4104407  
#> Iteration 2:  Log likelihood =   -5.41027  
#> Iteration 3:  Log likelihood =   -5.41027  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   7.59
#>                                                         Prob > chi2   = 0.0059
#> Log likelihood = -5.41027                               Pseudo R2     = 0.4123
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |   2.138812   .6520701     2.49   0.013     1.176691    3.887609
#>        _cons |   .0011243   .0003246   -23.52   0.000     .0006385    .0019798
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.
#> 
#> 
#> Iteration 0:  Log likelihood = -6.2456963  
#> Iteration 1:  Log likelihood = -6.1713863  
#> Iteration 2:  Log likelihood =  -6.171298  
#> Iteration 3:  Log likelihood =  -6.171298  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   4.01
#>                                                         Prob > chi2   = 0.0453
#> Log likelihood = -6.171298                              Pseudo R2     = 0.2450
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |    1.46824    .295728     1.91   0.057     .9893522     2.17893
#>        _cons |   .0049037   .0009267   -28.14   0.000     .0033858    .0071021
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.
#> 
#> 
#> Iteration 0:  Log likelihood = -6.1640322  
#> Iteration 1:  Log likelihood = -6.1203079  
#> Iteration 2:  Log likelihood = -6.1202768  
#> Iteration 3:  Log likelihood = -6.1202768  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   2.43
#>                                                         Prob > chi2   = 0.1189
#> Log likelihood = -6.1202768                             Pseudo R2     = 0.1658
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |    1.35606   .2748845     1.50   0.133     .9114509    2.017551
#>        _cons |   .0108317    .002047   -23.95   0.000     .0074789    .0156877
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.

Then we can use cifunction with multiple estimates:

cifunction 5.736638 2.138812 1.46824 1.35606, ///
  se(4.181258 .6520701 .295728 .2748845) eform ///
  fig(xscale(log) xlabel(.3 .5 1 2 5 10 20 50 100) ///
      legend(label(1 "35-44") label(2 "45-54") label(3 "55-64") ///
             label(4 "65-74") title("Age Group", size(small))) ///
      xtitle("Rate Ratio (Log Scale)"))
#> no; dataset in memory has changed since last saved
#> r(4);

Example 3: Comparing Crude and Pooled Estimates

Using case-control data to compare crude and pooled odds ratios:

#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#>                                                          Proportion
#>                  |   Exposed   Unexposed  |      Total      exposed
#> -----------------+------------------------+------------------------
#>            Cases |        10          36  |         46       0.2174
#>         Controls |         5          40  |         45       0.1111
#> -----------------+------------------------+------------------------
#>            Total |        15          76  |         91       0.1648
#>                  |                        |
#>                  |      Point estimate    |    [95% conf. interval]
#>                  |------------------------+------------------------
#>       Odds ratio |         2.222222       |      .69377    7.118025 (Woolf)
#>  Attr. frac. ex. |              .55       |      -.4414    .8595116 (Woolf)
#>  Attr. frac. pop |         .1195652       |
#>                  +-------------------------------------------------
#>                                chi2(1) =     1.87  Pr>chi2 = 0.1719
#> 
#> 
#>                                                          Proportion
#>                  |   Exposed   Unexposed  |      Total      exposed
#> -----------------+------------------------+------------------------
#>            Cases |        18          78  |         96       0.1875
#>         Controls |         7          86  |         93       0.0753
#> -----------------+------------------------+------------------------
#>            Total |        25         164  |        189       0.1323
#>                  |                        |
#>                  |      Point estimate    |    [95% conf. interval]
#>                  |------------------------+------------------------
#>       Odds ratio |         2.835165       |    1.123936    7.151794 (Woolf)
#>  Attr. frac. ex. |         .6472868       |    .1102698    .8601749 (Woolf)
#>  Attr. frac. pop |         .1213663       |
#>                  +-------------------------------------------------
#>                                chi2(1) =     5.18  Pr>chi2 = 0.0228
#> 
#> 
#> 
#> ------------------------------------------------------------------------------
#>              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
#> -------------+----------------------------------------------------------------
#>    Estimates |   2.835165   1.338429     2.21   0.027     1.123936    7.151794
#> ------------------------------------------------------------------------------
#> 
#> 
#> 
#> ------------------------------------------------------------------------------
#>              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
#> -------------+----------------------------------------------------------------
#>    Estimates |   2.222222   1.319891     1.34   0.179     .6937698    7.118025
#> ------------------------------------------------------------------------------

Then compare them with cifunction:

#> no; dataset in memory has changed since last saved
#> r(4);

Output from cifunction

The cifunction command produces and stores several variables when the saving() option is specified:

  • cilev: sequencing range of CIs (from 0 to 99.99)
  • plev: sequencing range of P-values (from 0.0001 to 1.0)
  • sval: S-values computed for respective plev values
  • lcl(#): computed lower confidence limits
  • ucl(#): computed upper confidence limits

References for cifunction

  • Birnbaum, A. 1961. Confidence curves: An omnibus technique for estimation and testing statistical hypotheses. Journal of the American Statistical Association 56: 246-249.
  • Folks, J. F. 1981. Ideas of Statistics. New York: John Wiley & Sons.
  • Greenland, S. 2019. Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values. The American Statistician 73(sup1): 106-114.
  • Sullivan, K. M., and Foster, D. 1990. Confidence curves versus confidence intervals. American Journal of Public Health 80(4): 452-453.

Cite R Packages

about
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> StataNow/MP 19.5 for Mac (Apple Silicon)
#> Revision 12 Aug 2026
#> Copyright 1985-2025 StataCorp LLC
#> 
#> Total physical memory: 48.01 GB
#> 
#> Stata license: Single-user 2-core , expiring  6 Feb 2027
#> Serial number: 501909358563
#>   Licensed to: Zad Rafi
#>

Session info

si <- sessionInfo()
print(si, RNG = TRUE, locale = TRUE)
#> R version 4.6.1 (2026-06-24)
#> Platform: aarch64-apple-darwin25.4.0
#> Running under: macOS Golden Gate 27.0
#> 
#> Matrix products: default
#> BLAS:   /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib 
#> LAPACK: /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib;  LAPACK version 3.12.1
#> 
#> Random number generation:
#>  RNG:     Mersenne-Twister 
#>  Normal:  Inversion 
#>  Sample:  Rejection 
#>  
#> locale:
#> [1] C.UTF-8/C.UTF-8/C.UTF-8/C/C.UTF-8/C.UTF-8
#> 
#> time zone: America/New_York
#> tzcode source: internal
#> 
#> attached base packages:
#>  [1] splines   grid      stats4    parallel  stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#>  [1] cli_3.6.6             texPreview_2.1.0      tinytex_0.60          rmarkdown_2.31        brms_2.23.0          
#>  [6] bootImpute_1.3.0      knitr_1.51            boot_1.3-32           gtsummary_2.5.1       reshape2_1.4.5       
#> [11] ProfileLikelihood_1.3 ImputeRobust_1.3-1    gamlss_5.5-0          gamlss.dist_6.1-1     gamlss.data_6.0-7    
#> [16] mvtnorm_1.4-2         performance_0.17.1    summarytools_1.1.5    tidybayes_3.0.7       htmltools_0.5.9      
#> [21] Statamarkdown_0.9.7   car_3.1-5             carData_3.0-6         qqplotr_0.0.7         ggcorrplot_0.3.0     
#> [26] mitml_0.4-5           pbmcapply_1.5.1       Amelia_1.8.3          Rcpp_1.1.2            blogdown_1.24        
#> [31] doParallel_1.0.17     iterators_1.0.14      foreach_1.5.2         lattice_0.23-1        bayesplot_1.15.0     
#> [36] wesanderson_0.3.7     VIM_7.0.0             colorspace_2.1-3      here_1.0.2            progress_1.2.3       
#> [41] loo_2.10.1            mi_1.3.1              Matrix_1.7-6          broom_1.0.13          yardstick_1.4.0      
#> [46] svglite_2.2.2         Cairo_1.7-0           cowplot_1.2.0         mgcv_1.9-4            nlme_3.1-170         
#> [51] xfun_0.60             broom.mixed_0.2.9.7   reticulate_1.46.0     kableExtra_1.4.1      posterior_1.7.0      
#> [56] checkmate_2.3.4       parallelly_1.48.0     miceFast_0.9.1        randomForest_4.7-1.2  missForest_1.6.1     
#> [61] miceadds_3.20-10      quantreg_6.1          SparseM_1.84-2        MCMCpack_1.7-1        MASS_7.3-66          
#> [66] coda_0.19-4.1         latex2exp_0.9.8       rstan_2.32.7          StanHeaders_2.32.10   lubridate_1.9.5      
#> [71] forcats_1.0.1         stringr_1.6.0         dplyr_1.2.1           purrr_1.2.2           readr_2.2.0          
#> [76] tibble_3.3.1          ggplot2_4.0.3         tidyverse_2.0.0       ggtext_0.1.2          concurve_3.0.0       
#> [81] showtext_0.9-8        showtextdb_3.0        sysfonts_0.8.9        future.apply_1.20.2   future_1.75.0        
#> [86] tidyr_1.3.2           magrittr_2.0.5        mice_3.19.0           rms_8.1-1             Hmisc_5.2-6          
#> 
#> loaded via a namespace (and not attached):
#>   [1] dichromat_2.0-1       nnet_7.3-21           TH.data_1.1-5         vctrs_0.7.3           digest_0.6.39        
#>   [6] png_0.1-9             shape_1.4.6.1         proxy_0.4-29          magick_2.9.1          withr_3.0.3          
#>  [11] survival_3.8-9        doRNG_1.8.6.3         MatrixModels_0.5-4    emmeans_2.0.4         systemfonts_1.3.2    
#>  [16] zoo_1.9-0             V8_8.2.0              ggdist_3.3.3          DEoptimR_1.2-0        Formula_1.2-6        
#>  [21] prettyunits_1.2.0     rematch2_2.1.2        httr_1.4.8            otel_0.2.0            globals_0.19.1       
#>  [26] rstudioapi_0.19.0     extremevalues_2.4.1   pan_2.0               generics_0.1.4        processx_3.9.0       
#>  [31] base64enc_0.1-6       curl_7.1.0            mitools_2.4           lgr_0.5.2             xtable_1.8-8         
#>  [36] svUnit_1.0.8          desc_1.4.3            pracma_2.4.6          evaluate_1.0.5        hms_1.1.4            
#>  [41] glmnet_5.0            rcartocolor_2.1.2     lmtest_0.9-40         palmerpenguins_0.1.1  robustbase_0.99-7    
#>  [46] matrixStats_1.5.0     svgPanZoom_0.3.4      class_7.3-24          pillar_1.11.1         caTools_1.18.4       
#>  [51] compiler_4.6.1        stringi_1.8.9         paradox_1.0.1         jomo_2.7-6            minqa_1.2.8          
#>  [56] plyr_1.8.9            crayon_1.5.3          abind_1.4-8           sp_2.2-3              rapportools_1.2      
#>  [61] twosamples_2.0.1      sandwich_3.1-3        whisker_0.4.1         codetools_0.2-20      multcomp_1.4-31      
#>  [66] textshaping_1.0.5     QuickJSR_1.10.0       e1071_1.7-17          gridtext_0.1.6        lme4_2.0-6           
#>  [71] fs_2.1.0              itertools_0.1-3       listenv_1.0.0         Rdpack_2.6.6          pkgbuild_1.4.8       
#>  [76] estimability_2.0.0    callr_3.8.0           tzdb_0.5.0            pkgconfig_2.0.3       tools_4.6.1          
#>  [81] rbibutils_2.4.1       viridisLite_0.4.3     DBI_1.3.0             fastmap_1.2.0         scales_1.4.0         
#>  [86] opdisDownsampling_1.6 insight_1.5.2         rpart_4.1.27          farver_2.1.2          moocore_0.3.2        
#>  [91] reformulas_0.4.4      yaml_2.3.12           foreign_0.8-91        lifecycle_1.0.5       bbotk_1.12.0         
#>  [96] backports_1.5.1       mlr3tuning_1.6.1      Brobdingnag_1.2-9     timechange_0.4.0      gtable_0.3.6         
#>  [ reached 'max' / getOption("max.print") -- omitted 42 entries ]
Back to top

Citation

BibTeX citation:
@misc{rafi2024,
  author = {Rafi, Zad and Panda, Sir},
  title = {Using {Stata:} {Producing} {Consonance} {Functions}},
  date = {2024-01-01},
  url = {https://lesslikely.com/statistics/stata},
  langid = {en-US},
  abstract = {A simple guide on how to produce consonance functions in
    Stata.}
}
]]> statistics https://lesslikely.com/statistics/stata.html Mon, 01 Jan 2024 05:00:00 GMT Simulation of a Two-Group Parallel-Arm RCT with Interim Analyses Zad Rafi https://lesslikely.com/statistics/interim_analyses.html

Recently Andrew Althouse informed me that he was going to simulate a two-group parallel-arm randomized trial with interim analyses using the rpact R package, so I offered to also help in constructing the R code to do so. He already has a number of R scripts on his GitHub repo for doing similar simulations, which can be viewed here and a number of tweets explaining these simulations. For this example, his goal was to simulate a trial where the outcome was binary and the probability of death for each group could be tuned in addition to:

  • the total number of participants
  • the number of interim analyses
  • the schedule of the interim analyses
  • the group-sequential design used

along with the usual trial analysis parameters such as:

  • the -level
  • the type of test (1-sided vs. 2-sided).

The goal was to be able to produce a table of various statistics such as:

  • the odds ratio
  • the confidence limits
  • the -value
  • the number of successes

for each of the interim analyses specified.

The function below is a reflection of our efforts to do so, and also returns several plots from the rpact package for the design that is chosen along with a plot comparing the design to other designs. In order to get similar results, you will need to load the R function first, and then simply enter the proper inputs. While there may be more efficient ways to write the code, for example using lapply() instead of for loops, we have chosen not to do so, and we have also tried to minimize the number of R packages necessary for the function to work but the following will be required:

You can quickly install and load both using:

Setting up the Function

#' @title Simulation of a Two-Group Parallel-Arm Trial With Interim Analyses
#' @docType Custom function for simulation from the rpact package
#' @author Andrew Althouse with edits by Zad Rafi
#' NOTE: If you want to confirm "type 1 error" under different stopping rules,
#' make death = in the two treatment arms (e.g. no treatment effect)
#' NOTE: I have set this one up to test the power for a treatment that would reduce mortality
#' from 40% in control group (1) to 30% in treatment group (2)
#' NOTE: Trial Design Parameters - Part 1
#' Here we will specify the basics: total N patients to enroll, and death rate for each treatment arm
#' NOTE: Trial Design Parameters - Part 2
#' Here we will define the interim analysis strategy and stopping rules
#' For this trial we will include provisions for efficacy stopping only (no pre-specified futility stopping)
#' We will use the rpact package to compute the stopping/success thresholds at the interim and final analysis
#' NOTE: Required packages: rpact and stringr
#' @param nSims # The number of simulations, the default is 1000
#' @param nPatients # here is where you specify the planned max number of patients you want included in each RCT
#' @param death1 # here is where you specify the event rate for patients receiving 'treatment 1' in these trial
#' @param death2 # here is where you specify the event rate for patients receiving 'treatment 2' in these trials
#' @param nLooks # here is where you put the number of looks that will take place (INCLUDING the final analysis)
#' @param analyses_scheduled # schedule of interim analyses
#' @param sided # Whether the test is 1-sided or 2-sided
#' @param alpha # Specified alpha level, the default is 0.05
#' @param informationRates #
#' @param trials # The total number of trials you wish to load in the table results.
#' @param typeOfDesign # The type of design.
#' @param seed # Argument to set the seed for the simulations
#' @return list of dataframes and plots
#' @example See below this code block
interim_sim <- function(nPatients = 1000, death1 = 0.4, death2 = 0.3,
                        nLooks = 4, analyses_scheduled = c(0.25, 0.50, 0.75, 1),
                        sided = 1, alpha = 0.05,
                        informationRates = analyses_scheduled,
                        typeOfDesign = "asOF", nSims = 1000, trials = 10,
                        seed = 1031) {

efficacy_thresholds <- numeric(nLooks)
design <- getDesignGroupSequential(
  sided = sided, alpha = alpha,
  informationRates = analyses_scheduled,
  typeOfDesign = typeOfDesign)
design_2 <- getDesignGroupSequential(typeOfDesign = "P") # Pocock
design_3 <- getDesignGroupSequential(typeOfDesign = "asP") # Alpha-spending Pocock
design_4 <- getDesignGroupSequential(typeOfDesign = "OF") # O'Brien-Fleming
designSet <- getDesignSet(designs = c(design, design_2,
                                      design_3, design_4),
                          variedParameters = "typeOfDesign")

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed(seed)
for (j in 1:nLooks) {
  efficacy_thresholds[j] <- design$stageLevels[j]
}

analyses_nPatients <- analyses_scheduled * nPatients
efficacy_thresholds
pb <- txtProgressBar(min = 0, max = nSims, initial = 0, style = 3)
trialnum <- numeric(nSims)
or <- data.frame(matrix(ncol = nLooks, nrow = nSims))
lcl <- data.frame(matrix(ncol = nLooks, nrow = nSims))
ucl <- data.frame(matrix(ncol = nLooks, nrow = nSims))
pval <- data.frame(matrix(ncol = nLooks, nrow = nSims))
success <- data.frame(matrix(ncol = nLooks, nrow = nSims))

strings <- c("OR_%d", "LCL_%d", "UCL_%d",
             "Pval_%d", "Success_%d")

colnames(or) <- sprintf("OR_%d", (1:nLooks))
colnames(lcl) <- sprintf("LCL_%d", (1:nLooks))
colnames(ucl) <- sprintf("UCL_%d", (1:nLooks))
colnames(pval) <- sprintf("Pval_%d", (1:nLooks))
colnames(success) <- sprintf("Success_%d", (1:nLooks))
overall_success <- numeric(nSims)
df <- data.frame(trialnum, or, lcl, ucl,
                 pval, success, overall_success)

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed(seed)
time <- system.time(
for (i in 1:nSims) {
  trialnum[i] <- i
  pid <- seq(1, nPatients, by = 1)
  treatment <- rep(1:2, nPatients / 2)
  deathprob <- numeric(nPatients)
  deathprob[treatment == 1] <- death1
  deathprob[treatment == 2] <- death2
  death <- rbinom(nPatients, 1, deathprob)
  trialdata <- data.frame(cbind(pid, treatment, death))

  for (j in 1:nLooks) {
    analysisdata <- subset(trialdata, pid <= analyses_nPatients[j])
    model <- glm(death ~ treatment,
                 family = binomial(link = "logit"),
                 data = analysisdata)
    or[i, j] <- exp(summary(model)$coefficients[2])
    lcl[i, j] <- exp(confint.default((model))[2, 1])
    ucl[i, j] <- exp(confint.default((model))[2, 2])
    pval[i, j] <- summary(model)$coefficients[8]
    success[i, j] <- ifelse(or[i, j] < 1 & pval[i, j] < efficacy_thresholds[j], 1, 0)
  }
  overall_success[i] <- 0
  for (j in 1:nLooks) {
    if (success[i, j] == 1) {
      overall_success[i] <- 1
    }
  }
    setTxtProgressBar(pb, i)
})

df <- data.frame(trialnum, or, lcl, ucl, pval,
                 success, overall_success)
simulation_results <- data.frame(matrix(vector(),
  nrow = nPatients, ncol = (length(df))))
colnames(simulation_results) <- c("trialnum", (do.call(
  rbind,
  lapply(1:length(strings),
    FUN = function(j) {
      (do.call(rbind, lapply(1:nLooks,
        FUN = function(i) ((sprintf(strings, i)))
      )[]))[, j]
    }
  )
)), "overall_success")
simulation_results[intersect(names(df), names(simulation_results))] <- (df[intersect(
  names(df),
  names(simulation_results)
)])
simulation_results <- as.data.frame(simulation_results)

outputs <- simulation_results[, c(-1, -length(simulation_results))]

cols <- as.character(1:nLooks)
rows <- as.character(1:length(strings))
interim <- matrix(
  nrow = length(strings), ncol = nLooks,
  dimnames = list((rows), (cols)),
  data = rep(0, (length(strings) * nLooks)))

for (i in cols) {
  interim[, i] <- str_subset(colnames(outputs), i)
}

colnames(interim) <- sprintf("Interim_Look_%d", (1:nLooks))
colnames(simulation_results)[1] <- c("TrialNum")
colnames(simulation_results)[length(simulation_results)] <- c("Overall_Success")
simulation_results_trials <- head(simulation_results, trials)
results <- list(
  summary(design),
  summary(time),
  plot(design, 1) +
    theme_light() +
    theme(
      plot.title = element_text(size = 14, face = "bold"),
      axis.title = element_text(size = 12, face = "bold")
    ),
  plot(designSet, type = 1) +
    theme_light() +
    theme(
      plot.title = element_text(size = 14, face = "bold"),
      axis.title = element_text(size = 12, face = "bold")
    ),
  table(overall_success),
  simulation_results_trials)

names(results) <- c(
  "Design Summary", "Time to complete simulation",
  "Main Design Plot", "Plot of Various Designs",
  "Table of Overall Success", "Simulation Results")

return(results)
}

A Simulated Example

results <- interim_sim(nPatients = 1000, death1 = 0.4, death2 = 0.3,
                       nLooks = 4, analyses_scheduled = c(0.25, 0.50, 0.75, 1),
                       sided = 1, alpha = 0.025, typeOfDesign = "asOF",
                       informationRates =  c(0.25, 0.50, 0.75, 1),
                       nSims = 1000, trials = 20, seed = 1031)

Examining the Results

results[1:5]
#> $`Design Summary`
#> 
#> $`Time to complete simulation`
#>    user  system elapsed 
#>    6.57    1.47    7.45 
#> 
#> $`Main Design Plot`
#> NULL
#> 
#> $`Plot of Various Designs`
#> NULL
#> 
#> $`Table of Overall Success`
#> overall_success
#>   0   1 
#> 134 866

table_results <- results[[6]]

table_results[ , 2:21] <- round((table_results[ , 2:21]), 3)

table_results$TrialNum <- row.names(table_results)
row.names(table_results) <- NULL

for (i in colnames(table_results)){
  table_results[,i] <- cell_spec(table_results[,i], color = "#777")
}

table_results <- table_results[c(colnames(table_results))]

To examine the results, I used the kableExtra package, though this is not necessary, and simply using the following script will suffice

table_results <- results[[6]]
View(table_results)
Interim_Look_1
Interim_Look_2
Interim_Look_3
Interim_Look_4
TrialNum OR_1 LCL_1 UCL_1 Pval_1 Success_1 OR_2 LCL_2 UCL_2 Pval_2 Success_2 OR_3 LCL_3 UCL_3 Pval_3 Success_3 OR_4 LCL_4 UCL_4 Pval_4 Success_4 Overall_Success
1 0.936 0.565 1.55 0.797 0 0.868 0.601 1.255 0.453 0 1.036 0.767 1.398 0.818 0 0.891 0.687 1.157 0.387 0 0
2 0.87 0.519 1.459 0.598 0 0.757 0.525 1.092 0.136 0 0.745 0.551 1.007 0.056 0 0.721 0.555 0.936 0.014 1 1
3 0.59 0.35 0.996 0.048 0 0.555 0.382 0.806 0.002 0 0.676 0.5 0.916 0.011 0 0.686 0.529 0.89 0.005 1 1
4 0.652 0.385 1.103 0.111 0 0.671 0.461 0.976 0.037 0 0.725 0.533 0.988 0.042 0 0.75 0.576 0.976 0.032 0 0
5 0.674 0.398 1.142 0.143 0 0.763 0.526 1.108 0.155 0 0.835 0.617 1.132 0.246 0 0.764 0.587 0.994 0.045 0 0
6 0.525 0.309 0.892 0.017 0 0.622 0.427 0.907 0.014 0 0.668 0.491 0.909 0.01 0 0.625 0.479 0.815 0.001 1 1
7 0.742 0.433 1.269 0.275 0 0.769 0.525 1.125 0.175 0 0.76 0.558 1.037 0.083 0 0.719 0.551 0.938 0.015 1 1
8 0.661 0.394 1.108 0.116 0 0.437 0.301 0.632 0 1 0.535 0.397 0.721 0 1 0.551 0.425 0.714 0 1 1
9 1.288 0.76 2.183 0.348 0 1.113 0.769 1.612 0.571 0 0.87 0.645 1.173 0.361 0 0.771 0.596 0.999 0.049 0 0
10 0.783 0.466 1.316 0.356 0 0.701 0.484 1.015 0.06 0 0.655 0.482 0.89 0.007 1 0.614 0.472 0.798 0 1 1
11 0.577 0.344 0.968 0.037 0 0.639 0.444 0.921 0.016 0 0.56 0.415 0.754 0 1 0.577 0.446 0.748 0 1 1
12 0.538 0.315 0.919 0.023 0 0.629 0.434 0.913 0.015 0 0.576 0.426 0.78 0 1 0.561 0.431 0.731 0 1 1
13 0.79 0.474 1.315 0.364 0 0.763 0.531 1.095 0.142 0 0.789 0.588 1.06 0.115 0 0.791 0.611 1.025 0.076 0 0
14 0.535 0.318 0.902 0.019 0 0.69 0.477 0.999 0.049 0 0.583 0.43 0.792 0.001 1 0.621 0.477 0.809 0 1 1
15 0.695 0.419 1.152 0.158 0 0.744 0.516 1.073 0.114 0 0.73 0.541 0.985 0.04 0 0.64 0.493 0.83 0.001 1 1
16 1.116 0.657 1.896 0.685 0 0.763 0.526 1.108 0.155 0 0.69 0.509 0.935 0.017 0 0.69 0.532 0.896 0.005 1 1
17 0.533 0.32 0.886 0.015 0 0.585 0.407 0.839 0.004 0 0.679 0.504 0.913 0.011 0 0.622 0.48 0.806 0 1 1
18 0.613 0.364 1.033 0.066 0 0.692 0.478 1 0.05 0 0.78 0.577 1.055 0.107 0 0.705 0.543 0.914 0.008 1 1
19 0.841 0.502 1.409 0.511 0 0.768 0.531 1.11 0.16 0 0.685 0.506 0.927 0.014 0 0.665 0.511 0.864 0.002 1 1
20 0.275 0.159 0.476 0 1 0.449 0.31 0.651 0 1 0.498 0.369 0.672 0 1 0.498 0.384 0.646 0 1 1
Abbreviations:
TrialNum: Trial Number | OR: Odds Ratio | LCL: Lower Confidence Level | UCL: Upper Confidence Level | Pval: P-value

Statistical Environment

The analyses were run on:

#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.6.1 (2026-06-24)
#>  os       macOS Golden Gate 27.0
#>  system   aarch64, darwin25.4.0
#>  ui       unknown
#>  language (EN)
#>  collate  en_US
#>  ctype    C.UTF-8
#>  tz       America/New_York
#>  date     2026-09-13
#>  pandoc   3.11 @ /opt/homebrew/bin/ (via rmarkdown)
#>  quarto   1.10.18 @ /usr/local/bin/quarto
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package           * version  date (UTC) lib source
#>  abind               1.4-8    2024-09-12 [1] CRAN (R 4.6.1)
#>  Amelia            * 1.8.3    2024-11-08 [1] CRAN (R 4.6.1)
#>  arm                 1.15-3   2026-04-15 [1] CRAN (R 4.6.1)
#>  arrayhelpers        1.1-2    2026-07-24 [1] CRAN (R 4.6.1)
#>  backports           1.5.1    2026-04-03 [1] CRAN (R 4.6.1)
#>  base              * 4.6.1    2026-06-24 [2] local
#>  base64enc           0.1-6    2026-02-02 [1] CRAN (R 4.6.1)
#>  bayesplot         * 1.16.0   2026-08-25 [1] CRAN (R 4.6.1)
#>  bitops              1.1-0    2026-07-30 [1] CRAN (R 4.6.1)
#>  blogdown          * 1.24     2026-06-19 [1] CRAN (R 4.6.1)
#>  boot              * 1.3-32   2025-08-29 [1] CRAN (R 4.6.1)
#>  bootImpute        * 1.3.0    2025-12-15 [1] CRAN (R 4.6.1)
#>  bridgesampling      1.2-1    2025-11-19 [1] CRAN (R 4.6.1)
#>  brms              * 2.23.0   2025-09-09 [1] CRAN (R 4.6.1)
#>  Brobdingnag         1.2-9    2022-10-19 [1] CRAN (R 4.6.1)
#>  broom             * 1.0.13   2026-05-14 [1] CRAN (R 4.6.1)
#>  broom.mixed       * 0.2.9.7  2026-02-17 [1] CRAN (R 4.6.1)
#>  Cairo             * 1.7-0    2025-10-29 [1] CRAN (R 4.6.1)
#>  car               * 3.1-5    2026-02-03 [1] CRAN (R 4.6.1)
#>  carData           * 3.0-6    2026-01-30 [1] CRAN (R 4.6.1)
#>  caTools             1.18.4   2026-07-20 [1] CRAN (R 4.6.1)
#>  checkmate         * 2.3.4    2026-02-03 [1] CRAN (R 4.6.1)
#>  class               7.3-24   2026-08-03 [1] CRAN (R 4.6.1)
#>  cli               * 3.6.6    2026-04-09 [1] CRAN (R 4.6.1)
#>  clipr               0.8.1    2026-05-25 [1] CRAN (R 4.6.1)
#>  cluster             2.1.8.3  2026-07-30 [1] CRAN (R 4.6.1)
#>  coda              * 0.19-4.1 2024-01-31 [1] CRAN (R 4.6.1)
#>  codetools           0.2-20   2024-03-31 [1] CRAN (R 4.6.0)
#>  colorspace        * 2.1-3    2026-07-12 [1] CRAN (R 4.6.1)
#>  compiler            4.6.1    2026-06-24 [2] local
#>  concurve          * 3.0.5    2026-09-13 [1] local
#>  cowplot           * 1.2.0    2025-07-07 [1] CRAN (R 4.6.1)
#>  crayon              1.5.3    2024-06-20 [1] CRAN (R 4.6.1)
#>  curl                8.0.0    2026-08-25 [1] CRAN (R 4.6.1)
#>  data.table          1.18.6.1 2026-08-24 [1] CRAN (R 4.6.1)
#>  datasets          * 4.6.1    2026-06-24 [2] local
#>  DBI                 1.3.0    2026-02-25 [1] CRAN (R 4.6.1)
#>  DEoptimR            1.2-1    2026-08-20 [1] CRAN (R 4.6.1)
#>  desc                1.4.3    2023-12-10 [1] CRAN (R 4.6.1)
#>  details             0.4.0    2025-02-09 [1] CRAN (R 4.6.0)
#>  dichromat           2.0-1    2026-07-22 [1] CRAN (R 4.6.1)
#>  digest              0.6.39   2025-11-19 [1] CRAN (R 4.6.1)
#>  distributional      0.9.0    2026-09-08 [1] CRAN (R 4.6.1)
#>  doParallel        * 1.0.17   2022-02-07 [1] CRAN (R 4.6.1)
#>  doRNG               1.8.6.3  2026-02-05 [1] CRAN (R 4.6.1)
#>  dplyr             * 1.2.1    2026-04-03 [1] CRAN (R 4.6.1)
#>  e1071               1.7-17   2025-12-18 [1] CRAN (R 4.6.1)
#>  emmeans             2.0.4    2026-07-15 [1] CRAN (R 4.6.1)
#>  estimability        2.0.0    2026-06-26 [1] CRAN (R 4.6.1)
#>  evaluate            1.0.5    2025-08-27 [1] CRAN (R 4.6.1)
#>  extremevalues       2.4.1    2024-12-17 [1] CRAN (R 4.6.1)
#>  farver              2.1.2    2024-05-13 [1] CRAN (R 4.6.1)
#>  fastmap             1.2.0    2024-05-15 [1] CRAN (R 4.6.1)
#>  forcats           * 1.0.1    2025-09-25 [1] CRAN (R 4.6.1)
#>  foreach           * 1.5.2    2022-02-02 [1] CRAN (R 4.6.1)
#>  foreign             0.8-91   2026-01-29 [1] CRAN (R 4.6.1)
#>  Formula             1.2-6    2026-08-03 [1] CRAN (R 4.6.1)
#>  fs                  2.1.0    2026-04-18 [1] CRAN (R 4.6.1)
#>  furrr               0.4.0    2026-03-31 [1] CRAN (R 4.6.1)
#>  future            * 1.75.0   2026-07-20 [1] CRAN (R 4.6.1)
#>  future.apply      * 1.20.2   2026-02-20 [1] CRAN (R 4.6.1)
#>  gamlss            * 5.5-0    2025-08-19 [1] CRAN (R 4.6.1)
#>  gamlss.data       * 6.0-7    2025-09-04 [1] CRAN (R 4.6.1)
#>  gamlss.dist       * 6.1-11   2026-09-10 [1] CRAN (R 4.6.1)
#>  generics            0.1.4    2025-05-09 [1] CRAN (R 4.6.1)
#>  ggcorrplot        * 0.3.0    2026-07-24 [1] CRAN (R 4.6.1)
#>  ggdist              3.3.3    2025-04-23 [1] CRAN (R 4.6.1)
#>  ggplot2           * 4.0.3    2026-04-22 [1] CRAN (R 4.6.1)
#>  ggtext            * 0.2.0    2026-08-28 [1] CRAN (R 4.6.1)
#>  glmnet              5.0      2026-05-04 [1] CRAN (R 4.6.1)
#>  globals             0.19.1   2026-03-13 [1] CRAN (R 4.6.1)
#>  glue                1.8.1    2026-04-17 [1] CRAN (R 4.6.1)
#>  graphics          * 4.6.1    2026-06-24 [2] local
#>  grDevices         * 4.6.1    2026-06-24 [2] local
#>  grid              * 4.6.1    2026-06-24 [2] local
#>  gridExtra           2.3.1    2026-06-25 [1] CRAN (R 4.6.1)
#>  gridtext            0.1.6    2026-02-19 [1] CRAN (R 4.6.0)
#>  gtable              0.3.6    2024-10-25 [1] CRAN (R 4.6.1)
#>  gtsummary         * 2.6.1    2026-09-04 [1] CRAN (R 4.6.1)
#>  here              * 1.0.2    2025-09-15 [1] CRAN (R 4.6.1)
#>  Hmisc             * 5.3-0    2026-09-06 [1] CRAN (R 4.6.1)
#>  hms                 1.1.4    2025-10-17 [1] CRAN (R 4.6.1)
#>  htmlTable           2.5.0    2026-04-22 [1] CRAN (R 4.6.1)
#>  htmltools         * 0.5.9    2025-12-04 [1] CRAN (R 4.6.1)
#>  htmlwidgets         1.6.4    2023-12-06 [1] CRAN (R 4.6.1)
#>  httr                1.4.9    2026-09-01 [1] CRAN (R 4.6.1)
#>  ImputeRobust      * 1.3-1    2018-11-30 [1] CRAN (R 4.6.1)
#>  inline              0.3.21   2025-01-09 [1] CRAN (R 4.6.1)
#>  insight             1.5.4    2026-09-05 [1] CRAN (R 4.6.1)
#>  iterators         * 1.0.14   2022-02-05 [1] CRAN (R 4.6.1)
#>  itertools           0.1-3    2014-03-12 [1] CRAN (R 4.6.1)
#>  jomo                2.7-6    2023-04-15 [1] CRAN (R 4.6.1)
#>  jsonlite            2.0.0    2025-03-27 [1] CRAN (R 4.6.1)
#>  kableExtra        * 1.4.1    2026-07-08 [1] CRAN (R 4.6.1)
#>  knitr             * 1.52     2026-09-06 [1] CRAN (R 4.6.1)
#>  labeling            0.4.3    2023-08-29 [1] CRAN (R 4.6.1)
#>  laeken              0.5.3    2024-01-25 [1] CRAN (R 4.6.1)
#>  latex2exp         * 0.9.8    2026-01-09 [1] CRAN (R 4.6.1)
#>  lattice           * 0.23-1   2026-08-12 [1] CRAN (R 4.6.1)
#>  lifecycle           1.0.5    2026-01-08 [1] CRAN (R 4.6.1)
#>  listenv             1.0.0    2026-06-22 [1] CRAN (R 4.6.1)
#>  lme4                2.0-6    2026-07-16 [1] CRAN (R 4.6.1)
#>  lmtest              0.9-40   2022-03-21 [1] CRAN (R 4.6.1)
#>  loo               * 2.10.1   2026-07-24 [1] CRAN (R 4.6.1)
#>  lubridate         * 1.9.5    2026-02-04 [1] CRAN (R 4.6.1)
#>  magick              2.9.1    2026-02-28 [1] CRAN (R 4.6.1)
#>  magrittr          * 2.0.5    2026-04-04 [1] CRAN (R 4.6.1)
#>  MASS              * 7.3-66   2026-07-15 [1] CRAN (R 4.6.1)
#>  Matrix            * 1.7-6    2026-07-25 [1] CRAN (R 4.6.1)
#>  MatrixModels        0.5-4    2025-03-26 [1] CRAN (R 4.6.1)
#>  matrixStats         1.5.0    2025-01-07 [1] CRAN (R 4.6.1)
#>  mcmc                0.9-8    2023-11-16 [1] CRAN (R 4.6.1)
#>  MCMCpack          * 1.7-1    2024-08-27 [1] CRAN (R 4.6.1)
#>  methods           * 4.6.1    2026-06-24 [2] local
#>  mgcv              * 1.9-4    2025-11-07 [1] CRAN (R 4.6.0)
#>  mi                * 1.3.1    2026-07-28 [1] CRAN (R 4.6.1)
#>  mice              * 3.19.0   2025-12-10 [1] CRAN (R 4.6.1)
#>  miceadds          * 3.20-10  2026-05-28 [1] CRAN (R 4.6.1)
#>  miceFast          * 0.9.1    2026-02-26 [1] CRAN (R 4.6.1)
#>  minqa               1.2.8    2024-08-17 [1] CRAN (R 4.6.1)
#>  missForest        * 1.6.1    2025-10-26 [1] CRAN (R 4.6.1)
#>  mitml             * 0.4-5    2023-03-08 [1] CRAN (R 4.6.1)
#>  mitools             2.7      2026-08-31 [1] CRAN (R 4.6.1)
#>  multcomp            1.4-32   2026-08-21 [1] CRAN (R 4.6.1)
#>  mvtnorm           * 1.4-2    2026-07-12 [1] CRAN (R 4.6.1)
#>  nlme              * 3.1-171  2026-09-01 [1] CRAN (R 4.6.1)
#>  nloptr              2.2.1    2025-03-17 [1] CRAN (R 4.6.1)
#>  nnet                7.3-21   2026-08-03 [1] CRAN (R 4.6.1)
#>  opdisDownsampling   1.7      2026-09-04 [1] CRAN (R 4.6.1)
#>  otel                0.2.0    2025-08-29 [1] CRAN (R 4.6.1)
#>  pan                 2.0      2026-06-30 [1] CRAN (R 4.6.1)
#>  pander              0.6.6    2025-03-01 [1] CRAN (R 4.6.1)
#>  parallel          * 4.6.1    2026-06-24 [2] local
#>  parallelly        * 1.48.0   2026-06-29 [1] CRAN (R 4.6.1)
#>  pbmcapply         * 1.5.1    2022-04-28 [1] CRAN (R 4.6.1)
#>  performance       * 0.18.2   2026-09-10 [1] CRAN (R 4.6.1)
#>  pillar              1.11.1   2025-09-17 [1] CRAN (R 4.6.1)
#>  pkgbuild            1.4.8    2025-05-26 [1] CRAN (R 4.6.1)
#>  pkgconfig           2.0.3    2019-09-22 [1] CRAN (R 4.6.1)
#>  plyr                1.8.9    2023-10-02 [1] CRAN (R 4.6.1)
#>  png                 0.1-9    2026-03-15 [1] CRAN (R 4.6.0)
#>  polspline           1.1.25   2024-05-10 [1] CRAN (R 4.6.1)
#>  posterior         * 1.7.0    2026-04-01 [1] CRAN (R 4.6.1)
#>  pracma              2.4.6    2025-10-22 [1] CRAN (R 4.6.1)
#>  prettyunits         1.2.0    2023-09-24 [1] CRAN (R 4.6.1)
#>  ProfileLikelihood * 1.3      2023-08-25 [1] CRAN (R 4.6.1)
#>  progress          * 1.2.3    2023-12-06 [1] CRAN (R 4.6.1)
#>  proxy               0.4-29   2025-12-29 [1] CRAN (R 4.6.1)
#>  purrr             * 1.2.2    2026-04-10 [1] CRAN (R 4.6.1)
#>  qqconf              1.3.2    2023-04-14 [1] CRAN (R 4.6.1)
#>  qqplotr           * 0.0.7    2025-09-05 [1] CRAN (R 4.6.1)
#>  quantreg          * 6.1      2025-03-10 [1] CRAN (R 4.6.1)
#>  QuickJSR            1.11.0   2026-08-21 [1] CRAN (R 4.6.1)
#>  R6                  2.6.1    2025-02-15 [1] CRAN (R 4.6.1)
#>  randomForest      * 4.7-1.2  2024-09-22 [1] CRAN (R 4.6.1)
#>  ranger              0.18.0   2026-01-16 [1] CRAN (R 4.6.1)
#>  rappdirs            0.3.4    2026-01-17 [1] CRAN (R 4.6.1)
#>  rapportools         1.2      2025-02-28 [1] CRAN (R 4.6.1)
#>  rbibutils           2.4.1    2026-01-21 [1] CRAN (R 4.6.1)
#>  RColorBrewer        1.1-3    2022-04-03 [1] CRAN (R 4.6.1)
#>  Rcpp              * 1.1.2    2026-07-05 [1] CRAN (R 4.6.1)
#>  RcppParallel        6.2.1    2026-08-27 [1] CRAN (R 4.6.1)
#>  Rdpack              2.6.6    2026-02-08 [1] CRAN (R 4.6.1)
#>  readr             * 2.2.0    2026-02-19 [1] CRAN (R 4.6.1)
#>  reformulas          0.4.4    2026-02-02 [1] CRAN (R 4.6.1)
#>  rematch2            2.1.2    2020-05-01 [1] CRAN (R 4.6.1)
#>  reshape2          * 1.4.5    2025-11-12 [1] CRAN (R 4.6.1)
#>  reticulate        * 1.47.0   2026-09-03 [1] CRAN (R 4.6.1)
#>  rlang               1.3.0    2026-07-05 [1] CRAN (R 4.6.1)
#>  rmarkdown         * 2.32     2026-09-01 [1] CRAN (R 4.6.1)
#>  rms               * 8.1-1    2026-02-18 [1] CRAN (R 4.6.1)
#>  rngtools            1.5.2    2021-09-20 [1] CRAN (R 4.6.1)
#>  robustbase          0.99-7   2026-02-05 [1] CRAN (R 4.6.1)
#>  rpact             * 4.4.0    2026-03-04 [1] CRAN (R 4.6.1)
#>  rpart               4.1.27   2026-03-27 [1] CRAN (R 4.6.0)
#>  rprojroot           2.1.1    2025-08-26 [1] CRAN (R 4.6.1)
#>  rstan             * 2.32.7   2025-03-10 [1] CRAN (R 4.6.1)
#>  rstantools          2.7.1    2026-09-04 [1] Github (stan-dev/rstantools@4d84659)
#>  rstudioapi          0.19.0   2026-06-11 [1] CRAN (R 4.6.1)
#>  S7                  0.2.2    2026-04-22 [1] CRAN (R 4.6.1)
#>  sandwich            3.1-3    2026-08-03 [1] CRAN (R 4.6.1)
#>  scales              1.4.0    2025-04-24 [1] CRAN (R 4.6.1)
#>  sessioninfo         1.2.4    2026-06-04 [1] CRAN (R 4.6.1)
#>  shape               1.4.6.1  2024-02-23 [1] CRAN (R 4.6.1)
#>  showtext          * 0.9-8    2026-03-21 [1] CRAN (R 4.6.1)
#>  showtextdb        * 3.0      2020-06-04 [1] CRAN (R 4.6.1)
#>  sp                  2.2-3    2026-07-19 [1] CRAN (R 4.6.1)
#>  SparseM           * 1.84-2   2024-07-17 [1] CRAN (R 4.6.1)
#>  splines           * 4.6.1    2026-06-24 [2] local
#>  StanHeaders       * 2.39.1   2026-09-02 [1] CRAN (R 4.6.1)
#>  Statamarkdown     * 1.0.0    2026-08-21 [1] CRAN (R 4.6.1)
#>  stats             * 4.6.1    2026-06-24 [2] local
#>  stats4            * 4.6.1    2026-06-24 [2] local
#>  stringi             1.8.9    2026-08-04 [1] CRAN (R 4.6.1)
#>  stringr           * 1.6.0    2025-11-04 [1] CRAN (R 4.6.1)
#>  summarytools      * 1.1.5    2026-06-25 [1] Github (dcomtois/summarytools@2b613cf)
#>  survival            3.8-12   2026-09-09 [1] CRAN (R 4.6.1)
#>  svglite           * 2.2.2    2025-10-21 [1] CRAN (R 4.6.1)
#>  svgPanZoom          0.3.4    2020-02-15 [1] CRAN (R 4.6.1)
#>  svUnit              1.0.8    2025-08-26 [1] CRAN (R 4.6.1)
#>  sysfonts          * 0.8.9    2024-03-02 [1] CRAN (R 4.6.1)
#>  systemfonts         1.3.2    2026-03-05 [1] CRAN (R 4.6.1)
#>  tcltk               4.6.1    2026-06-24 [2] local
#>  tensorA             0.36.2.1 2023-12-13 [1] CRAN (R 4.6.0)
#>  texPreview        * 2.1.0    2024-01-24 [1] CRAN (R 4.6.0)
#>  textshaping         1.0.5    2026-03-06 [1] CRAN (R 4.6.0)
#>  TH.data             1.1-5    2025-11-17 [1] CRAN (R 4.6.0)
#>  tibble            * 3.3.1    2026-01-11 [1] CRAN (R 4.6.1)
#>  tidybayes         * 3.0.7    2024-09-15 [1] CRAN (R 4.6.1)
#>  tidyr             * 1.3.2    2025-12-19 [1] CRAN (R 4.6.1)
#>  tidyselect          1.2.1    2024-03-11 [1] CRAN (R 4.6.1)
#>  tidyverse         * 2.0.0    2023-02-22 [1] CRAN (R 4.6.1)
#>  timechange          0.4.0    2026-01-29 [1] CRAN (R 4.6.1)
#>  tinytex           * 0.60     2026-06-16 [1] CRAN (R 4.6.1)
#>  tools               4.6.1    2026-06-24 [2] local
#>  twosamples          2.0.1    2023-06-23 [1] CRAN (R 4.6.1)
#>  tzdb                0.5.0    2025-03-15 [1] CRAN (R 4.6.1)
#>  utils             * 4.6.1    2026-06-24 [2] local
#>  V8                  8.2.0    2026-04-21 [1] CRAN (R 4.6.0)
#>  vcd                 1.4-14   2026-07-29 [1] CRAN (R 4.6.1)
#>  vctrs               0.7.3    2026-04-11 [1] CRAN (R 4.6.1)
#>  VIM               * 7.3.1    2026-09-11 [1] CRAN (R 4.6.1)
#>  viridisLite         0.4.3    2026-02-04 [1] CRAN (R 4.6.1)
#>  wesanderson       * 0.3.7    2023-10-31 [1] CRAN (R 4.6.1)
#>  whisker             0.4.1    2022-12-05 [1] CRAN (R 4.6.1)
#>  withr               3.0.3    2026-06-19 [1] CRAN (R 4.6.1)
#>  xfun              * 0.60     2026-07-09 [1] CRAN (R 4.6.1)
#>  xgboost             3.2.1.1  2026-03-18 [1] CRAN (R 4.6.1)
#>  xml2                1.6.0    2026-06-22 [1] CRAN (R 4.6.1)
#>  xtable              1.8-8    2026-02-22 [1] CRAN (R 4.6.1)
#>  yaml                2.3.12   2025-12-10 [1] CRAN (R 4.6.1)
#>  yardstick         * 1.4.0    2026-04-07 [1] CRAN (R 4.6.1)
#>  zoo                 1.9-0    2026-07-31 [1] CRAN (R 4.6.1)
#> 
#>  [1] /opt/homebrew/lib/R/4.6/site-library
#>  [2] /opt/homebrew/Cellar/r/4.6.1/lib/R/library
#>  * ── Packages attached to the search path.
#> 
#> ─ External software ──────────────────────────────────────────────────────────
#>  setting        value
#>  cairo          1.18.4
#>  cairoFT        2.14.3/2.18.1
#>  pango
#>  png
#>  jpeg
#>  tiff
#>  tcl            9.0
#>  curl           8.7.1
#>  zlib           1.2.12
#>  bzlib          1.0.8, 13-Jul-2019
#>  xz             5.8.3
#>  deflate
#>  zstd           1.5.7
#>  PCRE           10.48 2026-08-31
#>  ICU            78.1
#>  TRE            TRE 0.8.0 R_fixes (BSD)
#>  iconv          Apple or GNU libiconv 1.11 /usr/lib/libiconv.2.dylib
#>  readline       8.3
#>  BLAS           /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib
#>  lapack         /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib
#>  lapack_version 3.12.1
#> 
#> ─ Python configuration ───────────────────────────────────────────────────────
#>  Python is not available
#> 
#> ──────────────────────────────────────────────────────────────────────────────
#> $RNGkind
#> [1] "L'Ecuyer-CMRG" "Inversion"     "Rejection"

Article Citation

1

Back to top

References

1. Wassmer G, Pahlke F. (2018). “Rpact: Confirmatory adaptive clinical trial design and analysis.” doi: 10.32614/CRAN.package.rpact. https://doi.org/10.32614/CRAN.package.rpact.

Citation

BibTeX citation:
@misc{rafi2021,
  author = {Rafi, Zad and Rafi, Zad},
  title = {Simulation of a {Two-Group} {Parallel-Arm} {RCT} with
    {Interim} {Analyses}},
  date = {2021-02-01},
  url = {https://lesslikely.com/statistics/interim_analyses},
  langid = {en-US},
  abstract = {A simulation of a two-group parallel-arm randomized trial
    with interim analysis using the rpact package.}
}
]]> statistics https://lesslikely.com/statistics/interim_analyses.html Mon, 01 Feb 2021 05:00:00 GMT Tables, Graphs, and Computations from Rafi & Greenland (2020) Zad Rafi https://lesslikely.com/statistics/RG2020BMC.html

The following post provides some of the code that was used to construct the figures and tables from Rafi & Greenland, 20201. An enhanced PDF version of the paper can be found here. For further discussion of the computations, see the appendix of the main paper, along with our technical supplement.2

Disclaimer: I am responsible for all the code and mistakes below, and none of them can be attributed to my coauthors or my fellow package developers.

In order to recreate the functions, I would recommend installing the latest version of concurve from CRAN, as it has patched some issues with graphing when the outcome is binary. Use the script below to get the latest version and load the R package. A number of other R packages are also used in this post, which are listed below.

install.packages("concurve")
library("concurve")

Valid -values Are Uniform Under the Null Model

Here we show that valid -values have specific properties, when the null model is true. We first generate two variables (, ) that come from the same normal distribution with a of 0 and of 1, each with a total of 1000 observations. We assume that there is no relationship between these two variables. We run a simple t-test between and and iterate this 100000 times and compute 100000 -values to see the overall distribution of the -values, which we then plot using a histogram./

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
n.sim <- 100
t.sim <- numeric(n.sim)
n.samp <- 1000

for (i in 1:n.sim) {
    X <- rnorm(n.samp, mean = 0, sd = 1)
    Y <- rnorm(n.samp, mean = 0, sd = 1)
    df <- data.frame(X, Y)
    t <- t.test(X, Y, data = df)
    t.sim[i] <- t[[3]]
}

ggplot(NULL, aes(x = t.sim)) + geom_histogram(bins = 30, col = "black",
    fill = "#99c7c7", alpha = 0.25) + labs(title = "Distribution of P-values Under the Null",
    x = "P-value") + scale_x_continuous(breaks = seq(0, 1, 0.1)) +
    theme_bw()

This can also be shown using the TeachingDemos R package, which has a function dedicated to showing this phenomenon.

library("TeachingDemos")
RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
obs_p <- Pvalue.norm.sim(n = 1000, mu = 0, mu0 = 0, sigma = 1,
    sigma0 = 1, test = "t", alternative = "two.sided", alpha = 0.05,
    B = 1e+05)

ggplot(NULL, aes(x = obs_p)) + geom_histogram(bins = 30, col = "black",
    fill = "#99c7c7", alpha = 0.25) + labs(title = "Distribution of P-values Under the Null",
    x = "P-value") + scale_x_continuous(breaks = seq(0, 1, 0.1)) +
    theme_bw()

As you can see, when the null model is true, the distribution of -values is uniform. Valid -values are uniform under the null hypothesis and their corresponding -values are exponentially distributed. We run the same simulation as before, but then convert the obtained -values into -values, to see how they are distributed.

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
n.sim <- 100
t.sim <- numeric(n.sim)
n.samp <- 1000

for (i in 1:n.sim) {
    X <- rnorm(n.samp, mean = 0, sd = 1)
    Y <- rnorm(n.samp, mean = 0, sd = 1)
    df <- data.frame(X, Y)
    t <- t.test(X, Y, data = df)
    t.sim[i] <- t[[3]]
}

ggplot(NULL, aes(x = -log2(t.sim))) + geom_histogram(bins = 30,
    col = "black", fill = "#d46c5b", alpha = 0.5) + labs(title = "Distribution of S-values Under the Null",
    x = "S-value (Bits of Information)") + theme_bw()

Posterior Predictive -values

Despite the name, posterior predictive -values are not considered valid frequentist -values because they do not meet the uniformity criterion, instead they are pulled towards the parameter value 0.5. For further discussion, see our technical supplement along with Greenland (2019) for comprehensive theoretical discussions.2, 3

A quick excerpt from our main paper and the technical supplement explains why this is not the case,

As discussed in the Supplement, in Bayesian settings one may see certain -values that are not valid frequentist -values, the primary example being the posterior predictive -value4;5; unfortunately, the negative logs of such invalid -values do not measure surprisal at the statistic given the model, and so are not valid -values.

The decision rule “reject H if ” will reject H 100% of the time under sampling from a model M obeying H (i.e., the Type-1 error rate of the test will be ) provided the random variable corresponding to is valid (uniform under the model M used to compute it), but not necessarily otherwise6. This is one reason why frequentist writers reject invalid -values (such as posterior predictive -values, which highly concentrate around 0.50) and devote considerable technical coverage to uniform -values6;5;4. A valid -value (“U-value”) translates into an exponentially distributed -value with a mean of 1 nat or bits where is the base of the natural logs.

Uniformity is also central to the “refutational information” interpretation of the -value used here, for it is necessary to ensure that the -value from which is derived is in fact the percentile of the observed value of the test statistic in the distribution of the statistic under M, thus making small surprising under M and making the corresponding degree of surprise. Because posterior predictive -values do not translate into sampling percentiles of the statistic under the hypothesis (in fact, they are pulled toward 0.5 from the correct percentiles)5;4, the resulting negative log does not measure surprisal at the statistic given M, and so is not a valid -value in our terms.

And indeed, we can show this phenomenon below with simulations. Here we fit a simple Bayesian regression model with a weakly informative prior normal(0, 10) using rstanarm, where both the predictor and response variable come from the same distribution and have the same location and scale parameters (, . We then calculate the observed test statistics, along with their distributions, and convert them into posterior predictive -values and plot them using Bayesplot functions. Then, we iterate this 1000 times to examine the distribution of posterior predictive -values and compare them to standard -values that are known to be uniform.

But first, we’ll generate the distribution of the test statistic from one model.

library("bayesplot")
library("rstan")
library("rstanarm")
rstan_options(auto_write = TRUE)
options(mc.cores = 4)
teal <- c("#99c7c7")
color_scheme_set("teal")
RNGkind(kind = "L'Ecuyer-CMRG")
n.samp <- 100
X <- rnorm(n.samp, mean = 0, sd = 1)
Y <- rnorm(n.samp, mean = 0, sd = 1)
df <- data.frame(X, Y)
mod1 <- stan_glm(Y ~ X, data = df, chains = 1, cores = 4, refresh = 0,
    prior = normal(0, 10))
#> Error in `if (!prior_dist_name %in% unlist(ok_dists)) ...`:
#> ! argument is of length zero
yrep <- posterior_predict(mod1)
#> Error:
#> ! object 'mod1' not found

h <- ppc_stat(Y, yrep, stat = "median", freq = TRUE, binwidth = 0.01) +
    labs(title = "Distribution of Posterior Test Statistic") +
    theme_bw() + yaxis_text(on = TRUE)
#> Error:
#> ! object 'yrep' not found

values_all <- h[["plot_env"]][["T_yrep"]]
#> Error:
#> ! object 'h' not found
prob_to_find <- h[["plot_env"]][["T_y"]]
#> Error:
#> ! object 'h' not found
quantInv <- function(distr, value) ecdf(distr)(value)
posterior_p_value <- quantInv(values_all, prob_to_find)
#> Error:
#> ! object 'values_all' not found

l <- ppc_stat_2d(Y, yrep) + theme_bw() + labs(title = "Distribution of Posterior Test Statistic")
#> Error:
#> ! object 'yrep' not found

plot(h)
#> Error in `h()`:
#> ! error in evaluating the argument 'x' in selecting a method for function 'plot': object 'h' not found
plot(l)
#> Error in `h()`:
#> ! error in evaluating the argument 'x' in selecting a method for function 'plot': object 'l' not found
print(prob_to_find)
#> Error in `h()`:
#> ! error in evaluating the argument 'x' in selecting a method for function 'print': object 'prob_to_find' not found

The above is the posterior predictive test statistic for one model, along with it’s distribution. Now let’s iterate this process, convert them into posterior predictive -values, and generate their distribution.

rstan_options(auto_write = TRUE)
options(mc.cores = 4)
RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
n.sim <- 100
ppp <- numeric(n.sim)
n.samp <- 1000

quantInv <- function(distr, value) ecdf(distr)(value)

for (i in 1:length(ppp)) {
    X <- rnorm(n.samp, mean = 0, sd = 1)
    Y <- rnorm(n.samp, mean = 0, sd = 1)
    df <- data.frame(X, Y)
    mod1 <- stan_glm(Y ~ X, data = df, chains = 1, cores = 4,
        iter = 1000, refresh = 0, prior = normal(0, 10))
    yrep <- posterior_predict(mod1)
    h <- ppc_stat(Y, yrep, stat = "median")
    values_all <- h[["plot_env"]][["T_yrep"]]
    prob_to_find <- h[["plot_env"]][["T_y"]]
    posterior_p_value <- quantInv(values_all, prob_to_find)
    ppp[i] <- posterior_p_value
}
#> Error in `if (!prior_dist_name %in% unlist(ok_dists)) ...`:
#> ! argument is of length zero

ggplot(NULL, aes(x = ppp)) + geom_histogram(bins = 30, col = "black",
    fill = "#99c7c7", alpha = 0.25) + labs(title = "Distribution of Posterior Predictive P-values",
    subtitle = "From Simulation of 1000 Simple Linear Regressions",
    x = "Posterior Predictive P-value") + scale_x_continuous(breaks = seq(0,
    1, 0.1)) + theme_bw()

We can also show this in Stata 16.1, using their new bayesstats ppvalues command which generates posterior predictive -values.

clear
set seed 1031
set obs 100

generate x = rnormal(0, 1)
generate y = rnormal(0, 1)

* Bayesian linear regression of y on x, mirroring the rstanarm model.
* Stata parameterizes the normal() prior by VARIANCE, so rstanarm's
* normal(0, sd = 10) is normal(0, 100) here. {y:} puts the prior on all
* regression coefficients including the intercept -- bayesmh requires a
* prior for every parameter. The variance gets a proper inverse-gamma
* prior; a normal prior is invalid for {var} (its support includes
* negative values).
bayesmh y x, likelihood(normal({var}))  ///
    prior({y:}, normal(0, 100))         ///
    prior({var}, igamma(0.01, 0.01))    ///
    rseed(1031) mcmcsize(1000)          ///
    saving(bmh_sim, replace)

* Simulate replicated outcomes from the posterior predictive
* distribution. This must be saved to a DIFFERENT file than the
* bayesmh simulation dataset above.
bayespredict {_ysim}, rseed(1031) saving(bmh_pred, replace)

* Posterior predictive p-values for test statistics of the replicated
* outcome. (The R analog uses the median; @median is not a built-in
* bayespredict/bayesstats function, so we use the mean and variance,
* matching the worked example in [BAYES] bayesstats ppvalues.)
bayesstats ppvalues (tmean:@mean({_ysim})) (tvar:@variance({_ysim})) ///
    using bmh_pred
#> could not find feasible values
#> r(491);
#> 
#> 
#> 
#> 
#> Number of observations (_N) was 0, now 100.
#> 
#> 
#> 
#> 
#> Burn-in ...
#> Simulation ...
#> 
#> Model summary
#> -------------------------------------------------------------------------
#> > -----
#> Likelihood: 
#>   y ~ normal(xb_y,{var})
#> 
#> Priors: 
#>   {y:x _cons} ~
#>                 normal(0,100)                                            
#>                 >   (1)
#>         {var} ~ igamma(0.01,0.01)
#> -------------------------------------------------------------------------
#> > -----
#> (1) Parameters are elements of the linear form xb_y.
#> 
#> Bayesian normal regression                       MCMC iterations  =      
#> > 3,500
#> Random-walk Metropolis–Hastings sampling         Burn-in          =      
#> > 2,500
#>                                                  MCMC sample size =      
#> > 1,000
#>                                                  Number of obs    =      
#> >   100
#>                                                  Acceptance rate  =      
#> > .2428
#>                                                  Efficiency:  min =    .0
#> > 01044
#>                                                               avg =     .
#> > 04252
#> Log marginal-likelihood = -153.90987                          max =     .
#> > 08305
#> 
#> -------------------------------------------------------------------------
#> > -----
#>              |                                                Equal-taile
#> > d
#>              |      Mean   Std. dev.     MCSE     Median  [95% cred. inte
#> > rval]
#> -------------+-----------------------------------------------------------
#> > -----
#> y            |
#>            x |  .0115139   .1551555   .017026   .0145751  -.2995666   .34
#> > 70796
#>        _cons |  .1022613    .137549   .020865   .1348998  -.2513679   .34
#> > 55865
#> -------------+-----------------------------------------------------------
#> > -----
#>          var |  .9643251   .1503727    .14715   .9896135   .7582507   1.3
#> > 60154
#> -------------------------------------------------------------------------
#> > -----
#> 
#> file bmh_sim.dta saved.
#> 
#> 
#> Computing predictions ...
#> 
#> file bmh_pred.dta saved.
#> file bmh_pred.ster saved.
#> 
#> 
#> Posterior predictive summary   MCMC sample size =     1,000
#> 
#> -----------------------------------------------------------
#>            T |      Mean   Std. dev.  E(T_obs)  P(T>=T_obs)
#> -------------+---------------------------------------------
#>        tmean |  .0550687   .1146021   .0206952         .632
#>         tvar |  .9896307   .2164376   .9710317         .484
#> -----------------------------------------------------------
#> Note: P(T>=T_obs) close to 0 or 1 indicates lack of fit.
#> 
#> OMP: Warning #96: Cannot form a team with 2 threads, using 1 instead.
#> OMP: Hint Consider unsetting KMP_DEVICE_THREAD_LIMIT (KMP_ALL_THREADS), KMP_TEAMS_THREAD_LIMIT, and OMP_THREAD_LIMIT (if any are set).

As we can see from the R plot above, the test statistics are are generally concentrated around 0.5 and the distribution (as shown above via the R code) hardly resembles the uniform distribution of -value under the null model. Further, the base-2 log transformation of posterior predictive -values is not exponentially distributed, making them invalid -values in our terms.

ggplot(NULL, aes(x = (-log2(ppp)))) + geom_histogram(bins = 30,
    col = "black", fill = "#d46c5b", alpha = 0.25) + labs(title = "Distribution of -log2(Posterior Predictive P-values)",
    subtitle = "S-values Produced From Posterior Predictive P-values",
    x = "-log2(Posterior Predictive P-value)") + theme_bw()
#> Error in `geom_histogram()`:
#> ! Problem while computing stat.
#> ℹ Error occurred in the 1st layer.
#> Caused by error in `seq_len()`:
#> ! argument must be coercible to non-negative integer

Table 1: Some -values, -values, Maximum Likelihood Ratios, and Likelihood Ratio Statistics

Here, we look at some common -values, and how they correspond to other information statistics and likelihood measures.

pvalue <- c(0.99, 0.9, 0.5, 0.25, 0.1, 0.05, 0.025, 0.01, 0.005,
    1e-04, paste("5 sigma (~ 2.9 in 10 million)"), paste("1 in 100 million (GWAS)"),
    paste("6 sigma (~ 1 in a billion)"))

svalue <- round(c(-log2(c(0.99, 0.9, 0.5, 0.25, 0.1, 0.05, 0.025,
    0.01, 0.005, 1e-04)), 21.7, 26.6, 29.9), 2)

pvalue[1:10] <- formatC(pvalue[1:10], format = "e", digits = 2)

mlr <- round(c(1, 1.01, 1.26, 1.94, 3.87, 6.83, 12.3, 27.6, 51.4,
    1935, (5.2 * (10^5)), (1.4 * (10^7)), (1.3 * (10^8))), 2)

# likelihood ratio statistic

lr <- round(c(0.00016, 0.016, 0.45, 1.32, 2.71, 3.84, 5.02, 6.63,
    7.88, 15.1, 26.3, 32.8, 37.4), 2)

table1 <- data.frame(pvalue, svalue, mlr, lr)

colnames(table1) <- c("P-value (compatibility)", "S-value (bits)",
    "Maximum Likelihood Ratio", "Deviance Statistic 2ln(MLR)")

kbl(table1, format = "html", padding = 45, longtable = TRUE,
    color = "#777") %>%
    row_spec(row = 0, color = "#777", bold = T) %>%
    column_spec(column = 1:4, color = "#777", bold = F) %>%
    kable_classic(full_width = TRUE, html_font = "Cambria", lightable_options = "basic",
        font_size = 17, fixed_thead = list(enabled = F)) %>%
    footnote(general_title = "Abbreviations: ", title_format = "underline",
        general = c("Table 1 $P$-values                  and binary $S$-values, with corresponding maximum-likelihood ratios (MLR) and deviance (likelihood-ratio) statistics for a simple test hypothesis H under background assumptions A"))
P-value (compatibility) S-value (bits) Maximum Likelihood Ratio Deviance Statistic 2ln(MLR)
0.99 0.01 1.00e+00 0.00
0.9 0.15 1.01e+00 0.02
0.5 1.00 1.26e+00 0.45
0.25 2.00 1.94e+00 1.32
0.1 3.32 3.87e+00 2.71
0.05 4.32 6.83e+00 3.84
0.025 5.32 1.23e+01 5.02
0.01 6.64 2.76e+01 6.63
0.005 7.64 5.14e+01 7.88
1e-04 13.29 1.94e+03 15.10
5 sigma (~ 2.9 in 10 million) 21.70 5.20e+05 26.30
1 in 100 million (GWAS) 26.60 1.40e+07 32.80
6 sigma (~ 1 in a billion) 29.90 1.30e+08 37.40
Abbreviations:
Table 1 $P$-values and binary $S$-values, with corresponding maximum-likelihood ratios (MLR) and deviance (likelihood-ratio) statistics for a simple test hypothesis H under background assumptions A

For further discussion of 5 and 6 sigma cutoffs, see7.

Frequentist Interval Estimate Percentiles

Refer to Long-Run Coverage

Here we simulate a study where one group with 100 participants has an of 100 with a of 20 and the second group has the same number of participants but an average of 80 and a standard deviation of 20. We compare them using a Welch’s -test and generate 95% compatibility intervals several times, specifically 100 times, and then plot them. Since we know the mean difference is 20, we wish to see how often the interval estimates cover this true parameter value.

This shows that the intervals tend to vary simply as a result of randomness, and that the proper interpretation of the attached percentile is about long run coverage of the true parameter. However, this does not preclude interpretation of a single interval estimate from a study.8 As we write in the Replace unrealistic “confidence” claims with compatibility measures section of our paper.1

The fact that “confidence” refers to the procedure behavior, not the reported interval, seems to be lost on most researchers. Remarking on this subtlety, when Jerzy Neyman discussed his confidence concept in 1934 at a meeting of the Royal Statistical Society, Arthur Bowley replied, “I am not at all sure that the ‘confidence’ is not a confidence trick.”9. And indeed, 40 years later, Cox and Hinkley warned, “interval estimates cannot be taken as probability statements about parameters, and foremost is the interpretation ‘such and such parameter values are consistent with the data.’”8. Unfortunately, the word “consistency” is used for several other concepts in statistics, while in logic it refers to an absolute condition (of noncontradiction); thus, its use in place of “confidence” would risk further confusion.

To address the problems above, we exploit the fact that a 95% CI summarizes the results of varying the test hypothesis H over a range of parameter values, displaying all values for which > 0.0510 and hence < 4.32 bits3;11. Thus the CI contains a range of parameter values that are more compatible with the data than are values outside the interval, under the background assumptions3;12. Unconditionally (and thus even if the background assumptions are uncertain), the interval shows the values of the parameter which, when combined with the background assumptions, produce a test model that is “highly compatible” with the data in the sense of having less than 4.32 bits of information against it. We thus refer to CI as compatibility intervals rather than confidence intervals13;3;11; their abbreviation remains “CI.” We reject calling these intervals “uncertainty intervals,” because they do not capture uncertainty about background assumptions13.

Indeed, this also has to do with our paper on deemphasizing the model assumptions that are often behind common statistical outputs, and why it is necessary to treat them as uncertain, rather than given.14 For example, a 95% interval estimate is assumed to be free of bias in its construction, however, it’s coverage claims are no longer so, once there are systematic errors in play. Many of these common, classical statistical procedures are designed to deal with random variation, and less so with bias (a relatively new field, with shiny new methods).

Brown et al. (2017) Reanalysis

Taken from the Brown et al. data.15, 16

Here we take the reported statistics from the Brown et al. studies in order to run statistical tests of different alternative test hypotheses and use those results to construct various functions.

We calculate the standard errors from the point estimate and the confidence (compatibility) limits.

se <- log(2.59/0.997)/3.92

logUL <- log(2.59)
logLL <- log(0.997)

logpoint <- log(1.61)

logpoint + (1.96 * se)
#> [1] 0.954
logpoint - (1.96 * se)
#> [1] -0.0011

Table 2: -values, -values, and Likelihoods for Targeted Hypotheses About Hazard Ratios for Brown et al.

testhypothesis <- c("Halving of hazard", "No effect (null)",
    "Point estimate", "Doubling of hazard", "Tripling of hazard",
    "Quintupling of hazard")

hazardratios <- c(0.5, 1, 1.61, 2, 3, 5)
pvals <- c(1.6e-06, 0.05, 1, 0.37, 0.01, 3.2e-06)
svals <- round(-log2(pvals), 2)
mlr2 <- c((1 * 10^5), 6.77, 1, 1.49, 26.2, (5 * 10^4))
lr <- round(c(exp((((log(0.5/1.61))/se)^2)/(2)), exp((((log(1/1.61))/se)^2)/(2)),
    exp((((log(1.61/1.61))/se)^2)/(2)), exp((((log(2/1.61))/se)^2)/(2)),
    exp((((log(3/1.61))/se)^2)/(2)), exp((((log(5/1.61))/se)^2)/(2))),
    3)

LR <- formatC(lr, format = "e", digits = 2)

table2 <- data.frame(testhypothesis, hazardratios, pvals, svals,
    mlr2, LR)

colnames(table2) <- c("Test Hypothesis", "HR", "P-values", "S-values",
    "MLR", "LR")

kbl(table2, format = "html", padding = 45, longtable = TRUE,
    color = "#777") %>%
    row_spec(row = 0, color = "#777", bold = T) %>%
    column_spec(column = 1:6, color = "#777", bold = F) %>%
    kable_classic(full_width = TRUE, html_font = "Cambria", lightable_options = "basic",
        font_size = 15, fixed_thead = list(enabled = F))
Test Hypothesis HR P-values S-values MLR LR
Halving of hazard 0.50 0.00 19.25 1.00e+05 1.02e+05
No effect (null) 1.00 0.05 4.32 6.77e+00 6.77e+00
Point estimate 1.61 1.00 0.00 1.00e+00 1.00e+00
Doubling of hazard 2.00 0.37 1.43 1.49e+00 1.49e+00
Tripling of hazard 3.00 0.01 6.64 2.62e+01 2.62e+01
Quintupling of hazard 5.00 0.00 18.25 5.00e+04 5.03e+04

Plot the point estimate and 95% compatibility interval

label <- c("Brown et al. (2017)\n JAMA", "Brown et al. (2017)\n J Clin Psychiatry")

point <- c(1.61, 1.7)
lower <- c(0.997, 1.1)
upper <- c(2.59, 2.6)

df <- data.frame(label, point, lower, upper)

Here we plot the 95% compatibility interval estimate reported from the high-dimensional propensity score analysis.

ggplot(data = df, mapping = aes(x = label, y = point, ymin = lower,
    ymax = upper, group = label)) + geom_pointrange(color = "#007C7C",
    size = 1.5, alpha = 0.4) + geom_hline(yintercept = 1, lty = 3,
    color = "#d46c5b", alpha = 0.5) + coord_flip() + scale_y_log10(breaks = scales::pretty_breaks(n = 10),
    limits = c(0.8, 4)) + labs(title = "Association Between Serotonergic Antidepressant Exposure
               \nDuring Pregnancy and Child Autism Spectrum Disorder",
    subtitle = "Reported 95% Compatibility Intervals From Primary Results",
    x = "Study", y = "Hazard Ratio (JAMA) +
            Adjusted Pooled Odds Ratio (J Clin Psychiatry)") +
    annotate(geom = "text", x = 1.5, y = 1, size = 5, label = "Null parameter value of 1",
        color = "#d46c5b", alpha = 0.75) + annotate(geom = "text",
    x = 2.2, y = 1.8, size = 5, label = "Point Estimate = 1.61,
           Lower Limit = 0.997, Upper Limit = 2.59",
    color = "#000000", alpha = 0.75) + annotate(geom = "text",
    x = 1.2, y = 1.8, size = 5, label = "Point Estimate = 1.7,
           Lower Limit = 1.1, Upper Limit = 2.6",
    color = "#000000", alpha = 0.75) + theme_light() + theme(axis.text.y = element_text(angle = 90,
    hjust = 1, size = 13)) + theme(axis.text.x = element_text(size = 13)) +
    theme(title = element_text(size = 13))

Once again, the authors reported that15

In the 2837 pregnancies (7.9%) exposed to antidepressants, 2.0% (95% CI, 1.6%-2.6%) of children were diagnosed with autism spectrum disorder. Risk of autism spectrum disorder was significantly higher with serotonergic antidepressant exposure (4.51 exposed vs 2.03 unexposed per 1000 person-years; between-group difference, 2.48 95% CI, 2.33-2.62 per 1000 person-years) in crude (HR, 2.16 95% CI, 1.64-2.86) and multivariable-adjusted analyses (HR, 1.59 95% CI, 1.17-2.17) (Table 2). After inverse probability of treatment weighting based on the HDPS, the association was not significant (HR, 1.61 95% CI, 0.997-2.59) (Table 2).

and concluded with

In children born to mothers receiving public drug coverage in Ontario, Canada, in utero serotonergic antidepressant exposure compared with no exposure was not associated with autism spectrum disorder in the child. Although a causal relationship cannot be ruled out, the previously observed association may be explained by other factors.

We will show using the graphical and tabular summaries below, why a more nuanced summary such as,

“After HDPS adjustment for confounding, a 61% hazard elevation remained; however, under the same model, every hypothesis from no elevation up to a 160% hazard increase had > 0.05; Thus, while quite imprecise, these results are consistent with previous observations of a positive association between serotonergic antidepressant prescriptions and subsequent ASD. Because the association may be partially or wholly due to uncontrolled biases, further evidence will be needed for evaluating what, if any, proportion of it can be attributed to causal effects of prenatal serotonergic antidepressant use on ASD incidence.”

would have been far more appropriate, as we discuss in our paper.1

-value and -value Functions

In order to use this information to construct a -value function, we can use the concurve functions.

library("concurve")

We enter the reported point estimates and compatibility limits and produce all possible intervals + -values + -values. This is calculated assuming normal approximations with the curve_rev() function.

curve1 <- curve_rev(point = 1.61, LL = 0.997, UL = 2.59, measure = "ratio")

It is stored in the object curve1.

We can generate a table of the relevant statistics from this reanalysis, similar to the table from above, but this time using the concurve function, curve_table(), but this time it will give us interval estimates of various percentiles (%, %, %, %, etc.). These are in turn used to construct the entire confidence (compatibility) distribution or -value function.

kbl(curve_table(curve1[[1]]), format = "html", padding = 45,
    longtable = TRUE, color = "#777", row.names = F, col.names = colnames(curve_table(curve1[[1]]))) %>%
    row_spec(row = 0, color = "#777", bold = T) %>%
    column_spec(column = 1:7, color = "#777", bold = F) %>%
    kable_classic(full_width = TRUE, html_font = "Cambria", lightable_options = "basic",
        font_size = 17, fixed_thead = list(enabled = F)) %>%
    footnote(general_title = c("Table of Statistics for Various Interval Estimate Percentiles"),
        general = " ")
Lower Limit Upper Limit Interval Width Interval Level (%) CDF P-value S-value (bits)
1.490 1.74 0.250 25.0 0.625 0.750 0.415
1.366 1.90 0.531 50.0 0.750 0.500 1.000
1.217 2.13 0.914 75.0 0.875 0.250 2.000
1.178 2.20 1.021 80.0 0.900 0.200 2.322
1.134 2.29 1.152 85.0 0.925 0.150 2.737
1.079 2.40 1.325 90.0 0.950 0.100 3.322
0.999 2.60 1.596 95.0 0.975 0.050 4.322
0.933 2.78 1.846 97.5 0.988 0.025 5.322
0.860 3.02 2.155 99.0 0.995 0.010 6.644
Table of Statistics for Various Interval Estimate Percentiles

Figure 3: -value (Compatibility) Function

Plot the -value (Compatibility) function of the Brown et al. data

Now we plot the -value function from the data stored in curve1 using the ggcurve() function from concurve.

ggcurve(curve1[[1]], type = "c", measure = "ratio", nullvalue = c(1),
    levels = c(0.5, 0.75, 0.95)) + labs(title = expression(paste(italic(P),
    "-value (Compatibility) Function")), subtitle = expression(paste(italic(P),
    "-values for a range of hazard ratios (HR)")), x = "Hazard Ratio (HR)",
    y = expression(paste(italic(P), "-value"))) + geom_vline(xintercept = 1.61,
    lty = 1, color = "gray", alpha = 0.2) + geom_vline(xintercept = 2.59,
    lty = 1, color = "gray", alpha = 0.2) + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11))

It is practically the same as the published version from Rafi & Greenland, 20201.

curve1 <- curve_rev(point = 1.61, LL = 0.997, UL = 2.59, measure = "ratio")

curve2 <- curve_rev(point = 1.7, LL = 1.1, UL = 2.6, measure = "ratio")

Cumulative Confidence (Compatibility Distribution)

Although we do not cover this figure in our paper, it is easy to construct using concurve's ggcurve() function by specifying type as “cdf” to see the median estimate within the confidence distribution. The horizontal line that meets at the curve, is approximately the same as the maximum at the -value function/confidence (compatibility) curve.

ggcurve(curve1[[2]], type = "cdf", measure = "ratio", nullvalue = c(1)) +
    labs(title = "P-values for a range of hazard ratios (HR)",
        subtitle = "Association Between Serotonergic Antidepressant Exposure
       \nDuring Pregnancy and Child Autism Spectrum Disorder",
        x = "Hazard Ratio (HR)", y = "Cumulative Compatibility") +
    theme(plot.title = element_text(size = 12), plot.subtitle = element_text(size = 11),
        axis.title.x = element_text(size = 12), axis.title.y = element_text(size = 12),
        text = element_text(size = 11))

We will also calculate the likelihoods so that we can generate the corresponding likelihood functions.

lik2 <- curve_rev(point = 1.7, LL = 1.1, UL = 2.6, type = "l",
    measure = "ratio")

lik1 <- curve_rev(point = 1.61, LL = 0.997, UL = 2.59, type = "l",
    measure = "ratio")

We can also see how consistent these results are with previous studies conducted by the same research group, given the overlap of the functions, which can be compared using the plot_compare() function. Let’s compare the relative likelihood functions from both studies from this research group to see how consistent the results are.

plot_compare(data1 = lik1[[1]], data2 = lik2[[1]], type = "l1",
    measure = "ratio", nullvalue = TRUE, title = "Brown et al. 2017. J Clin Psychiatry. vs.
              \nBrown et al. 2017. JAMA.",
    subtitle = "J Clin Psychiatry: OR = 1.7, 1/6.83 LI: LL = 1.1, UL = 2.6
                         \nJAMA: HR = 1.61, 1/6.83 LI: LL = 0.997, UL = 2.59",
    xaxis = "Hazard Ratio / Odds Ratio") + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11))

and the -value functions.

plot_compare(data1 = curve1[[1]], data2 = curve2[[1]], type = "c",
    measure = "ratio", nullvalue = TRUE, title = "Brown et al. 2017. J Clin Psychiatry. vs.
              \nBrown et al. 2017. JAMA.",
    subtitle = "J Clin Psychiatry: OR = 1.7, 1/6.83 LI: LL = 1.1, UL = 2.6
              \nJAMA: HR = 1.61, 1/6.83 LI: LL = 0.997, UL = 2.59",
    xaxis = "Hazard Ratio / Odds Ratio") + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11))

Figure 4: -value (Suprisal) Function

Plot the -value (Surprisal) function of the Brown et al. data with ggcurve()

ggcurve(data = curve1[[1]], type = "s", measure = "ratio", nullvalue = c(1),
    title = "S-value (Surprisal) Function", subtitle = "S-Values (surprisals) for a range of hazard ratios (HR)",
    xaxis = "Hazard Ratio", yaxis1 = "S-value (bits of information)") +
    theme(plot.title = element_text(size = 12), plot.subtitle = element_text(size = 11),
        axis.title.x = element_text(size = 12), axis.title.y = element_text(size = 12),
        text = element_text(size = 11))

which is quite close to the figure from our paper.

Likelihood (Support) Functions

Here we provide the code to show how we constructed the likelihood functions from our paper, which are the supplementary figures found here (S1) and here (S2).

Calculate and Plot Likelihood (Support) Intervals.

Here we use the reported estimates to construct the -scores and standard errors, which are in turn used to compute the likelihoods.

hrvalues <- seq(from = 0.65, to = 3.98, by = 0.01)

se <- log(2.59/0.997)/3.92

zscore <- sapply(hrvalues, function(i) (log(i/1.61)/se))

Figure S1: Relative Likelihood Function

We then standardize all the likelihoods by their maximum at the likelihood function,17 to produce relative likelihoods, likelihood intervals, and their corresponding relative likelihood function.18

support <- exp((-zscore^2)/2)

likfunction <- data.frame(hrvalues, zscore, support)

ggplot(data = likfunction, mapping = aes(x = hrvalues, y = support)) +
    geom_line() + geom_vline(xintercept = 1, lty = 3, color = "#d46c5b",
    alpha = 0.75) + geom_hline(yintercept = (1/6.83), lty = 1,
    color = "#333333", alpha = 0.05) + geom_ribbon(aes(x = hrvalues,
    ymin = min(support), ymax = support), fill = "#d46c5b", alpha = 0.1) +
    labs(title = "Relative Likelihood Function", subtitle = "Relative likelihoods for a range of hazard ratios",
        x = "Hazard Ratio (HR)", y = "Relative Likelihood \n1/MLR") +
    annotate(geom = "text", x = 1.65, y = 0.2, label = "1/6.83 LI = 0.997, 2.59",
        size = 4, color = "#000000") + theme_bw() + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11)) +
    scale_x_log10(breaks = scales::pretty_breaks(n = 10)) + scale_y_continuous(expand = expansion(mult = c(0.01,
    0.0125)), breaks = seq(0, 1, 0.1))

The likelihood interval corresponds to the 95% compatibility interval, as shown by the figure from our paper.

Below we use the calculated -scores to construct the log-likelihood function which is the upward-concave parabola = 

support <- (zscore^2)/2

likfunction <- data.frame(hrvalues, zscore, support)

ggplot(data = likfunction, mapping = aes(x = hrvalues, y = support)) +
    geom_line() + geom_vline(xintercept = 1, lty = 3, color = "#d46c5b",
    alpha = 0.75) + geom_ribbon(aes(x = hrvalues, ymin = support,
    ymax = max(support)), fill = "#d46c5b", alpha = 0.1) + labs(title = "Log-Likelihood Function",
    subtitle = "Log-likelihoods for a range of hazard ratios",
    x = "Hazard Ratio (HR)", y = "ln(MLR)") + theme_bw() + theme(axis.title.x = element_text(size = 13),
    axis.title.y = element_text(size = 13)) + scale_x_log10(breaks = scales::pretty_breaks(n = 10)) +
    scale_y_continuous(expand = expansion(mult = c(0.01, 0.0125)),
        breaks = seq(0, 7, 1)) + theme(text = element_text(size = 15)) +
    theme(plot.title = element_text(size = 12), plot.subtitle = element_text(size = 12))

Figure S2: Deviance Function

Known as the deviance function, it is twice the log-likelihood, which maps to the -value function.

support <- (zscore^2)

likfunction <- data.frame(hrvalues, zscore, support)

ggplot(data = likfunction, mapping = aes(x = hrvalues, y = support)) +
    geom_line() + geom_vline(xintercept = 1, lty = 3, color = "#d46c5b",
    alpha = 0.75) + geom_hline(yintercept = 3.84, lty = 1, color = "#333333",
    alpha = 0.05) + geom_ribbon(aes(x = hrvalues, ymin = support,
    ymax = max(support)), fill = "#d46c5b", alpha = 0.1) + annotate(geom = "text",
    x = 1.65, y = 4.4, label = "1/6.83 LI = 0.997, 2.59", size = 4,
    color = "#000000") + theme_bw() + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11)) +
    scale_x_log10(breaks = scales::pretty_breaks(n = 10)) + scale_y_continuous(expand = expansion(mult = c(0.01,
    0.0125)), breaks = seq(0, 14, 2)) + labs(title = "Deviance Function",
    subtitle = "Deviance statistics for a range of hazard ratios",
    x = "Hazard Ratio (HR)", y = " Deviance Statistic \n2ln(MLR)")

And the version from our paper can be found here.

It is important to note that although we have calculated these likelihood functions manually here, they can also be generated easily using the curve_lik() function which takes inputs from the ProfileLikelihood R package. To see a further discussion, please see the following article, which gives several examples for a wide variety of models.

Further, we urge some caution. Although we endorse the construction and presentation of likelihood functions along with -value and -value functions, along with the tabulations, the use of pure likelihood methods has been highly controversial among some statisticians, with some going as far as to say that likelihood is blind although not all statisticians believe this, and have responded to such criticisms in kind, see discussants.19 Thus, we support providing both -value/-value and likelihood-based functions for a complete picture.

Indeed, we can see how they all easily map to one another when plotted side by side, with the following script.

library("cowplot")
plot_grid(confcurve, surprisalcurve, relsupportfunction, deviancefunction,
    ncol = 2, nrow = 2)

A clearer plot comparing the four functions is seen below (click on the image to view in full).

Additional adjustments that were made to the figures from Rafi & Greenland, 20201 were done using Adobe Illustrator and Photoshop.

All errors are ours, and we welcome critical feedback and reporting of errors. To report possible errors in our analyses, please post it as a comment below or as a bug here. We will compile them onto a public errata, if any errors or flaws are reported to us.

Statistical Package Citations

Please remember to cite the R packages if you use any of the R scripts from above. The citation for our1 can be found below in the References section or it can be downloaded here for a reference manager.

citation("concurve")
citation("TeachingDemos")
citation("ProfileLikelihood")
citation("rstan")
citation("rstanarm")
citation("bayesplot")
citation("knitr")
citation("kableExtra")
citation("ggplot2")
citation("cowplot")
citation("Statamarkdown")

Environment

The analyses were run on:

si <- sessionInfo()
print(si, RNG = TRUE, locale = TRUE)
#> R version 4.6.1 (2026-06-24)
#> Platform: aarch64-apple-darwin25.4.0
#> Running under: macOS Golden Gate 27.0
#> 
#> Matrix products: default
#> BLAS:   /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib 
#> LAPACK: /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib;  LAPACK version 3.12.1
#> 
#> Random number generation:
#>  RNG:     L'Ecuyer-CMRG 
#>  Normal:  Inversion 
#>  Sample:  Rejection 
#>  
#> locale:
#> [1] en_US/C.UTF-8/en_US/C/en_US/en_US
#> 
#> time zone: America/New_York
#> tzcode source: internal
#> 
#> attached base packages:
#>  [1] splines   grid      stats4    parallel  stats     graphics  grDevices
#>  [8] utils     datasets  methods   base     
#> 
#> other attached packages:
#>   [1] plotly_4.12.1         TSstudio_0.1.7        tfautograph_0.3.2    
#>   [4] tfdatasets_2.18.0     keras_2.16.1          tensorflow_2.20.0    
#>   [7] timetk_2.9.1          modeltime_1.3.5       workflowsets_1.1.1   
#>  [10] workflows_1.3.0       tune_2.1.0            tailor_0.1.0         
#>  [13] rsample_1.3.2         recipes_1.4.0         parsnip_1.6.0        
#>  [16] modeldata_1.6.0       infer_1.1.0           dials_1.4.4          
#>  [19] scales_1.4.0          tidymodels_1.5.0      xgboost_3.2.1.1      
#>  [22] prophet_1.1.7         rlang_1.3.0           astsa_2.5            
#>  [25] bayesforecast_1.0.5   smooth_4.5.2          greybox_2.0.8        
#>  [28] glmnet_5.0            forecast_9.0.2        fable_0.5.0          
#>  [31] fabletools_0.8.0      tsibbledata_0.4.1     tsibble_1.2.0        
#>  [34] rstanarm_2.32.2       texPreview_2.1.0      tinytex_0.60         
#>  [37] rmarkdown_2.32        brms_2.23.0           bootImpute_1.3.0     
#>  [40] knitr_1.52            boot_1.3-32           JuliaCall_0.17.6     
#>  [43] reshape2_1.4.5        ProfileLikelihood_1.3 ImputeRobust_1.3-1   
#>  [46] gamlss_5.5-0          gamlss.dist_6.1-11    gamlss.data_6.0-7    
#>  [49] mvtnorm_1.4-2         performance_0.18.2    summarytools_1.1.5   
#>  [52] tidybayes_3.0.7       htmltools_0.5.9       Statamarkdown_1.0.0  
#>  [55] car_3.1-5             carData_3.0-6         qqplotr_0.0.7        
#>  [58] ggcorrplot_0.3.0      mitml_0.4-5           pbmcapply_1.5.1      
#>  [61] Amelia_1.8.3          Rcpp_1.1.2            blogdown_1.24        
#>  [64] doParallel_1.0.17     iterators_1.0.14      foreach_1.5.2        
#>  [67] lattice_0.23-1        bayesplot_1.16.0      wesanderson_0.3.7    
#>  [70] VIM_7.3.1             colorspace_2.1-3      here_1.0.2           
#>  [73] progress_1.2.3        loo_2.10.1            mi_1.3.1             
#>  [76] Matrix_1.7-6          broom_1.0.13          yardstick_1.4.0      
#>  [79] svglite_2.2.2         Cairo_1.7-0           cowplot_1.2.0        
#>  [82] mgcv_1.9-4            nlme_3.1-171          xfun_0.60            
#>  [85] broom.mixed_0.2.9.7   reticulate_1.47.0     kableExtra_1.4.1     
#>  [88] posterior_1.7.0       checkmate_2.3.4       parallelly_1.48.0    
#>  [91] miceFast_0.9.1        randomForest_4.7-1.2  missForest_1.6.1     
#>  [94] miceadds_3.20-10      quantreg_6.1          SparseM_1.84-2       
#>  [97] MCMCpack_1.7-1        MASS_7.3-66           coda_0.19-4.1        
#> [100] latex2exp_0.9.8       rstan_2.32.7          StanHeaders_2.39.1   
#> [103] lubridate_1.9.5       forcats_1.0.1         stringr_1.6.0        
#> [106] dplyr_1.2.1           purrr_1.2.2           readr_2.2.0          
#> [109] tibble_3.3.1          ggplot2_4.0.3         tidyverse_2.0.0      
#> [112] ggtext_0.2.0          concurve_3.0.5        showtext_0.9-8       
#> [115] showtextdb_3.0        sysfonts_0.8.9        future.apply_1.20.2  
#> [118] future_1.75.0         tidyr_1.3.2           magrittr_2.0.5       
#> [121] mice_3.19.0           rms_8.1-1             Hmisc_5.3-0          
#> 
#> loaded via a namespace (and not attached):
#>   [1] igraph_2.3.3          Formula_1.2-6         rematch2_2.1.2       
#>   [4] tidyselect_1.2.1      bridgesampling_1.2-1  rngtools_1.5.2       
#>   [7] dichromat_2.0-1       png_0.1-9             cli_3.6.6            
#>  [10] arrayhelpers_1.1-2    textshaping_1.0.5     curl_8.0.0           
#>  [13] mime_0.13             evaluate_1.0.5        V8_8.2.0             
#>  [16] stringi_1.8.9         backports_1.5.1       desc_1.4.3           
#>  [19] qqconf_1.3.2          httpuv_1.6.17         rappdirs_0.3.4       
#>  [22] details_0.4.0         prodlim_2026.03.11    doRNG_1.8.6.3        
#>  [25] DT_0.34.0             DBI_1.3.0             withr_3.0.3          
#>  [28] reformulas_0.4.4      class_7.3-24          systemfonts_1.3.2    
#>  [31] tfruns_1.5.4          rprojroot_2.1.1       lmtest_0.9-40        
#>  [34] formatR_1.14          colourpicker_1.3.0    htmlwidgets_1.6.4    
#>  [37] fs_2.1.0              labeling_0.4.3        ranger_0.18.0        
#>  [40] DEoptimR_1.2-1        zoo_1.9-0             itertools_0.1-3      
#>  [43] svUnit_1.0.8          timechange_0.4.0      caTools_1.18.4       
#>  [46] extremevalues_2.4.1   data.table_1.18.6.1   timeDate_4052.112    
#>  [49] pan_2.0               clipr_0.8.1           yaml_2.3.12          
#>  [52] survival_3.8-12       crayon_1.5.3          tensorA_0.36.2.1     
#>  [55] RColorBrewer_1.1-3    later_1.4.8           codetools_0.2-20     
#>  [58] base64enc_0.1-6       shape_1.4.6.1         estimability_2.0.0   
#>  [61] foreign_0.8-91        DiceDesign_1.10       pkgconfig_2.0.3      
#>  [64] xml2_1.6.0            viridisLite_0.4.3     xtable_1.8-8         
#>  [67] plyr_1.8.9            httr_1.4.9            rbibutils_2.4.1      
#>  [70] tools_4.6.1           globals_0.19.1        hardhat_1.4.3        
#>  [73] pkgbuild_1.4.8        htmlTable_2.5.0       shinyjs_2.1.1        
#>  [76] crosstalk_1.2.2       twosamples_2.0.1      MatrixModels_0.5-4   
#>  [79] lme4_2.0-6            digest_0.6.39         furrr_0.4.0          
#>  [82] farver_2.1.2          tzdb_0.5.0            rapportools_1.2      
#>  [85] rpart_4.1.27          glue_1.8.1            mixtime_0.3.0        
#>  [88] fracdiff_1.5-4        generics_0.1.4        ggtime_1.0.0         
#>  [91] opdisDownsampling_1.7 statmod_1.5.2         arm_1.15-3           
#>  [94] minqa_1.2.8           mcmc_0.9-8            tcltk_4.6.1          
#>  [97] texreg_1.40           gower_1.0.2           mitools_2.7          
#> [100] gtools_3.9.5          gridExtra_2.3.1       shiny_1.14.0         
#> [103] lava_1.9.3            pander_0.6.6          threejs_0.3.4        
#> [106] rstudioapi_0.19.0     cluster_2.1.8.3       QuickJSR_1.11.0      
#> [109] whisker_0.4.1         rstantools_2.7.1      hms_1.1.4            
#> [112] anytime_0.3.13        jomo_2.7-6            xts_0.14.3           
#> [115] shinythemes_1.2.0     ipred_0.9-16          laeken_0.5.3         
#> [118] e1071_1.7-17          urca_1.3-4            TH.data_1.1-5        
#> [121] matrixStats_1.5.0     emmeans_2.0.4         abind_1.4-8          
#> [124] bitops_1.1-0          Rdpack_2.6.6          promises_1.5.0       
#> [127] inline_0.3.21         sandwich_3.1-3        proxy_0.4-29         
#> [130] compiler_4.6.1        prettyunits_1.2.0     distributional_0.9.0 
#> [133] listenv_1.0.0         insight_1.5.4         gridtext_0.1.6       
#> [136] R6_2.6.1              fastmap_1.2.0         multcomp_1.4-32      
#> [139] vecvec_1.3.0          vcd_1.4-14            ggdist_3.3.3         
#> [142] nnet_7.3-21           gtable_0.3.6          shinystan_2.7.0      
#> [145] miniUI_0.1.2          RcppParallel_6.2.1    polspline_1.1.25     
#> [148] lifecycle_1.0.5       S7_0.2.2              Brobdingnag_1.2-9    
#> [151] nloptr_2.2.1          dygraphs_1.1.1.6      vctrs_0.7.3          
#> [154] zeallot_0.2.0         robustbase_0.99-7     sp_2.2-3             
#> [157] pracma_2.4.6          pillar_1.11.1         magick_2.9.1         
#> [160] otel_0.2.0            jsonlite_2.0.0        svgPanZoom_0.3.4     
#> [163] markdown_2.0

Stata

about
version
#> could not find feasible values
#> r(491);
#> 
#> 
#> 
#> StataNow/MP 19.5 for Mac (Apple Silicon)
#> Revision 12 Aug 2026
#> Copyright 1985-2025 StataCorp LLC
#> 
#> Total physical memory: 48.01 GB
#> 
#> Stata license: Single-user 2-core , expiring  6 Feb 2027
#> Serial number: 501909358563
#>   Licensed to: Zad Rafi
#>                
#> 
#> version 19.5
#> 
#> OMP: Warning #96: Cannot form a team with 2 threads, using 1 instead.
#> OMP: Hint Consider unsetting KMP_DEVICE_THREAD_LIMIT (KMP_ALL_THREADS), KMP_TEAMS_THREAD_LIMIT, and OMP_THREAD_LIMIT (if any are set).

See also: What Makes a Sensitivity Analysis? — the hub piece tying this material to the broader cluster on assumptions, robustness, and what happens when models bend.

References

Back to top

References

1. Rafi Z, Greenland S. (2020). “Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise.” BMC Medical Research Methodology. 20:244. doi: 10.1186/s12874-020-01105-9.
2. Greenland S, Rafi Z. (2020). “Technical Issues in the Interpretation of S-values and Their Relation to Other Information Measures.” arXiv:200812991 [statME]. https://arxiv.org/abs/2008.12991.
3. Greenland S. (2019). “Valid P-values behave exactly as they should: Some misleading criticisms of P-values and their resolution with S-values.” The American Statistician. 73:106–114. doi: 10.1080/00031305.2018.1529625.
4. Bayarri MJ, Berger JO. (2000). “P Values for Composite Null Models.” Journal of the American Statistical Association. 95:1127–1142. doi: 10/dpvq8c.
5. Robins JM, van der Vaart A, Ventura V. (2000). “Asymptotic Distribution of P Values in Composite Null Models.” Journal of the American Statistical Association. 95:1143–1156. doi: 10/gg7krv.
6. Kuffner TA, Walker SG. (2019). “Why are P-values controversial?” The American Statistician. 73:1–3. doi: 10.1080/00031305.2016.1277161.
7. Cousins RD. (2017). “The Jeffreys paradox and discovery criteria in high energy physics.” Synthese. 194:395–432. doi: 10.1007/s11229-014-0525-z.
8. Cox DR, Hinkley DV. (1974). “Chapter 7, Interval estimation.” In: Theoretical Statistics. Chapman and Hall/CRC. p. 207–249. doi: 10.1201/b14832.
9. Bowley AL. (1934). “Discussion on Dr. Neyman’s Paper. P. 607 in: Neyman J. On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection (with discussion).” Journal of the Royal Statistical Society. 4:558–625. doi: 10.2307/2342192.
10. Cox DR. (2006). “Principles of Statistical Inference.” Cambridge University Press.
11. Amrhein V, Trafimow D, Greenland S. (2019). “Inferential statistics as descriptive statistics: There is no replication crisis if we don’t expect replication.” The American Statistician. 73:262–270. doi: 10.1080/00031305.2018.1543137.
12. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. (2016). “Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations.” European Journal of Epidemiology. 31:337–350. doi: 10.1007/s10654-016-0149-3.
13. Greenland S. (2019). “Are confidence intervals better termed ‘uncertainty intervals’? No: Call them compatibility intervals.” BMJ. 366. doi: 10.1136/bmj.l5381.
14. Greenland S, Rafi Z. (2020). “To Aid Scientific Inference, Emphasize Unconditional Descriptions of Statistics.” arXiv:190908583 [statME]. https://arxiv.org/abs/1909.08583.
15. Brown HK, Ray JG, Wilton AS, Lunsky Y, Gomes T, Vigod SN. (2017). “Association between serotonergic antidepressant use during pregnancy and autism spectrum disorder in children.” Journal of the American Medical Association. 317:1544–1552. doi: 10.1001/jama.2017.3415.
16. Brown HK, Hussain-Shamsy N, Lunsky Y, Dennis C-LE, Vigod SN. (2017). “The association between antenatal exposure to selective serotonin reuptake inhibitors and autism: A systematic review and meta-analysis.” The Journal of Clinical Psychiatry. 78:e48–e58. doi: 10.4088/JCP.15r10194.
17. Jewell NP. (2003). “Statistics for Epidemiology.” CRC Press.
18. Royall R. (1997). “Statistical Evidence: A Likelihood Paradigm.” CRC Press.
19. Davies PL. (2008). “Approximating data [with discussion and a rejoinder].” Journal of the Korean Statistical Society. 37:191–211. doi: 10/bn8cgk.
]]> https://lesslikely.com/statistics/RG2020BMC.html Sat, 12 Dec 2020 05:00:00 GMT What Makes a Sensitivity Analysis? Zad Rafi Sir Panda https://lesslikely.com/statistics/sensitivity.html

Cargo-Cult Uncertainty

Sensitivity analyses are an important part of statistical science and many other disciplines when conducted in a principle and systematic manner. However, in the published sensitivity analyses literature, there are many inconsistencies, misconceptions, and highly misleading findings from these analyses. A giant obstacle that prevents these issues from recurring is that they (sensitivity analysis techniques) are quite difficult to learn and often advanced statistical methods that even many statisticians have difficulty with.

Yet, the appearance of difficulty does not dissuade certain researchers away from adopting them for their own work, (so that they may give their colleagues and their stakeholders that they report to, the impression of rigor and methodological expertise), and when this is done mindlessly, researchers will often skip over learning the details and theory, and like many statistical procedures, they will rely on default settings built in the statistical software.

While there are many suites/commands/functions/libraries available to conduct such analyses, much of these procedures masquerade as meticulous sensitivity analyses and to the users, are often a formality to appease stakeholders and give researchers a false sense of confidence about what they are doing. And yet, their users have little to no idea what they’re actually doing. Thus, like many statistical procedures new and old, they too will inevitably be abused as they become more popular and as they are required in research reports.

As Stark & Saltelli1, along with many others such as Gigerenzer2 and Greenland3 have written in the past. Below is an excerpt where Stark describes that much of statistics is simply people masquerading as data analysts and being rewarded as experts despite not having a single clue what they’re doing.

In our experience, many applications of statistics are cargo-cult statistics: practitioners go through the motions of fitting models, computing p-values or confidence intervals, or simulating posterior distributions. They invoke statistical terms and procedures as incantations, with scant understanding of the assumptions or relevance of the calculations, or even the meaning of the terminology. This demotes statistics from a way of thinking about evidence and avoiding self-deception to a formal “blessing” of claims. The effectiveness of cargo-cult statistics is predictably uneven. But it is effective at getting weak work published - and is even required by some journals…

Here, Stark takes on the issue of widely accessible statistical software for academics, students, and analysts.

Statistical software does not help you know what to compute, nor how to interpret the result. It does not offer to explain the assumptions behind methods, nor does it flag delicate or dubious assumptions. It does not warn you about multiplicity or p-hacking. It does not check whether you picked the hypothesis or analysis after looking at the data, nor track the number of analyses you tried before arriving at the one you sought to publish - another form of multiplicity. The more “powerful” and “user-friendly” the software is, the more it invites cargo-cult statistics.1

Greenland3 focuses on how traditional statistical education is inadequate to deal with the messy and chaotic nature of the world and the missing data that is produced by it. Greenland writes that traditional statistical practice opens the room for several cognitive biases, making many reports and conclusions highly misleading. He provides some hope however, by pointing to the utility of sensitivity analyses for modeling bias and uncertainty.

I argue that current training in statistics and analytical methods is inadequate for addressing major sources of inference distortion, and that it should be expanded to cover the biased perceptual and thinking processes (cognitive biases) that plague research reports. As commonly misused, null-hypothesis significance testing (NHST) combines several cognitive problems to create highly distorted interpretations of study results. Interval estimation has proven highly vulnerable to the same problems. Sensitivity and bias analyses address model uncertainties by varying and relaxing assumptions, but (like Bayesian analyses) they are difficult to perform with proper accounting for prior information and are easily manipulated because they depend on specification of many models and parameters.

However, he then follows up to discuss how difficult sensitivity and bias analyses can be due to the plethora of decisions that the analyst must make when specifying parameters in the analysis. These judgements will differ from researcher to researcher, even in the same group, and such analyses have not been studied as widely as conventional statistical practice.

Even with realistic choices, the sensitivity of sensitivity and bias analyses must be evaluated (51). The plausibility of an estimated bias function is determined by intuitions, prejudices, and understanding of the applied context; those can vary dramatically across researchers, in turn leading to very different specifications and inferences even if they are anchored to the same conventional analysis. Adding to this problem, sensitivity and bias analyses are more difficult to perform correctly and more easily massaged toward preferred conclusions, because they require specification of many more equations and their parameters.

And unlike NHST, abuse of sensitivity and bias analysis is as yet barely studied because the pool of such analyses remains small and highly selective. It thus seems implausible that these analyses will increase replicability of inferences, although they can reveal how assumptions affect those inferences. (Here “replicability” is used according to recommendations of the American Statistical Association (52) to denote independent checks of reported results with new data; “reproducibility” then denotes checks of reported results using the original data and computer code.)

Indeed, this is one drawback of (relatively) newly adopted methods/procedures, they have not been studied long enough as say somethng such as a P-value, which has been studied for centuries. Even something as useful as exploring assumptions and uncertainty such as a principled sensitivity analysis is not free from this.

it seems Greenland’s predictions were accurate; in a large systematic review of published sensitivity analyses led by the mathematician Andrea Saltelli and his group, they found that highly cited papers in top impact factor journals rarely contained sensitivity analyses, and that many of these were poorly done and one-dimensional, in the sense of only varying one parameter at a time, which is hardly informative as a multi-dimensional analysis in which multiple parameters are varied at the same time.

Here, we will explore one example of how mindless sensitivity analyses are conducted within the context of clinical trials, although we will not explore multi-dimensional sensitivity analyses here. I make this distinction because the environment to conduct sensitivity/bias analyses differs substantially in trials where regulators are involved versus the environment in which many observational studies are analyzed and presented.

In medicine, sensitivity analyses (SA) are often conducted in trials and observational studies, but again, little thought is given to what they should entail.1 None of this is surprising given that they are not usually taught in traditional settings, although they have been discussed extensively within the statistical and epidemiological literature4–10 Yet, the other barrier that remains is that they are incredibly technical and will require the assistance of an analyst familiar with both the theory and the applications. It also does not help that the topic is vast and there are multiple new statistical methods being published every day.

Indeed, the Handbook of Missing Data Methodology,11 one of the most authoritative and comprehensive books to date on statistical methods to handle missing data in clinical trials, has nearly six chapters devoted to the topic of sensitivity analyses and different approaches to use. Unfortunately, at the rate that this field is advancing, this book may already be outdated, despite only coming out six years ago.

This article will focus on what a sensitivity analysis is often assumed to be based on practices in the literature, some differing points of view regarding sensitivity analyses (with an example of a highly promoted measure for conducting sensitivity analyses that has been very controversial, and a practical example of conducting principled(by principled, I mean non ad-hoc techniques, as James Carpenter would say) sensitivity analyses in the context of missing data in a randomized clinical trial.

Everything is Sensitive

Before we move forward, we must consider what a sensitivity analysis actually entails. The first word of the phrase is an obvious tell, it suggests an analysis that examines how sensitive or robust a result is. But a natural question is, sensitive specifically to what? What sort of perturbations and how relevant are they? Can I vary anything in the analysis to test the sensitivity of the result or assumption? For example, if I obtained an odds ratio of 1.7 from the primary analysis in my experiment and I conducted an extra analysis to see whether the odds ratio would change by a resampling method, is that a sensitivity analysis?

What if I did a completely different type of analysis, for example, suppose the primary analysis was a classical statistical test with an adjustment for multiple comparisons and I decided to run a hierarchical Bayesian regression with a spiked prior on the null, would that be a sensitivity analysis, given they are different frameworks and procedures (although being used for similar goals)? The possibilities of what can be varied are endless when a loose definition of sensitivity analysis is assumed. This is unique to every discipline and their culture. In epidemiology, sensitivity analyses and bias analyses which attempt to extensively explore assumptions and the robustness of the result are used synonymously, yet in the clinical trial world, this is not the case.

Frequent Misconceptions

Take for example in clinical trials and observational studies, in particular, it is common to see primary analyses often being intent-to-treat (ITT) analyses and sensitivity analyses being per-protocol analyses (PP).

I will not define these terms here and would encourage readers to consult other sources.

The following is a similar example from a high-profile trial published in the The Lancet in 2002,12

The Multicentre Aneurysm Screening Study group randomised 67,800 men to receive an invitation to an abdominal ultrasound scan or not [6]. Of those invited to receive an abdominal scan, 20% did not accept. The primary analysis was by intention to treat, thus estimating the effect of being randomised to abdominal ultrasound. Another analysis investigated the complier average causal effect, which considers what the (average) effect of treatment was in patients who would have adhered to protocol however they were randomised [7].

To many, this may seem perfectly fine, even great. The authors used the question they were primarily interested in as the main analysis, and then conducted an additional analysis to see if these results are consistent. Unfortunately, this is highly problematic as Morris et al. (2014)13 describes

These questions are different, and observing different results should not shake our confidence in either. The CACE analysis was a secondary analysis, not a sensitivity analysis.

It is common for authors to compare the results of intention-to-treat with per-protocol analysis; see for example [8, 9]. While it is hard to pin down the precise question of per-protocol analysis [10], this is clearly different to the question intention-to-treat addresses. Per-protocol analysis should not therefore be considered as a sensitivity analysis for intention-to-treat but as a secondary analysis, if at all.

Randomness & Uncertainty

Portable Sensitivity Analyses

These sorts of misunderstandings are so common and so prevalent throughout the literature, so it should come as no surprise that sensitivity analyses are rarely done, or they are done incorrectly. Although not in clinical trials, one particular controversial example has been the promotion and use of the -value (the ‘E’ apparently stands for ‘evidence’, but I am not sure.) within epidemiology to assess the amount of confounding necessary within an observational study result to practically reduce the effect estimate to something that is practically null.

The method and value has been adopted with open arms by many epidemiologists and health researchers given that it has simplified a task that is often arduous and requires extensive and careful thought in comparison to the traditional classical analysis. Yet, others have also been highly critical of it for a number of reasons, and although they see the value in promoting sensitivity analyses to more researchers around the world, many are concerned that this measure will also eventually go down the road of P-values, but for observational research.

Regulators & Sensitivity Analyses

As mentioned above, there are many possible ways to conduct a sensitivity analysis, especially if the phrase is used in a loose/vague way. Whether or not these are valid and principled approaches to conducting sensitivity analyses is another question. Luckily, many of us do not have to ponder day and night about the semantics about this because both The Handbook of Missing Data Methodology and the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) have given this topic much thought, and for the latter it is reflected by the fact that they recently created an entire new addition to their E9 guidance document (Statistical Principles for Clinical Trials) which has served as a reference for clinical trial statisticians for decades. In the addendum, titled Addendum on Estimands and Sensitivity Analysis In Clinical Trials, which has now been legally adopted by regulatory agencies around the world including the FDA and EMA, they elaborate on the concept of estimands and the role sensitivity analyses play.

To be clear, estimands are not new and have been discussed in the statistical literature since Tukey14 but the ICH working group’s addendum formalized its adoption in the clinical trial world and the importance of sensitivity analyses.15–17

Estimands & Sensitivity

A.5.2.1. Role of Sensitivity Analysis

Inferences based on a particular estimand should be robust to limitations in the data and deviations from the assumptions used in the statisticral model for the main estimator. This robustness is evaluated through a sensitivity analysis. Sensitivity analysis should be planned for the main estimators of all estimands that will be important for regulatory decision making and labelling in the product information. This can be a topic for discussion and agreement between sponsor and regulator.

The statistical assumptions that underpin the main estimator should be documented. One or more analyses, focused on the same estimand, should then be pre-specified to investigate these assumptions with the objective of verifying whether or not the estimate derived from the main estimator is robust to departures from its assumptions. This might be characterised as the extent of departures from assumptions that change the interpretation of the results in terms of their statistical or clinical significance (e.g. tipping point analysis).

Distinct from sensitivity analysis, where investigations are conducted with the intent of exploring robustness of departures from assumptions, other analyses that are conducted in order to more fully investigate and understand the trial data can be termed “supplementary analysis” (see Glossary; A.5.3.). Where the primary estimand(s) of interest is agreed between sponsor and regulator, the main estimator is pre-specified unambiguously, and the sensitivity analysis verifies that the estimate derived is reliable for interpretation, supplementary analyses should generally be given lower priority in assessment.

A.5.2.2. Choice of Sensitivity Analysis

When planning and conducting a sensitivity analysis, altering multiple aspects of the main analysis simultaneously can make it challenging to identify which assumptions, if any, are responsible for any potential differences seen. It is therefore desirable to adopt a structured approach, specifying the changes in assumptions that underlie the alternative analyses, rather than simply comparing the results of different analyses based on different sets of assumptions. The need for analyses varying multiple assumptions simultaneously should then be considered on a case by case basis. A distinction between testable and untestable assumptions may be useful when assessing the interpretation and relevance of different analyses.

ICH E9 (R1) Guideline

The need for sensitivity analysis in respect of missing data is established and retains its importance in this framework. Missing data should be defined and considered in respect of a particular estimand (see A.4.). The distinction between data that are missing in respect of a specific estimand and data that are not directly relevant to a specific estimand gives rise to separate sets of assumptions to be examined in sensitivity analysis.

they explicitly define a sensitivity analysis as being an analysis which:

  • realistically varies the assumptions from the primary analysis
  • still targets the same estimand
  • examines the robustness of the results to
  • assumption violations or departures
  • can possibly change the results/conclusions drawn

They contrast this with more extensive analyses that investigate these violations/departures of assumptions, and characterize those analyse as being supplementary rather than sensitivity. The latter is typically what is seen in rigorous epidemiological studies that employ quantitative bias analyses (a more specialized form of sensitivity analysis).

The ICH E9 addendum contains similar views that are echoed by the National Research Council’s advice on clinical trials (National Research Council 2010)18 regarding estimands and sensitivity

Recommendation 15: Sensitivity analysis should be part of the primary reporting of findings from clinical trials. Examining sensitivity to the assumptions about the missing data mechanism should be a mandatory component of reporting.

Now, back to the NEJM paper. The reason why an ITT analysis and a PP analysis cannot serve as primary and sensitivity analyses, respectively, is because they are targeting entirely different estimands.13 Thus, because they are answering completely different questions, they’re just two different analyses. And the additional PP analysis, although important to certain stakeholders, can be classified as a supplementary analysis or a non-sensitivity analysis.

Indeed, the following flowchart from Morris et al. (2014)13 is particularly useful as a guide to differentiate sensitivity analyses from non-sensitivity analyses.

Adapted sensitivity-analysis flowchart.

Adapted sensitivity-analysis flowchart.

Adapted flowchart from Morris et al. (2014).

Utility of Subject-Matter Experts

So what does a sensitivity analysis actually look like within the context of a clinical trial or experiment? Below, I walk through the analysis of a trial to give some examples.

Sensitivity in Clinical Trials

I’ll use a sample clinical trial dataset, which is described here:19

“This dataset is typical of diastolic blood pressure data measured in small clinical trials in hypertension from the mid-to-late 1960s and for approximately a decade thereafter.

During this time, hypertension was more severe, the number of effective treatments was relatively small, and the definition (DBP > 95 mmHg) of essential hypertension was not as stringent as it is now (DBP > 80 mmHg) - as seen in the 1967 report from the Veterans Administration Cooperative Study Group on Antihypertensive Agents VA Study Group (1967).

In Table 3.1, diastolic blood pressure (DBP) was measured (mmHg) in the supine position at baseline (i.e., DBP) before randomization… Patients’ age and sex were recorded at baseline and represent potential covariates. The primary objective in this chapter in the analysis of this dataset is to test whether treatment A (new drug) may be effective in lowering DBP as compared to B (placebo) and to describe changes in DBP across the times at which it was measured.”

So to recap, the primary investigators of this trial were interested to see whether treatment A was more effective in lowering DBP when compared to placebo.

Although we will not be using some of the scripts and project setup that I discussed in my statistical workflow post for the sake of efficiency and ease, I would urge and recommend others do so that it remains easy to catch errors. In particular, I would especially recommend using the following packages:

For the analyses in this post, see the R packages in the session info section at the end of this post, these will need to be loaded in order for the scripts below to work.

req_packs <- c("rms", "mice", "magrittr", "tidyr", "future",
    "showtext", "concurve", "ggtext", "tidyverse", "ggplot2",
    "parallel", "kableExtra", "cowplot", "Cairo", "svglite",
    "yardstick", "broom", "mi", "Hmisc", "here", "VIM", "wesanderson",
    "bayesplot", "lattice", "doParallel", "blogdown", "Amelia",
    "bootImpute", "brms", "rmarkdown", "tinytex", "texPreview",
    "ggcorrplot", "qqplotr", "car", "Statamarkdown", "rstan",
    "missForest", "htmltools", "tidybayes", "performance", "mvtnorm",
    "ImputeRobust", "gamlss", "ProfileLikelihood", "reshape2",
    "boot", "knitr", "MASS", "miceMNAR")

# Load all packages at once

lapply(req_packs, library, character.only = TRUE)

I will now simulate some of the data from this clinical trial.

# Simulate Trial Data

set.seed(seed = 1031,
         kind = "L'Ecuyer-CMRG",
         normal.kind = "Inversion",
         sample.kind = "Rejection")

BaselineDBP <- rnorm(500, 117, 2)
Group <- rbinom(500, 1, 0.5)
Age <- rnorm(500, 50, 7)
R1 <- rnorm(500, 0, 1)
R2 <- rnorm(500, 1, 0.5)

errors <- mvtnorm::rmvnorm(500,
  mean = c(0, 0),
  sigma = matrix(c(1, 0.5, 0.5, 1),
    nrow = 2, byrow = TRUE
  )
)

PostDBP <- BaselineDBP + (Age * 0.33) +
  (Group * 9.7) + errors[, 1]

I have made some minor adjustments to this dataset and also generated a new variable (Z) based on the existing ones in the dataset that is strongly correlated with the outcome of interest (PostDBP).

# Explore Data

str(BP_full)
#> 'data.frame':    500 obs. of  7 variables:
#>  $ PostDBP    : num  142 140 129 148 135 ...
#>  $ BaselineDBP: num  119 116 115 119 119 ...
#>  $ Group      : Factor w/ 2 levels "Drug X","Drug Y": 1 1 2 1 2 2 1 2 1 1 ...
#>  $ Age        : num  41.8 44.4 41.1 56.6 50.2 ...
#>  $ Z          : num  0.0141 -0.0629 -0.1101 0.0313 0.0495 ...
#>  $ R1         : num  0.523 1.021 -0.839 0.413 -0.175 ...
#>  $ R2         : num  0.451 0.35 0.897 1.227 1.166 ...
Patient Characteristics
Variable N Overall
N = 5001		 Drug X
N = 2581		 Drug Y
N = 2421
PostDBP 500 138.5 (133.4, 142.8) 142.5 (140.5, 144.6) 133.4 (131.3, 135.3)
BaselineDBP 500 116.88 (115.56, 118.22) 116.90 (115.53, 118.21) 116.81 (115.59, 118.23)
Age 500 49 (45, 55) 49 (45, 55) 50 (46, 55)
Z 500 0.00 (-0.04, 0.05) 0.00 (-0.04, 0.04) 0.00 (-0.05, 0.05)
R1 500 0.00 (-0.73, 0.56) 0.05 (-0.73, 0.60) -0.10 (-0.74, 0.52)
R2 500 0.96 (0.64, 1.28) 0.96 (0.59, 1.30) 0.97 (0.67, 1.28)
1 Median (Q1, Q3) 
summary(BP_full)
#>     PostDBP     BaselineDBP     Group          Age             Z                   R1                  R2        
#>  Min.   :124   Min.   :110   Drug X:258   Min.   :27.7   Min.   :-1.85e-01   Min.   :-3.205036   Min.   :-0.646  
#>  1st Qu.:133   1st Qu.:116   Drug Y:242   1st Qu.:45.3   1st Qu.:-4.44e-02   1st Qu.:-0.732386   1st Qu.: 0.645  
#>  Median :138   Median :117                Median :49.4   Median : 9.45e-05   Median :-0.000058   Median : 0.960  
#>  Mean   :138   Mean   :117                Mean   :49.7   Mean   : 0.00e+00   Mean   :-0.042800   Mean   : 0.962  
#>  3rd Qu.:143   3rd Qu.:118                3rd Qu.:54.7   3rd Qu.: 4.56e-02   3rd Qu.: 0.552206   3rd Qu.: 1.283  
#>  Max.   :151   Max.   :123                Max.   :68.4   Max.   : 2.02e-01   Max.   : 3.377526   Max.   : 2.480
PostDBP BaselineDBP Group Age Z R1 R2
142 119 Drug X 41.8 0.01 0.52 0.45
140 116 Drug X 44.4 -0.06 1.02 0.35
129 115 Drug Y 41.1 -0.11 -0.84 0.90
148 119 Drug X 56.6 0.03 0.41 1.23
135 119 Drug Y 50.2 0.05 -0.17 1.17
135 118 Drug Y 55.3 0.01 -0.89 0.77
143 117 Drug X 54.3 0.00 1.07 0.34
133 116 Drug Y 55.8 -0.05 -1.91 0.96
144 118 Drug X 46.9 0.08 1.03 1.75
150 119 Drug X 62.0 0.01 -0.94 2.48

Now, we can quickly explore the other characteristics of the dataset, and first we look at the rank correlations between the predictors and the response variable.

# Data Attributes

study_formula <- as.formula(PostDBP ~ .)

plot(spearman2(study_formula, BP_full),
  cex = 1,
  pch = 18, col = alpha("darkred", 0.75)
)

abline(v = 0, col = alpha(zred, 0.75), lty = 3)

Along with a correlation matrix between the variables of interest.

We will not be looking at the estimates from a fitted model yet. We are mainly looking at a few characteristics of this dataset before we move on to the next step.

The Threat of Missing Data

Now that we have familiarized ourselves with this dataset, we suddenly “notice” that there are missing data (which I actually generated below using a custom function) and suppose we didn’t actually know how the missing data were generated, we might suspect or start with the assumption that the data are missing at random (MAR).

We will explore a number of different methods used to handle missing data, including multiple imputation, selection models, and a number of ad-hoc methods commonly used by researchers handle missing data. However, before we move on to discussing some of these methods, we will quickly review concepts and terms regarding missing data mechanisms.

In the figure below, the three missing data mechanisms (MCAR, MAR, and MNAR) and ignorability (whether we need to model the mechanism of missing data) in relation to observed data (Y obs), missing data (Y mis), the missingness matrix (R), and their relationships (q; parameters that explain missingness, i.e., mechanism). The solid arrows, dotted arrows, and arrows with crosses represent “connection,” “possible connection,” and “no connection,” respectively.

The lines connecting ignorability and missingness group the three mechanisms into the two ignorability categories. Also no pure forms of MCAR, MAR, and MNAR exist, and all missingness can be considered as a form of MAR missingness; this is represented by the shaded continuum bar on the left. - Modified from Nakagawa and Freckleton (2011)

Missing-data mechanism flowchart.

Missing-data mechanism flowchart.

Flowchart from Ecological Statistics: Contemporary Theory and Applications.

Of course, here, we do have an idea of what the missing data mechanism is, but we shall assume the role of a data analyst who has just been given a dataset with missing values in the outcome from their clinical colleague.

# Generate Missing Data

set.seed(1031,
  kind = "L'Ecuyer-CMRG",
  normal.kind = "Inversion",
  sample.kind = "Rejection"
)
Ry <- ifelse(lesslikely() > 0, 1, 0)
PostDBP[Ry == 0] <- NA
BP_miss <- data.frame(
  PostDBP, BaselineDBP,
  Group, Age, Z, R1, R2
)
str(BP_miss)
#> 'data.frame':    500 obs. of  7 variables:
#>  $ PostDBP    : num  142 140 129 148 135 ...
#>  $ BaselineDBP: num  119 116 115 119 119 ...
#>  $ Group      : Factor w/ 2 levels "Drug X","Drug Y": 1 1 2 1 2 2 1 2 1 1 ...
#>  $ Age        : num  41.8 44.4 41.1 56.6 50.2 ...
#>  $ Z          : num  0.0141 -0.0629 -0.1101 0.0313 0.0495 ...
#>  $ R1         : num  0.523 1.021 -0.839 0.413 -0.175 ...
#>  $ R2         : num  0.451 0.35 0.897 1.227 1.166 ...

sum(is.na(BP_miss))
#> [1] 245

# Export data for Stata chunks: Group must be numeric and NAs must be blank,
# otherwise `import delimited` reads those columns as strings. Code Drug Y as 1
# so Stata's `i.group` matches R's factor contrast (base = Drug X).
dir.create("static/datasets", recursive = TRUE, showWarnings = FALSE)
BP_stata <- BP_miss
BP_stata$Group <- as.integer(BP_stata$Group == "Drug Y")
write.csv(BP_stata, "static/datasets/temp.csv", row.names = FALSE, na = "")

We will now attempt to visualize our dataset and examine what proportion of these data are missing.

missing_col <- c(zblue, zred)
aggr(BP_miss,
  col = alpha(missing_col, 0.60),
  plot = TRUE, prop = F, numbers = T
)

So now, we must examine this missing dataset and its characteristics and look for any systematic differences. We also examine the proportion of missingness and the influx and outflux patterns. This is important to assess how well connectected the missing data are to the observed data, explained by van Buuren here

The influx of a variable quantifies how well its missing data connect to the observed data on other variables. The outflux of a variable quantifies how well its observed data connect to the missing data on other variables. In general, higher influx and outflux values are preferred…

Influx and outflux are summaries of the missing data pattern intended to aid in the construction of imputation models. Keeping everything else constant, variables with high influx and outflux are preferred. Realize that outflux indicates the potential (and not actual) contribution to impute other variables. A variable with high may turn out to be useless for imputation if it is unrelated to the incomplete variables. On the other hand, the usefulness of a highly predictive variable is severely limited by a low .

More refined measures of usefulness are conceivable, e.g., multiplying by the average proportion of explained variance. Also, we could specialize to one or a few key variables to impute. Alternatively, analogous measures for could be useful. The further development of diagnostic summaries for the missing data pattern is a promising area for further investigation.

# Examine Flux Patterns

round(fico(BP_miss), 1)[2:6]
#> BaselineDBP       Group         Age           Z          R1 
#>         0.5         0.5         0.5         0.5         0.5
POBS Influx Outflux AINB AOUT FICO
PostDBP 0.51 0.45 0 1 0.00 0.00
BaselineDBP 1.00 0.00 1 0 0.08 0.49
Group 1.00 0.00 1 0 0.08 0.49
Age 1.00 0.00 1 0 0.08 0.49
Z 1.00 0.00 1 0 0.08 0.49
R1 1.00 0.00 1 0 0.08 0.49
R2 1.00 0.00 1 0 0.08 0.49

Next, We start off doing some preliminary analyses to eyeball differences between the observed data and the missing data.

#> [1] 245
pobs influx outflux ainb aout fico
PostDBP 0.51 0.45 0 1 0.00 0.00
BaselineDBP 1.00 0.00 1 0 0.08 0.49
Group 1.00 0.00 1 0 0.08 0.49
Age 1.00 0.00 1 0 0.08 0.49
Z 1.00 0.00 1 0 0.08 0.49
R1 1.00 0.00 1 0 0.08 0.49
R2 1.00 0.00 1 0 0.08 0.49

Having explored these data, we fit a preliminary model quickly to these missing data using the rlm() function from the MASS package and glance at what our estimates look like.

# Exploratory Model Estimates

eda_1 <- rlm(
  PostDBP ~ factor(Group) + BaselineDBP +
    Age + Z + R1 + R2,
  data = BP_miss,
  method = "MM", psi = psi.huber
)
Value Std. Error t value
(Intercept) 11.68 5.77 2.02
factor(Group)Drug Y -9.58 0.13 -71.57
BaselineDBP 0.98 0.05 19.86
Age 0.34 0.01 34.33
Z -0.93 1.48 -0.62
R1 -0.03 0.07 -0.43
R2 -0.13 0.13 -0.94
AIC AICc BIC RMSE Sigma
749 750 778 1.02 1.03

Because the MASS::rlm() function will automatically delete missing observations and do a complete-case analysis, we can gauge what our estimates would look like if we assume that the data are missing completely at random (MCAR), which is almost always unrealistic.

It is almost always more realistic to assume that the data are missing at random (MAR), . This can often be a reasonable assumption to start off with for the primary analysis.20

Multiple Imputation Workflow

For our main analysis, we will build an imputation model to essentially ‘predict’ the missing values in our dataset. This is because multiple imputation is a well-studied set of algorithms that are efficient and robust to deviances from normality.

Modern missing data methodologies include maximum-likelihood estimation (MLE) methods such as expectation–maximisation (EM) and multiple imputation (MI), both recommended for data which is MAR [3]. MI has been shown to be robust under departures from normality, in cases of low sample size, and when the proportion of missing data is high [2]. With complete outcome variables, MI is typically less computationally expensive than MLE, and MLE tends to be problem-specific with a different model being required for each analysis [8].

  • Plumpton CO, Morris T, Hughes DA, White IR. Multiple imputation of multiple multi-item scales when a full imputation model is infeasible. BMC Res Notes. 2016 Jan 26;9:45. doi: 10.1186/s13104-016-1853-5.

How exactly does multiple imputation work?

The standard multiple imputation procedure (Rubin, 1987; Schafer, 1997) replaces missing covariate data with values drawn from a set of specified imputation models based on the observed relationships between the covariates and outcome, typically under a missing at random (MAR) assumption (Little and Rubin, 1987). A key feature of the procedure is that it outputs a number, say M, of imputed datasets. A prediction model is then fitted to each imputed dataset to produce imputation-specific regression coefficients, which can be averaged using “Rubin’s rules” (Rubin, 1987) to provide pooled regression coefficients.

To extract predictions from such a prediction model and multiply imputed data, either the sets of imputation-specific regression coefficients or the pooled regression coefficients could be used, and we must decide how to handle individuals with multiply imputed covariates. Thus there are numerous combinations for constructing a set of predictions, including imputation-specific predictions (M predictions for each individual) and pooled predictions (an averaged prediction over imputed datasets for each individual). The differences between them and their advantages and drawbacks are unclear.

Wood, A.M., Royston, P. and White, I.R. (2015), The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data. Biom. J., 57: 614-632. https://doi.org/10.1002/bimj.201400004

We include all possible information in our imputation model, for example, that might mean entail specifying all the variables in our dataset as covariates, and impute the missing data using a set of reliable and well-studied imputation algorithms, and then analyze each dataset and pool the results. Our basic workflow will look like this:

Missingness workflow flowchart.

Missingness workflow flowchart.

Flowchart from Ecological Statistics: Contemporary Theory and Applications.

Concepts in Multiple Imputation

It can also be written using the following notation, which will be explained below.

Nagakawa writes:

There are three main steps in MI: imputation, analysis, and pooling (figure 4.5C). In the imputation step, you create copies of completed data set by using data imputation methods such as the EM/EMB algorithms or the MCMC procedure. In the analysis step, you run separate statistical analyses on each of data sets. Finally, in the pooling step, you aggregate sets of results to produce unbiased parameter and uncertainty estimates.

This aggregation process is done by the following equations (which are automatically calculated in R): where is the mean of (e.g., regression coefficients), which is a parameter estimated from the ith data set (), is the within-imputation variance calculated from the standard error associated with , is the between-imputation variance estimates, and is the total variance ( is the overall standard error for ).

This set of equations for combining estimates from sets of results is often referred to as Rubin’s rules, as it was developed by Rubin (1987). Statistical significance and confidence intervals (CIs) of pooled parameters are obtained by, where is the number of degrees of freedom used for t-tests or to obtain values and CI calculations, and is the significance level (e.g., 95% CI, = 0.05).:

Nagakawa then discusses the similarities between imputation algorithms and MCMC methods,

MCMC procedures (and also Gibbs sampling) are often called Bayesian methods (chapter 1) because their goal is to create the posterior distributions of parameters, but methods using MCMC have much wider applications than Bayesian statistics). The MCMC procedure, is akin to the EM algorithm (Schafer 1997) in that it uses a two-step iterative algorithm to find and .

The imputation step (I-step) uses stochastic regression with observed data. Next, the posterior step (P-step) uses this filled-in data set to construct the posterior distributions of and . Then, it uses a Monte Carlo method to sample a new set of and from these distributions. These new parameter estimates are used for the subsequent I-step. Iterations of the two steps create the Markov chain, which even actually converges into fully fledged posterior distributions of and (figure 4.6B). These distributions are, in turn, used for multiple imputation.

Now that we’ve quickly reviewed some of the concepts of multipe imputation and missing data, we can now begin to build our imputation model. This is generally done by building a model with all relevant covariates, however, this should be compared a reduced model with just the treatment group. This is because we want both the adjusted and unadjusted estimates. We will then compare these models using the likelihood-ratio test21 to see whether there are any substantial differences between the fully-adjusted and reduced model.

The imputation model for a particular parameter will generally resemble the estimation model being used.

When developing your imputation model, it is important to assess if your imputation model is “congenial” or consistent with your analytic model. Consistency means that your imputation model includes (at the very least) the same variables that are in your analytic or estimation model. This includes any transformations to variables that will be needed to assess your hypothesis of interest. This can include log transformations, interaction terms, or recodes of a continuous variable into a categorical form, if that is how it will be used in later analysis. The reason for this relates back to the earlier comments about the purpose of multiple imputation.

Since we are trying to reproduce the proper variance/covariance matrix for estimation, all relationships between our analytic variables should be represented and estimated simultaneously. Otherwise, you are imputing values assuming they have a correlation of zero with the variables you did not include in your imputation model. This would result in underestimating the association between parameters of interest in your analysis and a loss of power to detect properties of your data that may be of interest such as non-linearities and statistical interactions. - From the UCLA Statistical Consulting Website

Monte Carlo Error Analysis

However, before we begin to build the model, we must also have an idea of how many imputations we need to compute. To many, this may seem like a trivial and even nonsensical task when any individual can impute a large number of datasets. Indeed, it may seem that more imputations would give a more reliable and precise estimate, however, nothing in life is free, and running a imputation algorithm that is computationally demanding (and which can potentially take hours or days) is not feasible for everyone, hence the reason for “sample-size” calculations for imputation algorithms.

Multiple imputation is a stochastic procedure. Each time we reimpute our data, we get different sets of imputations because of the randomness of the imputation step, and therefore we get different multiple-imputation estimates. However, we want to be able to reproduce MI results. Of course, we can always set the random-number seed to ensure reproducibility by obtaining the same imputed values. However, what if we use a different seed? Would we not want our results to be similar regardless of what seed we use? This leads us to a notion we call statistical reproducibility—we want results to be similar across repeated uses of the same imputation procedure; that is, we want to minimize the simulation error associated with our results.

To assess the level of simulation error, White, Royston, and Wood (2011) propose to use a Monte Carlo error of the MI results, defined as the standard deviation of the results across repeated runs of the same imputation procedure using the same data. The authors suggest evaluating Monte Carlo error estimates not only for parameter estimates but also for other statistics, including p-values and confidence intervals, as well as MI statistics including RVI and FMI.

The issue with this is that without giving careful thought to how many imputations may be needed for a particular study, one may run the risk of imputing too few datasets and losing information, or they may waste resources imputing a nonsensical amount that may not be necessary. Indeed, the latter seems especially trivial, however, it may often yield no advantages whatsoever, and only incur costs as many authors have argued.

The argument is that “the additional resources that would be required to create and store more than a few imputations would not be well spent” (Schafer 1997, 107), and “in most situations there is simply little advantage to producing and analyzing more than a few imputed datasets” (Schafer and Olsen 1998, 549).

Therefore, we will examine certain characteristics of our dataset with missing values and use them in an analysis to determine how many datasets we should impute to efficiently achieve our goals. We will base this off a number of characteristics such as the fraction of missing information, the proportion of missing observations, and the losses that we are willing to incur.

We first start by running a “dry” or naive imputation of our data to quickly examine some of these characteristics. This imputed dataset will not be used for our primary analysis, it is simply serving to guide us in our main imputation model. This is also why we have set the number of iterations to 0.

Once again, we will include all the variables in our dataset, and use the predictive mean matching method with approximately 5 donors.

# Dry Imputation

form <- list(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2)
pred <- make.predictorMatrix(BP_miss)

init_imp <- mice::mice(
  BP_miss,
  method = "pmm",
  predictorMatrix = pred,
  m = 20,
  maxit = 0,
  seed = 1031,
  formulas = form,
  donors = 5,
  remove.collinear = TRUE,
  remove.constant = TRUE,
  printFlag = FALSE,
  visitSequence = "monotone"
)

head(init_imp$loggedEvents, 10)
#> NULL

Now that we have imputed our dataset, we may examine it for any issues before moving onto the next step of analyzing it.

Although it is only a dry run, there seems to be no issues with it. We now pool our estimates using Rubin’s rules.

init_res <- mice::pool(with(
  init_imp,
  rlm(
    PostDBP ~ factor(Group) +
      BaselineDBP + Age +
      Z + R1 + R2,
    method = "MM", psi = psi.huber
  )
))

init_res_sum <- summary(init_res)

colnames(init_res_sum) <- c(
  "Term", "Estimate", "SE",
  "Statistic", "df", "P-val"
)
ztable(init_res_sum)
Term Estimate SE Statistic df P-val
(Intercept) 41.77 17.18 2.43 NA NA
factor(Group)Drug Y -8.22 0.43 -19.26 NA NA
BaselineDBP 0.74 0.15 5.00 NA NA
Age 0.28 0.03 10.23 NA NA
Z 1.57 4.20 0.37 NA NA
R1 0.03 0.23 0.14 NA NA
R2 -0.08 0.36 -0.21 NA NA

Now that we have our vector, we can quickly examine certain characteristics such as the fraction of missing information (FIMO), the between-imputation variance and the within-imputation variance. The reason that we focus on these particular attributes is because they are crucial for several tasks in any statistical analysis plan, such as having enough power/precision, minimum long-run coverage, etc.

We use the FIMO from the dry imputation and a number of other criteria to determine how many imputations we need:

  • Some have suggested taking the FIMO and multiplying by 100 to obtain the number of imputed datasets needed

  • For our goals, we wish to ensure that the monte-carlo standard error from the imputations are less than 10% of the between-imputation standard error

  • We wish to minimize the monte-carlo error that is derived from dividing the chosen number of datasets to impute from the fraction of missing information so that the monte-carlo standard error is less than 0.01

  • We also wish to minimize the total variance so that the square-root of it is no greater than the ideal variance and the corresponding confidence interval width.

  • Furthermore, it is commonly advocated by missing data experts to impute the same or similar number of datasets as fractions of missing observations

We start by extracting the FMIs from our dry imputation and calculating all these numbers.

# Calculate Fraction of Missing Information
# `mice::pool()`/`summary.mipo()` have changed their returned columns across
# versions. Prefer the explicit `fmi` column when present; otherwise fall back
# to `lambda`, and finally compute it from Rubin's variance components.
extract_fmi <- function(model) {
  if (is(model, "mira")) {
    model <- mice::pool(model)
  }
  if (!is(model, "mipo")) {
    stop("Model must be multiply imputed.")
  }

  pooled <- model$pooled
  finite_max <- function(x) {
    x <- suppressWarnings(as.numeric(x))
    x <- x[is.finite(x)]
    if (length(x)) max(x) else NA_real_
  }

  fmi_value <- if ("fmi" %in% names(pooled)) finite_max(pooled$fmi) else NA_real_
  if (!is.na(fmi_value)) return(min(max(fmi_value, 0), 0.999))

  lambda_value <- if ("lambda" %in% names(pooled)) finite_max(pooled$lambda) else NA_real_
  if (!is.na(lambda_value)) return(min(max(lambda_value, 0), 0.999))

  if (all(c("b", "ubar") %in% names(pooled))) {
    m_count <- if (!is.null(model$m)) model$m else finite_max(pooled$m)
    riv <- (1 + 1 / m_count) * pooled$b / pooled$ubar
    lambda <- riv / (1 + riv)
    if ("df" %in% names(pooled)) {
      fmi_calc <- lambda + (2 / (pooled$df + 3))
    } else {
      fmi_calc <- lambda
    }
    fmi_value <- finite_max(fmi_calc)
    if (!is.na(fmi_value)) return(min(max(fmi_value, 0), 0.999))
  }

  stop("Could not extract or compute fraction of missing information from pooled model.")
}

fmi <- extract_fmi(init_res)

m <- fmi / 0.01 # FMI Imputations

mcse <- sqrt(max(init_res$pooled$b, na.rm = TRUE) / m) # MCSE

se <- max((init_res$pooled$ubar) / sqrt(500), na.rm = TRUE) # SE

What we find is that the fraction of missing information is nearly 50% which is very high and that in order to get a monte-carlo error of less than 0.01, we would need a minimum of at least 50 imputed datasets. We also calculate both the monte-carlo standard error and the standard error from the dry run and find that they are practically equivalent.

However, we must also explore other possibilities in terms of number of imputed datasets to reduce the monte-carlo errors so we run a quick function to do that, checking the effects of a number of imputed datasets on the width of the confidence interval, which is directly tied to the monte-carlo error.

Indeed, Von Hippel (2018) proposed a relationship between the fraction of missing information and the number of imputations needed to achieve a targeted CI length or monte-carlo error using a quadratic formula,

where and are the coefficient of variation (CV), summarizing the imputation variation in the SE estimates. We can graphically construct a function to depict this relationship and calculate how many imputations we will need based on some desired error and further display the distribution of this relationship by varying these parameters.

In the function below, adopted from von Hippel, and van Buuren,20 I vary both the coefficient of variation and the -level.

#' @title How Many Imputations?
#' @description Implements two-stage "how_many_imputations" from von Hippel (2018)
#' @param model Either a `mira` object (created by running a model on a data set which was imputed via "mice")
#' or a `mipo` object (creating by runing `pool()` on a `mira` object).
#' @param cv Desired precision of standard errors. Default to .05. (I.e., if the data were re-imputed, the
#' estimated standard errors would differ by no more than this amount.)
#' @param alpha Significance level for choice of "conservative" FMI.
#' @return The number of required imputations to obtain the `cv` level of precision.
#' @export
#' @importFrom methods is
#' @importFrom stats plogis qlogis qnorm
#' @importFrom mice pool
#' @references von Hippel, Paul T. (2018)
#' \sQuote{How Many Imputations Do You Need? A Two-stage Calculation Using a Quadratic Rule.},
#' \emph{Sociological Methods & Research} p.0049124117747303.
#'
powerimp <- function(model, cv = .05, alpha = .05) {
  if (is(model, "mira")) {
    model <- mice::pool(model)
  }
  if (!is(model, "mipo")) {
    stop("Model must be multiply imputed.")
  }
  fmi <- extract_fmi(model)
  z <- qnorm(1 - alpha / 2)
  fmiu <- plogis(qlogis(fmi) + z * sqrt(2 / model$m))
  ceiling(1 + 1 / 2 * (fmiu / cv)^2)
}

ci_width <- function() {
  cv_i <- seq(0.01, 0.20, 0.01)
  results1 <- lapply(
    (seq_along(cv_i) / 100),
    function(i) {
      powerimp(
        model = init_res,
        cv = i, alpha = 0.01
      )
    }
  )
  results2 <- lapply(
    (seq_along(cv_i) / 100),
    function(i) {
      powerimp(
        model = init_res,
        cv = i, alpha = 0.05
      )
    }
  )
  results3 <- lapply(
    (seq_along(cv_i) / 100),
    function(i) {
      powerimp(
        model = init_res,
        cv = i, alpha = 0.1
      )
    }
  )
  a_0.01 <- do.call(rbind, results1)
  df <- data.frame(cv_i, a_0.01)
  df$a_0.05 <- do.call(rbind, results2)
  df$a_0.1 <- do.call(rbind, results3)
  return(df)
}

some <- ci_width()

With this function, we can now graphically examine the effects of different parameter values on the monte-carlo errors from the imputations.

We can see above that as we aim for smaller monte-carlo errors, we need many more imputations to achieve our desired target.

Now that I have constructed and graphed three differing relationships between monte-carlo errors and number of imputations needed to achieve those errors, and varied the -levels, we can now move onto choosing a specific number of imputations to construct.

If I choose an level of 5%, which the maximum tolerable type-I error rate, and a coefficient of variation of 5%

# Estimate Number of Imputations

powerimp(model = init_res, cv = 0.10, alpha = 0.05)
#> [1] 24

powerimp(model = init_res, cv = 0.05, alpha = 0.05)
#> [1] 92

powerimp(model = init_res, cv = 0.01, alpha = 0.05)
#> [1] 2257

I need to construct approximately 124 imputations to achieve this coefficient of variation, which is what I will use for all the analyses from here on, and it is clear that it is far more reasonable than trying to impute 3000 datasets to achieve a tiny standard error.

However, I want to verify my calculations by running a similar estimation command that Stata provides. The documentation says the following regarding RVI and FMI:

Returning to the output, average RVI reports the average relative increase (averaged over all coefficients) in variance of the estimates because of the missing bmi values. A relative variance increase is an increase in the variance of the estimate because of the loss of information about the parameter due to nonresponse relative to the variance of the estimate with no information lost. The closer this number is to zero, the less effect missing data have on the variance of the estimate. Note that the reported RVI will be zero if you use mi estimate with the complete data or with missing data that have not been imputed. In our case, average RVI is small: 0.0312.

Largest FMI reports the largest of all the FMI about coefficient estimates due to nonresponse. This number can be used to get an idea of whether the specified number of imputations is sufficient for the analysis. A rule of thumb is that M ≥ 100 × FMI provides an adequate level of reproducibility of MI analysis. In our example, the largest FMI is 0.14 and the number of imputations, 20, exceeds the required number of imputations: 14

Although I cannot get the number of imputations directly from a command, I can run the mi estimate command, and look at the overall fraction of missing information as the number of imputations increases. If my previous calculations are consistent, I should look to increase the number of imputations until I achieve an FMI of 1.24 or less: 124/100

set linesize 255

clear all

set seed 1031

quietly import delimited "static/datasets/temp.csv", numericcols(2 3 5 6 7 8)

quietly mi set mlong

quietly mi register imputed baselinedbp postdbp age z group r1 r2

quietly mi impute chained (pmm, knn(5)) postdbp = baselinedbp age z r1 r2 group, burnin(5) add(120) rseed (1031) nomonotone

quietly mi estimate, saving(miest, replace): rreg postdbp i.group baselinedbp age z r1 r2

mi estimate using miest, mcerror level(95)

mi estimate, saving(miest, replace) vartable mcerror level(95) nocitable: rreg postdbp i.group baselinedbp age z r1 r2

mi estimate using miest, mcerror level(95)
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Multiple-imputation estimates                   Imputations       =        120
#> Robust regression                               Number of obs     =        500
#>                                                 Average RVI       =     0.7202
#>                                                 Largest FMI       =     0.4949
#>                                                 Complete DF       =        493
#> DF adjustment:   Small sample                   DF:     min       =     165.96
#>                                                         avg       =     202.37
#>                                                         max       =     274.66
#> Model F test:       Equal FMI                   F(   6,  431.4)   =    1183.09
#>                                                 Prob > F          =     0.0000
#> 
#> ------------------------------------------------------------------------------
#>      postdbp | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>      1.group |  -9.523045   .1168215   -81.52   0.000    -9.753025   -9.293066
#>              |   .0057905   .0025234     1.76   0.000     .0084985    .0067603
#>              |
#>  baselinedbp |    .980327   .0453156    21.63   0.000       .89096    1.069694
#>              |   .0026919   .0011629     0.57   0.000     .0038643    .0032359
#>              |
#>          age |   .3297468   .0088947    37.07   0.000      .312222    .3472716
#>              |   .0004879   .0002219     0.92   0.000     .0006042    .0007118
#>              |
#>            z |  -1.062159   1.509542    -0.70   0.483    -4.041587    1.917269
#>              |   .0945004   .0410673     0.07   0.041     .1227202    .1282877
#>              |
#>           r1 |   -.018845   .0725934    -0.26   0.795    -.1621706    .1244805
#>              |   .0046239   .0026182     0.06   0.049     .0075822    .0063906
#>              |
#>           r2 |  -.1529839   .1325282    -1.15   0.250    -.4145243    .1085565
#>              |   .0082342   .0037686     0.07   0.027     .0120192    .0102984
#>              |
#>        _cons |    11.6797   5.307458     2.20   0.029     1.213107     22.1463
#>              |   .3147167   .1363694     0.07   0.005     .3681237    .4611054
#> ------------------------------------------------------------------------------
#> Note: Values displayed beneath estimates are Monte Carlo error estimates.
#> 
#> 
#> Multiple-imputation estimates                   Imputations       =        120
#> Robust regression
#> 
#> Variance information
#> ------------------------------------------------------------------------------
#>              |        Imputation variance                             Relative
#>              |    Within   Between     Total       RVI       FMI    efficiency
#> -------------+----------------------------------------------------------------
#>      1.group |    .00959   .004024   .013647   .423052   .299522        .99751
#>              |   .000042   .000576   .000588   .060302   .030372       .000254
#>              |                                                  
#>  baselinedbp |   .001177    .00087   .002054   .745092   .430409       .996426
#>              |   5.1e-06   .000104   .000105   .088822   .029698       .000248
#>              |                                                  
#>          age |    .00005   .000029   .000079   .572671   .367007       .996951
#>              |   2.2e-07   3.9e-06   3.9e-06   .077358   .031978       .000267
#>              |                                                  
#>            z |   1.19815   1.07164   2.27872   .901867   .478054       .996032
#>              |   .005198   .122213   .123705   .102669   .028885       .000241
#>              |                                                  
#>           r1 |   .002683   .002566    .00527     .9643   .494902       .995893
#>              |   .000012   .000373   .000379   .139641   .037113       .000309
#>              |                                                  
#>           r2 |    .00936   .008136   .017564   .876541   .470898       .996091
#>              |   .000041    .00098   .000996    .10503   .030376       .000253
#>              |                                                  
#>        _cons |   16.1845   11.8856   28.1691   .740502   .428884       .996439
#>              |   .070211   1.42688    1.4443   .088798   .029842       .000249
#> ------------------------------------------------------------------------------
#> Note: Values displayed beneath estimates are Monte Carlo error estimates.
#> 
#> 
#> 
#> Multiple-imputation estimates                   Imputations       =        120
#> Robust regression                               Number of obs     =        500
#>                                                 Average RVI       =     0.7202
#>                                                 Largest FMI       =     0.4949
#>                                                 Complete DF       =        493
#> DF adjustment:   Small sample                   DF:     min       =     165.96
#>                                                         avg       =     202.37
#>                                                         max       =     274.66
#> Model F test:       Equal FMI                   F(   6,  431.4)   =    1183.09
#>                                                 Prob > F          =     0.0000
#> 
#> ------------------------------------------------------------------------------
#>      postdbp | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>      1.group |  -9.523045   .1168215   -81.52   0.000    -9.753025   -9.293066
#>              |   .0057905   .0025234     1.76   0.000     .0084985    .0067603
#>              |
#>  baselinedbp |    .980327   .0453156    21.63   0.000       .89096    1.069694
#>              |   .0026919   .0011629     0.57   0.000     .0038643    .0032359
#>              |
#>          age |   .3297468   .0088947    37.07   0.000      .312222    .3472716
#>              |   .0004879   .0002219     0.92   0.000     .0006042    .0007118
#>              |
#>            z |  -1.062159   1.509542    -0.70   0.483    -4.041587    1.917269
#>              |   .0945004   .0410673     0.07   0.041     .1227202    .1282877
#>              |
#>           r1 |   -.018845   .0725934    -0.26   0.795    -.1621706    .1244805
#>              |   .0046239   .0026182     0.06   0.049     .0075822    .0063906
#>              |
#>           r2 |  -.1529839   .1325282    -1.15   0.250    -.4145243    .1085565
#>              |   .0082342   .0037686     0.07   0.027     .0120192    .0102984
#>              |
#>        _cons |    11.6797   5.307458     2.20   0.029     1.213107     22.1463
#>              |   .3147167   .1363694     0.07   0.005     .3681237    .4611054
#> ------------------------------------------------------------------------------
#> Note: Values displayed beneath estimates are Monte Carlo error estimates.

The Primary Analysis

For the primary analysis, I will use the predictive-mean matching method and specify 5 donors (based on a number of simulation studies that have examined the effect of various donors specified)22, and fit a robust regression that includes all the variables within the dataset. Our imputation model will be

So we will now impute our datasets.

# Impute Missing Data

form <- list(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2)
pred <- make.predictorMatrix(BP_miss)

imp1 <- mice::mice(
  BP_miss,
  method = "pmm",
  predictorMatrix = pred,
  m = 120,
  maxit = 5,
  seed = 1031,
  formulas = form,
  ridge = 0,
  donors = 5,
  remove.collinear = TRUE,
  remove.constant = TRUE, allow.na = TRUE,
  printFlag = FALSE, visitSequence = "monotone"
)

head(imp1$loggedEvents, 10)
#> NULL

Now that we have imputed our datasets, we may examine them for any issues before moving onto the next step of analyzing it.

Our imputed datasets seem to look fine so far. We now fit our models to each of these datasets and combine them using Rubin’s rules.

# Primary Analysis

analysis1 <- with(imp1, rlm(
  PostDBP ~ (Group) +
    BaselineDBP + Age + Z + R1 + R2,
  method = "MM", psi = psi.huber
))
analysis2 <- with(imp1, rlm(PostDBP ~ (Group),
  method = "MM", psi = psi.huber
))

anova(analysis1, analysis2,
  method = "D3", use = "LR"
)
#>    test statistic df1 df2 dfcom p.value riv
#>  1 ~~ 2         0   5 600    NA       1 217
result1 <- mice::pool(analysis1)

results1_sum <- summary(result1)

colnames(results1_sum) <- c(
  "Term", "Estimate", "SE",
  "Statistic", "df", "P-val"
)
Term Estimate SE Statistic df P-val
(Intercept) 11.25 5.29 2.13 NA NA
GroupDrug Y -9.51 0.13 -75.89 NA NA
BaselineDBP 0.98 0.05 21.82 NA NA
Age 0.33 0.01 34.81 NA NA
Z -1.33 1.41 -0.94 NA NA
R1 -0.02 0.07 -0.32 NA NA
R2 -0.14 0.13 -1.09 NA NA

While we have our primary analysis estimates, we also continue to examine our imputations for any anomalies.

We’ve done our primary analysis, in which we imputed the missing data under the assumption of MAR, which is often a reasonable assumption to start with. Unfortunately, we cannot verify whether or not this assumption is true, but we can vary this missing data assumption and assume that the data are missing not at random (MNAR) , and that in the individuals with missing data, we can assume different numbers than those with complete data. Our likelihood-ratio test also suggests that there’s very little difference between the full model and the reduced model, so we generally will go with the full model.

SA I: Controlled Imputations

Now suppose we wish to handle these missing data under the assumption of MNAR, we would now be conducting a sensitivity analysis because we are still targeting the same estimand, but only varying the assumptions. The question / target remain the same. To handle missing data that we assume are MNAR, there are a number of different and complex approaches.

We start off with the -adjustment, controlled multiple imputation method,23 in which we assume that the group with missing observations differs systematically from the group with complete observations by a certain quantity. We produce a range of these quantities and add or subtract them from the imputed values and then conduct our analysis on this adjusted, imputed dataset. Here I will make the assumption that the average PostDBP estimate from the missing value group differs from at least 0 mmHg to at most 20 mmHg from the group with no missing observations.

# Controlled MI

ini <- mice(BP_miss,
  method = "pmm", maxit = 0,
  predictorMatrix = pred
)

ini$method["BaselineDBP"] <- ""
ini$method["PostDBP"] <- "pmm"
ini$method["Age"] <- ""
ini$method["R1"] <- ""
ini$method["Group"] <- ""
ini$method["R2"] <- ""
ini$method["Z"] <- ""
meth <- ini$method

delta <- seq(-20, 20, 5)

imp.all <- vector("list", length(delta))
post <- ini$post
for (i in 1:length(delta)) {
  d <- delta[i]
  cmd <- paste("imp[[j]][, i] <- imp[[j]][, i] +", d)
  post["PostDBP"] <- cmd
  imp <- mice::mice(
    BP_miss,
    method = meth,
    predictorMatrix = pred,
    m = 5,
    maxit = 1,
    donors = 5,
    ridge = 0,
    formulas = form,
    remove.collinear = TRUE,
    remove.constant = TRUE,
    allow.na = TRUE, post = post,
    printFlag = FALSE
  )
  imp.all[[i]] <- imp
}
for (i in 1:length(delta)) {
  imp.all[[i]][["data"]][["Group"]] <- as.factor(imp.all[[i]][["data"]][["Group"]])
}

Now that we have our imputed datasets that are modified by -adjustments, we can fit our models to each imputed dataset and combine the estimates using Rubin’s rules.

# Sensitivity Analysis 1

output <- sapply(imp.all,
  function(x) {
    mice::pool(with(
      x,
      rlm(
        PostDBP ~ factor(Group) +
          BaselineDBP + Age + Z + R1 + R2,
        method = "MM", psi = psi.huber
      )
    ))
  },
  simplify = "array", USE.NAMES = TRUE
)

a <- (as.data.frame(t(output)))$pooled
r <- array(rep(NA, length(delta) * 7),
  dim = c(length(delta), 7),
  dimnames = list(
    c(delta),
    c(
      "Intercept", "Group",
      "BaselineDBP", "Age",
      "Z", "R1", "R2"
    )
  )
)
for (i in 1:length(delta)) {
  r[i, ] <- cbind(a[[i]][["estimate"]])
}

r <- as.data.frame(r)
r <- data.frame(Delta = row.names(r), r, row.names = NULL)
Delta Intercept Group BaselineDBP Age Z R1 R2
-20 -112.03 -11.90 1.99 0.30 -3.18 0.09 -1.19
-15 -49.65 -10.64 1.46 0.31 -2.10 0.06 -0.60
-10 -25.15 -10.18 1.26 0.32 -1.17 0.06 -0.41
-5 -2.75 -9.80 1.09 0.32 -0.93 0.00 -0.25
0 13.07 -9.54 0.97 0.33 -0.59 -0.01 -0.13
5 32.79 -9.18 0.82 0.33 0.00 -0.07 -0.04
10 51.86 -8.84 0.67 0.33 -0.18 -0.09 0.05
15 80.67 -8.37 0.44 0.34 0.75 -0.11 0.18
20 138.66 -7.17 -0.05 0.35 1.00 -0.19 0.78

The table above gives the various ’s (the first column with values ranging from -30 - 30 in increments of 5) that were added to the multiply imputed datasets, and the corresponding estimates from the models fit to those datasets. So now we’ve conducted our first sensitivity analysis using the -adjustment, controlled multiple imputation method. This form of sensitivity analysis is also typically what is used and preferred by regulators.23

Regulators prefer simple methods that impute the missing outcomes under MAR, and then add an adjustment δ to the imputes, while varying δ over a plausible range and independently for each treatment group (Permutt 2016).

The most interesting scenarios will be those where the difference between the δ’s correspond to the size of the treatment effect in the completers. Contours of the p-values may be plotted on a graph as a function of the δ’s to assist in a tipping-point analysis (Liublinska and Rubin 2014).

And as expected, lower values gave us results that were consistent with the primary analysis, while the upper end of the extreme values gave us estimates that were substantially off the mark.

Next, we quickly examine the state of a random sample of these imputations to make sure they’re reliable.

Now we move onto our next and final sensitivity analysis.

SA II: Selection Models

To further explore the MNAR assumption, we could also use a Heckman selection model , as described by van Buuren here20

The selection model multiplies the marginal distribution in the population with the response weights . Both and are unknown, and must be specified by the user. The model where is normal and where is a probit model is known as the Heckman model. This model is widely used in economics to correct for selection bias.

We can implement this in R using the miceMNAR package, which is an extension to the mice R package.

# Selection Models
has_miceMNAR <- requireNamespace("miceMNAR", quietly = TRUE)
has_mnar_api <- FALSE

if (has_miceMNAR) {
  miceMNAR_exports <- getNamespaceExports("miceMNAR")
  has_mnar_api <- all(c("generate_JointModelEq", "MNARargument") %in% miceMNAR_exports)
}

if (!has_mnar_api) {
  message(
    "Skipping SA II (selection model): 'miceMNAR' with generate_JointModelEq/MNARargument is unavailable in this environment."
  )
  imp2 <- NULL
} else {
  JointModelEq <- miceMNAR::generate_JointModelEq(
    data = BP_miss,
    varMNAR = "PostDBP"
  )

  JointModelEq[, "PostDBP_var_sel"] <- c(0, 1, 1, 1, 1, 1, 1)
  JointModelEq[, "PostDBP_var_out"] <- c(0, 1, 1, 1, 0, 1, 1)

  arg <- miceMNAR::MNARargument(
    data = BP_miss,
    varMNAR = "PostDBP",
    JointModelEq = JointModelEq
  )

  imp2 <- mice(
    data = arg$data_mod,
    method = "pmm", seed = 1031,
    predictorMatrix = arg$predictorMatrix,
    JointModelEq = arg$JointModelEq,
    control = arg$control,
    remove.collinear = TRUE,
    remove.constant = FALSE, allow.na = FALSE,
    printFlag = FALSE, visitSequence = "monotone"
  )

  imp2$data$Group <- as.factor(imp2$data$Group)
}

Now we fit our models to these new datasets and pool them using Rubin’s rules.

# Sensitivity Analysis 2
if (is.null(imp2)) {
  analysis3 <- NULL
  result3 <- NULL
  results3_sum <- data.frame(
    Term = character(),
    Estimate = numeric(),
    SE = numeric(),
    Statistic = numeric(),
    df = numeric(),
    `P-val` = numeric()
  )
} else {
  analysis3 <- with(imp2, rlm(
    PostDBP ~ factor(Group) +
      BaselineDBP + Age + Z + R1 + R2,
    method = "MM", psi = psi.huber
  ))
  result3 <- mice::pool(analysis3)

  results3_sum <- summary(result3)
  colnames(results3_sum) <- c(
    "Term", "Estimate", "SE",
    "Statistic", "df", "P-val"
  )
}
Term Estimate SE Statistic df P-val
(Intercept) 10.12 6.14 1.65 NA NA
factor(Group)Drug Y -9.50 0.20 -47.69 NA NA
BaselineDBP 0.99 0.06 18.01 NA NA
Age 0.33 0.01 27.99 NA NA
Z -1.09 1.85 -0.59 NA NA
R1 -0.02 0.08 -0.30 NA NA
R2 -0.15 0.16 -0.96 NA NA

Now we examine our imputed values.

so now we have completed our second sensitivity analysis.

The results from both the primary analysis, the first sensitivity analysis, and the second seem somewhat consistent. However, with the first sensitivity analysis where we applied -adjustments, as the adjustments became larger, the coefficients changed drastically. However, such large -adjustments are not realistic and those that were smaller led to coefficients that were closer to the other two analyses.

Supplementary Analyses

The following analyses are what I mostly consider to be supplementary analyses that may further investigate some violations of assumptions but often go far beyond what would be necessary for a principled sensitivity analysis that would typically accompany a primary analysis.

First, I like to check that the results from my analyses are consistent across different statistical software So I may check to see that the results from my primary analysis from R are also similar to the results from another statistical software suite like Stata.

So in Stata, I would likely run something similar to the following in order to mimic the primary analysis in R.

set linesize 255

clear all

set seed 1031

import delimited "static/datasets/temp.csv", numericcols(2 3 5 6 7 8)

summarize

mi set mlong

mi register imputed baselinedbp postdbp age z group r1 r2

mi impute chained (pmm, knn(5)) postdbp = baselinedbp age z r1 r2 group, burnin(20) add(100) rseed (1031) savetrace(static/datasets/trace1, replace) nomonotone

mi estimate, mcerror: rreg postdbp i.group baselinedbp age z r1 r2

quietly mi passive: generate byte imputed = _mi_miss
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> (encoding automatically selected: ISO-8859-2)
#> (7 vars, 500 obs)
#> 
#> 
#>     Variable |        Obs        Mean    Std. dev.       Min        Max
#> -------------+---------------------------------------------------------
#>      postdbp |        255    138.3033    5.413468   123.8279   150.0574
#>  baselinedbp |        500    116.8569    1.997616   109.8634   123.0737
#>        group |        500        .484    .5002444          0          1
#>          age |        500    49.73008    6.938333   27.65241    68.4464
#>            z |        500    2.78e-13    .0626851  -.1845036   .2024892
#> -------------+---------------------------------------------------------
#>           r1 |        500   -.0428001    .9472443  -3.205036   3.377526
#>           r2 |        500    .9620217    .5061543  -.6455948   2.480457
#> 
#> 
#> (245 m=0 obs now marked as incomplete)
#> 
#> note: missing-value pattern is monotone.
#> 
#> Conditional models:
#>            postdbp: pmm postdbp baselinedbp age z r1 r2 group , knn(5)
#> 
#> Performing chained iterations ...
#> 
#> Multivariate imputation                     Imputations =      100
#> Chained equations                                 added =      100
#> Imputed: m=1 through m=100                      updated =        0
#> 
#> Initialization: monotone                     Iterations =     2000
#>                                                 burn-in =       20
#> 
#>            postdbp: predictive mean matching
#> 
#> ------------------------------------------------------------------
#>                    |               Observations per m             
#>                    |----------------------------------------------
#>           Variable |   Complete   Incomplete   Imputed |     Total
#> -------------------+-----------------------------------+----------
#>            postdbp |        255          245       245 |       500
#> ------------------------------------------------------------------
#> (Complete + Incomplete = Total; Imputed is the minimum across m
#>  of the number of filled-in observations.)
#> 
#> 
#> Multiple-imputation estimates                   Imputations       =        100
#> Robust regression                               Number of obs     =        500
#>                                                 Average RVI       =     0.7136
#>                                                 Largest FMI       =     0.5035
#>                                                 Complete DF       =        493
#> DF adjustment:   Small sample                   DF:     min       =     151.95
#>                                                         avg       =     188.67
#>                                                         max       =     233.67
#> Model F test:       Equal FMI                   F(   6,  421.4)   =    1181.06
#>                                                 Prob > F          =     0.0000
#> 
#> ------------------------------------------------------------------------------
#>      postdbp | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>      1.group |  -9.520968   .1207769   -78.83   0.000    -9.758918   -9.283017
#>              |   .0070306   .0035352     2.31   0.000     .0118962    .0076126
#>              |
#>  baselinedbp |   .9811365   .0448784    21.86   0.000     .8926164    1.069657
#>              |   .0028784   .0010755     0.54   0.000     .0037961    .0033965
#>              |
#>          age |    .328745   .0089894    36.57   0.000     .3110251    .3464649
#>              |   .0005493   .0002679     1.12   0.000      .000839    .0006919
#>              |
#>            z |  -1.017329   1.500306    -0.68   0.499    -3.979496    1.944838
#>              |   .1020678    .050788     0.06   0.041     .1658641    .1196211
#>              |
#>           r1 |  -.0169674   .0731975    -0.23   0.817    -.1615836    .1276488
#>              |   .0051452   .0023361     0.07   0.056     .0063651    .0075475
#>              |
#>           r2 |  -.1393538   .1309672    -1.06   0.289    -.3978634    .1191559
#>              |   .0087814   .0046722     0.07   0.032     .0138722    .0117957
#>              |
#>        _cons |   11.61969   5.244316     2.22   0.028     1.276334    21.96305
#>              |   .3346446   .1264252     0.08   0.005     .3897239    .4489929
#> ------------------------------------------------------------------------------
#> Note: Values displayed beneath estimates are Monte Carlo error estimates.
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> (Summaries of imputed values from -mi impute chained-)
#> 
#> 
#> Contains data from static/datasets/trace1.dta
#>  Observations:         2,100                  Summaries of imputed values from -mi impute chained-
#>     Variables:             4                  13 Sep 2026 04:16
#> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#> Variable      Storage   Display    Value
#>     name         type    format    label      Variable label
#> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#> iter            byte    %12.0g                Iteration numbers
#> m               byte    %12.0g                Imputation numbers
#> postdbp_mean    float   %9.0g                 Mean of postdbp
#> postdbp_sd      float   %9.0g                 Std. dev. of postdbp
#> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#> Sorted by: 
#> 
#> 
#> 
#> reshaping m=0 data ...
#> (j = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 8
#> > 7 88 89 90 91 92 93 94 95 96 97 98 99 100)
#> 
#> Data                               Long   ->   Wide
#> -----------------------------------------------------------------------------
#> Number of observations            2,100   ->   21          
#> Number of variables                   4   ->   201         
#> j variable (100 values)               m   ->   (dropped)
#> xij variables:
#>                            postdbp_mean   ->   postdbp_mean1 postdbp_mean2 ... postdbp_mean100
#>                              postdbp_sd   ->   postdbp_sd1 postdbp_sd2 ... postdbp_sd100
#> -----------------------------------------------------------------------------
#> 
#> assembling results ...
#> 
#> 
#> Time variable: iter, 0 to 20
#>         Delta: 1 unit
#> 
#> 
#> 
#> file ()print() not found
#> r(601);
#> 
#> r(601);

As you can see from above, I imported the missing data I generated in R into Stata, and then used it to multiply impute the datasets using the predictive mean matching method with 5 donors. I chose the exact same number of variables to include in the imputation model as I did with the primary and sensitivity analyses. Just like the primary analysis, I also used robust regression, rreg in Stata, to analyze the datasets. The numbers seem to be fairly consistent with those from the primary analysis and the some of the results from the sensitivity analyses.

I will now conduct another supplementary analysis, in which I fit a Bayesian regression model using a t-distribution so that it is analogous to the other robust regression models I have utilized so far, and I will specify a weakly informative prior. I will fit one model using the brm_multiple() function which will fit the model to each of the imputed datasets from before, while I fit another Bayesian model which will impute values during the model fitting process and compare the results.

# Quantile Regression  --------------------------

implist <- mitml::mids2mitml.list(imp1)

vcov.rq <- function(object, ...) {
  summary(object,
    se = "nid",
    covariance = TRUE
  )$cov
}

fit1 <- with(implist, rq(PostDBP ~ factor(Group) + BaselineDBP +
  Age + Z + R1 + R2, tau = 0.5))
mitml::testEstimates(fit1)
#> 
#> Call:
#> 
#> mitml::testEstimates(model = fit1)
#> 
#> Final parameter estimates and inferences obtained from 120 imputed data sets.
#> 
#>                      Estimate Std.Error   t.value        df   P(>|t|)       RIV       FMI 
#> (Intercept)            10.092     6.610     1.527   519.639     0.127     0.918     0.481 
#> factor(Group)Drug Y    -9.433     0.157   -59.956   622.364     0.000     0.777     0.439 
#> BaselineDBP             0.991     0.056    17.617   522.900     0.000     0.912     0.479 
#> Age                     0.336     0.011    31.319   690.038     0.000     0.710     0.417 
#> Z                      -1.397     1.755    -0.796   594.157     0.426     0.810     0.449 
#> R1                     -0.054     0.091    -0.600   403.495     0.549     1.189     0.545 
#> R2                     -0.137     0.160    -0.859   451.123     0.391     1.056     0.516 
#> 
#> Unadjusted hypothesis test as appropriate in larger samples.

confint(mitml::testEstimates(fit1))
#>                      2.5 % 97.5 %
#> (Intercept)         -2.894 23.078
#> factor(Group)Drug Y -9.742 -9.124
#> BaselineDBP          0.880  1.101
#> Age                  0.315  0.357
#> Z                   -4.843  2.049
#> R1                  -0.233  0.124
#> R2                  -0.452  0.177
# Bayesian Regression  --------------------------

BP_miss$Group <- as.factor(BP_miss$Group)

brm_form2 <- bf(
  PostDBP | mi() ~ factor(Group) + BaselineDBP +
    Age + Z + R1 + R2, quantile = 0.50
)

prior1 <- get_prior(brm_form2, data = BP_miss, knots = 4,
                    family = asym_laplace(link = 'identity'))

b_2 <- brm(formula = brm_form2, data = BP_miss, prior = prior1,
           family = asym_laplace(link = 'identity'), sample_prior = 'yes',
           control = list(max_treedepth = 10, adapt_delta = 0.80),
           iter = 2000, refresh = 0, backend = "rstan", chains = 4,
           algorithm = "sampling", knots = 4, future = FALSE, cores = 8L,
           warmup = 500, seed = 1031)
# `post` is already the mice post-processing vector above, so use a distinct name
bayes_post <- if (exists("b_2")) summary(b_2) else NULL

if (is.null(bayes_post)) {
  cat("Bayesian model summary skipped: `b_2` was not fitted in this environment.\n")
} else {
  bayes_post
}
#> Bayesian model summary skipped: `b_2` was not fitted in this environment.

We can also compare the results we get from brms/Stan with MCMCpack.

posterior_95 <- MCMCquantreg(
  PostDBP ~ factor(Group) + BaselineDBP +
    Age + Z + R1 + R2,
  data = BP_miss,
  tau = 0.95, mcmc = 5000, chains = 4,
  thin = 10, seed = 1031
)
summary(posterior_95)
#> 
#> Iterations = 1001:5991
#> Thinning interval = 10 
#> Number of chains = 1 
#> Sample size per chain = 500 
#> 
#> 1. Empirical mean and standard deviation for each variable,
#>    plus standard error of the mean:
#> 
#>                        Mean      SD Naive SE Time-series SE
#> (Intercept)         10.2441 19.8021  0.88558        1.00205
#> factor(Group)Drug Y -9.6451  0.4329  0.01936        0.02600
#> BaselineDBP          1.0197  0.1688  0.00755        0.00855
#> Age                  0.3108  0.0324  0.00145        0.00156
#> Z                   -3.0856  5.6527  0.25279        0.27623
#> R1                  -0.0277  0.2696  0.01206        0.01383
#> R2                  -0.4040  0.3937  0.01761        0.01950
#> 
#> 2. Quantiles for each variable:
#> 
#>                        2.5%    25%     50%    75%  97.5%
#> (Intercept)         -27.035 -2.560  9.2822 22.020 49.797
#> factor(Group)Drug Y -10.572 -9.916 -9.6297 -9.350 -8.846
#> BaselineDBP           0.679  0.918  1.0263  1.130  1.338
#> Age                   0.253  0.290  0.3094  0.332  0.375
#> Z                   -14.438 -6.933 -3.3466  1.069  7.636
#> R1                   -0.550 -0.214 -0.0152  0.146  0.507
#> R2                   -1.189 -0.646 -0.3728 -0.158  0.312

Next, we examine the results from the Bayesian model in which values were imputed during model fitting.

posterior_summary(b_2)[1:8]
(post <- posterior_samples(b_2))

They seem to be similar to the results we’ve been seeing so far.

Indeed, it is no surprise that the Bayesian model with a weakly dispersed prior would give similar results to the base rlm() function. We now look at robust methods and resampling and the results they give.

Although this is currently under the supplementary analysis section, I would very much consider these next two analyses to be sensitivity analyses. The first method utilized here is known as the random indicator method, and it is an experimental method that essentially combines a selection model and a pattern-mixture model and is designed to handle data that are missing not at random. As van Buuren writes20

The random indicator method (Jolani 2012)24 is an experimental iterative method that redraws the missing data indicator under a selection model, and imputes the missing data under a pattern-mixture model, with the objective of estimating δ from the data under relaxed assumptions. Initial simulation results look promising. The algorithm is available as the ri method in mice.

The script below utilizes this method (random indicator), but bootstraps the missing dataset before the imputation procedure, and then after multiply imputing the datasets, obtains estimates using maximum likelihood multiple (MLMI) imputation, an alternative and more efficient method to the more popular posterior draw multiple imputation (PDMI). More about MLMI can be found here.25

# Bootstrap Inference

bootMice(BP_miss, method = "ri", maxit = 5,
         predictorMatrix = pred, nBoot = 96, nImp = 2,
         seed = 1031, formulas = form, nCores = 8L,
         ridge = 0, remove.collinear = TRUE,
         remove.constant = TRUE, allow.na = TRUE,
         printFlag = FALSE, visitSequence = "monotone") -> imps

analyseImp <- function(inputData) {
  return(coef(rlm(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2,
    data = inputData, method = "MM", psi = psi.huber
  )))
}

analyseImp1 <- function(inputData) {
  return(coef(glm(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2,
    data = inputData, family = gaussian()
  )))
}

ests <- bootImputeAnalyse(imps, analyseImp, quiet = TRUE)
ests1 <- bootImputeAnalyse(imps, analyseImp1, quiet = TRUE)

ests <- as.data.frame(ests)

colnames(ests) <- c("Estimate", "Var",
                    "CI.ll", "CI.ul", "df")

rownames(ests) <- c("Intercept", "Group", "BaselineDBP",
                    "Age", "Z", "R1", "R2")
ests1 <- as.data.frame(ests1)

colnames(ests1) <- c("Estimate", "Var",
                     "CI.ll", "CI.ul", "df")

rownames(ests1) <- c("Intercept", "Group", "BaselineDBP",
                     "Age", "Z", "R1", "R2")
Estimate Var CI.ll CI.ul df
Intercept 10.72 16.51 2.19 19.25 18.3
Group -9.62 0.02 -9.89 -9.34 48.0
BaselineDBP 0.99 0.00 0.91 1.06 19.6
Age 0.34 0.00 0.32 0.36 55.4
Z -1.00 2.32 -4.06 2.05 51.2
R1 -0.02 0.01 -0.19 0.15 64.3
R2 -0.14 0.02 -0.39 0.11 44.8
Estimate Var CI.ll CI.ul df
Intercept 10.37 16.67 1.84 18.89 19.7
Group -9.63 0.02 -9.89 -9.36 47.5
BaselineDBP 0.99 0.00 0.92 1.06 20.7
Age 0.33 0.00 0.31 0.35 57.3
Z -1.02 2.21 -4.00 1.96 50.1
R1 -0.02 0.01 -0.18 0.15 62.6
R2 -0.15 0.01 -0.39 0.09 43.3

For our second supplementary analysis, We will utilize the GAMLSS method, which is an extension of the generalized linear model and the generalized additive model. For the vast majority of the imputations we have conducted, we have drawn values from an assumed normal distribution. Thus, GAMLSS allows us to be far more flexible with our assumptions. However, it is also worth noting that general normal imputations are robust against violations of the assumption of normality

In general, normal imputations appear to be robust against violations of normality. Demirtas, Freels, and Yucel (2008) found that flatness of the density, heavy tails, non-zero peakedness, skewness and multimodality do not appear to hamper the good performance of multiple imputation for the mean structure in samples n > 400, even for high percentages (75%) of missing data in one variable. The variance parameter is more critical though, and could be off-target in smaller samples.

Regardless, we will be using the GAMLSS method and also imputing from a t-distribution, which is much more robust to outliers.

gamt <- mice(BP_miss, method = "gamlssTF",
             predictorMatrix = pred,
             m = 5, maxit = 1, seed = 1031,
             cores = cores, formulas = form, ridge = 0,
             remove.collinear = TRUE, remove.constant = TRUE,
             allow.na = TRUE, printFlag = FALSE)

gt1 <- with(gamt, rlm(PostDBP ~ factor(Group) +
  BaselineDBP + Age +
  Z + R1 + R2))

resultsgt <- mice::pool(gt1)
ztable(summary(resultsgt))
term estimate std.error statistic df p.value
(Intercept) 13.23 4.95 2.67 NA NA
factor(Group)Drug Y -9.52 0.13 -76.02 NA NA
BaselineDBP 0.97 0.04 23.48 NA NA
Age 0.32 0.01 35.44 NA NA
Z -0.62 1.39 -0.44 NA NA
R1 -0.02 0.07 -0.34 NA NA
R2 -0.14 0.10 -1.35 NA NA

Having done that, I shall try one more supplementary analysis which piqued my curiosity. I recently came across a few papers that concluded that random forests outperformed other imputation algorithms in recovering the true parameter values.

Generally, imputation algorithms are evaluated based on a general simulation design which often resembles the following:

  1. Sampling mechanism only. The basic simulation steps are: choose , take samples , fit the complete-data model, estimate and and calculate the outcomes aggregated over .
  1. Sampling and missing data mechanisms combined. The basic simulation steps are: choose , take samples , generate incomplete data , impute, estimate and and calculate outcomes aggregated over and .
  1. Missing data mechanism only. The basic simulation steps are: choose , generate incomplete data , impute, estimate and and calculate outcomes aggregated over .

These designs generally lead to evaluating a number of metrics which include the following:

  • Raw Bias: where is bias and is the difference between the estimate and the truth , which may also be expressed as a percent
  • Coverage Rate: which is simply the proportion of times the intervals actually contain the true parameter value
  • Average Width: the width of the interval estimates, with longer widths indicating less information
  • Root mean squared error (RMSE)

The authors primarily based their conclusions on the root mean squared error (RMSE). However, this is problematic for a number of reasons; imputation is not the same as prediction, and so the same loss functions are not as useful; indeed, the RMSE is highly misleading as a measure of performance.

van Buuren explains here:

It is well known that the minimum RMSE is attained by predicting the missing by the linear model with the regression weights set to their least squares estimates. According to this reasoning the “best” method replaces each missing value by its most likely value under the model. However, this will find the same values over and over, and is single imputation.

This method ignores the inherent uncertainty of the missing values (and acts as if they were known after all), resulting in biased estimates and invalid statistical inferences. Hence, the method yielding the lowest RMSE is bad for imputation. More generally, measures based on similarity between the true and imputed values do not separate valid from invalid imputation methods.

Having set up our simulation design, we can now test the performances between regression imputation norm.predict and and regular regression imputation norm and we will be examining the metrics above to evaluate performance. Regression imputation generally gives very biased results given that it uses its predictions to impute the missing values. Thus, I will show here how RMSE is highly misleading as a metric to evaluate the performance of imputation algorithms.

# Evaluate MI Algorithms

create.data <- function(beta = 1, sigma2 = 1, n = 500,
                        run = 1) {
  set.seed(seed = run)
  x <- rnorm(n)
  y <- beta * x + rnorm(n, sd = sqrt(sigma2))
  cbind(x = x, y = y)
}

make.missing <- function(data, p = 0.5) {
  rx <- rbinom(nrow(data), 1, p)
  data[rx == 0, "x"] <- NA
  data
}

test.impute <- function(data, m = 5, method = "norm", ...) {
  imp <- mice(data,
    method = method, m = m,
    print = FALSE, cores = cores, ...
  )
  fit <- with(imp, lm(y ~ x))
  tab <- summary(mice::pool(fit), "all", conf.int = TRUE)
  as.numeric(tab["x", c("estimate", "2.5 %", "97.5 %")])
}

rmse <- function(truedata, imp, v = "x") {
  mx <- is.na(mice::complete(imp, 0))[, v]
  mse <- rep(NA, imp$m)
  for (k in seq_len(imp$m)) {
    filled <- mice::complete(imp, k)[mx, v]
    true <- truedata[mx, v]
    mse[k] <- mean((filled - true)^2)
  }
  sqrt(mean(mse))
}

mae <- function(truedata, imp, v = "x") {
  mx <- is.na(mice::complete(imp, 0))[, v]
  mse <- rep(NA, imp$m)
  for (k in seq_len(imp$m)) {
    filled <- mice::complete(imp, k)[mx, v]
    true <- truedata[mx, v]
    mse[k] <- mean((filled - true))
  }
  abs(mean(mse))
}

simulate1 <- function(runs = 10) {
  res <- array(NA, dim = c(2, runs, 1))
  dimnames(res) <- list(
    c("norm.predict", "norm"),
    as.character(1:runs),
    "MAE"
  )
  for (run in 1:runs) {
    truedata <- create.data(run = run)
    data <- make.missing(truedata)
    imp <- mice(data,
      method = "norm.predict",
      m = 1, cores = cores,
      print = FALSE
    )
    res[1, run, ] <- mae(truedata, imp)
    imp <- mice(data, method = "norm", print = FALSE)
    res[2, run, ] <- mae(truedata, imp)
  }
  res
}

simulate2 <- function(runs = 10) {
  res <- array(NA, dim = c(2, runs, 1))
  dimnames(res) <- list(
    c("norm.predict", "norm"),
    as.character(1:runs),
    "RMSE"
  )
  for (run in 1:runs) {
    truedata <- create.data(run = run)
    data <- make.missing(truedata)
    imp <- mice(data,
      method = "norm.predict",
      m = 1, cores = cores,
      print = FALSE
    )
    res[1, run, ] <- rmse(truedata, imp)
    imp <- mice(data, method = "norm", print = FALSE)
    res[2, run, ] <- rmse(truedata, imp)
  }
  res
}

Our simulation above will generate a dataset, delete values from the dataset and then impute the missing values for that dataset using regression imputation and stochastic regression imputation. We will use two metrics to evaluate the performance of these two algorithms, RMSE and MAE. Once again, it is worth keeping in mind that regression imputation generally performs very poorly when it comes to imputation and that stochastic regression imputation performs better.

set.seed(1031,
  kind = "L'Ecuyer-CMRG",
  normal.kind = "Inversion",
  sample.kind = "Rejection"
)
res1 <- simulate1(1000)
res_1 <- apply(res1, c(1, 3),
  mean,
  na.rm = TRUE
)
res2 <- simulate2(1000)
res_2 <- apply(res2, c(1, 3),
  mean,
  na.rm = TRUE
)
ztable(cbind(res_1, res_2))

What this simulation clearly shows is that despite regression imputation norm.predict giving highly biased results as usual relative to the truth, it still has a much lower RMSE than stochastic regression imputation, which both had a higher RMSE and a closer distance to the correct parameter values. Once again, this is not surprising given that imputation is not the same thing as prediction and therefore the loss functions cannot be used synonymously. However, when it came to raw bias and mean absolute error, stochastic regression imputation outperformed regression imputation which had a much larger mean absolute error and raw bias.

This is why the most informative metrics to focus on when evaluating an imputation algorithm include information/width, sufficient long-run coverage, and less misleading metrics such as mean absolute error and mean absolute percentage error.

The authors specifically used the missForest package. I tried using the package, but encountered too many difficulties so I simply specified the rf method in mice, and worked from there.

# MI Using Random Forest

form <- list(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2)

rf <- mice::mice(
  BP_miss,
  method = "rf",
  ntree = 20,
  predictorMatrix = pred,
  m = 10,
  maxit = 5,
  seed = 1031,
  formulas = form,
  ridge = 0,
  remove.collinear = TRUE,
  remove.constant = TRUE,
  allow.na = TRUE,
  printFlag = FALSE,
  visitSequence = "monotone"
)

rf1 <- with(rf, rlm(PostDBP ~ factor(Group) +
  BaselineDBP + Age +
  Z + R1 + R2,
  method = "MM", psi = psi.huber
))

results5 <- mice::pool(rf1)

results5_sum <- summary(results5)

colnames(results5_sum) <- c(
  "Term", "Estimate", "SE",
  "Statistic", "df", "P-val"
)
ztable(results5_sum)
Term Estimate SE Statistic df P-val
(Intercept) 33.94 9.94 3.42 NA NA
factor(Group)Drug Y -9.45 0.18 -53.97 NA NA
BaselineDBP 0.80 0.09 9.38 NA NA
Age 0.29 0.02 17.77 NA NA
Z 0.02 2.58 0.01 NA NA
R1 0.02 0.09 0.22 NA NA
R2 -0.02 0.18 -0.10 NA NA

Full Analysis Set

After we run all those analyses, we can finally check our numbers from the full dataset with no missing observations to see how our approaches to handle the missing data have performed. We first look at the fully adjusted model fit with rlm().

# Full Analysis Set

final_full_mod <- rlm(
  PostDBP ~ factor(Group) + BaselineDBP +
    Age + Z + R1 + R2,
  data = BP_full,
  method = "MM", psi = psi.huber
)
Value Std. Error t value
(Intercept) 9.19 3.72 2.47
factor(Group)Drug Y -9.66 0.09 -106.85
BaselineDBP 1.00 0.03 31.68
Age 0.33 0.01 50.17
Z -2.02 1.01 -2.00
R1 0.01 0.05 0.21
R2 -0.01 0.09 -0.09
Term VIF VIF_CI_low VIF_CI_high SE_factor Tolerance Tolerance_CI_low Tolerance_CI_high
factor(Group) 1.01 1.00 489.74 1.00 0.99 0.00 1.00
BaselineDBP 1.97 1.75 2.26 1.40 0.51 0.44 0.57
Age 1.02 1.00 4.09 1.01 0.98 0.24 1.00
Z 1.98 1.75 2.27 1.41 0.51 0.44 0.57
R1 1.01 1.00 33.65 1.01 0.99 0.03 1.00
R2 1.01 1.00 1603.17 1.00 0.99 0.00 1.00

We can also graphically inspect our assumptions.

Then we look at the unadjusted model, where the difference is the group that the participants were randomized to.

# Unadjusted Model

final_reduced_mod <- rlm(PostDBP ~ factor(Group),
  data = BP_full,
  method = "MM", psi = psi.huber
)

final_reduced_mod_sum <- summary(final_reduced_mod)[["coefficients"]]

colnames(final_reduced_mod_sum) <- c(
  "Value",
  "Std. Error",
  "t-value"
)
Value Std. Error t-value
(Intercept) 142.6 0.20 728.8
factor(Group)Drug Y -9.2 0.28 -32.7

We can also compare the two to see if there are any substantial differences.

Name Model AIC AIC_wt AICc AICc_wt BIC BIC_wt RMSE Sigma
final_full_mod rlm 1420 1 1420 1 1453 1 0.98 0.99
final_reduced_mod rlm 2556 0 2557 0 2569 0 3.10 3.11

While the results have mostly been consistent, I also wanted to run a few additional analyses of my own to see whether these effect estimates were consistent I first started by constructing the confidence distribution using the [concurve] package to see the range of parameter values consistent with the data given the background model assumptions.26, 27

# Compatibility of Values

fit1 <- glm(
  PostDBP ~ factor(Group) + BaselineDBP +
    Age + Z + R1 + R2,
  data = BP_full, family = gaussian()
)

summary(fit1)
#> 
#> Call:
#> glm(formula = PostDBP ~ factor(Group) + BaselineDBP + Age + Z + 
#>     R1 + R2, family = gaussian(), data = BP_full)
#> 
#> Coefficients:
#>                     Estimate Std. Error t value Pr(>|t|)    
#> (Intercept)          9.02736    3.65879    2.47    0.014 *  
#> factor(Group)Drug Y -9.64556    0.08906 -108.30   <2e-16 ***
#> BaselineDBP          1.00567    0.03120   32.23   <2e-16 ***
#> Age                  0.32696    0.00645   50.69   <2e-16 ***
#> Z                   -1.97391    0.99551   -1.98    0.048 *  
#> R1                   0.01071    0.04711    0.23    0.820    
#> R2                  -0.02493    0.08799   -0.28    0.777    
#> ---
#> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#> 
#> (Dispersion parameter for gaussian family taken to be 0.983)
#> 
#>     Null deviance: 15519.32  on 499  degrees of freedom
#> Residual deviance:   484.49  on 493  degrees of freedom
#> AIC: 1419
#> 
#> Number of Fisher Scoring iterations: 2

fit2 <- glm(PostDBP ~ factor(Group),
  data = BP_full, family = gaussian()
)

summary(fit2)
#> 
#> Call:
#> glm(formula = PostDBP ~ factor(Group), family = gaussian(), data = BP_full)
#> 
#> Coefficients:
#>                     Estimate Std. Error t value Pr(>|t|)    
#> (Intercept)          142.596      0.193   737.3   <2e-16 ***
#> factor(Group)Drug Y   -9.263      0.278   -33.3   <2e-16 ***
#> ---
#> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#> 
#> (Dispersion parameter for gaussian family taken to be 9.65)
#> 
#>     Null deviance: 15519.3  on 499  degrees of freedom
#> Residual deviance:  4805.8  on 498  degrees of freedom
#> AIC: 2556
#> 
#> Number of Fisher Scoring iterations: 2

curve1 <- curve_gen(fit1,
  var = "factor(Group)Drug Y", method = "lm",
  log = FALSE, steps = 1000, table = TRUE
)

Lower Limit Upper Limit Interval Width Interval Level (%) CDF P-value S-value (bits)
-9.67 -9.62 0.057 25.0 0.625 0.750 0.415
-9.71 -9.59 0.120 50.0 0.750 0.500 1.000
-9.75 -9.54 0.205 75.0 0.875 0.250 2.000
-9.76 -9.53 0.228 80.0 0.900 0.200 2.322
-9.77 -9.52 0.256 85.0 0.925 0.150 2.737
-9.79 -9.50 0.293 90.0 0.950 0.100 3.322
-9.82 -9.47 0.349 95.0 0.975 0.050 4.322
-9.85 -9.45 0.399 97.5 0.988 0.025 5.322
-9.88 -9.42 0.459 99.0 0.995 0.010 6.644
Table of Statistics for Various Interval Estimate Percentiles

Indeed, our confidence distribution ends up being very similar to our Bayesian posterior distribution for the treatment effect. Under certain situations, the maximum likelihood estimate ends up also being equivalent to the median unbiased estimate. In this case, it happens to be very similar to the maximum a posteriori estimate.

#> Posterior density plot skipped: `b_2` was not fitted in this environment.

We can also check to see that our confidence distribution and posterior distribution are consistent with the profile-likelihood function, which is free of nuisance parameters. The advantage of profiling is allowing us to recover more accurate parameter estimates when there are several nuisance parameters or messy data. Below are implementations in both R and Stata, except this time we do not specify errors from the t-distribution.

analyseImp3 <- function(inputData, i) {
  dt <- inputData[i, ]
  (coef(glm(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2,
    data = dt, family = gaussian()
  ))[2])
}

bootdist <- boot::boot(
  data = BP_full, statistic = analyseImp3,
  R = 4000, sim = "ordinary", stype = "i",
  parallel = "multicore", ncpus = 8
)

sd <- boot::boot.ci(bootdist, conf = c((1:99) / 100), type = "bca")[4]

A <- sd$bca

qplot((c(A[, 4], A[, 5])), geom = "density", fill = I("#3f8f9b44")) +
  theme_less() +
  labs(
    title = "bCA Bootstrap Density Plot",
    x = "'Group' Parameter Value (Treatment Effect mmHg)",
    y = "Density"
  )

We now focus on profiling on the treatment effect and calculating the profile deviance function. We then multiply our deviance by -0.5 to obtain the profile log-likelihood function.

# Profile Likelihood

prof <- profile(fit1)
disp <- attr(prof, "summary")$dispersion
mindev <- attr(prof, "original.fit")$deviance

dev1 <- prof[[1]]$tau^2
dev2 <- dev1 * disp + mindev

tmpf <- function(x, n) {
  data.frame(
    par = n, tau = x$tau,
    deviance = x$tau^2 * disp + mindev,
    x$par.vals, check.names = FALSE
  )
}

pp <- do.call(rbind, mapply(tmpf, prof, names(prof),
  SIMPLIFY = FALSE
))

pp2 <- melt(pp, id.var = 1:3)
pp3 <- subset(pp2, par == variable, select = -variable)
pp4 <- pp3[-c(1:13), ]

pp4$loglik <- pp4$deviance * -0.5

set linesize 255
clear all
set seed 1031
import delimited "static/datasets/temp.csv", numericcols(2 3 5 6 7 8)
capture which pllf
if _rc == 0 {
foreach var of varlist  group baselinedbp age z r1 r2 {
grstyle init
grstyle color background white, opacity(.1)
quietly pllf glm postdbp group baselinedbp age z r1 r2, profile(`var')
graph copy Graph `var'
}
graph combine  group baselinedbp age z r1 r2
graph export "loglikfunction.svg", replace
}
else {
display as text "Skipping profile-likelihood plots: the -pllf- command is not installed."
}
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> (encoding automatically selected: ISO-8859-2)
#> (7 vars, 500 obs)
#> 
#> 
#> 
#> Skipping profile-likelihood plots: the -pllf- command is not installed.
Lower Limit Upper Limit CI Width Level (%) CDF P-value S-value
-9.67 -9.62 0.057 25.0 0.625 0.750 0.415
-9.71 -9.59 0.120 50.0 0.750 0.500 1.000
-9.75 -9.54 0.205 75.0 0.875 0.250 2.000
-9.76 -9.53 0.228 80.0 0.900 0.200 2.322
-9.77 -9.52 0.256 85.0 0.925 0.150 2.737
-9.79 -9.50 0.293 90.0 0.950 0.100 3.322
-9.82 -9.47 0.349 95.0 0.975 0.050 4.322
-9.85 -9.45 0.399 97.5 0.988 0.025 5.322
-9.88 -9.42 0.459 99.0 0.995 0.010 6.644
Table of Statistics for Various Interval Estimate Percentiles

I then compared the results of the full analysis set to the results from the primary analysis and sensitivity analyses.

#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 11.359786 -9.5274391 .98342353 .32837421 -1.3210281 -.02229379 -.12617403

[concurve]: R: #R Stata: #stata

Examining the Analyses

Evaluation of Imputation Algorithms
PMM Heckman Stata Bayes GAMLSS RI RF True
(Intercept) 11.25 10.12 11.36 NA 13.23 10.37 33.94 9.19
factor(Group)Drug Y -9.51 -9.50 -9.53 NA -9.52 -9.63 -9.45 -9.66
BaselineDBP 0.98 0.99 0.98 NA 0.97 0.99 0.80 1.00
Age 0.33 0.33 0.33 NA 0.32 0.33 0.29 0.33
Z -1.33 -1.09 -1.32 NA -0.62 -1.02 0.02 -2.02
R1 -0.02 -0.02 -0.02 NA -0.02 -0.02 0.02 0.01
R2 -0.14 -0.15 -0.13 NA -0.14 -0.15 -0.02 -0.01
MPE -203.93 -214.28 -171.96 NA -194.38 -224.45 -53.76 0.00
Huber Loss 0.26 0.13 0.28 NA 0.64 0.17 3.69 0.00
MAE 0.44 0.32 0.45 NA 0.83 0.34 3.89 0.00
MSD -0.39 -0.26 -0.41 NA -0.77 -0.29 -3.82 0.00
Metric Abbreviations:
MPE = Mean Percentage Error; MAE = Mean Absolute Error; MSD = Mean Signed Deviation
MI Abbreviations:
PMM = Predictive Mean Matching; Heckman = Heckman Selection Model; RI = Random Indicator; RF = Random Forest; Bayes = Imputation During Model Fitting

We can see that that the overall treatment effect is approximately 9.7-9.9, and for most of the principled methods we used to handle the missing data, we did not obtain similar estimates. For example, both the primary analysis and the controlled multiple imputation methods did not yield the correct treatment effect estimate in the adjusted model.

In fact, nearly all of the methods used except for the Heckman selection model, yielded overestimates of the treatment effect. Even though such analyses can be substantially improved with more specification and more information, the reality is that data that are assumed to be MNAR are difficult to handle and that no method can guarantee the correct answer. Indeed, we knew the missing data model / mechanism, via our custom function

# Missing Data Mechanism

lesslikely <- function() {
  set.seed(seed = 1031,
           kind = "L'Ecuyer-CMRG",
           normal.kind = "Inversion",
           sample.kind = "Rejection"
  )
  logistic <- function(x) exp(x) / (1 + exp(x))
  1 - rbinom(n, 1, logistic(errors[, 2]))
}

Ry <- ifelse(lesslikely() > 0, 1, 0)
PostDBP[Ry == 0] <- NA

for MNAR we set

Thus, we obtain the following model:

Even the random indicator method, which has shown much promise in simulation studies24 and offers the advantage of not having to specify a pattern-mixture or selection model could not recover the correct treatment effect estimate either, showing the difficulty of handling missing data and the limitations of automated approaches.

I hope it is clear from the examples above that it is quite easy to do any analyses with missing data and use a variety of ad-hoc approaches or even principled approaches to explore whether or not the results you’ve obtained are trustworthy or at least even give that impression, but doing a careful and principled one is quite difficult and requires much thought that will take into account real departures from assumptions and whether or not those will lead to substantial changes in results and conclusions/decisions.

Acknowledgements

I’d like to thank Dr. Tim Morris and Dr. Andrew Althouse for helpful comments on an earlier version of this piece. My acknowledgment of their help does not imply endorsement of my views by these colleagues or vice versa, and I remain solely responsible for the views expressed herein. All errors are my own. To point out such errors, please leave a comment below or use the contact form to get in touch.

Statistical Environments

R Environment

#> ─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.6.1 (2026-06-24)
#>  os       macOS Golden Gate 27.0
#>  system   aarch64, darwin25.4.0
#>  ui       unknown
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       America/New_York
#>  date     2026-09-13
#>  pandoc   3.11 @ /opt/homebrew/bin/ (via rmarkdown)
#>  quarto   1.10.18 @ /Applications/quarto/bin/quarto
#> 
#> ─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  package           * version  date (UTC) lib source
#>  Amelia            * 1.8.3    2024-11-08 [1] CRAN (R 4.6.1)
#>  bayesplot         * 1.16.0   2026-08-25 [1] CRAN (R 4.6.1)
#>  blogdown          * 1.24     2026-06-19 [1] CRAN (R 4.6.1)
#>  boot              * 1.3-32   2025-08-29 [1] CRAN (R 4.6.1)
#>  bootImpute        * 1.3.0    2025-12-15 [1] CRAN (R 4.6.1)
#>  brms              * 2.23.0   2025-09-09 [1] CRAN (R 4.6.1)
#>  broom             * 1.0.13   2026-05-14 [1] CRAN (R 4.6.1)
#>  broom.mixed       * 0.2.9.7  2026-02-17 [1] CRAN (R 4.6.1)
#>  Cairo             * 1.7-0    2025-10-29 [1] CRAN (R 4.6.1)
#>  car               * 3.1-5    2026-02-03 [1] CRAN (R 4.6.1)
#>  carData           * 3.0-6    2026-01-30 [1] CRAN (R 4.6.1)
#>  checkmate         * 2.3.4    2026-02-03 [1] CRAN (R 4.6.1)
#>  cli               * 3.6.6    2026-04-09 [1] CRAN (R 4.6.1)
#>  coda              * 0.19-4.1 2024-01-31 [1] CRAN (R 4.6.1)
#>  colorspace        * 2.1-3    2026-07-12 [1] CRAN (R 4.6.1)
#>  concurve          * 3.0.5    2026-09-13 [1] local
#>  cowplot           * 1.2.0    2025-07-07 [1] CRAN (R 4.6.1)
#>  doParallel        * 1.0.17   2022-02-07 [1] CRAN (R 4.6.1)
#>  dplyr             * 1.2.1    2026-04-03 [1] CRAN (R 4.6.1)
#>  forcats           * 1.0.1    2025-09-25 [1] CRAN (R 4.6.1)
#>  foreach           * 1.5.2    2022-02-02 [1] CRAN (R 4.6.1)
#>  futile.logger     * 1.4.9    2025-12-29 [1] CRAN (R 4.6.0)
#>  future            * 1.75.0   2026-07-20 [1] CRAN (R 4.6.1)
#>  future.apply      * 1.20.2   2026-02-20 [1] CRAN (R 4.6.1)
#>  gamlss            * 5.5-0    2025-08-19 [1] CRAN (R 4.6.1)
#>  gamlss.data       * 6.0-7    2025-09-04 [1] CRAN (R 4.6.1)
#>  gamlss.dist       * 6.1-11   2026-09-10 [1] CRAN (R 4.6.1)
#>  ggcorrplot        * 0.3.0    2026-07-24 [1] CRAN (R 4.6.1)
#>  ggplot2           * 4.0.3    2026-04-22 [1] CRAN (R 4.6.1)
#>  ggtext            * 0.2.0    2026-08-28 [1] CRAN (R 4.6.1)
#>  gtsummary         * 2.6.1    2026-09-04 [1] CRAN (R 4.6.1)
#>  here              * 1.0.2    2025-09-15 [1] CRAN (R 4.6.1)
#>  Hmisc             * 5.3-0    2026-09-06 [1] CRAN (R 4.6.1)
#>  htmltools         * 0.5.9    2025-12-04 [1] CRAN (R 4.6.1)
#>  ImputeRobust      * 1.3-1    2018-11-30 [1] CRAN (R 4.6.1)
#>  iterators         * 1.0.14   2022-02-05 [1] CRAN (R 4.6.1)
#>  kableExtra        * 1.4.1    2026-07-08 [1] CRAN (R 4.6.1)
#>  knitr             * 1.52     2026-09-06 [1] CRAN (R 4.6.1)
#>  latex2exp         * 0.9.8    2026-01-09 [1] CRAN (R 4.6.1)
#>  lattice           * 0.23-1   2026-08-12 [1] CRAN (R 4.6.1)
#>  loo               * 2.10.1   2026-07-24 [1] CRAN (R 4.6.1)
#>  lubridate         * 1.9.5    2026-02-04 [1] CRAN (R 4.6.1)
#>  magrittr          * 2.0.5    2026-04-04 [1] CRAN (R 4.6.1)
#>  MASS              * 7.3-66   2026-07-15 [1] CRAN (R 4.6.1)
#>  Matrix            * 1.7-6    2026-07-25 [1] CRAN (R 4.6.1)
#>  MCMCpack          * 1.7-1    2024-08-27 [1] CRAN (R 4.6.1)
#>  mgcv              * 1.9-4    2025-11-07 [1] CRAN (R 4.6.0)
#>  mi                * 1.3.1    2026-07-28 [1] CRAN (R 4.6.1)
#>  mice              * 3.19.0   2025-12-10 [1] CRAN (R 4.6.1)
#>  miceadds          * 3.20-10  2026-05-28 [1] CRAN (R 4.6.1)
#>  miceFast          * 0.9.1    2026-02-26 [1] CRAN (R 4.6.1)
#>  missForest        * 1.6.1    2025-10-26 [1] CRAN (R 4.6.1)
#>  mitml             * 0.4-5    2023-03-08 [1] CRAN (R 4.6.1)
#>  mvtnorm           * 1.4-2    2026-07-12 [1] CRAN (R 4.6.1)
#>  nlme              * 3.1-171  2026-09-01 [1] CRAN (R 4.6.1)
#>  parallelly        * 1.48.0   2026-06-29 [1] CRAN (R 4.6.1)
#>  pbmcapply         * 1.5.1    2022-04-28 [1] CRAN (R 4.6.1)
#>  performance       * 0.18.2   2026-09-10 [1] CRAN (R 4.6.1)
#>  posterior         * 1.7.0    2026-04-01 [1] CRAN (R 4.6.1)
#>  ProfileLikelihood * 1.3      2023-08-25 [1] CRAN (R 4.6.1)
#>  progress          * 1.2.3    2023-12-06 [1] CRAN (R 4.6.1)
#>  purrr             * 1.2.2    2026-04-10 [1] CRAN (R 4.6.1)
#>  qqplotr           * 0.0.7    2025-09-05 [1] CRAN (R 4.6.1)
#>  quantreg          * 6.1      2025-03-10 [1] CRAN (R 4.6.1)
#>  randomForest      * 4.7-1.2  2024-09-22 [1] CRAN (R 4.6.1)
#>  Rcpp              * 1.1.2    2026-07-05 [1] CRAN (R 4.6.1)
#>  readr             * 2.2.0    2026-02-19 [1] CRAN (R 4.6.1)
#>  reshape2          * 1.4.5    2025-11-12 [1] CRAN (R 4.6.1)
#>  reticulate        * 1.47.0   2026-09-03 [1] CRAN (R 4.6.1)
#>  rmarkdown         * 2.32     2026-09-01 [1] CRAN (R 4.6.1)
#>  rms               * 8.1-1    2026-02-18 [1] CRAN (R 4.6.1)
#>  rstan             * 2.32.7   2025-03-10 [1] CRAN (R 4.6.1)
#>  showtext          * 0.9-8    2026-03-21 [1] CRAN (R 4.6.1)
#>  showtextdb        * 3.0      2020-06-04 [1] CRAN (R 4.6.1)
#>  SparseM           * 1.84-2   2024-07-17 [1] CRAN (R 4.6.1)
#>  StanHeaders       * 2.39.1   2026-09-02 [1] CRAN (R 4.6.1)
#>  Statamarkdown     * 1.0.0    2026-08-21 [1] CRAN (R 4.6.1)
#>  stringr           * 1.6.0    2025-11-04 [1] CRAN (R 4.6.1)
#>  svglite           * 2.2.2    2025-10-21 [1] CRAN (R 4.6.1)
#>  sysfonts          * 0.8.9    2024-03-02 [1] CRAN (R 4.6.1)
#>  texPreview        * 2.1.0    2024-01-24 [1] CRAN (R 4.6.0)
#>  tibble            * 3.3.1    2026-01-11 [1] CRAN (R 4.6.1)
#>  tidybayes         * 3.0.7    2024-09-15 [1] CRAN (R 4.6.1)
#>  tidyr             * 1.3.2    2025-12-19 [1] CRAN (R 4.6.1)
#>  tidyverse         * 2.0.0    2023-02-22 [1] CRAN (R 4.6.1)
#>  tinytex           * 0.60     2026-06-16 [1] CRAN (R 4.6.1)
#>  tryCatchLog       * 1.3.3    2025-12-04 [1] CRAN (R 4.6.0)
#>  VIM               * 7.3.1    2026-09-11 [1] CRAN (R 4.6.1)
#>  wesanderson       * 0.3.7    2023-10-31 [1] CRAN (R 4.6.1)
#>  xfun              * 0.60     2026-07-09 [1] CRAN (R 4.6.1)
#>  xtable            * 1.8-8    2026-02-22 [1] CRAN (R 4.6.1)
#>  yardstick         * 1.4.0    2026-04-07 [1] CRAN (R 4.6.1)
#> 
#>  [1] /opt/homebrew/lib/R/4.6/site-library
#>  [2] /opt/homebrew/Cellar/r/4.6.1/lib/R/library
#>  * ── Packages attached to the search path.
#> 
#> ─ External software ────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting        value
#>  cairo          1.18.4
#>  cairoFT        2.14.3/2.18.1
#>  pango
#>  png
#>  jpeg
#>  tiff
#>  tcl            9.0
#>  curl           8.7.1
#>  zlib           1.2.12
#>  bzlib          1.0.8, 13-Jul-2019
#>  xz             5.8.3
#>  deflate
#>  zstd           1.5.7
#>  PCRE           10.48 2026-08-31
#>  ICU            78.1
#>  TRE            TRE 0.8.0 R_fixes (BSD)
#>  iconv          Apple or GNU libiconv 1.11 /usr/lib/libiconv.2.dylib
#>  readline       8.3
#>  BLAS           /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib
#>  lapack         /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib
#>  lapack_version 3.12.1
#> 
#> ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> Quarto: 1.10.18
#> pandoc: 3.11
#> knitr engines: awk, bash, coffee, gawk, groovy, haskell, lein, mysql, node, octave, perl, php, psql, Rscript, ruby, sas, scala, sed, sh, stata, zsh, asis, asy, block, block2, bslib, c, cat, cc, comment, css, ditaa, dot, embed, eviews, exec, fortran, fortran95, go, highlight, js, julia, python, R, Rcpp, sass, scss, sql, stan, targets, tikz, verbatim, ojs, mermaid, glue, glue_sql, gluesql, details, texpreview

Stata Environment

set linesize 255
display %tcMonth_dd,CCYY_hh:MM_am now()
about
ado dir
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#>  September 13,2026 4:18 am
#> 
#> 
#> StataNow/MP 19.5 for Mac (Apple Silicon)
#> Revision 12 Aug 2026
#> Copyright 1985-2025 StataCorp LLC
#> 
#> Total physical memory: 48.01 GB
#> 
#> Stata license: Single-user 2-core , expiring  6 Feb 2027
#> Serial number: 501909358563
#>   Licensed to: Zad Rafi
#>                
#> 
#> 
#> [1] package parallel from http://fmwww.bc.edu/repec/bocode/p
#>       'PARALLEL': module for Parallel Computing
#> 
#> [2] package svalue from http://fmwww.bc.edu/RePEc/bocode/s
#>       'SVALUE': module for computing and graphically displaying S-values against their respective P-values
#> 
#> [3] package cifunction from http://fmwww.bc.edu/RePEc/bocode/c
#>       'CIFUNCTION': module for computing and graphically displaying all possible confidence intervals around a point estimate
#> 
#> [4] package getregstats from http://fmwww.bc.edu/RePEc/bocode/g
#>       'GETREGSTATS': module for computing all values in a regression table when only the coefficient and one other statistic is available
#> 
#> [5] package gr0072_2 from http://www.stata-journal.com/software/sj25-4
#>       SJ25-4 gr0072_2. Update: Nice axis labels for ...
#> 
#> [6] package grstyle from http://fmwww.bc.edu/repec/bocode/g
#>       'GRSTYLE': module to customize the overall look of graphs
#> 
#> [7] package palettes from http://fmwww.bc.edu/repec/bocode/p
#>       'PALETTES': module to provide color palettes, symbol palettes, and line pattern palettes

References

Back to top

References

1. Stark PB, Saltelli A. (2018). “Cargo-cult statistics and scientific crisis.” Significance. 15:40–43. doi: 10.1111/j.1740-9713.2018.01174.x.
2. Gigerenzer G. (2018). “Statistical Rituals: The Replication Delusion and How We Got There.” Advances in Methods and Practices in Psychological Science. 1:198–218. doi: 10.1177/2515245918771329.
3. Greenland S. (2017). “Invited commentary: The need for cognitive science in methodology.” American Journal of Epidemiology. 186:639–645. doi: 10.1093/aje/kwx259.
4. Greenland S. (2005). “Multiple-bias modelling for analysis of observational data.” Journal of the Royal Statistical Society Series A (Statistics in Society). 168:267–306. doi: 10.1111/j.1467-985X.2004.00349.x.
5. Greenland S, Lash TL. (2008). “Bias analysis.” In: Rothman KJ, Greenland S, Lash TL, editors. Modern Epidemiology. 3rd edition. Lippincott Williams & Wilkins. p. 345–380.
6. Lash TL, Fox MP, Fink AK. (2009). “Applying Quantitative Bias Analysis to Epidemiologic Data.” Springer New York.
7. Lash TL, Fox MP, MacLehose RF, Maldonado G, McCandless LC, Greenland S. (2014). “Good practices for quantitative bias analysis.” International Journal of Epidemiology. 43:1969–1985. doi: 10.1093/ije/dyu149.
8. Lash TL, Ahern TP, Collin LJ, Fox MP, MacLehose RF. (2021). “Bias Analysis Gone Bad.” American Journal of Epidemiology. doi: 10.1093/aje/kwab072.
9. Gustafson P. (2021). “Invited Commentary: Toward Better Bias Analysis.” American Journal of Epidemiology. doi: 10.1093/aje/kwab068.
10. Greenland S. (2021). “Dealing with the Inevitable Deficiencies of Bias Analysis and All Analyses.” American Journal of Epidemiology. doi: 10.1093/aje/kwab069.
11. Molenberghs G, Fitzmaurice G, Kenward MG, Tsiatis A, Verbeke G. (2014). “Handbook of Missing Data Methodology.” CRC Press.
12. Scott RaP. (2002). “The Multicentre Aneurysm Screening Study (MASS) into the effect of abdominal aortic aneurysm screening on mortality in men: A randomised controlled trial.” The Lancet. 360:1531–1539. doi: 10.1016/S0140-6736(02)11522-4.
13. Morris TP, Kahan BC, White IR. (2014). “Choosing sensitivity analyses for randomised trials: principles.” BMC Medical Research Methodology. 14:11. doi: 10.1186/1471-2288-14-11.
14. Mosteller F, Tukey JW. (1987). “Data analysis including statistics.” In: The Collected Works of John W. Tukey: Philosophy and Principles of Data Analysis 1965-1986. CRC Press.
15. Akacha M, Bretz F, Ohlssen D, Rosenkranz G, Schmidli H. (2017). “Estimands and Their Role in Clinical Trials.” Statistics in Biopharmaceutical Research. 9:268–271. doi: 10.1080/19466315.2017.1302358.
16. Permutt T. (2020). “Do Covariates Change the Estimand?” Statistics in Biopharmaceutical Research. 12:45–53. doi: 10.1080/19466315.2019.1647874.
17. Mitroiu M, Oude Rengerink K, Teerenstra S, Pétavy F, Roes KCB. (2020). “A narrative review of estimands in drug development and regulatory evaluation: Old wine in new barrels?” Trials. 21:671. doi: 10.1186/s13063-020-04546-1.
18. Little RJ, D’Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, et al. (2012). “The Prevention and Treatment of Missing Data in Clinical Trials.” New England Journal of Medicine. 367:1355–1360. doi: 10.1056/NEJMsr1203730.
19. Chen D-G(Din), Peace KE, Zhang P. (2017). “Clinical trial data analysis using r and SAS.” 2nd edition. Chapman; Hall/CRC. doi: 10.1201/9781315155104.
20. Buuren S van. (2018). “Flexible Imputation of Missing Data, Second Edition.” CRC Press.
21. Meng X-L, Rubin DB. (1992). “Performing Likelihood Ratio Tests with Multiply-Imputed Data Sets.” Biometrika. 79:103–111. doi: 10.2307/2337151.
22. Schenker N, Taylor J. (1996). “Partially parametric techniques for multiple imputation.” Computational Statistics & Data Analysis. 22:425–446. doi: 10.1016/0167-9473(95)00057-7.
23. Permutt T. (2016). “Sensitivity analysis for missing data in regulatory submissions.” Statistics in Medicine. 35:2876–2879. doi: 10.1002/sim.6753.
24. Jolani S. (2012). “Dual Imputation Strategies for Analyzing Incomplete Data.”
25. von Hippel PT, Bartlett J. (2019). “Maximum likelihood multiple imputation: Faster imputations and consistent standard errors without posterior draws.” arXiv:12100870 [stat]. https://arxiv.org/abs/1210.0870.
26. Rafi Z, Greenland S. (2020). “Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise.” BMC Medical Research Methodology. 20:244. doi: 10.1186/s12874-020-01105-9.
27. Greenland S, Rafi Z. (2020). “To Aid Scientific Inference, Emphasize Unconditional Descriptions of Statistics.” arXiv:190908583 [statME]. https://arxiv.org/abs/1909.08583.

Citation

BibTeX citation:
@online{rafi2020,
  author = {Rafi, Zad and Panda, Sir},
  title = {What {Makes} a {Sensitivity} {Analysis?}},
  date = {2020-12-12},
  url = {https://lesslikely.com/statistics/sensitivity.html},
  langid = {en-US}
}
]]> statistics https://lesslikely.com/statistics/sensitivity.html Sat, 12 Dec 2020 05:00:00 GMT Medicine Is Being Treated with Snake-Oil Statistics Zad Rafi Andrew Gelman Aleksi Reito Sander Greenland https://lesslikely.com/statistics/snakeoilstats.html Suggested Running Head: Snake Oil Statistics

Statistics has helped medicine move away from an eminence-based framework, where subject-matter experts decided what worked and what didn’t, towards an evidence-based one. Among the techniques deployed, null-hypothesis statistical testing (NHST) is the most common, where the word “null” is invariably taken by users to mean that the only hypotheses tested are those of “no association” or “no effect” (sometimes labeled “nil hypotheses”). Despite objections to them extending throughout the past century1, these tests are supposed to serve as a safeguard against researchers fooling themselves and others, as well as serving as a central component of experimental design and analysis.

Unfortunately, these tests are easily subverted into playing the reverse role of providing a badge of approval, allowing researchers to make stronger claims than warranted by valid statistical analyses. Confidence intervals have been extensively promoted to address this problem, but they too have been subverted by being treated as if they are only testing null hypotheses. The consequences for evidence-based medicine have been dire.

There are three commonly stated principles (shown in Figure 1) of evidence-based medicine2:

  1. reliance on statistically significant results (and thus NHST) from randomized controlled trials,
  2. balancing of costs, benefits, and uncertainties in decision making, and
  3. combining clinical expertise with external evidence to tailor treatments for individuals.

Figure 1

Unfortunately, the use of NHST can get in the way of the movement toward an evidence-based framework. This may sound paradoxical, given that one of the foundations of evidence-based medicine is hypothesis testing based on randomized controlled trials, deemed by many to be the most reliable forms of evidence2. One problem is that reliance on statistical significance (principle 1) may conflict with the other principles: In conflict with balancing costs and benefits under uncertainty (principle 2), statistical significance or non-significance is typically used to replace uncertainty with certainty3, 4 — indeed, researchers are encouraged to do this and it is even forced by some journals5.

Reliance on statistical significance can interact badly with expertise and background evidence (principle 3), often leading to incoherent attempts at resolution. For example, researchers may do separate analyses for men and women, often with no biological basis for expecting other than small differences in effects (if any). In doing so they often find that one group (usually the larger one, which is usually men) show a “significant” effect while the other does not. They then misreport this difference in “significance” as if it represented a significant difference in effect between groups when in fact it only reflects a difference in the group sizes, plus random differences in group-specific P-values. Such misinterpretations fool researchers into thinking that the data support targeting treatments at the group showing “significance” and mislead clinicians into tailoring treatment plans for individuals based on nothing more than random variation.

The unfortunate reality is that estimating effects for individuals or population subsets requires far more sophistication than basic testing procedures6. It can take several times more patients to estimate variation in effects (“interactions”) than average effects7. Given that few studies are large enough to estimate main effects of interest, it will typically be impossible to obtain reliable estimates of effect variation even if the study is otherwise flawless. That problem should be dealt with under principle (2) by recognizing that subgroups will have very imprecise estimates, leading to enormous uncertainties about who if anyone should be targeted for or excluded from treatment. Unfortunately, the prevalent misunderstandings of classical statistics have trained most researchers, editors, and reviewers to demand statistical significance and certainty as a prerequisite for publication8 and decision making. Applying those demands within subgroups all but guarantees that distorted impressions of patient-specific effects will follow.

Through neglect of basic education, the statistics profession is partly responsible for these issues, but there is nothing new about medicine’s desire for certainty. That desire is part of human nature and thus existed long before the adoption of statistical methods. The resulting demands for certainty in reported results have provided ample opportunities for overconfident researchers to rise to prominence. Physicians have long capitalized on several opportunities with their sophisticated knowledge of physiology and anatomy, and used their medical authority to argue what treatments worked, shape public policy, and design clinical guidelines, maintaining a form of medicine characterized as “eminence based.”

The landscape changed when advances in quantitative methods eventually reached medicine9, leading to a newfound demand for scientific rigor and making many individuals wary of the claims of subject-matter experts10. These demands may be some of the biggest contributors to medicine’s adoption of statistical methods and its movement towards an evidence-based framework. Unfortunately, in its pursuit for objectivity, medicine was sold snake oil statistics and as a result, its desire for objectivity and rigor backfired.

Figure 2: Snake oil statistics. Methodological interventions that have been oversold by early adopters as cure alls despite no supporting evidence of utility in a particular application.

To see the reach of these problems, one simply needs to look wherever a new drug has been considered effective only if it has been shown statistically significantly better than a control in one or more randomized clinical trials (a standard that applies to FDA new-drug approvals, though not to medical devices, which can be cleared through other pathways). The largest enforcers of these methods have been regulatory agencies that wish to minimize treatments that do not work (false positives) and minimize adverse events from treatments. One should ask: what specifically convinced these agencies that statistical significance and in particular NHST was the best analysis criterion to achieve these goals?

In a nonexistent ideal world, the regulatory agencies looked at several statistical methods, tested each of them in many settings to see how well they identified true benefits and harms while avoiding false conclusions, and from that determined NHST performed best out of all options – with periodic updates as new methods appeared. Of course, history shows something else entirely, even indicating that adoption of NHST was mainly a result of political desperation11, 12. To understand the harsh reality, we must go back to the mid-20th century, when pharmaceutical companies submitted new drug applications to the FDA that often lacked any protocols and statistical analysis plans, making the entire drug approval process chaotic. By the end of the 1960s, the FDA had become desperate to standardize the drug-approval process and make it scientifically rigorous, especially with impending pressures from the Drug Amendments of 1962, which demanded rigorous evidence for drug approval.

At the same time, applied researchers were looking for rigorous ways to summarize experiments with numerical quantities. Their search struck gold with the works of the prominent statisticians, Ronald A. Fisher13–15, Jerzy Neyman16, and Egon Pearson17, who created and popularized powerful statistical tools for researchers that lacked statistical training. Eventually, researchers across many disciplines adopted NHST and the now-infamous 0.05 cutoff. Soon, the FDA followed and incorporated these methods into its regulatory process, without any formal debate about the methods’ utility or evidence12.

Figure 3: Austin Bradford Hill. The statistician and epidemiologist who conducted the first randomized controlled trial in medicine.

It thus appears that embracement of the NHST paradigm arose from expediently following the ascendant trend in research rather than a critical evaluation of various options emerging in the same period. Since its adoption into the regulatory process, this framework has been rigorously enforced by the FDA as the gold standard in the approval process.

We find it ironic that the gatekeepers of evidence-based medicine — regulatory agencies, journal editors and reviewers, and medical researchers – continue to insist on enforcement of a statistical framework that was never critically examined for its utility in medicine12. This failure makes it understandable why it can be credibly argued that conventional statistical methods have done more harm than good in medical science. These methods, including null-hypothesis tests and so-called “confidence” intervals are a set of decision-making tools designed for tightly controlled randomized experiments in which the treatment effects (if any) can be distinguished from all other causal effects, and can be distinguished from random error (noise) by increasing the study size in an affordable manner. Examples of such scenarios arise in agricultural research where the experimental units are plants or plots, and industrial quality control, where the experimental unit is a part or product. The tools were originally developed for these environments, in which (compared to clinical studies) experimenters have almost godlike control over the selection of units and their subsequent experiences, aided by having a rather short follow-up period and low cost per experimental unit. And then, the decisions to be made are relatively simple and easily monitored, e.g., change a fertilizer formulation or manufacturing tolerance17. Experimental psychology is similar in terms of control, cost, and even lower cost-benefit consequences.

In clinical environments, however, the cost per experimental unit (the patient) is far higher, creating severe limits on study size and thus noise reduction, and the costs and benefits of decisions can be enormous. At the same time, there is far less control of extraneous selection and confounding effects: Physicians and their patients can and do selectively refuse to participate or cease to adhere to assigned treatment protocols, and may drop out for unknown reasons. Meanwhile, direct physical control of the patient environment is extremely limited or nonexistent, especially when follow-up extends beyond hospital stay. And then, amplifying these limits, the required decisions are often complex and of highly uncertain consequence, yet may be pivotal to the experimental results and clinical decisions, e.g., when to withdraw treatment from patients apparently experiencing side effects or when to switch treatments for nonresponsive patients.

These vast differences haven’t stopped medical researchers from using conventional methods as if they were operating in a tightly controlled environment, treating ambiguous trial results as if they supported decisive conclusions despite obvious uncertainties and potentially devastating consequences. The usual depictions of this problem involves researchers trying to “game” statistical methods so that they show “significant” effects18, 19; while such “significance questing” is a real problem, warnings about it usually ignore or dismiss opposite behavior in which showing “nonsignificance” will facilitate publication in prestigious medical journals, especially where so-called “replication failure” has become a hot topic3. Adding to these distortions is the publication bias that results when researchers or editors deem results unworthy of submission or acceptance because they are just not interesting because they fail to report a “discovery” or only replicate what is “known.” This desire for novelty or publicity, along with demands for certainty, has distorted the medical literature with dubious, inflated effects20 and misleading claims of replication failure based on fundamentally ambiguous results21, 22.

Responses to the problems

In response to the ongoing abuse of statistical testing, the American Statistical Association released a statement in 2016 cautioning against the fixating on P-values and statistical significance23. Three years later, the organization published an issue titled “A World Beyond P < 0.05” with 43 commentaries from statistical experts on how to improve statistical inference, with or without P-values24. The issue was accompanied by a highly discussed commentary in Nature titled “Scientists Rise up Against Statistical Significance” that discouraged mindless automation and dichotomization of statistical results21, and was supported by signatures of some 800 applied researchers.

While these calls are impressive and a growing number of journals — including the New England Journal of Medicine and, in modified form, JAMA — have updated their statistical-reporting policies, most regulatory agencies and the bulk of the medical literature continue to fixate on statistical significance5, 25. Correspondingly, we can expect medical researchers to keep cutting corners to achieve or remove statistical significance18, 19 or misinterpret ambiguous results as if definitive, a practice that is often labeled as “spin”26.

We can interpret the persistence of null hypothesis significance testing (NHST) in two ways. One story is that the value of NHST is recognized by real-world decision makers, despite the carping of ivory-tower critics such as the authors of the present article. The other story is that the counterproductive nature of NHST has been denied by the medical establishment5, 25, and that better alternatives are available12. There are legitimate practical concerns behind both these perspectives. On one hand, active researchers and regulators have legitimate concerns about working on or approving treatments that do not work. On the other hand, many well-publicized examples have made it clear that NHST can easily lead to overconfidence and erroneous inferences, as seen in discussions that treat statistically significance as demonstrating presence of effects and nonsignificance as demonstrating absence of effects.

Methodologists propose new methods to address NHST in their field

Fear of false positives has dominated most discussions of scientific rigor18, 27, 28, often relegating false negatives to more technical discussions of power. Although false positives can be costly, so can missing clinically meaningful effects21. Take postmarketing surveillance for pharmaceuticals; in such situations, serious adverse events from drugs are often underreported and have in some cases been actively suppressed29, 30, as seen in Cochrane reviews of unpublished trial data, leading to a scarcity of data and resulting in studies that will never be able to show a “significant” effect because of the lack of resources and time. If a study comparing adverse events from those taking an approved drug and some control group is unable to show statistical significance, even when the estimated effect is important and plausible, the results will typically be confused and used for evidence of absence21, 31, and prescribing is unlikely to be curtailed, leading to continued harm.

The confusion of statistical nonsignificance with evidence of absence will remain a problem in fields where effects are often small and yet studies powered to detect them are infeasible. Most studies in surgical science rarely have more than a dozen participants. Again, this makes it incredibly difficult to achieve statistical significance, even when clinically important effects are likely. In such fields, adopting the NHST framework for analysis all but guarantees failure to detect those effects. As a result, researchers have looked to other methods — including Bayes factors, second-generation P-values, and post-hoc power calculations — to circumvent the bad hand that was dealt to them32–34. Unfortunately, such transitions are not always helpful: while reasonably defensible alternatives such as interval testing and posterior intervals do exist, several of the most-publicized proposals introduce their own errors, and in fact, some may actually result in more errors than before.

For example, a group of surgeons recently published several statistical recommendations on what to do if a surgical study result was nonsignificant35–37, providing hope to researchers who have had their inquiries halted by that result. But the recommendations are not only statistically invalid and clinically misleading, and thus have been discouraged by statisticians for nearly two decades38, 39. In a similar tale, two sports scientists published a statistical method40 which was supposed to improve the classification of individual treatment responses by reducing the influence of random error in small studies. Unfortunately, reducing the influence of random error in a valid manner requires improvement of study design, including increase in study size, so unsurprisingly the proposed method has unacceptably poor statistical properties41.

The efforts behind new methods and recommendations are a response to an arbitrary dichotomy that has been imposed upon researchers by the medical and scientific establishment: decide whether a result is “positive” or “negative”, with no allowance for what is usually the most reasonable interpretation – ambiguous or indecisive. Unfortunately, conventional statistics has aggravated this problem by offering methods like NHST which produce only dichotomous answers. These methods which swept research sciences in the mid-20th century and are now firmly rooted in tradition, as if that tradition is the best we can do. But it isn’t.

The problem with upending this tradition however is that there is no consensus about what to replace it with, a problem that only grows as alternatives continue to proliferate. A idealistic (and we think naïve) view would simply allow authors to choose alternatives of their liking. The problem with this anarchic approach is that many of the alternatives are themselves misleading and even defective or at best inferior to other methods in demonstrable ways. Yet seeing these problems requires not only sufficient technical expertise to evaluate methods, but also a willingness to see them – a willingness that should not be assumed for originators and adopters of the method. Unfortunately, mere publication of a method in a peer-reviewed journal is a faulty indicator of method validity. That is especially so if publication is not in a statistics journal, for in that case it means only that peer-review was by referees who may have had little of the special technical expertise needed for thorough evaluation.

Solutions

There are no simple solutions or universal guidelines that medical researchers can always use to improve scientific rigor within their area of work. We nonetheless offer some recommendations and resources that we believe may be useful to those who recognize problems in their field and wish for some sort of guidance:

  1. Collaborate with well-qualified statisticians to design and conduct studies that are rigorous, efficient, and cost effective42, 43. Look to these statistical collaborators for guidance about honestly reporting uncertainty, not certainty19.
  2. If possible, pool resources to run larger studies that are likely to be more precise and informative than individual studies, which may simply waste resources and offer little yield44. However, more data is not synonymous with more information: Increasing study size may be detrimental if it entails a reduction in data quality45.
  3. Aim to be more descriptive and less inferential22. Accept uncertainty and the anxiety that comes with it4, along with the idea that no one study warrants conclusions about the true nature of a phenomenon, a delusion that is not even true in particle physics! And if a phenomenon is subtle, even several studies may be insufficient for valid conclusions beyond “more research is needed.”
  4. When making real world decisions, use all information available to you, balancing costs, benefits, and uncertainties46, rather than basing decisions on whether a single numerical value is above or below an arbitrary cutoff.
  5. Be mindful of cognitive biases that may distort your conclusions throughout the study and the many cognitive biases that will afflict you, your colleagues, and your collaborators3, 47, 48.
  6. Use statistical methods that have been reviewed and validated by the statistical community beyond their developers and promoters. That validation is provided not only by publication in statistical journals, but also by applications that can be judged as having reached sound, well-cautioned conclusions in context. Especially, beware of any method that (like NHST) claims to offer firm conclusions based on purely numeric comparisons.
Back to top

References

1. Boring EG. (1919). “Mathematical vs. Scientific significance.” Psychological Bulletin. 16:335–338. doi: 10.1037/h0074554.
2. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. (1996). “Evidence based medicine: What it is and what it isn’t.” BMJ (Clinical research ed). 312:71–72. doi: 10.1136/bmj.312.7023.71.
3. Greenland S. (2017). “Invited commentary: The need for cognitive science in methodology.” American Journal of Epidemiology. 186:639–645. doi: 10.1093/aje/kwx259.
4. McShane BB, Gal D, Gelman A, Robert C, Tackett JL. (2019). “Abandon Statistical Significance.” The American Statistician. 73:235–245. doi: 10.1080/00031305.2018.1527253.
5. Bauchner H, Golub RM, Fontanarosa PB. (2019). “Reporting and Interpretation of Randomized Clinical Trials.” Journal of the American Medical Association. 322:732–735. doi: 10.1001/jama.2019.12056.
6. Senn S. (2018). “Statistical pitfalls of personalized medicine.” Nature. 563:619–621. doi: 10.1038/d41586-018-07535-2.
7. Greenland S. (1983). “Tests for interaction in epidemiologic studies: A review and a study of power.” Statistics in Medicine. 2:243–251. doi: 10.1002/sim.4780020219.
8. Rosenthal R. (1979). “The file drawer problem and tolerance for null results.” Psychol Bull. 86:638. doi: 10.1037/0033-2909.86.3.638.
9. Council MR. (1948). “Streptomycin Treatment of Pulmonary Tuberculosis.” British Medical Journal. 2:769–782.
10. Goodman SN. (2019). “Why is Getting Rid of P-Values So Hard? Musings on Science and Statistics.” The American Statistician. 73:26–30. doi: 10.1080/00031305.2018.1558111.
11. Kennedy-Shaffer L. (2017). “When the Alpha is the Omega: P-Values, ‘Substantial Evidence,’ and the 0.05 Standard at FDA.” Food Drug Law J. 72:595–635.
12. Ruberg SJ, Jr FEH, Gamalo-Siebers M, LaVange L, Lee JJ, Price K, et al. (2019). “Inference and Decision Making for 21st-Century Drug Development and Approval.” The American Statistician. 73:319–327. doi: 10.1080/00031305.2019.1566091.
13. Fisher R. (1955). “Statistical Methods and Scientific Induction.” Journal of the Royal Statistical Society Series B (Methodological). 17:69–78. doi: 10.1111/j.2517-6161.1955.tb00180.x.
14. Fisher RA. (1925). “Statistical Methods for Research Workers.” Oliver and Boyd: Edinburgh.
15. Fisher RA. (1935). “The Design of Experiments.” Oxford, England: Oliver & Boyd.
16. Neyman J, Pearson ES. (1933). “On the Problem of the Most Efficient Tests of Statistical Hypotheses.” Philosophical Transactions of the Royal Society of London Series A, Containing Papers of a Mathematical or Physical Character. 231:289–337. doi: 10.1098/rsta.1933.0009.
17. Pearson ES. (1933). “A Survey of the Uses of Statistical Method in the Control and Standardization of the Quality of Manufactured Products.” Journal of the Royal Statistical Society. 96:21–75. doi: 10.2307/2341869.
18. Simmons JP, Nelson LD, Simonsohn U. (2011). “False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.” Psychological Science. 22:1359–1366. doi: 10.1177/0956797611417632.
19. Wang MQ, Yan AF, Katz RV. (2018). “Researcher requests for inappropriate analysis and reporting: A U.S. Survey of consulting biostatisticians.” Annals of Internal Medicine. 169:554. doi: 10.7326/M18-1230.
20. Dickersin K, Berlin JA. (1992). “Meta-analysis: State-of-the-Science.” Epidemiologic Reviews. 14:154–176. doi: 10.1093/oxfordjournals.epirev.a036084.
21. Amrhein V, Greenland S, McShane B. (2019). “Scientists rise up against statistical significance.” Nature. 567:305. doi: 10.1038/d41586-019-00857-9.
22. Amrhein V, Trafimow D, Greenland S. (2019). “Inferential statistics as descriptive statistics: There is no replication crisis if we don’t expect replication.” The American Statistician. 73:262–270. doi: 10.1080/00031305.2018.1543137.
23. Wasserstein RL, Lazar NA. (2016). “The ASA Statement on p-Values: Context, Process, and Purpose.” The American Statistician. 70:129–133. doi: 10.1080/00031305.2016.1154108.
24. Wasserstein RL, Schirm AL, Lazar NA. (2019). “Moving to a world beyond ‘p 0.05’.” The American Statistician. 73:1–19. doi: 10.1080/00031305.2019.1583913.
25. Harrington D, D’Agostino RB, Gatsonis C, Hogan JW, Hunter DJ, Normand S-LT, et al. (2019). “New guidelines for statistical reporting in the journal.” New England Journal of Medicine. 381:285–286. doi: 10.1056/NEJMe1906559.
26. Khan MS, Lateef N, Siddiqi TJ, Rehman KA, Alnaimat S, Khan SU, et al. (2019). “Level and Prevalence of Spin in Published Cardiovascular Randomized Clinical Trial Reports With Statistically Nonsignificant Primary Outcomes: A Systematic Review.” JAMA Network Open. 2:e192622–e192622. doi: 10.1001/jamanetworkopen.2019.2622.
27. Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers E, Berk R, et al. (2017). “Redefine statistical significance.” Nature Human Behaviour. 2:6–10. doi: 10.1038/s41562-017-0189-z.
28. Ioannidis JPA. (2018). “The Proposal to Lower P Value Thresholds to .005.” JAMA. 319:1429–1430. doi: 10.1001/jama.2018.1536.
29. Vandenbroucke JP, Psaty BM. (2008). “Benefits and risks of drug treatments: How to combine the best evidence on benefits with the best data about adverse effects.” JAMA. 300:2417–2419. doi: 10.1001/jama.2008.723. https://doi.org/10.1001/jama.2008.723.
30. Doshi P, Jefferson T, Del Mar C. (2012). “The imperative to share clinical study reports: Recommendations from the Tamiflu experience.” PLoS Medicine. 9:e1001201. doi: 10.1371/journal.pmed.1001201. https://doi.org/10.1371/journal.pmed.1001201.
31. Altman DG, Bland JM. (1995). “Absence of evidence is not evidence of absence.” BMJ (Clinical research ed). 311:485. doi: 10.1136/bmj.311.7003.485.
32. Gigerenzer G, Marewski JN. (2015). “Surrogate Science: The Idol of a Universal Method for Scientific Inference.” Journal of Management. 41:421–440. doi: 10.1177/0149206314547522.
33. van Ravenzwaaij D, Monden R, Tendeiro JN, Ioannidis JPA. (2019). “Bayes factors for superiority, non-inferiority, and equivalence designs.” BMC Medical Research Methodology. 19:71. doi: 10.1186/s12874-019-0699-7.
34. Simonsohn U, Nelson LD, Simmons JP. (2014). “P-Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results.” Perspect Psychol Sci. 9:666–681. doi: 10.1177/1745691614553988.
35. Bababekov YJ, Stapleton SM, Mueller JL, Fong ZV, Chang DC. (2018). “A Proposal to Mitigate the Consequences of Type 2 Error in Surgical Science.” Annals of Surgery. 267:621. doi: 10.1097/SLA.0000000000002547.
36. Bababekov YJ, Hung Y-C, Hsu Y-T, Udelsman BV, Mueller JL, Lin H-Y, et al. (2019). “Is the Power Threshold of 0.8 Applicable to Surgical Science? The Underpowered Study.” Journal of Surgical Research. 241:235–239. doi: 10.1016/j.jss.2019.03.062.
37. Bababekov YJ, Chang DC. (2019). “Post Hoc Power: A Surgeon’s First Assistant in Interpreting ‘Negative’ Studies.” Annals of Surgery. doi: 10.1097/SLA.0000000000002914.
38. Greenland S. (2012). “Nonsignificance plus high power does not imply support for the null over the alternative.” Annals of Epidemiology. 22:364–368. doi: 10.1016/j.annepidem.2012.02.007.
39. Hoenig JM, Heisey DM. (2001). “The Abuse of Power.” The American Statistician. 55:19–24. doi: 10.1198/000313001300339897.
40. Dankel SJ, Loenneke JP. (2019). “A Method to Stop Analyzing Random Error and Start Analyzing Differential Responders to Exercise.” Sports Medicine. doi: 10.1007/s40279-019-01147-0.
41. Tenan M, Vigotsky AD, Caldwell AR. (2019). “On the Statistical Properties of the Dankel-Loenneke Method.” SportRxiv. doi: 10.31236/osf.io/8ndhg.
42. Gelman A, Carlin J. (2014). “Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors.” Perspectives on Psychological Science. 9:641–651. doi: 10.1177/1745691614551642.
43. Rothman KJ, Greenland S. (2018). “Planning study size based on precision rather than power.” Epidemiology. 29:599–603. doi: 10.1097/EDE.0000000000000876.
44. Moshontz H, Campbell L, Ebersole CR, IJzerman H, Urry HL, Forscher PS, et al. (2018). “The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network:” Advances in Methods and Practices in Psychological Science. doi: 10.1177/2515245918797607.
45. Cox DR, Donnelly CA. (2011). “Principles of Applied Statistics.” Cambridge University Press.
46. Parmigiani G, Inoue L. (2009). “Decision Theory: Principles and Approaches.” John Wiley & Sons.
47. Gigerenzer G. (2004). “Mindless statistics.” The Journal of Socio-Economics. 33:587–606. doi: 10.1016/j.socec.2004.09.033.
48. Stark PB, Saltelli A. (2018). “Cargo-cult statistics and scientific crisis.” Significance. 15:40–43. doi: 10.1111/j.1740-9713.2018.01174.x.

Citation

BibTeX citation:
@online{rafi2020,
  author = {Rafi, Zad and Rafi, Zad and Gelman, Andrew and Reito, Aleksi
    and Greenland, Sander},
  title = {Medicine {Is} {Being} {Treated} with {Snake-Oil}
    {Statistics}},
  date = {2020-11-11},
  url = {https://lesslikely.com/statistics/snakeoilstats.html},
  langid = {en-US}
}
]]> statistics https://lesslikely.com/statistics/snakeoilstats.html Wed, 11 Nov 2020 05:00:00 GMT Quality Control in Statistical Analyses Zad Rafi https://lesslikely.com/statistics/statistical-quality-control.html

Experienced statisticians and data analysts are familiar with stories where a coding error has led to an entire conclusion changing, or even a retraction.[@aboumatarNoticeRetractionAboumatar2019] It’s the sort of stuff that keeps people up at night. Unfortunately, not many of us think about these sorts of scenarios until we realize it’s very possible that it could happen to any of us.

To me, it seems that many of these issues could be avoided by having a principled data management and statistical workflow, and making it as transparent, open, and reproducible as possible. I’d like to quickly go over a few things that I’ve found helpful over the years, and I’ll first start with data management and data entry and then move onto analysis workflows. I largely consider this to be a living document, and I’m sure many people who will read this will have far better suggestions, so please leave them down below in the comments!

Before I go on, I want to emphasize that backing up your data, scripts, and using version control is extremely important. There is no debate about this. It’s necessary so that other collaborators/colleagues can inspect your work and catch potential mistakes or see overall progress, but more importantly, it will prevent you from losing your data in a disaster, and it’ll help you catch your own mistakes, since you’ll be the most familiar with the data and scripts.

Data Entry & Management

A nice paper that I’d like to review is the one by Broman & Woo, 2018 on how to manage your data when working with spreadsheets.[@bromanDataOrganizationSpreadsheets2018] The sad reality is that even though spreadsheets like Microsoft Excel or Google Sheets are available everywhere, and easy to use, there are many risks when working with spreadsheets, just ask any statistician who works in genetics or any bioinformatician.[@ziemannGeneNameErrors2016]

One of the most fatal errors occurred recently when a group of researchers lost thousands of documented COVID cases because they entered data for each case as a column instead of a row, and Excel has a limit on how many columns and rows it can handle (1,048,576 rows and 16,384 columns, according to Microsoft), as a result, most of these cases were lost, resulting in an enormous waste of resources due to a careless and ignorant mistake, highlighting the dangers of recklessly inputting data and conducting statistical analyses. There is no doubt that reviewing principles of good data management and workflow are essential to any data analyst. I’d like to touch on some of the most important points of Broman & Woo, 2018 paper before moving onto some other “principles” I’d like to share:

General Principle: Be Consistent With Everything

  • When labeling missing values (NA, N/A, etc.):

    • Good: Choose one method (NA, N/A, etc.) and stick with it.

    • Bad: Leaving cells empty to indicate missingness.

  • When labeling response variables:

    • Good: If you label a response variable consistently, for example, fat-free-mass in every instance to refer to the same variable.

    • Bad: If you label a response variable fat-free-mass in one script/sheet, and ffm/fat_free_mass in another.

  • When formatting variable predictor names:

    • Good: predictor_1_week12, predictor_2_week12, response_variable_week12. This is consistent, with the order of the names, and dates/weeks, so it is easier to organize, inspect, and clean. Same thing applies to labeling response variables and pretty much all variables.

    • Bad: When you label one variable week12_predictor_1, the next predictor_2_week12, and the last, 12_week_response. Just no.

  • When formatting dates:

    • Good: Use one format, YYYY-MM-DD, consistently.

    • Bad: Everything else.

  • When filling out cells:

    • Good: One cell = one thing. No more, no less.

    • Bad: One cell = multiple entries or no entries at all.

  • When keeping track of the data:

    • Good: Create a comprehensive data dictionary so anyone can look at it and understand the spreadsheets/dataframes/variables.

    • Bad: Expecting yourself and others to figure it out based on the variable names that you thought were brilliant.

  • When using Excel or Google Sheet’s shiny features:

    • Good: Avoid using any of them, any formulas, highlighting, italicizing, bolding, etc.

    • Bad: If you’re the type of person to create charts in Excel.

  • Planning for disasters:

    • Good: Always, always, backup your files, save them in .txt files. Keep backups of those. And most importantly, use version control!

    • Bad: You don’t really plan for disasters and go with the flow.

  • Last but not least, when entering cases:

    • Good: Rows are for cases and observations, and columns are for variables and characteristics. Please do not switch these up!

    • Bad: Using them interchangeably.

Setting Up a Principled Statistical Workflow

Here’s what I’ve been doing for many years and what seems to work for me (none of these ideas are originally mine, and I actually picked them up over the years from others’ advice, which will be linked below).

Setting up the Project Folder

When setting up a folder specific for a project (if you’re not doing this, you absolutely need to), first, I will create a folder with the title of my project, and usually have an RStudio project set up in there.

Disclaimer: while much of this advice will be familiar and easy to understand for those who use R, I think the general principles are widely applicable, especially for those who use scripts in their statistical software).

This folder will contain many other folders (more on that below), so the structure will end up being a bit complex.

workflow

It may be a bit difficult to read the words within the circles, but clicking on the diagram should enlargen it and allow you to zoom in, giving you a sense of how I organize all these scripts and files.

If this ends up confusing you, you can just scroll all the way down to see an image of what the folder structure looks like.

Instead of constantly changing the working directory to each subfolder when I need to do something inside that subfolder for a particular project or analysis, I set the project folder as the working directory only once, and then run the following:

# Very seriously consider installing the `here` package

# https://cran.r-project.org/package=here

library("here")

# This and the scripts below take
# the directory used and set it as the top

set_here()

here()

This not only sets the working directory, but gives you far more control over how you can save your files from any place within the hierarchical folder structure. It will create a file called .here inside the Main Project folder, which will indicate that this is the top level of the hierarchy.

Then, I will usually have created a README.Rmd file in the Main Project folder with updates on what I’ve done and what I still need to work on, so I will remember and so my collaborators can see it too (when it has been pushed to GitHub or other version control repos).

Next, I’ll set up a Data folder inside the Main Project folder. This is where all my data/spreadsheets/.txt files and data dictionaries will go. The original data files will stay inside this folder, while I create two other subsubfolders inside this Data folder called Transformed and Models. In Transformed, I will typically save .rds files that were a result of cleaning and transforming the data, including imputing missing data. I will touch more on exactly how I do that later below. A Models subfolder will obviously store fitted models and any validation/sensitivity analyses of those models.[@morrisChoosingSensitivityAnalyses2014]

Now, back to the Main Project folder, I’ll set up another folder within it called R. This will be where all of my .R scripts/files live. I will number them sequentially along with a title for a specific purpose, like so

  • main.R

  • 01-functions.R

  • 02-cleaning.R

  • 03-inspection.R

  • 04-analysis.R

  • 04.5-validation.R

  • 05-sensitivity.R

  • 06-tables.R

Setting up the Functions Script

All potential R packages and custom functions that I need will belong in 01-functions.R. This .R file will only serve this purpose and nothing else. As you’ve guessed by now, the other .R files will be doing the same, they have very specific purposes and are organized to reflect this.

An example of the first script can be found below.

# Necessary Packages ------------------------------------

req_packs <- c("rms", "concurve", "mice",
               "tidyverse", "parallel",
               "bayesplot", "projpred",
               "Hmisc", "loo", "rstan", "here",
               "tryCatchLog", "futile.logger",
               "patchwork", "corrplot", "beepr",
               "summarytools", "broom", "wakefield",
               "boot", "mfp", "knitr", "flextable",
               "MASS", "lme4", "brms", "miceMNAR")

# Load all packages at once

lapply(req_packs, library,
       character.only = TRUE)

# Loggings messages
options(keep.source = TRUE)
options("tryCatchLog.write.error.dump.file" = TRUE)

# Set seed for random number generator

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed(1031) # My birthday

# RStan Settings  -------------------------------------

theme_set(theme_bw())
color_scheme_set("red")
rstan_options(auto_write = TRUE)
options(mc.cores = 4)

has_build_tools(debug = TRUE)

# Stan Settings  ---------------------------------------

dotR <- file.path(Sys.getenv("HOME"), ".R")
if (!file.exists(dotR)) dir.create(dotR)
M <- file.path(dotR, "Makevars")
if (!file.exists(M)) file.create(M)
cat("\nCXX14FLAGS=-O3 -march=native -mtune=native -fPIC",
  "CXX14=g++",
  file = M, sep = "\n", append = TRUE)

The script above loads all the necessary R packages each time it is called, along with the specified functions, so most of the other .R files will depend upon this one.

However, I will not be running each of these .R files/scripts individually, line by line, or by selecting all the lines in an .R file and running the script.

Instead, after I’ve created all these files (figuring out what I need to do carefully and writing it down and annotating it), every single .R file except for main.R will have the following script, which I will explain later below (or some iteration of this script to match the name of the file) at the beginning:

Setting up the Error Catching Script

library("here")

set_here()

source(here("R", "01-functions.R"))

library(futile.logger)
library(tryCatchLog)

# Keeps source code file name and line number tracking
options(keep.source = TRUE)
options("tryCatchLog.write.error.dump.file" = TRUE)

# Logs messages into a file
flog.appender(appender.file("my_app.log"))
flog.threshold(INFO)

# Function that catches any messages
tryCatchLog(source("script.R"))

You’ll notice several things. First, I’m once again calling the here package and telling it that I’m working within this folder (R), and then I’m calling the 01-functions.R file by using the source() function, but also notice how the source() function is followed by a here() function/argument. This here() function allows you to fully control files in a specific folder from anywhere else, without having to actually be in that folder. So, suppose I was in the Main Project -> Data -> Models folder and I was saving my work there (in the Models folder), which is pretty far from the R folder, I can still, using functions like source(), save(), etc., call or manipulate files from a totally different folder by specifying the hierarchy using here(). This can also be done in other ways, but those are far more cumbersome, and not flexible.

This is how I always call the necessary packages and functions I need from every .R script, simply by using the source() function and using here() to direct it to the 01-functions.R file in the R folder. Now let’s look at the next few lines.

flog.appender(appender.file("script.log"))
flog.threshold(INFO)

tryCatchLog(source("script.R"))

This script is designed to catch any warnings that occur inside the .R script and save them to a .log file in another subfolder within the R folder called Errors. So here’s an example of a full script for a mix of data generation and inspection, so that you can see it in action.

Example of Data Loading & Inspection

library("here")

set_here()

source(here("R", "01-functions.R")) # Loads all R functions

# First we need some data

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed(1031) # My birthday

# Suppose you wanted to simulate a clinical trial
library("simstudy")

# Simulating a Clinical Trial --------------------------

def <- defData(varname = "male", dist = "binary",
               formula = .5 , id="cid")
def <- defData(def, varname = "over65",
               dist = "binary",
               formula = "-1.7 + .8 * male",
               link = "logit")
def <- defData(def,varname = "baseDBP",
               dist = "normal",
                 formula = 70, variance = 40)
def <- defData(def, varname = "age",
               dist = "normal",
               formula = 10,
               variance = 2)
def <- defData(def, varname = "visits",
               dist = "poisson",
    formula = "1.5 - 0.2 * age + 0.5 * male",
    link = "log")
def <- defData(def, varname = "weight",
               dist = "normal",
                 formula = 60, variance = 10)
dtstudy <- genData(500, def)
study <- trtAssign(dtstudy, nTrt = 2,
                   balanced = FALSE, grpName = "rxGrp")
study <-  as.data.frame(study)
study$iq <- iq <- wakefield::iq(500)
study$height <- height <- wakefield::height(500)
study$income <- income <- wakefield::income(500,
                              digits = 2, name = "Income")
study$SAT_score <- wakefield::sat(500, mean = 1500,
                 sd = 100, min = 0, max = 2400,
                 digits = 0, name = "SAT")
study$visits <- as.numeric(study$visits)
# Add noisy predictors
study[, 11:20] <- lapply(1:10, FUN = function(i)
                         (rnorm(n = 500,
                                mean = 1, sd = 2)))
varname <- sprintf("x%d", (11:20))
colnames(study)[11:20] <- varname

study1 <- trtAssign(dtstudy, n = 2, balanced = TRUE,
                    strata = c("male", "over65"),
    grpName = "rxGrp")

study1

# Inspecting Data  -------------------------------------

summarytools::view(dfSummary(study))

study$over65 <- as.factor(study$over65)
study$male <- as.factor(study$male)
study$rxGrp <- as.factor(study$rxGrp)

study_formula <- as.formula(baseDBP ~ .)

plot(spearman2(study_formula, study))
abline(v = 0)

x_1 <- model.matrix(~ ., data = study)[, -1]

# Initial Data Analysis  ------------------------------

(cormatrix <- (cor(x_1)))

col <- colorRampPalette(c("#BB4444", "#EE9988",
                          "#FFFFFF",
                          "#77AADD", "#4477AA"))

corrplot(cormatrix, method = "color",
         col = col(200),
         tl.col = "black", tl.cex = 0.40,
         addCoef.col = "black", number.cex = .35)


clus1 <- varclus(x_1, similarity = "h")
clus2 <- varclus(x_1, similarity = "s")
clus3 <- varclus(x_1, similarity = "p")

plot(clus1, ylab = "Hoeffing's D statistic",
     lwd = 2.5, lty = 1, cex = 0.75)
title("Hierarchical Cluster Analysis")


plot(clus2, lwd = 2.5, lty = 1, cex = 0.75)
plot(clus3, lwd = 2.5, lty = 1, cex = 0.75)

Pre <- names(study)
(fmla <- as.formula(paste("~",
                          paste(Pre,
                                collapse = "+"))))

# Redundancy analysis

redun <- redun(fmla,
  data = study,
  r2 = 0.90, type = "ordinary", allcat = FALSE,
  tlinear = FALSE, iterms = FALSE, pr = TRUE)

The above script simulated some fake data for a hypothetical clinical trial (usually we use this first script to load data from a .csv or .txt file) and then we used several functions to inspect the dataframe for the structure and distributions of the variables and missing data, etc. We also attempted to properly code the variables. Now, in order to capture any possible errors, we will have to run this entire script from another R script instead of running it line by line or selecting all the code and running it.

We will do this from a Main source script, which will allow us to use the tryCatchLog() function.

Example of the Main Source Script

library("here")
library("tryCatchLog")
library("futile.logger")

set_here() # Load and set here

here() # Call here

options(keep.source = TRUE)
options("tryCatchLog.write.error.dump.file" = TRUE)

# Script loads all necessary functions
flog.appender(appender.file(
  here("Errors", "01-functions.log")))
tryCatchLog(source(here("R", "01-functions.R")))

# Line script to clean the data
flog.appender(appender.file(
  here("Errors", "02-cleaning.log")))
tryCatchLog(source(here("R", "02-cleaning.R")))

# Line script below to inspect the data
flog.appender(appender.file(
  here("Errors", "03-inspection.log")))
tryCatchLog(source(here("R", "03-inspection.R")))

# Load saved objects from cleanings
flog.appender(appender.file(
  here("Errors", "03.5-load_data.log")))
tryCatchLog(source(here("R", "03.5-load_data.R")))

# Main analysis script
flog.appender(appender.file(
  here("Errors", "04-analysis.log")))
tryCatchLog(source(here("R", "04-analysis.R")))

# Below is the validation of main analysis results
flog.appender(appender.file(
  here("Errors", "04.5-validation.log")))
tryCatchLog(source(here("R", "04.5-validation.R")))

# Test the sensitivity of the results by
# varying assumptions/multiple parameters
flog.appender(appender.file(
  here("Errors", "05-sensitivity.log")))
tryCatchLog(source(here("R", "05-sensitivity.R")))

# Generate tables and figures of all your results
flog.appender(appender.file(
  here("Errors", "06-tables.log")))
tryCatchLog(source(here("R", "06-tables.R")))

# Export these into actual files that can be
# inserted into papers, reports, etc.
flog.appender(appender.file(
  here("Errors", "07-export.log")))
tryCatchLog(source(here("R", "07-export.R")))

If anything went wrong during the simulations and their inspections in the first script from above, it would have been captured by the error-catching scripts that we have set up, which would be placed within a .log file in the Errors folder, which is inside the R folder.

Setting the Seed

Also, note that we have set the seed using a specific random number generator algorithm. This is not that important, however, it is important to correctly set the seed for reproducing any random numbers generated. This requires even more careful thought when running a simulation study, in which the states of the simulation need to be saved for a particular repetition, and so that streams do not overlap.[@morrisUsingSimulationStudies2019]

I will admit that the error script is not perfect at capturing warnings, but for most things, if something wrong occurs when a script is run, the function tryCatchLog() and its .log file usually catches the warning, so I would know that something went wrong and what specifically.

However, some may be skeptical of this approach and ask why I need to catch these errors and save them if I ran the scripts? Why not just scroll up and look at the console to see the errors or warnings?

Many reasons:

  • Suppose the analyses take a very long time, and are computationally intensive, and R/RStudio crashes, you may not be able to figure out what went wrong.
  • Suppose the analysis completed but because there was so much occurring in the console (for example, sampling from a posterior) that you missed the warnings/errors and got results that you don’t know have issues.

Thus, having this system set up to catch errors in every script also helps us automate the entire workflow once we’ve carefully set it up. After each .R script has one of these message-catching scripts, we can run our entire analysis from start to finish from main.R with one command. This not only makes your workspace organized, but also makes it easier for others to reproduce your work if you share your files and folders via a repository.

So, now the Error folder will contain .txt files with any warning messages or errors.

Typically, when I’m setting up a new project, I will also add a few other folders such as an Outputs folder, where I will save tables and graphs, and a Report folder, where I might be working on a manuscript.

Now that I’ve mentioned my hierarchy and how I catch errors, I can mention some other things that I believe are absolutely heinous practices.

Many people often will run an entire analysis and have several objects/vectors stored in their Global Environment in R and RStudio. Suppose they’re taking a break, finished with the analysis, turning off the computer, or stepping away for whatever reason, what they’ll typically do is click the save button on the top right pane and “Save the workspace image”. Indeed, if they don’t do that themselves, the IDE RStudio will explicitly ask them if they wish to save their workspace image.

This is a horrific practice and you should never do this because it can lead to several problems such as:

  • having several saved objects and packages conflict with one another once they’re all loaded together at once

  • having giant workspace images, that will probably cause R, and especially RStudio to constantly crash

  • not allowing you to load very specific objects that you need at a time while leaving everything else

I would say that it is a good idea to never save the workspace image, ever. It is also quite easy to turn off. Simply go the RStudio menu and click Preferences, and in the General section, you will see the following options:

It is essential to uncheck everything under R Sessions, Workspace, and History, so that you do not set your self up for a future disaster.

Instead of using these convenient but highly problematic options, always use the saveRDS() base R function, where the first argument is the object in your environment that you want to save, and the second argument is the path where you want to save it, which of course should be using the here() function.

Now suppose I decided to use multiple imputation because I had a dataset with many missing values, and suppose I had a grasp of the missing data mechanism, and so I used something like the not-at-random fully conditional specification approach (NARFCS)[@tompsettUseNotatrandomFully2018] to attempt to handle data that I believe to be missing not at random (MNAR), and I wanted to save the imputed dataset, and only load the imputed dataset next time. This is how I would do it (in the vaguest way possible). First, I’ll start with the dataset that we generated above and generate missing values via different missing data mechanisms using the ampute() function from the mice package.[@buurenMiceMultivariateImputation2011]

Important: This is all assuming that one has carefully constructed an imputation model, and that multiple imputation is the most appropriate solution. As many others have pointed out, simply running a script like mice(data) in R without careful thought to the imputation model, is a very, very bad idea. Missing data is an extremely complex topic and I recommend everyone carefully read the works[@buurenFlexibleImputationMissing2018] of missing data researchers such as:

Example of a Multiple Imputation Script

source(here("R", "01-functions.R"))

library("beepr")

# Package that uses multiple imputation
# by chained equations

library("mice")


# Generate Missing Data  --------------------------------

study_mnar <- ampute(data = study[, 5:10],
                     prop = 0.65, mech = "MNAR")
mnar_patterns <- study_mnar$patterns
mnar_weight <- study_mnar$weight
study_mnar_rerun <- ampute(data = study[, 5:10],
                           prop = 0.5,
                           mech = "MNAR",
                           patterns = mnar_patterns,
                           weights = mnar_weight)
study_mar <- ampute(data = study[, 2:4], prop = 0.5,
                    mech = "MAR")
mar_patterns <- study_mar$patterns
mar_weight <- study_mar$weight
study_mar_rerun <- ampute(data = study[, 2:4],
                          prop = 0.5,
                          mech = "MAR",
                          patterns = mar_patterns,
                          weights = mar_weight)

study_mar_rerun[[11]]$over65 %>%
  as.factor()
study_mar_rerun[[11]]$male %>%
  as.factor()
study_mar_rerun[[11]]$rxGrp %>%
  as.factor()

df <- study_mnar_rerun$amp
df2 <- study_mar_rerun$amp

# Inspect Data

str(df) # Inspect data frame
summary(df) # High level summary of dataframe
sum(is.na(df)) # Look at missing values etc.
colSums(is.na(df))
summarytools::view(dfSummary(df))

str(df2) # Inspect data frame
summary(df2) # High level summary of dataframe
sum(is.na(df2)) # Look at missing values etc.
colSums(is.na(df2))
summarytools::view(dfSummary(df2))

# Prepare Data for Imputation -------------------------

temp <- df
temp2 <- df2

missing_plot(temp)

missing_plot(temp2)

# Set-up predictor matrix

predMatrix <- make.predictorMatrix(temp)
predMatrix2 <- make.predictorMatrix(temp2)

# Set-up predictor matrix for unidentifiable part:
predSens <- matrix(rep(0, 6), ncol = 6, nrow = 6)
colnames(predSens) <- paste(":", names(temp), sep = "")
rownames(predSens) <- names(temp)

# Set-up list with sensitivity parameter values

pSens <- rep(list(list("")), ncol(temp))
names(pSens) <- names(temp)
pSens[["rxGrp"]] <- list(-.4)
pSens[["baseDBP"]] <- list(-.4)
pSens[["male"]] <- list(-.4)
pSens[["over65"]] <- list(-.2)

# MICE RI Imputation  ----------------------------------

z1 <- parlmice(data = temp, method = "ri",
              predictorMatrix = predMatrix,
              m = 8, maxit = 1,
              cluster.seed = 1031, n.core = 4,
              n.imp.core = 2, cl.type = "FORK",
              ridge = 1e-04, remove.collinear = TRUE,
              remove.constant = FALSE, allow.na = TRUE)

z2 <- parlmice(data = temp2, method = "pmm",
              predictorMatrix = predMatrix2,
              m = 8, maxit = 1,
              cluster.seed = 1031, n.core = 4,
              ridge = 1e-04, remove.collinear = TRUE,
              n.imp.core = 2, cl.type = "FORK")

zfinal <- cbind(z1, z2)

# Heckman Imputation  ----------------------------------

study$baseDBP <- temp$baseDBP
JointModelEq<-generate_JointModelEq(data=temp,
                                    varMNAR="baseDBP")
JointModelEq[,"baseDBP_var_sel"] <- c(0,1,1,1,1,1)
JointModelEq[,"baseDBP_var_out"] <- c(0,1,1,1,1,1)

arg <- MNARargument(data = temp, varMNAR = "baseDBP",
                    JointModelEq = JointModelEq)

arg$method["age"] <- "ri"
arg$method["visits"] <- "ri"
arg$method["weight"] <- "ri"
arg$method["iq"] <- "ri"
arg$method["height"] <- "ri"

imp1 <- parlmice(data = arg$data_mod,
                  method = arg$method,
                  predictorMatrix = arg$predictorMatrix,
                  JointModelEq = arg$JointModelEq,
                  control = arg$control,
                  maxit = 1, m = 8,
                  cluster.seed = 1031, n.core = 4,
                  n.imp.core = 2, cl.type = "FORK")

analysis1 <- with(zfinal, glm(baseDBP ~ age + visits + weight + iq + height + rxGrp + male + over65))
result1 <- pool(analysis1)
summary(result1)

z2 <- parlmice(data = temp2, method = "pmm",
              predictorMatrix = predMatrix2,
              m = 8, maxit = 1,
              cluster.seed = 1031, n.core = 4,
              n.imp.core = 2, cl.type = "FORK")

heckman <- cbind(imp1, z2)

beep(3) # I will touch on this function below.

saveRDS(zfinal, here("Main Project", "Data",
                     "zfinal.rds"))

Here is the workflow truly showing its advantages, saveRDS() is saving a very specific object (zfinal, the imputed dataset) into the Data folder by using the hierarchical structure of the folder which can easily be navigated using here(). Notice how to save something and use here() to specify the location, the first argument is Main Project, followed by Data, and then comes the object name (zfinal.rds).

Now I have saved my imputed dataset as an object called zfinal.rds and it is saved in the Data folder even though I am hypothetically working from the R folder.

Also notice that I use the beep() function from the beepr R package, which can be quite handy in letting you know if a particular script or analysis is done.

Next time, if I cleared my entire global environment and wanted to only load the imputed dataset, I would simply have to just click on the zfinal.rds file in the Data folder and it would load into the environment or I could use the following command,

Saving and Reading Data Carefully

# The function readRDS() and saveRDS()
# are possibly the most important base R functions

zfinal <- readRDS("zfinal.rds",
             here("Main Project", "Data",
                  "zfinal.rds"))

This gives us full control over the environment, and it’s also why I encourage people to turn off the automatic prompt to save the workspace that RStudio gives you by default (I am certain even Hadley Wickham has said the same on multiple occasions).

So if I now wanted to include the imputed dataset into one of my .R scripts that I am trying to automate with one command, whether for reproducibility, or speed, etc., I could do it with absolute control.

This is also how I save my models and how I validate them, which is especially handy because running those can also take an excruciatingly long time.

Here’s an example of a script that would be in the 04-analysis.R script, I’m leaving out the warning-catching script now to avoid making this too long. I don’t need to load the brms package because I’m already calling it from the first .R script, 01-functions.R (but I’m just showing it for now to avoid confusion), and then I am saving it in the Models folder which is found in the Data folder. And again, the beep() function would let me know when the script is over, and scripts like this can often take very long, due to the computational intensity.

The script below is an analysis script whose results we plan on saving once completed. Due to the arguments I have set (iterations, warmups, chains, and using an imputed dataset), it can often take a long time before it is finished, and if I did happen to do other tasks while I was waiting on that, the output would automatically save to the Models folder via saveRDS(), and I would also receive a notification via the beep() function, preventing me from wasting any time.

Example Modeling Script

# Calling all packages, please report for duty

source(here("R", "01-functions.R"))

library("brms")

# This is a script I used previously
# where I used Bayes for regularization
# Although these are all generic options

# Prior Information  --------------------------------

sample_z <- complete(zfinal, 1)

n_1 <- nrow(sample_z) # of observations

k_1 <- (ncol(sample_z) - 1) # of predictors

p0_1 <- 2 # Prior for the number of relevant variables

tau0_1 <- p0_1 / (k_1 - p0_1) * 1 / sqrt(n_1) # tau scale

# Regularized horseshoe prior
hs_prior <- set_prior("horseshoe(scale_global = tau0_1,
                      scale_slab = 1)", class = "b")

library("future")
plan(multiprocess)

# Regularized Bayesian quantile
# regression using regularized horseshoe prior
brms::brm_multiple(
  bf(baseDBP ~ age + visits + weight + iq + height + rxGrp + male + over65,
     quantile = 0.50),
     data = zfinal, prior = hs_prior,
     family = asym_laplace(), sample_prior = TRUE,
     seed = 1031, future = F,
     iter = 1000, warmup = 500, chains = 4,
     cores = 4, thin = 1, combine = TRUE,
     control = list(max_treedepth = 10,
                    adapt_delta = 0.80)) -> pen_model_1

# Principled way of saving results
saveRDS(pen_model_1,
        here("Main Project", "Data", "Models",
             "pen_model_1.rds"))

beep(3)

So I’ve saved the full contents of the fitted model (pen_model_1), which are usually several gigabytes large in size, within the Models folder. Once again, notice how the here() function allows you to work from a subfolder like R, which is inside the Main Project folder, and allows you to specify the hierarchy, and where you are within it, and save to whichever folder you wish, in this case Models, which is inside of Data. Now, I will typically conduct model checks and make sure it is performing well, and not misspecified,

source(here("R", "01-functions.R"))

# Inspect RHat values ---------------------------

rhats_1 <- pen_model_1$rhats

rhats_1_df <- (rhats_1 <= 1.1)
((as.numeric(table(rhats_1_df)) /
    (nrow(rhats_1_df) * ncol(rhats_1_df)) * 100))

rhats_1_vec <- as.numeric(unlist(c(
  rhats_1[, 1:ncol(rhats_1)])))

pdf(here("Outputs", "Figures", "Diagnostics",
         "OUTCOME_COMP_rhat.pdf"))
mcmc_rhat_hist(rhats_1_vec) +
  ggtitle("Model 1 Chain Convergence")
dev.off()

# Inspect Effective Sample Size ----------------------

mcmc_neff_hist(rhats_1_vec) +
  ggtitle("Effective Sample Size")

# Inspect Divergences ---------------------------

pdf(here("Outputs", "Figures",
         "Diagnostics", "OUTCOME_1.pdf"))
mcmc_nuts_divergence(nuts_params(pen_model_1),
                     log_posterior(pen_model_1))
dev.off()

# Examine Residual Plots ----------------------------

df_resid_stan_1 <- data.frame(fitted(pen_model_1)[, 1],
                              residuals(pen_model_1)[, 1])

pdf(here("Outputs", "Figures",
         "Diagnostics", "OUTCOME_1.pdf"))
ggplot(df_resid_stan_1,
       aes(sample = residuals(pen_model_1)[, 1])) +
  geom_qq() +
  geom_qq_line() +
  ggtitle("Model 1 Residuals")
dev.off()

(sum1 <- (posterior_summary(pen_model_1))[2:8, ])

mcmc_intervals(pen_model_1, point_est = "median",
               prob = 0.95, prob_outer = 0,
               pars = parnames(pen_model_1)) +
  ggplot2::scale_y_discrete() +
  theme(axis.text=element_text(size=13),
        axis.title=element_text(size=13),
        plot.title = element_text(size=15,
                                  face = "bold")) +
  annotate("rect",
    xmin = -0.05, xmax = 0.05, ymin = 0, ymax = 7,
    fill = "darkred", alpha = 0.075) +
  annotate("segment",
    x = -0.05, xend = -0.05, y = 0, yend = 7,
    colour = "#990000", alpha = 0.4,
    size = .75, linetype = 3) +
  annotate("segment",
    x = 0.05, xend = 0.05, y = 0, yend = 7,
    colour = "#990000", alpha = 0.4,
    size = .75, linetype = 3) +
  ggtitle("Change in Outcome 1")
dev.off()

pdf(here("Outputs", "Figures", "OUTCOME_1.pdf"))
mcmc_hist(pen_model_1)

# Posterior Predictive Checks

pdf(here("Outputs", "Figures", "Diagnostics",
         "OUTCOME_1check.pdf"))
pp_check(pen_model_1, nsamples = 250) +
  ggtitle("Posterior Predictive Check: Model 1")
dev.off()

# Shows dens_overlay plot by default

bayes_rsq1 <- bayes_R2(pen_model_1)

bayes_rsq1 <- print(median(bayes_rsq1))

write.csv(sum1, here("Outputs", "Tables",
                     "OUTCOME_1.csv"))

Once I have fit my model and done some initial checks, I usually conduct some more thorough checks using k-fold cross-validation or nested cross-validation, or bootstrap optimism.

A script of this can be found below.

source(here("R", "01-functions.R"))

# 10-Fold Cross-Validations -------------------------

kfold_1 <- kfold(pen_model_1, K = 10,
                 save_fits = TRUE, cores = 4)
saveRDS(kfold_1, here("Data", "Models",
                      "Validation", "kfold_1.rds"))
kfp_1 <- kfold_predict(kfold_1)
saveRDS(kfp_1, here("Data", "Models",
                    "Validation", "kfp_1.rds"))
kfold_rmse(y = kfp_1$y,
           yrep = kfp_1$yrep,
           type = "rmse",
           reps = 5000,
cores = 4)[[2]][["t"]]) -> kfold_rmse_1

kfold_rmse_1 <- median(kfold_rmse_1)

saveRDS(kfold_rmse_1,
        here("Data", "Models",
             "Validation", "kfold_rmse_1.rds"))

Once I’ve written all my analysis and validation scripts, I’ll typically create a subfolder in the Outputs folder called Figures and Tables and save them appropriately with the help here().

Example Table Script

source(here("R", "01-functions.R"))

# Read in Validation Objects ------------------------

kfold_1 <- readRDS("kfold_1.rds",
                   here("Main Project", "Data",
                        "Models", "Validation",
                        "kfold_1.rds"))
kfold_2 <- readRDS("kfold_1.rds",
                   here("Main Project", "Data",
                        "Models", "Validation",
                        "kfold_2.rds"))
kfp_1 <- readRDS("kfp_1.rds",
                 here("Main Project", "Data",
                      "Models", "Validation",
                      "kfp_1.rds"))
kfp_2 <- readRDS("kfp_1.rds",
                 here("Main Project", "Data",
                      "Models","Validation",
                      "kfp_2.rds"))

# Extract Estimates from Objects ---------------------

KFOLDIC_1 <- c(kfold_1[["estimates"]][3],
                     kfold_2[["estimates"]][3])
KFOLDIC_SE_1 <- c(kfold_1[["estimates"]][3, 2],
                        kfold_2[["estimates"]][3, 2])
BAYES_R2 <- c(bayes_rsq1, bayes_rsq2), 2)
KFOLD_ELPD_1 <- c(kfold_1[["estimates"]][1],
                        kfold_2[["estimates"]][1])
KFOLD_ELPD_SE_1 <- c(kfold_1[["estimates"]][1, 2],
                           kfold_2[["estimates"]][1, 2])

Outcome <- c("OUTCOME_1", "OUTCOME_2")

# Construct Data Frames ------------------------------

table4 <- data.frame(Outcome, BAYES_R2, KFOLD_ELPD_1,
                     KFOLD_ELPD_SE_1, KFOLDIC_1,
                     KFOLDIC_SE_1)

colnames(table4) <- c("Outcome", "Bayes R^2",
                      "10-fold CV ELPD",
                      "10-fold CV ELPD SE",
                      "10-fold CV IC",
                      "10-fold CV IC SE")

# Export as Tables ----------------------------------

write.csv(table4, here("Main Project","Outputs",
                       "Tables", "Table4.csv"))

table5 <- data.frame(Outcome, KFOLD_ELPD_1,
                     KFOLD_ELPD_SE_1,
                     KFOLDIC_1, KFOLDIC_SE_1)
colnames(table5) <- c("Outcome", "10-fold CV ELPD",
                      "10-fold CV ELPD SE",
                      "10-fold CV IC",
                      "10-fold CV IC SE")
write.csv(table5,
          here("Main Project","Outputs", "Tables",
               "Table5.csv"))

# Save Tables From Sensitivity Analyses --------------

write.csv(sens_table_OUTCOME_1,
          here("Main Project","Outputs", "Tables",
               "Table7_OUTCOME_1.csv"))
write.csv(sens_table_OUTCOME_2,
          here("Main Project","Outputs", "Tables",
               "Table7_OUTCOME_2.csv"))

Then, I’ll go back to the README.Rmd file, make notes of the hierarchy of the entire folder and subfolders, and what’s been completed, and what still needs to be done.

Now, suppose I deleted all of my saved models, imputed data, graphs, and tables, (which would be a devastating blow!) and only had the original dataset and organized scripts. Anyone could go into my project and simply highlight the entire main.R file and run it, and they would get all the same outputs and models, assuming a seed was set in advance, which is essential. Although, it might take them a long time to get the same outputs (if computationally intensive)… they would likely be able to get all the exact same results.

Possible Drawbacks of this Approach

Every proposal tends to have drawbacks or limitations, and those who do not explicitly tell you about them are misleading you. In the approach I have outlined above, one major drawback is that it takes a somewhat long time to set up and therefore could deter many statisticians and data analysts. The other drawback is that by loading all your functions and libraries in the first 01_functions.R script, there is potential for conflict between functions, therefore you’ll have to be mindful of what packages you need and when. Indeed, I ran into this issue when loading the brms package, which had several functions that conflicted with the base R functions. So please be mindful of this.

Further, and most importantly, I have no evidence that this approach can truly prevent errors, it is simply a belief I have as a result of my own experiences and suggesting them to others, and hearing positive comments. So please, take that into account, although I would be grateful if you were able to share your positive or even negative experiences using this approach below.

Please, please annotate your scripts, as I have done so several times above to explain some of the functions. This is not only helpful for others who are trying to go through your code and understand it, but also for yourself. If you end up forgetting what function does what, and what steps you were taking, you will be in for a really frustrating time, and possibly prone to making serious mistakes.

Here’s what my project folders tend to look like (figure below). And here is a template GitHub repo to show what the structure typically looks like.

It should go without saying that sharing your data (if possible) and code will help you and your collaborators catch errors before it’s too late, but also, sharing them with others/the public after a project is done will also help catch possible errors that you/your collaborators/reviewers didn’t catch, which may mislead other researchers and also result in a much more stressful situation for you.

And last but not least, it is also essential to give extensive details on which version of R you used and which packages you used and their versions. This is because many scripts that once worked on previous versions of the software may no longer give the same results or may not even work at all. In order for someone to reproduce your results, they’ll need to know the environment on which you ran your analyses on. You can provide these details easily simply by running the script sessionInfo(), which should be at the end of every R script you run.

Helpful Resources

Environment

The analyses were run on:

R version 4.6.0 (2026-04-24)
Platform: aarch64-apple-darwin25.4.0
Running under: macOS Tahoe 26.5.1

Matrix products: default
BLAS:   /opt/homebrew/Cellar/openblas/0.3.33/lib/libopenblasp-r0.3.33.dylib 
LAPACK: /opt/homebrew/Cellar/r/4.6.0/lib/R/lib/libRlapack.dylib;  LAPACK version 3.12.1

Random number generation:
 RNG:     Mersenne-Twister 
 Normal:  Inversion 
 Sample:  Rejection 
 
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/New_York
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] htmlwidgets_1.6.4 compiler_4.6.0    fastmap_1.2.0     cli_3.6.6         htmltools_0.5.9   tools_4.6.0       parallel_4.6.0   
 [8] otel_0.2.0        yaml_2.3.12       rmarkdown_2.31    knitr_1.51        data.table_1.18.4 jsonlite_2.0.0    xfun_0.59        
[15] digest_0.6.39     rlang_1.2.0       evaluate_1.0.5

See also: What Makes a Sensitivity Analysis? — the hub piece tying this material to the broader cluster on assumptions, robustness, and what happens when models bend.

References

Back to top ]]> statistics https://lesslikely.com/statistics/statistical-quality-control.html Sat, 13 Jun 2020 04:00:00 GMT Book Review: Regression and Other Stories by Gelman, Hill, and Vehtari Zad Rafi https://lesslikely.com/statistics/regression-stories.html

Over a decade ago, Andrew Gelman and Jennifer Hill gave applied researchers a comprehensive book (Data Analysis Using Regression and Multilevel/Hierarchical Models) on fitting simple and complex statistical models in R both from a classical framework and a Bayesian one. Now they’re back with an updated version and a new author (Aki Vehtari).

Much has changed in applied statistics since 2006 (when the book was first released). The primary software used at the time and in the book to fit Bayesian models was BUGS (Bayesian inference Using Gibbs Sampling).

However, both BUGS and some of the R code in the first edition are now outdated. The new edition updates the R code and contains intuitive instructions on how to fit simple and complex models using the probabilistic programming language, Stan (also developed by Gelman and colleagues), which is now used in several fields (even for studying wine!).

Indeed, running a Bayesian regression model in R is now as simple as

# I use the sample PlantGrowth dataset in R
library("rstanarm")
library("bayesplot")
pg <- PlantGrowth
model1 <- stan_glm(weight ~ group, 
                   data = pg, refresh = 0)
summary(model1)

Model Info:
 function:     stan_glm
 family:       gaussian [identity]
 formula:      weight ~ group
 algorithm:    sampling
 sample:       4000 (posterior sample size)
 priors:       see help('prior_summary')
 observations: 30
 predictors:   3

Estimates:
              mean   sd   10%   50%   90%
(Intercept)  5.0    0.2  4.8   5.0   5.3 
grouptrt1   -0.4    0.3 -0.7  -0.4   0.0 
grouptrt2    0.5    0.3  0.1   0.5   0.9 
sigma        0.6    0.1  0.5   0.6   0.8 

Fit Diagnostics:
           mean   sd   10%   50%   90%
mean_PPD 5.1    0.2  4.9   5.1   5.3  

The mean_ppd is the sample average posterior predictive distribution of the outcome variable (for details see help('summary.stanreg')).

MCMC diagnostics
              mcse Rhat n_eff
(Intercept)   0.0  1.0  2844 
grouptrt1     0.0  1.0  2882 
grouptrt2     0.0  1.0  2642 
sigma         0.0  1.0  3279 
mean_PPD      0.0  1.0  3354 
log-posterior 0.0  1.0  1645 

For each parameter, mcse is Monte Carlo standard error, n_eff is a crude measure of effective sample size, and Rhat is the potential scale reduction factor on split chains (at convergence Rhat=1).
plot(model1)

Another key difference between the first edition and the new edition is that the 2006 book attempted to cover several topics at once. It contained instructions on how to fit simple models in a classical framework all the way up to multilevel models in a Bayesian framework. The new edition attempts to reduce this information overload by splitting itself into two volumes.

The first volume (Regression and Other Stories) covers fitting simple and complex models using R and Stan, and is oriented towards the applied researcher or statistician, who wants a smooth introduction to fitting Bayesian models using Stan without diving into much theory or math.

A draft copy of the table of contents in the new edition can be found here, though it’s very likely that the published edition will have some changes.

The book does not cover much of multilevel modeling, which is reserved for the second volume, Advanced Regression and Multilevel Models (planned to be released in the next year or two).

Make no mistake, although both of these books are unlikely to touch on a serious amount of theory or math, they are not books that can be read without serious engagement and practice. Every chapter contains enough math for the reader to understand the concepts being discussed with exercises at the end to solidify these concepts.

The chapter exercises are incredibly similar to these exam questions that Gelman created for his Applied Regression class.

Question: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, solution to 15.

I suspect that many of the commenters of the blog who had some difficulty with these questions would have had an easier time had they had the opportunity to read the book.

The new edition also covers several news stories from the past few years (some of which long-time blog readers will be familiar with) and gives readers a set of tools to think critically about these stories and how proper statistical thinking could’ve prevented mishaps. In addition, it incorporates concepts that Gelman and colleagues have developed and solidified over the years, since the first edition was published, such as the concept of Type-M and Type-S errors.

Overall, the book is quite comprehensive and will leave the reader with a rich set of tools to think critically about statistics and to fit models in the real world. I look forward to grabbing a hard copy once the book is out, which seems to be in the summer to fall of 2020.

Update: Looks like the book is out!

Back to top ]]> https://lesslikely.com/statistics/regression-stories.html Tue, 11 Jun 2019 04:00:00 GMT