Less Likely https://lesslikely.com/posts.html Statistical science, applied to medicine and nutrition — the quiet arithmetic of being wrong less often. https://lesslikely.com/images/HomeLogo-static.png Less Likely https://lesslikely.com/posts.html 92 144 quarto-1.10.18 Wed, 16 Sep 2026 04:00:00 GMT The P-value Is Not a Verdict: An Interactive Consonance Curve Zad Rafi https://lesslikely.com/statistics/consonance-explorer.html A single P-value reports one number about one hypothesis. The P-value function reports the same test statistic applied to every candidate parameter value at once — and it contains every compatibility interval at every level as horizontal slices.1 Nothing below is new mathematics. It is the same normal approximation you already use, drawn in full instead of evaluated at one point.

1 Slice the curve at height and read off the x-coordinates: those are the 95% limits. Slice at and you have the 99% limits. The interval was never a separate calculation.

The display is old. Poole called for it in 19871, Sullivan and Foster catalogued its uses in 19902, and it has sat in Modern Epidemiology ever since. What has changed is that it now costs nothing to render, and that we have a better vocabulary for the vertical axis: compatibility rather than confidence, and surprisal rather than significance3.

TipDefinitions, once

A P-value function (equivalently, a consonance or compatibility curve) plots, for every candidate parameter value , the P-value from testing against the data under the full set of analysis assumptions.

An S-value is the Shannon transform : the number of consecutive heads from a fair coin that would be exactly as surprising as the observed test statistic, if and every other assumption were correct.

A 95% compatibility interval is the set of whose P-value exceeds 0.05 — a horizontal slice through the curve, not a separate object.

The explorer

Move the sliders. Everything recomputes in the browser — no server, no round trip.

Calibrating bits

The S-value exists because “P = 0.05” carries no intuition and “significant” carries the wrong one. Bits do carry intuition, because everyone has a physical model of coin flipping. Drag μ above and watch the coins.

The S-value measures surprise at the whole model, not at alone. Every bit is also a bit against the assumed absence of uncontrolled confounding, selection effects, measurement error, and model form. A large S-value says the data are hard to reconcile with the conjunction of and everything else you assumed; it does not tell you which conjunct to abandon. In observational work, the “everything else” is usually the weaker link.

A worked example: one data set, four analyses

Brown and colleagues studied serotonergic antidepressant use in pregnancy and autism spectrum disorder in 35,906 Ontario births, of which 2,837 pregnancies (7.9%) were exposed4. They reported four hazard ratios from the same cohort under progressively stronger confounding control. This is the example reanalysed at length in3, and the working code for that paper lives here.

Look at the point estimates before you look at anything else.

Three of the four point estimates are 1.59, 1.60 and 1.61. By the conventional reading, one of them is a finding and two of them are nothing. What actually changed between them is not the estimate — it is the width of the interval, which grows from a ratio of 1.9 to 5.4 across the three adjusted analyses as each trades precision for stronger confounding control. The verdicts flip; the estimate does not move.

The published conclusion was that exposure “was not associated with” autism. That sentence is doing work the arithmetic will not support: the IPT-weighted analysis puts against the null, 4.31 bits of surprise, and its interval runs to 2.59.

Uncheck the crude analysis and the remaining three curves are near-concentric: same peak, different width. That is the shape of a precision problem, not a disagreement about effect size. Now test a value other than the null:

Set the slider to 1.5 and all four analyses agree: = 6.6, 0.5, 0.4 and 0.2 bits. Three of the four find a 50% increase in hazard less surprising than a single coin flip. The analysis that was “not significant” and the analysis that was “significant” are, against this hypothesis, saying nearly the same thing.

Testing HR = 1 is a legitimate question. Reporting only that test, and only its dichotomised outcome, discards the answer to every other question the same data address — including questions the reader may care about more, such as whether an effect large enough to change prescribing can be ruled out. On these data it cannot: the IPT-weighted 95% interval reaches 2.59, and even the 50% interval runs from 1.37 to 1.90.

What more data would do

Precision and parameter value are separate things that a single P-value fuses into one number. Hold the IPT-weighted estimate fixed and vary only the information: the standard error scales as when the sample size is multiplied by .

The dashed ghost is the study as published. Push to 4 and the null falls well outside the 95% interval without the estimate having moved a hair. Pull down to 0.3 and the same estimate becomes compatible with almost everything between 0.7 and 3.9.

This is worth sitting with, because it is the mechanism behind most “failure to replicate” reporting. A study that reports and a study that reports can have identical point estimates and differ only in how many people they enrolled. Neither number is a property of the effect.

ImportantThe one thing this widget cannot show

Scaling the standard error models random error only. Multiplying the sample size by 16 does nothing to confounding, selection, or measurement error — those biases do not shrink with , and past some sample size they dominate entirely. A very narrow curve centred in the wrong place is the most confident way to be wrong. If you want that quantified rather than assumed away, the tool is bias analysis, not more subjects.

Do two estimates conflict?

“Study A was significant, study B was not, so they disagree” is a comparison of two verdicts, not of two estimates. The comparison that answers the question is a curve in its own right: under independence, the log ratio-of-ratios has standard error , and the null of no difference sits at a ratio of 1.

Study A is the explorer at the top of this page. Set study B here.

With the defaults — A at 1.61 () and B at 1.05 () — the two studies land on opposite sides of the conventional threshold. Their ratio of ratios is 1.53, 95% interval 0.90 to 2.61, and the difference carries 3.1 bits of information: fewer than the 4.3 bits the 0.05 convention itself demands, and fewer than four coin flips. Two verdicts that disagree, one comparison that does not.

WarningThis does not apply to the four Brown analyses

The overlay in the previous section shows four analyses of the same subjects. Their estimates are strongly correlated, so badly overstates the standard error of their difference and the resulting P-value is not interpretable. Use this widget for genuinely independent studies. For nested analyses of one data set, the honest summary is the overlay itself: same peak, different width.

A P-value and an S-value are the same quantity on different scales, so a dual axis is technically defensible. It is still a bad idea: readers reliably misread which curve belongs to which axis, and the visual slope of the P-value curve near the tails is uninformative precisely where the S-value is most informative. Two stacked panels sharing an x-axis cost one extra inch of page and remove the ambiguity.

  • No shading of “significant” regions. The point of the curve is that there is no cliff at 0.05.
  • No default null at 1. The μ slider starts at 1 because that is the conventional test value, not because it is privileged.
  • No P-value below about 1e-300. Double precision runs out. The implementation stays accurate to roughly z = 37, which is far past the point where the normal approximation itself is the binding constraint.
  • No normality diagnostics. Every curve here assumes the log estimate is approximately normal with the reported standard error. That is the same assumption the published interval already made; the curve does not add it, it only makes it visible across the whole range.

Interactive table

The same idea applied to tabular results — searchable and sortable by clicking column headers, which is most of what makes a long results table usable. These are the Brown analyses again, with each row carrying its own P-value against a hypothesis you choose.

How to report this

Nothing above requires a new statistic, a new threshold, or a new software stack. It requires four sentences you can paste into a results section:

  1. Report the estimate and interval without the word “significant” — “HR 1.61, 95% compatibility interval 0.997 to 2.59”.
  2. Report the P-value as a number, not as an inequality against 0.05, and say what hypothesis it tests.
  3. Report at least one P-value for a non-null hypothesis that matters clinically or practically. If the data cannot distinguish the null from a 60% increase, that is the finding.
  4. Say that the interval and P-value are conditional on the entire model, not just on the parameter.

If a reviewer asks for the significance verdict back, the curve is the reply: it contains their verdict as one horizontal slice, plus every slice they did not ask for.

Reader infrastructure that is already on

Three things people associate with Gwern’s pages are Quarto defaults and need no code at all:

  • Citation popups. Hover any @citekey reference.
  • Footnote popups. Hover the marker on 2 above.
  • Cross-reference popups. Hover any @fig- or @tbl- reference.

2 The convention that 0.05 marks a boundary is worth pricing in bits: it is 4.32 bits of information against the tested hypothesis, or roughly the surprise of four consecutive heads. Few people would abandon a belief on four coin flips.

All three are controlled by citations-hover, footnotes-hover and crossrefs-hover, which default to true. The margin placement of footnotes here comes from reference-location: margin in this document’s front matter.

References

1. Poole C. (1987). “Beyond the confidence interval.” American Journal of Public Health. 77:195–199. doi: 10.2105/ajph.77.2.195.
2. Sullivan KM, Foster DA. (1990). “Use of the confidence interval function.” Epidemiology. 1:39–42. doi: 10.1097/00001648-199001000-00009.
3. Rafi Z, Greenland S. (2020). “Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise.” BMC Medical Research Methodology. 20:244. doi: 10.1186/s12874-020-01105-9.
4. Brown HK, Ray JG, Wilton AS, Lunsky Y, Gomes T, Vigod SN. (2017). “Association between serotonergic antidepressant use during pregnancy and autism spectrum disorder in children.” JAMA. 317:1544–1552. doi: 10.1001/jama.2017.3415.
Back to top ]]> statistics interactive https://lesslikely.com/statistics/consonance-explorer.html Wed, 16 Sep 2026 04:00:00 GMT Constructing Likelihood Functions in R, Stata, Python, Julia, MATLAB, and SQL Zad Rafi https://lesslikely.com/statistics/likelihood-polyglot.html

Three constructions, six languages, one inferential object. The likelihood function, its deviance transform, the P-value (compatibility) function, and the S-value function are coordinate views of the same thing; a script that can build one can build them all. Each language section below carries the same self-checks:

  • the from-scratch MLE must reproduce the packaged estimator,
  • the likelihood peak must land on the known estimator, and
  • the support interval must reproduce the 95% interval, since .

A silent run means every check passed. The running example for the summary-statistic construction is the Brown et al. (2017) JAMA HDPS-adjusted hazard ratio, HR (0.997, 2.59).

However, there are also variations of likelihood which, of which one we touch on. To see a clear expalanation of the differences between these various likelihood, see1, 2

Maximum likelihood from scratch

Fit a normal linear model and a logistic regression by writing the log-likelihood yourself and maximizing it, then check against the built-in estimator. The one trick used everywhere: parameterize with so the scale parameter stays positive during optimization.

Base R needs no helper package: optim on a hand-written negative log-likelihood, checked against lm and glm.

set.seed(1031)

# ---- Part A: normal linear model ----
n <- 200
x <- rnorm(n)
y <- 2 + 0.75*x + rnorm(n, sd = 1.5)
X <- cbind(1, x)

negloglik_normal <- function(theta, y, X) {
  beta <- theta[1:2]; log_sigma <- theta[3]
  -sum(dnorm(y, mean = X %*% beta, sd = exp(log_sigma), log = TRUE))
}

fit <- optim(c(0, 0, 0), negloglik_normal, y = y, X = X, method = "BFGS")
b_ml <- fit$par[2]; sigma_ml <- exp(fit$par[3])

ols <- lm(y ~ x)
sigma_ols_nscaled <- sqrt(sum(residuals(ols)^2) / n)   # ML sigma uses n, not n-k

stopifnot(abs(b_ml - coef(ols)[["x"]]) < 1e-4,
          abs(sigma_ml - sigma_ols_nscaled) < 1e-3)
cat("Part A validation PASSED: optim MLE reproduces lm\n")
#> Part A validation PASSED: optim MLE reproduces lm

# ---- Part B: logistic regression ----
n2 <- 300
x2 <- rnorm(n2)
y2 <- as.numeric(runif(n2) < plogis(-0.4 + 0.9*x2))
X2 <- cbind(1, x2)

negloglik_logit <- function(beta, y, X) {
  xb <- X %*% beta
  sum(log1p(exp(-xb))*y + log1p(exp(xb))*(1 - y))
}

fit2 <- optim(c(0, 0), negloglik_logit, y = y2, X = X2, method = "BFGS")
glmfit <- glm(y2 ~ x2, family = binomial())

stopifnot(abs(fit2$par[2] - coef(glmfit)[["x2"]]) < 1e-4,
          abs(-fit2$value - as.numeric(logLik(glmfit))) < 1e-4)
cat("Part B validation PASSED: optim MLE reproduces glm\n")
#> Part B validation PASSED: optim MLE reproduces glm

The lf evaluator receives one linear predictor per equation and fills in the observation-level log likelihood (see [R] ml).

#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> Number of observations (_N) was 0, now 200.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Initial:      Log likelihood =  -806.4418
#> Alternative:  Log likelihood = -450.70073
#> Rescale:      Log likelihood = -429.09515
#> Rescale eq:   Log likelihood = -374.61873
#> Iteration 0:  Log likelihood = -374.61873  
#> Iteration 1:  Log likelihood = -348.89609  
#> Iteration 2:  Log likelihood = -343.02889  
#> Iteration 3:  Log likelihood = -342.99237  
#> Iteration 4:  Log likelihood = -342.99235  
#> 
#>                                                         Number of obs =    200
#>                                                         Wald chi2(1)  =  72.77
#> Log likelihood = -342.99235                             Prob > chi2   = 0.0000
#> 
#> ------------------------------------------------------------------------------
#>            y | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#> mu           |
#>            x |   .8119551   .0951852     8.53   0.000     .6253954    .9985147
#>        _cons |   1.868002   .0954381    19.57   0.000     1.680947    2.055057
#> -------------+----------------------------------------------------------------
#> lnsigma      |
#>        _cons |   .2960232        .05     5.92   0.000      .198025    .3940214
#> ------------------------------------------------------------------------------
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Part A validation PASSED: ml reproduces regress
#> 
#> 
#> Number of observations (_N) was 0, now 300.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Initial:      Log likelihood = -207.94415
#> Alternative:  Log likelihood =  -205.2231
#> Rescale:      Log likelihood = -204.28183
#> Iteration 0:  Log likelihood = -204.28183  
#> Iteration 1:  Log likelihood = -187.36811  
#> Iteration 2:  Log likelihood = -187.34499  
#> Iteration 3:  Log likelihood = -187.34499  
#> 
#>                                                         Number of obs =    300
#>                                                         Wald chi2(1)  =  28.32
#> Log likelihood = -187.34499                             Prob > chi2   = 0.0000
#> 
#> ------------------------------------------------------------------------------
#>            y | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>            x |   .7507449   .1410829     5.32   0.000     .4742275    1.027262
#>        _cons |  -.3434135    .124075    -2.77   0.006     -.586596    -.100231
#> ------------------------------------------------------------------------------
#> 
#> 
#> 
#> 
#> Part B validation PASSED: ml reproduces logit
import numpy as np
from scipy import optimize
from scipy.stats import norm
import statsmodels.api as sm

rng = np.random.default_rng(1031)

# ---- Part A: normal linear model ----
n = 200
x = rng.normal(0, 1, n)
y = 2 + 0.75 * x + rng.normal(0, 1.5, n)
X = sm.add_constant(x)

def negloglik_normal(theta, y, X):
    beta, log_sigma = theta[:-1], theta[-1]
    return -np.sum(norm.logpdf(y, loc=X @ beta, scale=np.exp(log_sigma)))

fit = optimize.minimize(negloglik_normal, np.zeros(3), args=(y, X), method="BFGS")
b_ml, sigma_ml = fit.x[1], np.exp(fit.x[2])

ols = sm.OLS(y, X).fit()
sigma_ols_nscaled = np.sqrt(ols.ssr / n)   # ML sigma uses n, not n-k

assert abs(b_ml - ols.params[1]) < 1e-4
assert abs(sigma_ml - sigma_ols_nscaled) < 1e-3
print("Part A validation PASSED: scipy MLE reproduces OLS")
#> Part A validation PASSED: scipy MLE reproduces OLS

# ---- Part B: logistic regression ----
n = 300
x2 = rng.normal(0, 1, n)
y2 = (rng.uniform(size=n) < 1/(1 + np.exp(-(-0.4 + 0.9*x2)))).astype(float)
X2 = sm.add_constant(x2)

def negloglik_logit(beta, y, X):
    xb = X @ beta
    return np.sum(np.logaddexp(0, -xb)*y + np.logaddexp(0, xb)*(1 - y))

fit2 = optimize.minimize(negloglik_logit, np.zeros(2), args=(y2, X2), method="BFGS")
logit = sm.Logit(y2, X2).fit(disp=0)

assert abs(fit2.x[1] - logit.params[1]) < 1e-4
assert abs(-fit2.fun - logit.llf) < 1e-4
print("Part B validation PASSED: scipy MLE reproduces Logit")
#> Part B validation PASSED: scipy MLE reproduces Logit

Note: current Optim.jl rejects the old autodiff = :forward symbol (it now wants an ADTypes object); the default finite-difference gradient is version-robust and plenty here. The trailing semicolons are load-bearing for the output, not the math: Julia echoes the value of every top-level expression, so without them each randn(rng, n) would dump a 200-element vector into the page. Three kinds of line are deliberately left unsuppressed, because what they echo is worth reading – the DataFrames (which self-truncate with ⋮ and a rows-omitted note), the fitted models (lm/glm print a coefficient table with standard errors and intervals, which is what the assertions below check), and the bare p on a plotting chunk’s last line, which is what emits the figure.

using Distributions, Optim, GLM, DataFrames, Random, Statistics

rng = MersenneTwister(1031);

# ---- Part A: normal linear model ----
n = 200;
x = randn(rng, n);
y = 2 .+ 0.75 .* x .+ 1.5 .* randn(rng, n);
X = hcat(ones(n), x);

function negloglik_normal(theta)
    beta, log_sigma = theta[1:2], theta[3]
    -sum(logpdf.(Normal.(X * beta, exp(log_sigma)), y))
end;

fit = optimize(negloglik_normal, zeros(3), BFGS());
b_ml, sigma_ml = Optim.minimizer(fit)[2], exp(Optim.minimizer(fit)[3]);

df = DataFrame(y = y, x = x)
#> 200×2 DataFrame
#>  Row │ y          x
#>      │ Float64    Float64
#> ─────┼────────────────────────
#>    1 │  2.51681    1.53733
#>    2 │  2.49614    0.67567
#>    3 │  3.53854    0.529915
#>    4 │  2.40127   -0.469345
#>    5 │  2.47939   -0.344731
#>    6 │  1.25291    0.00596521
#>    7 │ -0.12643   -1.52318
#>    8 │  2.20185    0.390072
#>   ⋮  │     ⋮           ⋮
#>  194 │  1.6941     1.13584
#>  195 │  0.352342   1.1385
#>  196 │  1.16013   -0.839467
#>  197 │  4.71371    1.00806
#>  198 │  0.659784  -1.24059
#>  199 │  1.914     -0.300155
#>  200 │  0.488223  -0.759049
#>               185 rows omitted
ols = lm(@formula(y ~ x), df)
#> StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Vector{Float64}}, GLM.DensePredChol{Float64, CholeskyPivoted{Float64, Matrix{Float64}, Vector{Int64}}}}, Matrix{Float64}}
#> 
#> y ~ 1 + x
#> 
#> Coefficients:
#> ────────────────────────────────────────────────────────────────────────
#>                 Coef.  Std. Error      t  Pr(>|t|)  Lower 95%  Upper 95%
#> ────────────────────────────────────────────────────────────────────────
#> (Intercept)  2.06123     0.111722  18.45    <1e-44   1.84091     2.28154
#> x            0.820263    0.111287   7.37    <1e-11   0.600803    1.03972
#> ────────────────────────────────────────────────────────────────────────
sigma_ols_nscaled = sqrt(sum(residuals(ols) .^ 2) / n);

@assert abs(b_ml - coef(ols)[2]) < 1e-4
@assert abs(sigma_ml - sigma_ols_nscaled) < 1e-3
println("Part A validation PASSED: Optim MLE reproduces lm")
#> Part A validation PASSED: Optim MLE reproduces lm

# ---- Part B: logistic regression ----
n2 = 300;
x2 = randn(rng, n2);
y2 = Float64.(rand(rng, n2) .< 1 ./ (1 .+ exp.(-(-0.4 .+ 0.9 .* x2))));
X2 = hcat(ones(n2), x2);

log1pexp(u) = u > 35 ? u : log1p(exp(u));
negloglik_logit(beta) =
    sum(log1pexp.(-(X2 * beta)) .* y2 .+ log1pexp.(X2 * beta) .* (1 .- y2));

fit2 = optimize(negloglik_logit, zeros(2), BFGS());
df2 = DataFrame(y = y2, x = x2)
#> 300×2 DataFrame
#>  Row │ y        x
#>      │ Float64  Float64
#> ─────┼─────────────────────
#>    1 │     1.0   0.64402
#>    2 │     0.0  -0.883275
#>    3 │     0.0  -0.746231
#>    4 │     0.0   2.32986
#>    5 │     1.0   1.89985
#>    6 │     0.0  -0.931408
#>    7 │     1.0  -0.316581
#>    8 │     1.0   0.801148
#>   ⋮  │    ⋮         ⋮
#>  294 │     1.0  -0.198019
#>  295 │     0.0  -0.0022127
#>  296 │     1.0   0.10344
#>  297 │     0.0  -0.831018
#>  298 │     1.0   0.177091
#>  299 │     0.0  -1.0159
#>  300 │     0.0  -0.478225
#>            285 rows omitted
logit = glm(@formula(y ~ x), df2, Binomial(), LogitLink())
#> StatsModels.TableRegressionModel{GeneralizedLinearModel{GLM.GlmResp{Vector{Float64}, Binomial{Float64}, LogitLink}, GLM.DensePredChol{Float64, CholeskyPivoted{Float64, Matrix{Float64}, Vector{Int64}}}}, Matrix{Float64}}
#> 
#> y ~ 1 + x
#> 
#> Coefficients:
#> ─────────────────────────────────────────────────────────────────────────
#>                  Coef.  Std. Error      z  Pr(>|z|)  Lower 95%  Upper 95%
#> ─────────────────────────────────────────────────────────────────────────
#> (Intercept)  -0.385155    0.127502  -3.02    0.0025  -0.635055  -0.135256
#> x             0.881431    0.144892   6.08    <1e-08   0.597447   1.16541
#> ─────────────────────────────────────────────────────────────────────────

@assert abs(Optim.minimizer(fit2)[2] - coef(logit)[2]) < 1e-4
@assert abs(-Optim.minimum(fit2) - loglikelihood(logit)) < 1e-4
println("Part B validation PASSED: Optim MLE reproduces glm")
#> Part B validation PASSED: Optim MLE reproduces glm

fminunc is the Optimization Toolbox analogue of optim/minimize; with no toolbox, fminsearch (Nelder–Mead, base MATLAB) reaches the same optimum more slowly. Same parameterization as everywhere else.

rng(1031, 'twister');

% ---- Part A: normal linear model ----
n = 200;
x = randn(n, 1);
y = 2 + 0.75*x + 1.5*randn(n, 1);
X = [ones(n,1) x];

% theta = [b0; b1; log_sigma]
negloglik_normal = @(th) -sum(log(normpdf(y, X*th(1:2), exp(th(3)))));

opts  = optimset('TolX', 1e-10, 'TolFun', 1e-10, 'MaxFunEvals', 1e4);
fit   = fminsearch(negloglik_normal, [0; 0; 0], opts);
b_ml     = fit(2);
sigma_ml = exp(fit(3));

b_ols = X \ y;                        % backslash = least squares
resid = y - X*b_ols;
sigma_ols_nscaled = sqrt(sum(resid.^2) / n);   % ML sigma uses n, not n-k

assert(abs(b_ml - b_ols(2))            < 1e-4)
assert(abs(sigma_ml - sigma_ols_nscaled) < 1e-3)
disp('Part A validation PASSED: fminsearch MLE reproduces backslash')

% ---- Part B: logistic regression ----
n2 = 300;
x2 = randn(n2, 1);
y2 = double(rand(n2,1) < 1 ./ (1 + exp(-(-0.4 + 0.9*x2))));
X2 = [ones(n2,1) x2];

% log1p(exp(u)) via a numerically safe branch, as in the other languages
log1pexp = @(u) (u > 35).*u + (u <= 35).*log1p(exp(min(u, 35)));
negloglik_logit = @(b) sum(log1pexp(-(X2*b)).*y2 + log1pexp(X2*b).*(1 - y2));

fit2 = fminsearch(negloglik_logit, [0; 0], opts);

% glmfit needs the Statistics and Machine Learning Toolbox; it prepends the
% intercept itself, so pass x2 rather than X2.
b_glm = glmfit(x2, y2, 'binomial', 'link', 'logit');

assert(abs(fit2(2) - b_glm(2)) < 1e-4)
disp('Part B validation PASSED: fminsearch MLE reproduces glmfit')
#> Part A validation PASSED: fminsearch MLE reproduces backslash
#> Part B validation PASSED: fminsearch MLE reproduces glmfit

In SQL the closed forms are the ML estimates – REGR_SLOPE and REGR_INTERCEPT – and grid search plays the role of the optimizer. The chunks below execute at render time against an in-process DuckDB seeded in the setup chunk, so every number on this page is query output rather than a transcription. The dialect is ANSI where it counts; the Snowflake originals differ only in how rows get generated (GENERATOR/SEQ4 for generate_series, UNIFORM for random).

-- The data is simulated in the setup chunk above (n = 200,
-- y = 2 + 0.75x + N(0, 1.5)) via Box-Muller, since neither DuckDB nor
-- Snowflake ships a normal deviate. Note the two-stage CTE: you cannot write
-- AVG(POWER(y - REGR_INTERCEPT(y,x) OVER () ..., 2)) -- an aggregate may not
-- contain a window call. Compute the coefficients first, then the residual
-- variance against them.
WITH coef AS (
  SELECT regr_slope(y, x) AS b1, regr_intercept(y, x) AS b0
  FROM mytable
)
SELECT c.b1 AS b1_mle,
       c.b0 AS b0_mle,
       -- ML sigma^2 (divides by n, not n-k):
       avg(power(d.y - c.b0 - c.b1 * d.x, 2)) AS sigma2_mle
FROM mytable d CROSS JOIN coef c
GROUP BY c.b1, c.b0;
1 records
b1_mle b0_mle sigma2_mle
0.595 1.84 2.46

The estimates are real query output. Pulling the same rows into R confirms SQL is computing the maximum likelihood fit and not something adjacent to it:

d   <- DBI::dbGetQuery(con, "SELECT * FROM mytable")
ols <- lm(y ~ x, data = d)
sql <- DBI::dbGetQuery(con, "
  WITH coef AS (SELECT regr_slope(y,x) AS b1, regr_intercept(y,x) AS b0 FROM mytable)
  SELECT c.b1, c.b0, avg(power(d.y - c.b0 - c.b1*d.x, 2)) AS s2
  FROM mytable d CROSS JOIN coef c GROUP BY c.b1, c.b0;")

stopifnot(abs(sql$b1 - coef(ols)[["x"]]) < 1e-12,
          abs(sql$s2 - sum(residuals(ols)^2)/nrow(d)) < 1e-12)
cat(sprintf("SQL b1 = %.12f\n lm b1 = %.12f\n", sql$b1, coef(ols)[["x"]]))
#> SQL b1 = 0.595368434861
#>  lm b1 = 0.595368434861
cat("validation PASSED: REGR_SLOPE == lm, to machine precision\n")
#> validation PASSED: REGR_SLOPE == lm, to machine precision

Likelihood function from summary statistics

A point estimate and a 95% interval on the ratio scale are enough to reconstruct the whole function: recover the standard error from the interval width, lay down a grid, and transform one per grid point.

point <- 1.61; LL <- 0.997; UL <- 2.59
se <- log(UL / LL) / (2 * 1.96)

hr <- seq(0.5, 4.0, length.out = 600)
z  <- (log(hr) - log(point)) / se

support  <- exp(-(z^2) / 2)
deviance <- z^2
pvalue   <- 2 * (1 - pnorm(abs(z)))
svalue   <- -log2(pvalue)

inside <- hr[support >= 1/6.83]
stopifnot(abs(min(inside) - LL) < 0.02, abs(max(inside) - UL) < 0.02)
cat("validation PASSED: 1/6.83 LI == 95% CI\n")
#> validation PASSED: 1/6.83 LI == 95% CI

plot(hr, support, type = "l", log = "x", col = "#007C7C", lwd = 2,
     xaxt = "n", xlab = "Hazard Ratio (HR)", ylab = "Relative likelihood",
     main = "Relative Likelihood Function")
axis(1, at = c(0.5, 0.75, 1, 1.5, 2, 3, 4))
abline(v = 1, lty = 3, col = "#d46c5b")
abline(h = 1/6.83, col = "gray", lty = 2)

#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> Number of observations (_N) was 0, now 200.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Initial:      Log likelihood =  -806.4418
#> Alternative:  Log likelihood = -450.70073
#> Rescale:      Log likelihood = -429.09515
#> Rescale eq:   Log likelihood = -374.61873
#> Iteration 0:  Log likelihood = -374.61873  
#> Iteration 1:  Log likelihood = -348.89609  
#> Iteration 2:  Log likelihood = -343.02889  
#> Iteration 3:  Log likelihood = -342.99237  
#> Iteration 4:  Log likelihood = -342.99235  
#> 
#>                                                         Number of obs =    200
#>                                                         Wald chi2(1)  =  72.77
#> Log likelihood = -342.99235                             Prob > chi2   = 0.0000
#> 
#> ------------------------------------------------------------------------------
#>            y | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#> mu           |
#>            x |   .8119551   .0951852     8.53   0.000     .6253954    .9985147
#>        _cons |   1.868002   .0954381    19.57   0.000     1.680947    2.055057
#> -------------+----------------------------------------------------------------
#> lnsigma      |
#>        _cons |   .2960232        .05     5.92   0.000      .198025    .3940214
#> ------------------------------------------------------------------------------
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> check: ML slope  =  0.811955  OLS slope  =  0.811955
#> 
#> check: ML sigma  =  1.344501  OLS sigma (n-scaled) =  1.344501
#> 
#> 
#> 
#> Part A validation PASSED: ml reproduces regress
#> 
#> 
#> Number of observations (_N) was 0, now 300.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Initial:      Log likelihood = -207.94415
#> Alternative:  Log likelihood =  -205.2231
#> Rescale:      Log likelihood = -204.28183
#> Iteration 0:  Log likelihood = -204.28183  
#> Iteration 1:  Log likelihood = -187.36811  
#> Iteration 2:  Log likelihood = -187.34499  
#> Iteration 3:  Log likelihood = -187.34499  
#> 
#>                                                         Number of obs =    300
#>                                                         Wald chi2(1)  =  28.32
#> Log likelihood = -187.34499                             Prob > chi2   = 0.0000
#> 
#> ------------------------------------------------------------------------------
#>            y | Coefficient  Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>            x |   .7507449   .1410829     5.32   0.000     .4742275    1.027262
#>        _cons |  -.3434135    .124075    -2.77   0.006     -.586596    -.100231
#> ------------------------------------------------------------------------------
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> check: ml slope =  0.750745  logit slope =  0.750745
#> 
#> check: ml ll    =  -187.344987  logit ll    =  -187.344987
#> 
#> 
#> 
#> Part B validation PASSED: ml reproduces logit
#> 
#> 
#> All maximum likelihood constructions validated.

The inverse route: sweeping levels instead of parameters with cifunction

Everything above builds the curve by sweeping the parameter axis: fix a grid of candidate values, compute one (or one refit) per value, and read off a P-value. There is a second route to the same object that works from the other side: sweep the confidence level from 0 to 99.99% and compute the interval limits at each level. Plotting the limits against the levels traces exactly the same curve – the two constructions are inverse functions of each other, since the P-value function evaluated at a level’s limit returns that level’s .

A confidence interval function (also called a confidence curve, P-value function, or consonance function) displays every confidence interval around an estimate simultaneously. Ariel Linden’s cifunction Stata module implements the level-sweeping construction directly; the R equivalent is concurve. Install once from SSC:

ssc install cifunction
ssc install getregstats   // helper: recover SEs from p-values or CI limits

One convention to respect: with eform, cifunction takes se() on the ratio scale (the delta-method SE, ), does its computation on the log scale, and back-transforms. Same Brown et al. example as above, with the saved output asserted against both the reported interval and the manual z-grid construction: ## Python

import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt

point, LL, UL = 1.61, 0.997, 2.59
se = np.log(UL / LL) / (2 * 1.96)

hr = np.linspace(0.5, 4.0, 600)
z = (np.log(hr) - np.log(point)) / se

support  = np.exp(-(z**2) / 2)
deviance = z**2
pvalue   = 2 * (1 - norm.cdf(np.abs(z)))
svalue   = -np.log2(pvalue)

inside = hr[support >= 1/6.83]
assert abs(inside.min() - LL) < 0.02 and abs(inside.max() - UL) < 0.02
print("validation PASSED: 1/6.83 LI == 95% CI")
#> validation PASSED: 1/6.83 LI == 95% CI

fig, ax = plt.subplots(figsize=(7, 4.5))
ax.plot(hr, support, color="#007C7C")
ax.axvline(1, ls=":", color="#d46c5b"); ax.axhline(1/6.83, color="gray", alpha=0.4)
ax.set_xscale("log"); ax.set_xticks([0.5, 0.75, 1, 1.5, 2, 3, 4])
ax.set_xlabel("Hazard Ratio (HR)"); ax.set_ylabel("Relative likelihood")
ax.set_title("Relative Likelihood Function")
plt.show()

using Distributions, Plots
gr();

point, LL, UL = 1.61, 0.997, 2.59;
se = log(UL / LL) / (2 * 1.96);

hr = range(0.5, 4.0; length = 600);
z = (log.(hr) .- log(point)) ./ se;

support  = exp.(-(z .^ 2) ./ 2);
deviance = z .^ 2;
pvalue   = 2 .* (1 .- cdf.(Normal(), abs.(z)));
svalue   = .-log2.(pvalue);

inside = hr[support .>= 1/6.83];
@assert abs(minimum(inside) - LL) < 0.02 && abs(maximum(inside) - UL) < 0.02
println("validation PASSED: 1/6.83 LI == 95% CI")
#> validation PASSED: 1/6.83 LI == 95% CI

p = plot(hr, support; xscale = :log10, legend = false,
         xlabel = "Hazard Ratio (HR)", ylabel = "Relative likelihood",
         title = "Relative Likelihood Function");
hline!(p, [1/6.83]; color = :gray, alpha = 0.4);
vline!(p, [1.0]; linestyle = :dot, color = :red);
p

normcdf needs the Statistics Toolbox; the erfc form below is base MATLAB and gives the identical two-sided tail, so this tab runs either way.

point = 1.61; LL = 0.997; UL = 2.59;
se = log(UL / LL) / (2 * 1.96);

hr = linspace(0.5, 4.0, 600)';
z  = (log(hr) - log(point)) / se;

support  = exp(-(z.^2) / 2);
deviance = z.^2;
% two-sided p; erfc form avoids the Statistics Toolbox dependency
pvalue   = erfc(abs(z) / sqrt(2));
svalue   = -log2(pvalue);

inside = hr(support >= 1/6.83);
assert(abs(min(inside) - LL) < 0.02 && abs(max(inside) - UL) < 0.02)
disp('validation PASSED: 1/6.83 LI == 95% CI')

figure;
semilogx(hr, support, 'LineWidth', 2, 'Color', [0 0.486 0.486]);
xticks([0.5 0.75 1 1.5 2 3 4]);
xline(1,      ':', 'Color', [0.831 0.424 0.357]);
yline(1/6.83, '--', 'Color', [0.5 0.5 0.5]);
xlabel('Hazard Ratio (HR)'); ylabel('Relative likelihood');
title('Relative Likelihood Function');
#> validation PASSED: 1/6.83 LI == 95% CI

Neither Snowflake nor DuckDB ships erf(), so uses the Abramowitz–Stegun 26.2.17 polynomial ( per tail). SQL does not plot; it returns a table, and something else draws it – Power BI, or in this case R.

WITH params AS (
  SELECT 1.61::DOUBLE AS point,
         ln(2.59 / 0.997) / (2 * 1.96) AS se
),
grid AS (
  SELECT 0.5 + (i - 1) * (4.0 - 0.5) / 599 AS hr
  FROM generate_series(1, 600) AS t(i)
),
lik AS (
  SELECT g.hr,
         (ln(g.hr) - ln(p.point)) / p.se                     AS z,
         exp(-power((ln(g.hr) - ln(p.point)) / p.se, 2) / 2) AS support,
         power((ln(g.hr) - ln(p.point)) / p.se, 2)           AS deviance
  FROM grid g CROSS JOIN params p
),
pv AS (
  SELECT hr, z, support, deviance,
         2 * ( exp(-z*z/2) / sqrt(2*pi()) *
               ( 0.319381530 * (1/(1 + 0.2316419*abs(z)))
               - 0.356563782 * power(1/(1 + 0.2316419*abs(z)), 2)
               + 1.781477937 * power(1/(1 + 0.2316419*abs(z)), 3)
               - 1.821255978 * power(1/(1 + 0.2316419*abs(z)), 4)
               + 1.330274429 * power(1/(1 + 0.2316419*abs(z)), 5) ) ) AS pvalue
  FROM lik
)
SELECT hr, support, deviance, pvalue, -ln(pvalue)/ln(2) AS svalue
FROM pv ORDER BY hr LIMIT 5;
5 records
hr support deviance pvalue svalue
0.500 0 23.1 0 19.3
0.506 0 22.6 0 18.9
0.512 0 22.2 0 18.6
0.518 0 21.7 0 18.3
0.523 0 21.3 0 17.9

All four coordinate views come out of that one query. The interval it returns is against the reported , and the Abramowitz–Stegun polynomial tracks R’s pnorm to :

cur <- DBI::dbGetQuery(con, "
  WITH params AS (SELECT 1.61::DOUBLE AS point, ln(2.59/0.997)/(2*1.96) AS se),
  grid AS (SELECT 0.5 + (i-1)*(4.0-0.5)/599 AS hr FROM generate_series(1,600) AS t(i)),
  lik AS (SELECT g.hr, (ln(g.hr)-ln(p.point))/p.se AS z,
                 exp(-power((ln(g.hr)-ln(p.point))/p.se,2)/2) AS support,
                 power((ln(g.hr)-ln(p.point))/p.se,2) AS deviance
          FROM grid g CROSS JOIN params p),
  pv AS (SELECT hr, z, support, deviance,
           2 * ( exp(-z*z/2)/sqrt(2*pi()) *
             ( 0.319381530*(1/(1+0.2316419*abs(z)))
             - 0.356563782*power(1/(1+0.2316419*abs(z)),2)
             + 1.781477937*power(1/(1+0.2316419*abs(z)),3)
             - 1.821255978*power(1/(1+0.2316419*abs(z)),4)
             + 1.330274429*power(1/(1+0.2316419*abs(z)),5) ) ) AS pvalue
         FROM lik)
  SELECT hr, support, deviance, pvalue, -ln(pvalue)/ln(2) AS svalue
  FROM pv ORDER BY hr;")

inside <- cur$hr[cur$support >= 1/6.83]
exact  <- 2 * (1 - pnorm(abs((log(cur$hr) - log(1.61)) /
                             (log(2.59/0.997)/(2*1.96)))))
stopifnot(abs(min(inside) - 0.997) < 0.02, abs(max(inside) - 2.59) < 0.02,
          max(abs(cur$pvalue - exact)) < 1e-6)
cat(sprintf("1/6.83 interval: [%.4f, %.4f]\n", min(inside), max(inside)))
#> 1/6.83 interval: [1.0025, 2.5918]
cat(sprintf("max |Abramowitz-Stegun - pnorm| = %.2e\n",
            max(abs(cur$pvalue - exact))))
#> max |Abramowitz-Stegun - pnorm| = 1.49e-07
cat(sprintf("S-value at HR = 1: %.2f bits\n",
            cur$svalue[which.min(abs(cur$hr - 1))]))
#> S-value at HR = 1: 4.27 bits

op <- par(mfrow = c(2, 2), mar = c(4, 4, 2.5, 1))
panel <- function(y, ylab, main, href = NA) {
  plot(cur$hr, y, type = "l", log = "x", col = "#007C7C", lwd = 2,
       xaxt = "n", xlab = "Hazard Ratio", ylab = ylab, main = main)
  axis(1, at = c(0.5, 1, 2, 4))
  abline(v = 1, lty = 3, col = "#d46c5b")
  if (!is.na(href)) abline(h = href, col = "gray", lty = 2)
}
panel(cur$support,  "Relative likelihood", "Support",   1/6.83)
panel(cur$deviance, "Deviance",            "Deviance",  3.84)
panel(cur$pvalue,   "P-value",             "P-value (compatibility)", 0.05)
panel(cur$svalue,   "Bits of information", "S-value",   4.32)

par(op)

The dotted vertical line marks HR . Reading the bottom-right panel: the S-value there is 4.27 bits – about as much evidence against the null as five straight heads, which is not much.

Profile likelihood for a regression coefficient

The offset trick: to fix the coefficient of at , move into an offset and refit the model without . The refit re-maximizes the intercept and the other covariates at every grid point – that is what makes it a profile likelihood rather than a slice through the likelihood at the MLEs.

set.seed(1031)
n <- 300
x <- rnorm(n)
w <- rnorm(n)
y <- as.numeric(runif(n) < plogis(-0.3 + 0.8*x + 0.4*w))

full  <- glm(y ~ x + w, family = binomial())
bhat  <- coef(full)[["x"]]
sehat <- sqrt(diag(vcov(full)))[["x"]]

# offset trick: fix the coefficient of x at b and refit without x, so the
# intercept and w are re-maximized at every grid point
grid <- seq(bhat - 4*sehat, bhat + 4*sehat, length.out = 121)
ll <- vapply(grid, function(b)
  as.numeric(logLik(glm(y ~ w, family = binomial(), offset = b*x))),
  numeric(1))

support  <- exp(ll - max(ll))
deviance <- -2*(ll - max(ll))

stopifnot(abs(grid[which.max(support)] - bhat) < diff(grid)[1])
cat("validation PASSED: profile peak == full-model MLE\n")
#> validation PASSED: profile peak == full-model MLE

si <- range(grid[support >= 1/6.83])
cat(sprintf("1/6.83 support interval: [%.4f, %.4f]   Wald 95%% CI: [%.4f, %.4f]\n",
            si[1], si[2], bhat - 1.96*sehat, bhat + 1.96*sehat))
#> 1/6.83 support interval: [0.5593, 1.1148]   Wald 95% CI: [0.5459, 1.1090]

plot(grid, support, type = "l", col = "#007C7C", lwd = 2,
     xlab = "Coefficient of x (log-odds scale)",
     ylab = "Relative likelihood",
     main = "Profile Likelihood Function")
abline(v = bhat, lty = 2, col = "gray40")
abline(h = 1/6.83, col = "gray", lty = 2)

#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> 
#> Number of observations (_N) was 0, now 300.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> (file /tmp/S_70849.000001 not found)
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> profile peak =  0.748198   full-model MLE =  0.748198
#> 
#> 
#> 
#> 1/6.83 support interval: [0.4730, 1.0430]   Wald 95% CI: [0.4593, 1.0371]
#> 
#> 
#> 
#> Part B validation PASSED: profile peak and support interval check out
#> 
#> 
#> 
#> 
#> file profile_likelihood_logit.svg saved as SVG format
#> 
#> 
#> All likelihood-function constructions validated.

import numpy as np
import statsmodels.api as sm
import matplotlib.pyplot as plt

rng = np.random.default_rng(1031)
n = 300
x = rng.normal(0, 1, n)
w = rng.normal(0, 1, n)
y = (rng.uniform(size=n) < 1/(1 + np.exp(-(-0.3 + 0.8*x + 0.4*w)))).astype(float)

X_full = sm.add_constant(np.column_stack([x, w]))
full = sm.GLM(y, X_full, family=sm.families.Binomial()).fit()
bhat, sehat = full.params[1], full.bse[1]
X_nuis = sm.add_constant(w)

grid = np.linspace(bhat - 4*sehat, bhat + 4*sehat, 121)
ll = np.array([
    sm.GLM(y, X_nuis, family=sm.families.Binomial(), offset=b*x).fit().llf
    for b in grid
])
support = np.exp(ll - ll.max())

assert abs(grid[np.argmax(support)] - bhat) < grid[1] - grid[0]
print("validation PASSED: profile peak == full-model MLE")
#> validation PASSED: profile peak == full-model MLE

fig, ax = plt.subplots(figsize=(7, 4.5))
ax.plot(grid, support, color="#007C7C")
ax.axvline(bhat, ls="--", color="gray")
ax.axhline(1/6.83, color="gray", alpha=0.4)
ax.set_xlabel("Coefficient of x (log-odds)")
ax.set_ylabel("Relative likelihood")
ax.set_title("Profile Likelihood Function")
plt.show()

using Distributions, GLM, DataFrames, Random, Plots

rng = MersenneTwister(1031);
n = 300;
x = randn(rng, n);
w = randn(rng, n);
y = Float64.(rand(rng, n) .< 1 ./ (1 .+ exp.(-(-0.3 .+ 0.8 .* x .+ 0.4 .* w))));
df = DataFrame(y = y, x = x, w = w)
#> 300×3 DataFrame
#>  Row │ y        x            w
#>      │ Float64  Float64      Float64
#> ─────┼──────────────────────────────────
#>    1 │     0.0   1.53733      0.212606
#>    2 │     1.0   0.67567      2.18329
#>    3 │     0.0   0.529915    -0.411984
#>    4 │     0.0  -0.469345     0.243801
#>    5 │     0.0  -0.344731     0.181979
#>    6 │     1.0   0.00596521   0.102878
#>    7 │     0.0  -1.52318     -0.446261
#>    8 │     1.0   0.390072    -1.61528
#>   ⋮  │    ⋮          ⋮           ⋮
#>  294 │     1.0  -1.21183      0.133128
#>  295 │     1.0   0.35636     -0.373798
#>  296 │     1.0   0.125155     0.561362
#>  297 │     0.0  -1.63194     -0.0784788
#>  298 │     0.0  -0.464039    -0.214666
#>  299 │     1.0   1.21094     -0.392618
#>  300 │     1.0  -1.2263      -0.228791
#>                         285 rows omitted

full = glm(@formula(y ~ x + w), df, Binomial(), LogitLink())
#> StatsModels.TableRegressionModel{GeneralizedLinearModel{GLM.GlmResp{Vector{Float64}, Binomial{Float64}, LogitLink}, GLM.DensePredChol{Float64, CholeskyPivoted{Float64, Matrix{Float64}, Vector{Int64}}}}, Matrix{Float64}}
#> 
#> y ~ 1 + x + w
#> 
#> Coefficients:
#> ─────────────────────────────────────────────────────────────────────────
#>                  Coef.  Std. Error      z  Pr(>|z|)  Lower 95%  Upper 95%
#> ─────────────────────────────────────────────────────────────────────────
#> (Intercept)  -0.378416    0.133353  -2.84    0.0045  -0.639782  -0.117049
#> x             1.07021     0.160245   6.68    <1e-10   0.756132   1.38428
#> w             0.475207    0.130953   3.63    0.0003   0.218545   0.73187
#> ─────────────────────────────────────────────────────────────────────────
bhat, sehat = coef(full)[2], stderror(full)[2];

grid = range(bhat - 4*sehat, bhat + 4*sehat; length = 121);
ll = [loglikelihood(glm(@formula(y ~ w), df, Binomial(), LogitLink();
                        offset = b .* x)) for b in grid];
support = exp.(ll .- maximum(ll));

@assert abs(grid[argmax(support)] - bhat) < grid[2] - grid[1]
println("validation PASSED: profile peak == full-model MLE")
#> validation PASSED: profile peak == full-model MLE

p = plot(grid, support; legend = false,
         xlabel = "Coefficient of x (log-odds)",
         ylabel = "Relative likelihood",
         title = "Profile Likelihood Function");
vline!(p, [bhat]; linestyle = :dash, color = :gray);
hline!(p, [1/6.83]; color = :gray, alpha = 0.4);
p

glmfit takes an 'Offset' name–value pair, so the offset trick carries over directly: fix the coefficient of at , push into the offset, and refit on alone so the intercept and are re-maximized at every point.

rng(1031, 'twister');
n = 300;
x = randn(n, 1);
w = randn(n, 1);
y = double(rand(n,1) < 1 ./ (1 + exp(-(-0.3 + 0.8*x + 0.4*w))));

% full fit: MLE and Wald SE define the grid
[b_full, ~, stats] = glmfit([x w], y, 'binomial', 'link', 'logit');
bhat  = b_full(2);
sehat = stats.se(2);

grid = linspace(bhat - 4*sehat, bhat + 4*sehat, 121)';
ll   = zeros(size(grid));
for k = 1:numel(grid)
    % offset fixes b*x; only the intercept and w are re-estimated
    [~, dev] = glmfit(w, y, 'binomial', 'link', 'logit', ...
                      'Offset', grid(k)*x);
    ll(k) = -dev / 2;   % glmfit returns deviance = -2*loglik (up to a constant)
end

support  = exp(ll - max(ll));
deviance = -2*(ll - max(ll));

[~, imax] = max(support);
assert(abs(grid(imax) - bhat) < grid(2) - grid(1))
disp('validation PASSED: profile peak == full-model MLE')

si = [min(grid(support >= 1/6.83)), max(grid(support >= 1/6.83))];
fprintf('1/6.83 support interval: [%.4f, %.4f]   Wald 95%% CI: [%.4f, %.4f]\n', ...
        si(1), si(2), bhat - 1.96*sehat, bhat + 1.96*sehat);

figure;
plot(grid, support, 'LineWidth', 2, 'Color', [0 0.486 0.486]);
xline(bhat,   '--', 'Color', [0.4 0.4 0.4]);
yline(1/6.83, '--', 'Color', [0.5 0.5 0.5]);
xlabel('Coefficient of x (log-odds scale)');
ylabel('Relative likelihood');
title('Profile Likelihood Function');
#> validation PASSED: profile peak == full-model MLE
#> 1/6.83 support interval: [0.4246, 0.9635]   Wald 95% CI: [0.4116, 0.9579]

For the binomial family glmfit’s deviance is up to an additive constant that does not depend on the parameters, so it cancels in ll - max(ll) – which is all the relative likelihood needs.

SQL cannot iteratively refit a GLM (no IRLS), so a logistic profile is out of scope. But for the normal linear model the conditional MLEs of the intercept and given a fixed slope are closed-form, so profiling collapses to one AVG per candidate slope – a GROUP BY, not a refit loop. The profile-grid MLE matches REGR_SLOPE exactly here – the peak and the closed form agree to the last printed digit, since both are computed from the same rows. (Dialect landmine: Snowflake rejects correlated scalar subqueries here; the CROSS JOIN + GROUP BY below is the working idiom.)

-- One AVG per candidate slope: profiling collapses to a GROUP BY because the
-- conditional MLEs of the intercept and sigma^2 given a fixed slope are
-- closed-form. No refit loop, no IRLS.
WITH closed AS (
  SELECT regr_slope(y, x) AS b1, count(*) AS n FROM mytable
),
m AS (
  SELECT avg(x) AS xbar, avg(y) AS ybar FROM mytable
),
grid AS (
  SELECT c.b1 - 0.3 + (g.i - 1) * 0.6 / 400 AS b_cand
  FROM generate_series(1, 401) AS g(i) CROSS JOIN closed c
),
prof AS (
  SELECT g.b_cand,
         -(c.n / 2.0) *
         ln(avg(power(d.y - (m.ybar - g.b_cand * m.xbar)
                          - g.b_cand * d.x, 2))) AS prof_loglik
  FROM grid g
  CROSS JOIN mytable d
  CROSS JOIN m
  CROSS JOIN closed c
  GROUP BY g.b_cand, c.n
)
SELECT b_cand,
       exp(prof_loglik - max(prof_loglik) OVER ()) AS support
FROM prof ORDER BY b_cand LIMIT 5;
5 records
b_cand support
0.295 0.012
0.297 0.013
0.298 0.013
0.300 0.014
0.301 0.014

That returns all 401 grid points (truncated to five above). SQL has no plotting layer, so hand the result set to whatever draws your charts – here, R:

prof <- DBI::dbGetQuery(con, "
  WITH closed AS (SELECT regr_slope(y,x) AS b1, count(*) AS n FROM mytable),
  m AS (SELECT avg(x) AS xbar, avg(y) AS ybar FROM mytable),
  grid AS (SELECT c.b1 - 0.3 + (g.i-1) * 0.6/400 AS b_cand
           FROM generate_series(1,401) AS g(i) CROSS JOIN closed c),
  prof AS (
    SELECT g.b_cand,
           -(c.n/2.0) * ln(avg(power(d.y - (m.ybar - g.b_cand*m.xbar)
                                         - g.b_cand*d.x, 2))) AS prof_loglik
    FROM grid g CROSS JOIN mytable d CROSS JOIN m CROSS JOIN closed c
    GROUP BY g.b_cand, c.n)
  SELECT b_cand, exp(prof_loglik - max(prof_loglik) OVER ()) AS support
  FROM prof ORDER BY b_cand;")

b1_sql <- sql$b1
peak   <- prof$b_cand[which.max(prof$support)]
si     <- range(prof$b_cand[prof$support >= 1/6.83])
ci     <- confint(ols, "x", level = 0.95)

# the two self-checks from the top of the post
stopifnot(abs(peak - b1_sql) < diff(prof$b_cand)[1])
cat(sprintf("profile peak = %.10f   REGR_SLOPE = %.10f\n", peak, b1_sql))
#> profile peak = 0.5953684349   REGR_SLOPE = 0.5953684349
cat(sprintf("1/6.83 support interval: [%.4f, %.4f]\n", si[1], si[2]))
#> 1/6.83 support interval: [0.3989, 0.7919]
cat(sprintf("lm 95%% CI:               [%.4f, %.4f]\n", ci[1], ci[2]))
#> lm 95% CI:               [0.3974, 0.7933]

plot(prof$b_cand, prof$support, type = "l", col = "#007C7C", lwd = 2,
     xlab = "Candidate slope", ylab = "Relative likelihood",
     main = "Profile Likelihood Computed Entirely in SQL")
abline(v = b1_sql, lty = 2, col = "gray40")
abline(h = 1/6.83, col = "gray", lty = 2)
segments(si[1], 1/6.83, si[2], 1/6.83, col = "#d46c5b", lwd = 3)
text(b1_sql, 0.55, "REGR_SLOPE", cex = 0.75, col = "gray30", pos = 4)
text(mean(si), 1/6.83 - 0.055, "1/6.83 support interval",
     cex = 0.75, col = "#d46c5b")

The support interval drawn in orange and the Wald interval from lm agree to about three decimals, which is the same identity the R, Python, and Julia sections check – here with every number produced by a GROUP BY.

Existing R tools as alternatives

Everything above builds the likelihood by hand. In R you usually do not have to: several packages will do the maximizing and profiling for you, and their output reshapes into the same five-column object concurve plots. The one helper below is all the glue needed. See the concurve vignettes on likelihood tools and profile likelihoods for the longer treatment.

library(concurve)

# Reshape any (parameter grid, log-likelihood) pair into concurve's format.
as_concurve_lik <- function(values, loglik) {
  loglik_rel <- loglik - max(loglik)
  support    <- exp(loglik_rel)
  df <- data.frame(
    values        = values,
    likelihood    = support,
    loglikelihood = loglik_rel,
    support       = support,
    deviancestat  = -loglik_rel
  )
  class(df) <- c("data.frame", "concurve")
  df
}

set.seed(123)
x <- rnorm(50, mean = 9.3, sd = 3.2)
c(mean = mean(x), sd = sd(x))
#> mean   sd 
#> 9.41 2.96

Base R, no dependencies. optim finds the MLE and the Hessian gives Wald standard errors. To profile , substitute the conditional MLE of at each grid point rather than holding fixed.

nll_normal <- function(theta, x) {
  mu <- theta[1]; sigma <- theta[2]
  -sum(dnorm(x, mean = mu, sd = sigma, log = TRUE))
}

fit <- optim(c(mu = 8, sigma = 3), nll_normal, x = x,
             method = "BFGS", hessian = TRUE)
fit$par
#>    mu sigma 
#>  9.41  2.93
sqrt(diag(solve(fit$hessian)))
#>    mu sigma 
#> 0.415 0.293

n <- length(x)
mu_grid <- seq(fit$par["mu"] - 3, fit$par["mu"] + 3, length.out = 4000)
loglik_mu <- vapply(mu_grid, function(m) {
  sigma2_hat <- mean((x - m)^2)          # conditional MLE of sigma^2 given mu
  -(n / 2) * (log(2 * pi) + log(sigma2_hat) + 1)
}, numeric(1))

lik_optim <- as_concurve_lik(mu_grid, loglik_mu)
stopifnot(abs(mu_grid[which.max(loglik_mu)] - fit$par[["mu"]]) < 1e-2)

ggcurve(lik_optim, type = "l1", nullvalue = round(fit$par[["mu"]], 2),
        xaxis = expression(mu),
        title = "Likelihood for the Mean (via optim profiling)")

maxLik maximizes the log-likelihood directly, with no sign flipping, and prints a summary with standard errors.

library(maxLik)

loglik_normal <- function(theta, x) {
  sum(dnorm(x, mean = theta[1], sd = sqrt(theta[2]), log = TRUE))
}
theta_mle <- maxLik(loglik_normal, start = c(mu = 8, sig2 = 9), x = x)
summary(theta_mle)
#> --------------------------------------------
#> Maximum Likelihood estimation
#> Newton-Raphson maximisation, 7 iterations
#> Return code 8: successive function values within relative tolerance limit (reltol)
#> Log-Likelihood: -125 
#> 2  free parameters
#> Estimates:
#>      Estimate Std. error t value Pr(> t)    
#> mu      9.410      0.415   22.66 < 2e-16 ***
#> sig2    8.602      1.712    5.02 5.1e-07 ***
#> ---
#> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#> --------------------------------------------

loglik_mu2 <- vapply(mu_grid, function(m) {
  loglik_normal(c(m, mean((x - m)^2)), x = x)
}, numeric(1))

stopifnot(abs(coef(theta_mle)[[1]] - fit$par[["mu"]]) < 1e-3)

ggcurve(as_concurve_lik(mu_grid, loglik_mu2), type = "l1",
        nullvalue = round(coef(theta_mle)[[1]], 2),
        xaxis = expression(mu),
        title = "Likelihood for the Mean (via maxLik)")

bbmle::mle2 is the only route here that profiles for you. profile() returns the signed square-root deviance , so the relative log-likelihood is .

library(bbmle)

nll <- function(mu, sigma) -sum(dnorm(x, mean = mu, sd = sigma, log = TRUE))
m <- mle2(nll, start = list(mu = 8, sigma = 3), data = list(x = x))

pr    <- profile(m)
pr_mu <- as.data.frame(pr)
pr_mu <- pr_mu[pr_mu$param == "mu", ]

mu_vals <- pr_mu[["par.vals.mu"]]
if (is.null(mu_vals)) mu_vals <- pr_mu[["mu"]]

lik_bbmle <- as_concurve_lik(mu_vals, -0.5 * pr_mu$z^2)
stopifnot(abs(coef(m)[["mu"]] - fit$par[["mu"]]) < 1e-3)

ggcurve(lik_bbmle, type = "l1", nullvalue = round(coef(m)[["mu"]], 2),
        xaxis = expression(mu),
        title = "Profile Likelihood for the Mean (via bbmle)")

For a binary outcome carrying many nuisance parameters, cond gives exact conditional likelihood inference: it conditions the nuisance parameters out instead of profiling them.

library(cond)
data(babies)

mod_babies <- glm(cbind(r1, r2) ~ day + lull - 1,
                  family = binomial, data = babies)
cond_fit <- cond(mod_babies, offset = lullyes)
summary(cond_fit)
#> 
#>  Formula:  cbind(r1, r2) ~ day + lull - 1
#>  Family:  binomial
#>  Offset:  lullyes
#> 
#>           Estimate   Std. Error 
#> uncond.        1.43        0.734
#> cond.          1.27        0.689
#> 
#> Confidence intervals
#> --------------------
#>  level = 95 %
#>                                            lower two-sided upper
#> Wald pivot                              -0.00651            2.87
#> Wald pivot (cond. MLE)                  -0.08020            2.62
#> Likelihood root                          0.12300            3.09
#> Modified likelihood root                 0.01070            2.76
#> Modified likelihood root (cont. corr.)  -0.15200            3.10
#> 
#> Diagnostics:
#> ----------- 
#>   INF    NP 
#> 0.076 0.289 
#> 
#>  Approximation based on 20 points
plot(cond_fit, which = 2)

The pitfall: slicing is not profiling

Fixing a nuisance parameter instead of re-maximizing it is the most common way to get this wrong. Holding here yields a likelihood far too narrow, because the real is about 3.2 — the interval it implies is roughly a third of the honest width:

loglik_slice <- vapply(mu_grid, function(m) -0.5 * sum((x - m)^2), numeric(1))
lik_slice    <- as_concurve_lik(mu_grid, loglik_slice)

w <- function(o) {
  s <- curve_support(o, 6.83)
  s$upper.limit - s$lower.limit
}
c(slice_width = w(lik_slice), profile_width = w(lik_optim))
#>   slice_width profile_width 
#>         0.554         1.658
stopifnot(w(lik_slice) < w(lik_optim))

ggcurve(lik_slice, type = "l1", nullvalue = round(fit$par[["mu"]], 2),
        xaxis = expression(mu),
        title = "WRONG: mean likelihood with sigma^2 fixed at 1",
        subtitle = "Too narrow -- the real sigma is about 3.2, not 1")

Profile likelihood from ProfileLikelihood

profilelike.glm profiles one coefficient of a GLM over a grid, and curve_lik turns that straight into a concurve object — which then gives all four coordinate views from the top of this post off a single fit. The self-check: the profile’s peak must reproduce the coefficient from the ordinary glm.

library(ProfileLikelihood)
data(dataglm)

xx <- profilelike.glm(y ~ x1 + x2, data = dataglm, profile.theta = "group",
                      lo.theta = -1.5, hi.theta = 4.2,
                      family = binomial(link = "logit"), length = 500, round = 2)
lik_pl <- curve_lik(xx, data = dataglm)

si <- curve_support(lik_pl[[1]], 6.83)
si
k support.level lower.limit upper.limit mle
6.83 0.146 0.098 2.91 1.36 

# the profile peak must land on the full-model MLE
mod_pl <- glm(y ~ x1 + x2 + group, data = dataglm,
              family = binomial(link = "logit"))
stopifnot(abs(si$mle - coef(mod_pl)[["group"]]) < 0.01)
cat("validation PASSED: profile peak == glm coefficient\n")
#> validation PASSED: profile peak == glm coefficient

The same object, four ways — relative likelihood, log-likelihood, likelihood, and deviance:

ggcurve(lik_pl[[1]], type = "l1", nullvalue = TRUE)

ggcurve(lik_pl[[1]], type = "l2")

ggcurve(lik_pl[[1]], type = "l3")

ggcurve(lik_pl[[1]], type = "d")

Profile versus Wald, and where the 1/6.83 interval lands

curve_lik_glm profiles a coefficient directly. Comparing it against the Wald normal approximation on the same grid shows the skew the Wald interval discards — and the Wald support interval reproduces its own consonance interval, which is the identity from the top of this post:

mod <- glm(am ~ mpg, family = binomial, data = mtcars)
est <- coef(mod)[["mpg"]]
se  <- sqrt(diag(vcov(mod)))[["mpg"]]

lik_prof <- curve_lik_glm(mod, "mpg", steps = 200)
grid     <- lik_prof[[1]]$values
lik_wald <- as_concurve_lik(grid, dnorm(grid, mean = est, sd = se, log = TRUE))

cons <- curve_analytic(estimate = est, se = se, dist = "z")
ci95 <- cons[[1]][which.min(abs(cons[[1]]$intrvl.level - 0.95)),
                  c("lower.limit", "upper.limit")]

out <- rbind(
  `profile 1/6.83 support` = unlist(curve_support(lik_prof[[1]], 6.83)[c("lower.limit", "upper.limit")]),
  `Wald    1/6.83 support` = unlist(curve_support(lik_wald,      6.83)[c("lower.limit", "upper.limit")]),
  `Wald    95% consonance` = unlist(ci95)
)
out
#>                        lower.limit upper.limit
#> profile 1/6.83 support      0.1218       0.588
#> Wald    1/6.83 support      0.0818       0.532
#> Wald    95% consonance      0.0819       0.532

# Wald support interval and Wald consonance interval agree to ~1e-3
stopifnot(max(abs(out[2, ] - out[3, ])) < 1e-3)
cat("validation PASSED: Wald 1/6.83 support == Wald 95% consonance\n")
#> validation PASSED: Wald 1/6.83 support == Wald 95% consonance

The profile curve leans right of the symmetric Wald curve, and that lean is precisely what a Wald interval throws away:

plot_compare(lik_prof[[1]], lik_wald, type = "l1",
             xaxis = expression(beta[mpg]))

Profile likelihood for a regression coefficient

The offset trick: to fix the coefficient of at , move into an offset and refit the model without . The refit re-maximizes the intercept and the other covariates at every grid point – that is what makes it a profile likelihood rather than a slice through the likelihood at the MLEs.

version 16
clear all
set seed 1031

set obs 300
generate x = rnormal(0, 1)
generate w = rnormal(0, 1)                       // nuisance covariate
generate y = runiform() < invlogit(-0.3 + 0.8*x + 0.4*w)

* full fit: the MLE and Wald SE define the grid
quietly logit y x w
scalar bhat  = _b[x]
scalar sehat = _se[x]

* grid: MLE +/- 4 SEs, 121 points (each point costs one -logit- refit)
local npts  = 121
local blo   = bhat - 4*sehat
local bstep = 8*sehat / (`npts' - 1)

tempname results
tempfile profile
postfile `results' double(beta ll) using "`profile'", replace

forvalues i = 1/`npts' {
    local b = `blo' + (`i' - 1)*`bstep'
    capture drop off
    quietly generate double off = `b'*x
    quietly logit y w, offset(off)
    post `results' (`b') (e(ll))
}
postclose `results'

use "`profile'", clear

* normalize into support / deviance
quietly summarize ll
generate support  = exp(ll - r(max))
generate deviance = -2*(ll - r(max))

* --- validation 1: profile peak sits at the full-model MLE ---
quietly summarize beta if support == 1
display as text _n "profile peak = " as result %9.6f r(mean)  ///
    as text "   full-model MLE = " as result %9.6f bhat
assert abs(r(mean) - bhat) < `bstep'   // within one grid step

* --- validation 2: 1/6.83 support interval ~ Wald 95% CI ---
quietly summarize beta if support >= 1/6.83
display as text "1/6.83 support interval: [" as result %6.4f r(min)   ///
    as text ", " as result %6.4f r(max) as text "]"                    ///
    as text "   Wald 95% CI: [" as result %6.4f bhat - 1.96*sehat      ///
    as text ", " as result %6.4f bhat + 1.96*sehat as text "]"
* profile and Wald agree closely here (smooth likelihood, moderate n);
* they diverge exactly as much as the likelihood is skewed
assert abs(r(min) - (bhat - 1.96*sehat)) < 3*`bstep'
assert abs(r(max) - (bhat + 1.96*sehat)) < 3*`bstep'
display as result "validation PASSED: profile peak and support interval check out"

* --- plots ---
twoway (line support beta),                                            ///
    xline(`=bhat', lpattern(dash) lcolor(gs10))                        ///
    xline(0, lpattern(dot) lcolor(red))                                ///
    yline(0.1464, lcolor(gs12))                                        ///
    title("Profile Likelihood Function for a Logistic Coefficient")    ///
    subtitle("Nuisance parameters re-maximized at each grid point")    ///
    xtitle("Coefficient of x (log-odds scale)")                        ///
    ytitle("Relative Likelihood") name(prof_lik, replace)

twoway (line deviance beta),                                           ///
    xline(`=bhat', lpattern(dash) lcolor(gs10))                        ///
    yline(3.84, lcolor(gs12))                                          ///
    title("Profile Deviance Function")                                 ///
    subtitle("Horizontal line at 3.84; crossings = 95% profile CI")    ///
    xtitle("Coefficient of x (log-odds scale)")                        ///
    ytitle("Profile Deviance") name(prof_dev, replace)

graph combine prof_lik prof_dev, cols(2)
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> Number of observations (_N) was 0, now 300.
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> (file /tmp/S_76071.000001 not found)
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> profile peak =  0.748198   full-model MLE =  0.748198
#> 
#> 
#> 
#> 1/6.83 support interval: [0.4730, 1.0430]   Wald 95% CI: [0.4593, 1.0371]
#> 
#> 
#> 
#> validation PASSED: profile peak and support interval check out

Validation status

Language Runtime validated Notes
Python yes – all asserts pass scipy/statsmodels/matplotlib
Julia yes – all asserts pass (numerics) Plots.jl output not exercised
MATLAB yes – executes at render via matlab -batch fminsearch is base MATLAB; glmfit/normcdf need the Statistics Toolbox
SQL (DuckDB, live) yes – executes at render; asserts vs lm/pnorm agreement to machine precision; plotting delegated to R
SQL (Snowflake) yes – all three parts run live there same logic, dialect differs in row generation
Stata syntax verified against [R] ml, [P] postfile run once locally; asserts are the safety net
Stata cifunction asserts against reported CI + z-grid construction SSC module; conventions per the cifunction guide

Package Citations

citation("Statamarkdown")
#> To cite package ‘Statamarkdown’ in publications use:
#> 
#>   Hemken D, Palmer T (2026). _Statamarkdown: 'Stata' Markdown_. doi:10.32614/CRAN.package.Statamarkdown
#>   <https://doi.org/10.32614/CRAN.package.Statamarkdown>. R package version 1.0.0,
#>   <https://CRAN.R-project.org/package=Statamarkdown>.
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {Statamarkdown: 'Stata' Markdown},
#>     author = {Doug Hemken and Tom Palmer},
#>     year = {2026},
#>     note = {R package version 1.0.0},
#>     url = {https://CRAN.R-project.org/package=Statamarkdown},
#>     doi = {10.32614/CRAN.package.Statamarkdown},
#>   }
citation("reticulate")
#> To cite package ‘reticulate’ in publications use:
#> 
#>   Kalinowski T, Ushey K, Allaire J, Tang Y (2026). _reticulate: Interface to 'Python'_.
#>   doi:10.32614/CRAN.package.reticulate <https://doi.org/10.32614/CRAN.package.reticulate>. R package version 1.47.0,
#>   <https://CRAN.R-project.org/package=reticulate>.
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {reticulate: Interface to 'Python'},
#>     author = {Tomasz Kalinowski and Kevin Ushey and JJ Allaire and Yuan Tang},
#>     year = {2026},
#>     note = {R package version 1.47.0},
#>     url = {https://CRAN.R-project.org/package=reticulate},
#>     doi = {10.32614/CRAN.package.reticulate},
#>   }
citation("JuliaCall")
#> To cite package ‘JuliaCall’ in publications use:
#> 
#>   Li C (2019). “JuliaCall: an R package for seamless integration between R and Julia.” _The Journal of Open Source
#>   Software_, *4*(35), 1284. doi:10.21105/joss.01284 <https://doi.org/10.21105/joss.01284>.
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Article{JuliaCall,
#>     author = {Changcheng Li},
#>     title = {{JuliaCall}: an {R} package for seamless integration between {R} and {Julia}},
#>     journal = {The Journal of Open Source Software},
#>     publisher = {The Open Journal},
#>     year = {2019},
#>     volume = {4},
#>     number = {35},
#>     pages = {1284},
#>     doi = {10.21105/joss.01284},
#>   }

Statistical Environments

R Environment

#> ─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.6.1 (2026-06-24)
#>  os       macOS Golden Gate 27.0
#>  system   aarch64, darwin25.4.0
#>  ui       unknown
#>  language (EN)
#>  collate  C
#>  ctype    C.UTF-8
#>  tz       America/New_York
#>  date     2026-09-13
#>  pandoc   3.11 @ /opt/homebrew/bin/ (via rmarkdown)
#>  quarto   1.10.18 @ /usr/local/bin/quarto
#> 
#> ─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  package           * version    date (UTC) lib source
#>  abind               1.4-8      2024-09-12 [1] CRAN (R 4.6.1)
#>  Amelia            * 1.8.3      2024-11-08 [1] CRAN (R 4.6.1)
#>  arm                 1.15-3     2026-04-15 [1] CRAN (R 4.6.1)
#>  arrayhelpers        1.1-2      2026-07-24 [1] CRAN (R 4.6.1)
#>  backports           1.5.1      2026-04-03 [1] CRAN (R 4.6.1)
#>  base              * 4.6.1      2026-06-24 [2] local
#>  base64enc           0.1-6      2026-02-02 [1] CRAN (R 4.6.1)
#>  bayesplot         * 1.16.0     2026-08-25 [1] CRAN (R 4.6.1)
#>  bbmle             * 1.0.26     2026-08-31 [1] CRAN (R 4.6.1)
#>  bdsmatrix           1.3-7      2024-03-02 [1] CRAN (R 4.6.1)
#>  bitops              1.1-0      2026-07-30 [1] CRAN (R 4.6.1)
#>  blogdown          * 1.24       2026-06-19 [1] CRAN (R 4.6.1)
#>  boot              * 1.3-32     2025-08-29 [1] CRAN (R 4.6.1)
#>  bootImpute        * 1.3.0      2025-12-15 [1] CRAN (R 4.6.1)
#>  bridgesampling      1.2-1      2025-11-19 [1] CRAN (R 4.6.1)
#>  brms              * 2.23.0     2025-09-09 [1] CRAN (R 4.6.1)
#>  Brobdingnag         1.2-9      2022-10-19 [1] CRAN (R 4.6.1)
#>  broom             * 1.0.13     2026-05-14 [1] CRAN (R 4.6.1)
#>  broom.mixed       * 0.2.9.7    2026-02-17 [1] CRAN (R 4.6.1)
#>  Cairo             * 1.7-0      2025-10-29 [1] CRAN (R 4.6.1)
#>  car               * 3.1-5      2026-02-03 [1] CRAN (R 4.6.1)
#>  carData           * 3.0-6      2026-01-30 [1] CRAN (R 4.6.1)
#>  caTools             1.18.4     2026-07-20 [1] CRAN (R 4.6.1)
#>  checkmate         * 2.3.4      2026-02-03 [1] CRAN (R 4.6.1)
#>  class               7.3-24     2026-08-03 [1] CRAN (R 4.6.1)
#>  cli               * 3.6.6      2026-04-09 [1] CRAN (R 4.6.1)
#>  clipr               0.8.1      2026-05-25 [1] CRAN (R 4.6.1)
#>  cluster             2.1.8.3    2026-07-30 [1] CRAN (R 4.6.1)
#>  coda              * 0.19-4.1   2024-01-31 [1] CRAN (R 4.6.1)
#>  codetools           0.2-20     2024-03-31 [1] CRAN (R 4.6.0)
#>  colorspace        * 2.1-3      2026-07-12 [1] CRAN (R 4.6.1)
#>  compiler            4.6.1      2026-06-24 [2] local
#>  concurve          * 3.0.5      2026-09-13 [1] local
#>  cond              * 1.2-4      2025-05-25 [1] CRAN (R 4.6.1)
#>  cowplot           * 1.2.0      2025-07-07 [1] CRAN (R 4.6.1)
#>  crayon              1.5.3      2024-06-20 [1] CRAN (R 4.6.1)
#>  curl                8.0.0      2026-08-25 [1] CRAN (R 4.6.1)
#>  data.table          1.18.6.1   2026-08-24 [1] CRAN (R 4.6.1)
#>  datasets          * 4.6.1      2026-06-24 [2] local
#>  DBI               * 1.3.0      2026-02-25 [1] CRAN (R 4.6.1)
#>  DEoptimR            1.2-1      2026-08-20 [1] CRAN (R 4.6.1)
#>  desc                1.4.3      2023-12-10 [1] CRAN (R 4.6.1)
#>  details             0.4.0      2025-02-09 [1] CRAN (R 4.6.0)
#>  dichromat           2.0-1      2026-07-22 [1] CRAN (R 4.6.1)
#>  digest              0.6.39     2025-11-19 [1] CRAN (R 4.6.1)
#>  distributional      0.9.0      2026-09-08 [1] CRAN (R 4.6.1)
#>  doParallel        * 1.0.17     2022-02-07 [1] CRAN (R 4.6.1)
#>  doRNG               1.8.6.3    2026-02-05 [1] CRAN (R 4.6.1)
#>  dplyr             * 1.2.1      2026-04-03 [1] CRAN (R 4.6.1)
#>  duckdb              1.5.5      2026-07-25 [1] CRAN (R 4.6.1)
#>  e1071               1.7-17     2025-12-18 [1] CRAN (R 4.6.1)
#>  emmeans             2.0.4      2026-07-15 [1] CRAN (R 4.6.1)
#>  estimability        2.0.0      2026-06-26 [1] CRAN (R 4.6.1)
#>  evaluate            1.0.5      2025-08-27 [1] CRAN (R 4.6.1)
#>  extremevalues       2.4.1      2024-12-17 [1] CRAN (R 4.6.1)
#>  farver              2.1.2      2024-05-13 [1] CRAN (R 4.6.1)
#>  fastmap             1.2.0      2024-05-15 [1] CRAN (R 4.6.1)
#>  forcats           * 1.0.1      2025-09-25 [1] CRAN (R 4.6.1)
#>  foreach           * 1.5.2      2022-02-02 [1] CRAN (R 4.6.1)
#>  foreign             0.8-91     2026-01-29 [1] CRAN (R 4.6.1)
#>  Formula             1.2-6      2026-08-03 [1] CRAN (R 4.6.1)
#>  fs                  2.1.0      2026-04-18 [1] CRAN (R 4.6.1)
#>  furrr               0.4.0      2026-03-31 [1] CRAN (R 4.6.1)
#>  future            * 1.75.0     2026-07-20 [1] CRAN (R 4.6.1)
#>  future.apply      * 1.20.2     2026-02-20 [1] CRAN (R 4.6.1)
#>  gamlss            * 5.5-0      2025-08-19 [1] CRAN (R 4.6.1)
#>  gamlss.data       * 6.0-7      2025-09-04 [1] CRAN (R 4.6.1)
#>  gamlss.dist       * 6.1-11     2026-09-10 [1] CRAN (R 4.6.1)
#>  generics            0.1.4      2025-05-09 [1] CRAN (R 4.6.1)
#>  ggcorrplot        * 0.3.0      2026-07-24 [1] CRAN (R 4.6.1)
#>  ggdist              3.3.3      2025-04-23 [1] CRAN (R 4.6.1)
#>  ggplot2           * 4.0.3      2026-04-22 [1] CRAN (R 4.6.1)
#>  ggtext            * 0.2.0      2026-08-28 [1] CRAN (R 4.6.1)
#>  glmnet              5.0        2026-05-04 [1] CRAN (R 4.6.1)
#>  globals             0.19.1     2026-03-13 [1] CRAN (R 4.6.1)
#>  glue                1.8.1      2026-04-17 [1] CRAN (R 4.6.1)
#>  graphics          * 4.6.1      2026-06-24 [2] local
#>  grDevices         * 4.6.1      2026-06-24 [2] local
#>  grid              * 4.6.1      2026-06-24 [2] local
#>  gridExtra           2.3.1      2026-06-25 [1] CRAN (R 4.6.1)
#>  gridtext            0.1.6      2026-02-19 [1] CRAN (R 4.6.0)
#>  gtable              0.3.6      2024-10-25 [1] CRAN (R 4.6.1)
#>  gtsummary         * 2.6.1      2026-09-04 [1] CRAN (R 4.6.1)
#>  here              * 1.0.2      2025-09-15 [1] CRAN (R 4.6.1)
#>  Hmisc             * 5.3-0      2026-09-06 [1] CRAN (R 4.6.1)
#>  hms                 1.1.4      2025-10-17 [1] CRAN (R 4.6.1)
#>  htmlTable           2.5.0      2026-04-22 [1] CRAN (R 4.6.1)
#>  htmltools         * 0.5.9      2025-12-04 [1] CRAN (R 4.6.1)
#>  htmlwidgets         1.6.4      2023-12-06 [1] CRAN (R 4.6.1)
#>  httr                1.4.9      2026-09-01 [1] CRAN (R 4.6.1)
#>  ImputeRobust      * 1.3-1      2018-11-30 [1] CRAN (R 4.6.1)
#>  inline              0.3.21     2025-01-09 [1] CRAN (R 4.6.1)
#>  insight             1.5.4      2026-09-05 [1] CRAN (R 4.6.1)
#>  iterators         * 1.0.14     2022-02-05 [1] CRAN (R 4.6.1)
#>  itertools           0.1-3      2014-03-12 [1] CRAN (R 4.6.1)
#>  jomo                2.7-6      2023-04-15 [1] CRAN (R 4.6.1)
#>  jsonlite            2.0.0      2025-03-27 [1] CRAN (R 4.6.1)
#>  JuliaCall           0.17.6     2024-12-07 [1] CRAN (R 4.6.1)
#>  kableExtra        * 1.4.1      2026-07-08 [1] CRAN (R 4.6.1)
#>  knitr             * 1.52       2026-09-06 [1] CRAN (R 4.6.1)
#>  labeling            0.4.3      2023-08-29 [1] CRAN (R 4.6.1)
#>  laeken              0.5.3      2024-01-25 [1] CRAN (R 4.6.1)
#>  latex2exp         * 0.9.8      2026-01-09 [1] CRAN (R 4.6.1)
#>  lattice           * 0.23-1     2026-08-12 [1] CRAN (R 4.6.1)
#>  lifecycle           1.0.5      2026-01-08 [1] CRAN (R 4.6.1)
#>  listenv             1.0.0      2026-06-22 [1] CRAN (R 4.6.1)
#>  lme4                2.0-6      2026-07-16 [1] CRAN (R 4.6.1)
#>  lmtest              0.9-40     2022-03-21 [1] CRAN (R 4.6.1)
#>  loo               * 2.10.1     2026-07-24 [1] CRAN (R 4.6.1)
#>  lubridate         * 1.9.5      2026-02-04 [1] CRAN (R 4.6.1)
#>  magick              2.9.1      2026-02-28 [1] CRAN (R 4.6.1)
#>  magrittr          * 2.0.5      2026-04-04 [1] CRAN (R 4.6.1)
#>  MASS              * 7.3-66     2026-07-15 [1] CRAN (R 4.6.1)
#>  Matrix            * 1.7-6      2026-07-25 [1] CRAN (R 4.6.1)
#>  MatrixModels        0.5-4      2025-03-26 [1] CRAN (R 4.6.1)
#>  matrixStats         1.5.0      2025-01-07 [1] CRAN (R 4.6.1)
#>  maxLik            * 1.5-2.2    2025-12-29 [1] CRAN (R 4.6.0)
#>  mcmc                0.9-8      2023-11-16 [1] CRAN (R 4.6.1)
#>  MCMCpack          * 1.7-1      2024-08-27 [1] CRAN (R 4.6.1)
#>  methods           * 4.6.1      2026-06-24 [2] local
#>  mgcv              * 1.9-4      2025-11-07 [1] CRAN (R 4.6.0)
#>  mi                * 1.3.1      2026-07-28 [1] CRAN (R 4.6.1)
#>  mice              * 3.19.0     2025-12-10 [1] CRAN (R 4.6.1)
#>  miceadds          * 3.20-10    2026-05-28 [1] CRAN (R 4.6.1)
#>  miceFast          * 0.9.1      2026-02-26 [1] CRAN (R 4.6.1)
#>  minqa               1.2.8      2024-08-17 [1] CRAN (R 4.6.1)
#>  miscTools         * 0.6-30     2026-01-20 [1] CRAN (R 4.6.1)
#>  missForest        * 1.6.1      2025-10-26 [1] CRAN (R 4.6.1)
#>  mitml             * 0.4-5      2023-03-08 [1] CRAN (R 4.6.1)
#>  mitools             2.7        2026-08-31 [1] CRAN (R 4.6.1)
#>  multcomp            1.4-32     2026-08-21 [1] CRAN (R 4.6.1)
#>  mvtnorm           * 1.4-2      2026-07-12 [1] CRAN (R 4.6.1)
#>  nlme              * 3.1-171    2026-09-01 [1] CRAN (R 4.6.1)
#>  nloptr              2.2.1      2025-03-17 [1] CRAN (R 4.6.1)
#>  nnet                7.3-21     2026-08-03 [1] CRAN (R 4.6.1)
#>  numDeriv            2016.8-1.1 2019-06-06 [1] CRAN (R 4.6.1)
#>  opdisDownsampling   1.7        2026-09-04 [1] CRAN (R 4.6.1)
#>  otel                0.2.0      2025-08-29 [1] CRAN (R 4.6.1)
#>  pan                 2.0        2026-06-30 [1] CRAN (R 4.6.1)
#>  parallel          * 4.6.1      2026-06-24 [2] local
#>  parallelly        * 1.48.0     2026-06-29 [1] CRAN (R 4.6.1)
#>  pbmcapply         * 1.5.1      2022-04-28 [1] CRAN (R 4.6.1)
#>  performance       * 0.18.2     2026-09-10 [1] CRAN (R 4.6.1)
#>  pillar              1.11.1     2025-09-17 [1] CRAN (R 4.6.1)
#>  pkgbuild            1.4.8      2025-05-26 [1] CRAN (R 4.6.1)
#>  pkgconfig           2.0.3      2019-09-22 [1] CRAN (R 4.6.1)
#>  plyr                1.8.9      2023-10-02 [1] CRAN (R 4.6.1)
#>  png                 0.1-9      2026-03-15 [1] CRAN (R 4.6.0)
#>  polspline           1.1.25     2024-05-10 [1] CRAN (R 4.6.1)
#>  posterior         * 1.7.0      2026-04-01 [1] CRAN (R 4.6.1)
#>  pracma              2.4.6      2025-10-22 [1] CRAN (R 4.6.1)
#>  prettyunits         1.2.0      2023-09-24 [1] CRAN (R 4.6.1)
#>  ProfileLikelihood * 1.3        2023-08-25 [1] CRAN (R 4.6.1)
#>  progress          * 1.2.3      2023-12-06 [1] CRAN (R 4.6.1)
#>  proxy               0.4-29     2025-12-29 [1] CRAN (R 4.6.1)
#>  purrr             * 1.2.2      2026-04-10 [1] CRAN (R 4.6.1)
#>  qqconf              1.3.2      2023-04-14 [1] CRAN (R 4.6.1)
#>  qqplotr           * 0.0.7      2025-09-05 [1] CRAN (R 4.6.1)
#>  quantreg          * 6.1        2025-03-10 [1] CRAN (R 4.6.1)
#>  QuickJSR            1.11.0     2026-08-21 [1] CRAN (R 4.6.1)
#>  R6                  2.6.1      2025-02-15 [1] CRAN (R 4.6.1)
#>  randomForest      * 4.7-1.2    2024-09-22 [1] CRAN (R 4.6.1)
#>  ranger              0.18.0     2026-01-16 [1] CRAN (R 4.6.1)
#>  rbibutils           2.4.1      2026-01-21 [1] CRAN (R 4.6.1)
#>  RColorBrewer        1.1-3      2022-04-03 [1] CRAN (R 4.6.1)
#>  Rcpp              * 1.1.2      2026-07-05 [1] CRAN (R 4.6.1)
#>  RcppParallel        6.2.1      2026-08-27 [1] CRAN (R 4.6.1)
#>  Rdpack              2.6.6      2026-02-08 [1] CRAN (R 4.6.1)
#>  readr             * 2.2.0      2026-02-19 [1] CRAN (R 4.6.1)
#>  reformulas          0.4.4      2026-02-02 [1] CRAN (R 4.6.1)
#>  rematch2            2.1.2      2020-05-01 [1] CRAN (R 4.6.1)
#>  reshape2          * 1.4.5      2025-11-12 [1] CRAN (R 4.6.1)
#>  reticulate        * 1.47.0     2026-09-03 [1] CRAN (R 4.6.1)
#>  rlang               1.3.0      2026-07-05 [1] CRAN (R 4.6.1)
#>  rmarkdown         * 2.32       2026-09-01 [1] CRAN (R 4.6.1)
#>  rms               * 8.1-1      2026-02-18 [1] CRAN (R 4.6.1)
#>  rngtools            1.5.2      2021-09-20 [1] CRAN (R 4.6.1)
#>  robustbase          0.99-7     2026-02-05 [1] CRAN (R 4.6.1)
#>  rpart               4.1.27     2026-03-27 [1] CRAN (R 4.6.0)
#>  rprojroot           2.1.1      2025-08-26 [1] CRAN (R 4.6.1)
#>  rstan             * 2.32.7     2025-03-10 [1] CRAN (R 4.6.1)
#>  rstantools          2.7.1      2026-09-04 [1] Github (stan-dev/rstantools@4d84659)
#>  rstudioapi          0.19.0     2026-06-11 [1] CRAN (R 4.6.1)
#>  S7                  0.2.2      2026-04-22 [1] CRAN (R 4.6.1)
#>  sandwich            3.1-3      2026-08-03 [1] CRAN (R 4.6.1)
#>  scales              1.4.0      2025-04-24 [1] CRAN (R 4.6.1)
#>  sessioninfo         1.2.4      2026-06-04 [1] CRAN (R 4.6.1)
#>  shape               1.4.6.1    2024-02-23 [1] CRAN (R 4.6.1)
#>  showtext          * 0.9-8      2026-03-21 [1] CRAN (R 4.6.1)
#>  showtextdb        * 3.0        2020-06-04 [1] CRAN (R 4.6.1)
#>  sp                  2.2-3      2026-07-19 [1] CRAN (R 4.6.1)
#>  SparseM           * 1.84-2     2024-07-17 [1] CRAN (R 4.6.1)
#>  splines           * 4.6.1      2026-06-24 [2] local
#>  StanHeaders       * 2.39.1     2026-09-02 [1] CRAN (R 4.6.1)
#>  Statamarkdown     * 1.0.0      2026-08-21 [1] CRAN (R 4.6.1)
#>  statmod           * 1.5.2      2026-05-17 [1] CRAN (R 4.6.1)
#>  stats             * 4.6.1      2026-06-24 [2] local
#>  stats4            * 4.6.1      2026-06-24 [2] local
#>  stringi             1.8.9      2026-08-04 [1] CRAN (R 4.6.1)
#>  stringr           * 1.6.0      2025-11-04 [1] CRAN (R 4.6.1)
#>  survival          * 3.8-12     2026-09-09 [1] CRAN (R 4.6.1)
#>  svglite           * 2.2.2      2025-10-21 [1] CRAN (R 4.6.1)
#>  svgPanZoom          0.3.4      2020-02-15 [1] CRAN (R 4.6.1)
#>  svUnit              1.0.8      2025-08-26 [1] CRAN (R 4.6.1)
#>  sysfonts          * 0.8.9      2024-03-02 [1] CRAN (R 4.6.1)
#>  systemfonts         1.3.2      2026-03-05 [1] CRAN (R 4.6.1)
#>  tensorA             0.36.2.1   2023-12-13 [1] CRAN (R 4.6.0)
#>  texPreview        * 2.1.0      2024-01-24 [1] CRAN (R 4.6.0)
#>  textshaping         1.0.5      2026-03-06 [1] CRAN (R 4.6.0)
#>  TH.data             1.1-5      2025-11-17 [1] CRAN (R 4.6.0)
#>  tibble            * 3.3.1      2026-01-11 [1] CRAN (R 4.6.1)
#>  tidybayes         * 3.0.7      2024-09-15 [1] CRAN (R 4.6.1)
#>  tidyr             * 1.3.2      2025-12-19 [1] CRAN (R 4.6.1)
#>  tidyselect          1.2.1      2024-03-11 [1] CRAN (R 4.6.1)
#>  tidyverse         * 2.0.0      2023-02-22 [1] CRAN (R 4.6.1)
#>  timechange          0.4.0      2026-01-29 [1] CRAN (R 4.6.1)
#>  tinytex           * 0.60       2026-06-16 [1] CRAN (R 4.6.1)
#>  tools               4.6.1      2026-06-24 [2] local
#>  twosamples          2.0.1      2023-06-23 [1] CRAN (R 4.6.1)
#>  tzdb                0.5.0      2025-03-15 [1] CRAN (R 4.6.1)
#>  utils             * 4.6.1      2026-06-24 [2] local
#>  V8                  8.2.0      2026-04-21 [1] CRAN (R 4.6.0)
#>  vcd                 1.4-14     2026-07-29 [1] CRAN (R 4.6.1)
#>  vctrs               0.7.3      2026-04-11 [1] CRAN (R 4.6.1)
#>  VIM               * 7.3.1      2026-09-11 [1] CRAN (R 4.6.1)
#>  viridisLite         0.4.3      2026-02-04 [1] CRAN (R 4.6.1)
#>  wesanderson       * 0.3.7      2023-10-31 [1] CRAN (R 4.6.1)
#>  whisker             0.4.1      2022-12-05 [1] CRAN (R 4.6.1)
#>  withr               3.0.3      2026-06-19 [1] CRAN (R 4.6.1)
#>  xfun              * 0.60       2026-07-09 [1] CRAN (R 4.6.1)
#>  xgboost             3.2.1.1    2026-03-18 [1] CRAN (R 4.6.1)
#>  xml2                1.6.0      2026-06-22 [1] CRAN (R 4.6.1)
#>  xtable            * 1.8-8      2026-02-22 [1] CRAN (R 4.6.1)
#>  yaml                2.3.12     2025-12-10 [1] CRAN (R 4.6.1)
#>  yardstick         * 1.4.0      2026-04-07 [1] CRAN (R 4.6.1)
#>  zoo                 1.9-0      2026-07-31 [1] CRAN (R 4.6.1)
#> 
#>  [1] /opt/homebrew/lib/R/4.6/site-library
#>  [2] /opt/homebrew/Cellar/r/4.6.1/lib/R/library
#>  * ── Packages attached to the search path.
#> 
#> ─ External software ────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting        value
#>  cairo          1.18.4
#>  cairoFT        2.14.3/2.18.1
#>  pango
#>  png
#>  jpeg
#>  tiff
#>  tcl            9.0
#>  curl           8.7.1
#>  zlib           1.2.12
#>  bzlib          1.0.8, 13-Jul-2019
#>  xz             5.8.3
#>  deflate
#>  zstd           1.5.7
#>  PCRE           10.48 2026-08-31
#>  ICU            78.1
#>  TRE            TRE 0.8.0 R_fixes (BSD)
#>  iconv          Apple or GNU libiconv 1.11 /usr/lib/libiconv.2.dylib
#>  readline       8.3
#>  BLAS           /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib
#>  lapack         /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib
#>  lapack_version 3.12.1
#> 
#> ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> $RNGkind
#> [1] "Mersenne-Twister" "Inversion"        "Rejection"

Stata Environment

set linesize 255
display %tcMonth_dd,CCYY_hh:MM_am now()
about
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#>  September 13,2026 5:39 am
#> 
#> 
#> StataNow/MP 19.5 for Mac (Apple Silicon)
#> Revision 12 Aug 2026
#> Copyright 1985-2025 StataCorp LLC
#> 
#> Total physical memory: 48.01 GB
#> 
#> Stata license: Single-user 2-core , expiring  6 Feb 2027
#> Serial number: 501909358563
#>   Licensed to: Zad Rafi
#>

References

Back to top

References

1. Cole SR, Chu H, Greenland S. (2013). “Maximum likelihood, profile likelihood, and penalized likelihood: A primer.” American Journal of Epidemiology. 179:252–260. doi: 10.1093/aje/kwt245. https://doi.org/10.1093/aje/kwt245.
2. Cox DR. (2006). “Principles of Statistical Inference.” Cambridge University Press.

Citation

BibTeX citation:
@online{rafi2026,
  author = {Rafi, Zad and Rafi, Zad},
  title = {Constructing {Likelihood} {Functions} in {R,} {Stata,}
    {Python,} {Julia,} {MATLAB,} and {SQL}},
  date = {2026-09-16},
  url = {https://lesslikely.com/statistics/likelihood-polyglot.html},
  langid = {en-US}
}
]]> statistics https://lesslikely.com/statistics/likelihood-polyglot.html Wed, 16 Sep 2026 04:00:00 GMT How to Be a Good Collaborator to a Statistician and a Data Engineer Zad Rafi https://lesslikely.com/statistics/good-collaborator.html

Most failed projects are not failures due to technical matters, but failures of a good collaboration. They fail earlier, in the part nobody writes up: a question that was never pinned down, a column whose meaning changed in 2023, a “quick pull” that quietly became the number in the board deck. The statistician gets blamed for the estimate and the data engineer gets blamed for the pipeline, but the decisive errors were usually committed by well-meaning people who thought they were being helpful by not bothering anyone.

The influential statistician and geneticist, Ronald A. Fisher put it elegantly, in his 1938 presidential address to the Indian Statistical Congress:

consulting the statistician after the experiment is finished is often merely asking him to perform a post mortem examination — he can perhaps say what the experiment died of.1.

This is a piece about the habits that keep the patient alive. It is deliberately two-sided. Collaborators owe analysts specific things; analysts owe collaborators specific things back, and “the data wasn’t clean” is not a professional deliverable.

Why This Is Harder Than It Looks

The intuition that a well-specified question has one obvious analysis is wrong, and it is wrong by a measurable amount.

In the best-known demonstration, 29 teams comprising 61 analysts were given the same dataset and the same question — whether soccer referees give more red cards to dark-skin-toned players. The 29 analyses used 21 unique combinations of covariates. Estimated effects ranged from an odds ratio of 0.89 to 2.93, median 1.31; 20 teams (69%) reported a statistically significant positive effect and 9 (31%) did not. Neither analysts’ prior beliefs nor their expertise nor peer ratings of analysis quality explained the spread2.

The same pattern shows up in neuroimaging. Seventy teams analysed one fMRI dataset against nine pre-specified hypotheses. No two teams chose identical workflows. The resulting variation in hypothesis-test outcomes was substantial even between teams whose intermediate statistical maps were highly correlated3.

The lesson is not that analysis is arbitrary. It is that the analysis is underdetermined by the question, and the gap gets filled by defensible-looking judgment calls — the same garden of forking paths that makes untested model assumptions so consequential. Every one of those calls is a place where your context — what the variable actually measures, which exclusions are clinically or commercially sensible, what decision the number feeds — either arrives in time or doesn’t.

NoteThe Load-Bearing Claim

The variation in results across competent analysts is large enough that your contribution as a collaborator — supplying context that narrows the defensible choice set — is not a courtesy. It is part of the estimate.

What You Owe the Statistician

In Research Settings

Bring them in at design, not at submission. The single highest-leverage change is temporal. Kass and colleagues put “statistical analysis is more than a set of computations” and “plan ahead, really ahead” near the top of their ten rules for effective statistical practice, precisely because analysis choices are constrained by design choices that have already been locked in by the time most people call4. A statistician consulted at design can change the sample size. A statistician consulted at submission can only change the adjectives.

State the estimand before the method. Not “can you run a regression,” but: what quantity, in what population, under what contrast, with what handling of intercurrent events. If you cannot say it in a sentence, that is the finding, and it is worth knowing before you collect data.

Disclose everything you already looked at. Selective reporting is not a moral failing you have to confess; it is a fact about the data-generating process that changes what the numbers mean. An analyst who learns on submission day that you already ran eleven subgroup comparisons has been handed a different problem than the one they thought they were solving.

Settle authorship and credit at the start. The ASA’s Ethical Guidelines for Statistical Practice — eight principles across 72 elements, approved in 2022 — put responsibilities to stakeholders and integrity of data and methods on the same footing5. Statisticians are expected to refuse to produce misleading analyses. That obligation is much easier to honour when they are a named collaborator than when they are an unacknowledged service provider being paid by the hour.

Accept that the honest answer may be “this design can’t answer that.” The most valuable thing a statistician does is sometimes to tell you your study is not powered to detect what you care about. Treating this as obstruction rather than information is how you end up with an underpowered study and a strained relationship.

In Industry and Analytics Settings

Bring the decision, not the query. “Pull last quarter’s same-store sales by region” is a specification. “We’re deciding whether to close three underperforming stores and I need to know whether the recent dip is real” is a problem. The second lets the analyst tell you the first request would have misled you — because, say, the comp-store definition excludes the exact stores you’re evaluating.

Define the metric once, in writing, and stop redefining it in meetings. Most “the numbers don’t match” escalations are two correct calculations of two different things. A metric definition should name its grain, its filters, its date logic, its denominator, and its owner.

Say what happens if the number is wrong. An estimate for a monthly newsletter and an estimate feeding a pricing change deserve different levels of scrutiny, and analysts cannot calibrate effort they cannot see. Tell them the blast radius.

Give real deadlines and real slack. “EOD” for a request that requires joining three source systems is not a deadline; it is a request for an unvalidated guess with your name on it.

What You Owe the Data Engineer

Data engineers are the ones who find out that your business logic is inconsistent, usually at 2 a.m. Treat their constraints as physics, not preferences.

Send the reproduction, not the vibe. “The dashboard looks off” costs an engineer hours. “Row for store 412, 2026-07-14, shows fuel gallons of 0 but the POS extract shows 8,431 — here’s the query” costs minutes.

A defect report has four parts, and “something is wrong” is none of them. Telling an engineer or statistician that a number is wrong hands them a search problem with no search criteria. A usable report specifies:

  1. What is wrong — the exact rows, cells, or figures, with the query or filter that reproduces them.
  2. Why it is wrong — the business or domain fact that contradicts it: “these stores were acquired in March, so they can’t have January comp sales.”
  3. What correct looks like — the expected value or behaviour, with the independent source it comes from (the POS extract, the vendor invoice, the trial protocol).
  4. How to build the correct logic — the rule in implementable terms: “exclude stores from the comp base until they have 13 full months of history, keyed on acquisition_date, not open_date.”

The fourth part is the one collaborators most often skip, because it feels like doing the engineer’s job. It is the opposite: you are the only person who knows the rule. The engineer can translate a rule into SQL in an afternoon; they cannot conjure the rule itself, and if forced to guess, you have just delegated your business logic to the person with the least context for it — which is how the definition-drift problem below gets minted in the first place.

Announce upstream changes before they ship. A renamed field, a new store format, a changed promotion code, a vendor migration — every one of these breaks something downstream. The cost of a two-line heads-up is roughly zero. The cost of discovering it via a silently wrong report is not.

Understand that “just add a column” is rarely just adding a column. It may mean a backfill, a grain change, a contract change with every downstream consumer, and a reconciliation against numbers people have already quoted externally.

Stop building shadow pipelines. The spreadsheet you maintain because the official table was inconvenient will eventually be cited in a meeting as a source of truth, and someone will have to reconcile it. If the official table is wrong, file that as a defect rather than routing around it.

Treat data quality as a shared problem, not a service-desk ticket. In dbt Labs’ 2025 practitioner survey (459 respondents, collected October–December 2024 — a self-selected industry sample, not a probability sample), poor data quality was the most commonly cited challenge, named by more than 56% of respondents6. Data quality originates in operational systems and business processes, which means it is mostly not within the data team’s control, which means it will not be fixed by escalating to the data team.

What Statisticians and Data Engineers Owe Back

The obligations run both directions, and this is where the genre usually goes soft.

Learn the domain well enough to be wrong intelligently. Vance and Smith’s ASCCR framework — Attitude, Structure, Content, Communication, Relationship — is drawn from directing statistical collaboration labs across more than 900 direct projects and supervision of 4,000 more, and it puts Attitude and Relationship on equal footing with Content7. The framework exists because technical correctness delivered incomprehensibly has an effect size of zero.

Report uncertainty as uncertainty. Give the interval and the assumptions it rests on. Treating a P-value as a binary verdict rather than as continuous evidence throws away most of what the analysis produced — the case for S-values and the full compatibility/consonance function rather than a single dichotomized interval.

Say what would change your mind. Before running the analysis, state which results would be surprising and what you would do about them. This is the analyst-side equivalent of pre-registration and it is free.

Make it reproducible or don’t ship it. Given the demonstrated spread across analysts2, 3, an analysis whose choices aren’t recorded is not a result — it’s an anecdote with decimal places. Code in version control, the query saved next to the number, the exclusions written down.

Push back early and in writing. An analyst who accepts an impossible framing in the meeting and privately resents it for six weeks has chosen the worst available option.

Explain the constraint, not just the refusal. “That’s a three-week change because the fact table’s grain is transaction-line and you’re asking for basket-level” is a sentence a business partner can act on. “That’s not possible” is not.

Everything Runs on Assumptions — Ledger Them, Don’t Launder Them

Every model is a bundle of assumptions: linearity, exchangeability, missingness mechanisms, the proposition that the measurement instrument measures the thing. But so is everything else in the collaboration. You assume the chair holds, the laptop boots, the warehouse job ran last night, the engineer understood the ticket, the stakeholder meant what the ticket says. There is no assumption-free work — there is only work whose assumptions are written down and work whose assumptions are discovered later, at a worse time, by someone angrier.

The failure mode is not making assumptions. It is laundering them — letting them pass silently into the result so the final number looks cleaner than the process that produced it. The analyst-variability results above2, 3 are precisely this: dozens of defensible assumption sets, each invisible in its own write-up, visible only when laid side by side.

So the practice, for both sides of the table, is a two-column discipline. For each assumption, state:

  1. The assumption, in one falsifiable sentence.
  2. What is different if it’s wrong — which conclusions survive, which die, and what you’d check first.

A short worked example:

Assumption If it’s wrong
The nightly load completed before the snapshot Yesterday’s figures are partial; totals are biased downward; check the job log before anything else
acquisition_date marks when a store enters the comp base Comp growth mixes organic and acquired stores; the trend is uninterpretable; re-key the base
Missingness in loyalty ID is unrelated to basket size Basket-level estimates are biased toward small transactions; bound the effect or model the missingness
The stakeholder’s “sales” means net of returns Every reconciliation against finance fails by the return rate; re-confirm the definition

Two things happen when you do this. First, most of the second column turns out to be checkable in minutes — the ledger converts vague dread into a short verification queue. Second, disagreements relocate to where they belong: not “your number is wrong” but “I don’t accept row three,” which is a conversation two professionals can actually finish. An assumption stated with its failure consequence is a hypothesis; an assumption left implicit is a landmine with your name on the deed.

An Intake Template

Put this at the top of every request. If it cannot be filled in, the work isn’t ready to start.

Field Why it matters
Decision What action changes based on the answer
Question One sentence, in domain language
Estimand / metric Quantity, population, contrast, grain, denominator
Population & exclusions Who’s in, who’s out, and why
Time window Including how “current” is defined
Known data issues Migrations, backfills, definition changes
Prior looks What has already been run or explored
Blast radius External? Regulatory? Internal only?
Deadline & slack Real date, and what happens if it slips
Owner Who signs off on the definition
Assumption ledger Each assumption, and what changes if it’s wrong

Two Checklists

Before You Ask

Before You Deliver

The Failure Modes Worth Naming

The post-mortem request. Design is locked, data are collected, and the ask is to find something. See Fisher, above.

The definition drift. Two teams, two correct numbers, one metric name. Costs more meeting-hours annually than any modelling error.

The heroic pull. An engineer bypasses the pipeline under deadline pressure; the one-off becomes load-bearing; nobody remembers the filters six months later.

The silent constraint. The analyst knows the design cannot support the claim, says so once, softly, in a meeting, and is not heard. Write it down. Writing it down is the whole technique.

None of this is sophisticated. That is rather the point: the returns to being a good collaborator come almost entirely from unglamorous, cheap, early habits, and almost not at all from anyone being cleverer.

TipOne-Line Summary

Bring the decision, define the metric, disclose the prior looks, warn about upstream changes — and, on the other side, explain the constraint, report the uncertainty, and make it reproducible.

References

1. Fisher RA. (1938). “Presidential address to the first indian statistical congress.” Sankhyā: The Indian Journal of Statistics. 4:14–17.
2. Silberzahn R, Uhlmann EL, Martin DP, Anselmi P, Aust F, Awtrey E, et al. (2018). “Many analysts, one data set: Making transparent how variations in analytic choices affect results.” Advances in Methods and Practices in Psychological Science. 1:337–356. doi: 10.1177/2515245917747646.
3. Botvinik-Nezer R, Holzmeister F, Camerer CF, Dreber A, Huber J, Johannesson M, et al. (2020). “Variability in the analysis of a single neuroimaging dataset by many teams.” Nature. 582:84–88. doi: 10.1038/s41586-020-2314-9.
4. Kass RE, Caffo BS, Davidian M, Meng X-L, Yu B, Reid N. (2016). “Ten simple rules for effective statistical practice.” PLOS Computational Biology. 12:e1004961. doi: 10.1371/journal.pcbi.1004961.
5. American Statistical Association. (2022). “Ethical guidelines for statistical practice.” https://www.amstat.org/docs/default-source/amstat-documents/EthicalGuidelines.pdf.
6. dbt Labs. (2025). “The 2025 state of analytics engineering report.” https://www.getdbt.com/resources/state-of-analytics-engineering-2025.
7. Vance EA, Smith HS. (2019). “The ASCCR frame for learning essential collaboration skills.” Journal of Statistics Education. 27:265–274. doi: 10.1080/10691898.2019.1687370.
Back to top

Citation

BibTeX citation:
@online{rafi2026,
  author = {Rafi, Zad and Rafi, Zad},
  title = {How to {Be} a {Good} {Collaborator} to a {Statistician} and a
    {Data} {Engineer}},
  date = {2026-08-11},
  url = {https://lesslikely.com/statistics/good-collaborator.html},
  langid = {en-US}
}
]]> statistics https://lesslikely.com/statistics/good-collaborator.html Tue, 11 Aug 2026 04:00:00 GMT The Octagon Is a One-Trial Experiment Zad Rafi https://lesslikely.com/statistics/mma-single-event.html Jon Jones was, by almost any measure, the most complete mixed martial artist ever assembled. In 2023 he returned from a three-year layoff to fight Ciryl Gane for the heavyweight championship. Jones won by first-round submission. It was not particularly close.

Now imagine you had never seen Jones fight. You watched exactly one round of that contest and were asked: what is the probability Jones wins this match?

If you’re a frequentist, you have a problem. Frequency-based probability is formally defined as a limiting ratio — the number of times an event occurs divided by the number of opportunities, as the number of opportunities goes to infinity. A single fight is one trial. You cannot take a limit over one data point. So strictly speaking, the frequentist cannot assign a probability to this specific contest.

This is not a minor inconvenience. It is one of the oldest and most discussed objections in the philosophy of probability: the single-case problem, treated at length by Hájek (2009) and Gillies (2000).

But here is the thing: most people watching that fight thought Jones was a heavy favorite, and they were right. And they were right in a way that a frequentist framework, properly applied, can actually explain — just not in the way the objection implies.

The standard critique goes like this: frequentism requires repeatable experiments. A unique event — this fight, this night, with this version of these two people — can’t be repeated. Therefore frequentism is useless for one-off events. Bayesians, by contrast, treat probability as a degree of belief updated on evidence, which works fine for singular events.

The critique is not wrong as far as it goes. But it conflates two things: the probability of a fight type and the probability of this specific fight.

When a model estimates that Jones wins a matchup like his bout with Gane with, say, a 70% probability, it is not claiming something about a unique unrepeatable event. It is claiming something about a class of contests — fights between a wrestler of Jones’s profile and a striker of Gane’s profile, under roughly these conditions. The probability is a statement about the reference class, not the individual.

This is exactly von Mises’s original framing: probability is always about collectives, not single instances. What we need is the right collective — and choosing that collective is the hard part.

The empirical floor: what the betting market knows

Betting markets in MMA aggregate the predictions of many informed people, and their track record gives us a useful empirical anchor.

The numbers in this section come from market data, not controlled experiments. They describe the aggregate accuracy of odds-setters and bettors, not the underlying probability of any single fight.

Across all UFC fights, favorites win somewhere in the range of 60 to 65 percent of the time — meaning underdogs pull off upsets in more than a third of all fights. That 35-to-40 percent upset rate is notably higher than in most major team sports.

Heavy favorites do better: UFC odds priced between -400 and -900 (implying an 80-to-90 percent win probability) have returned an 88 to 93 percent win rate since 2013. At the other end, fights where odds land between +100 and -122 — roughly a coin flip — have resolved in favor of the nominal favorite only 51 percent of the time.

The market, in other words, does predict fight outcomes with meaningful accuracy when there is a genuine skill gap. It fails — or at least flattens — when the gap is small.

Skill is real; one fight is still noisy

Here is the central claim, stated plainly: a fighter being more skilled than their opponent does not mean they win every fight. It means their probability of winning a single fight is greater than 0.5, perhaps substantially so. But a 70% win probability is also a 30% loss probability.

This is not a trivial point. It is, in miniature, the same confusion that afflicts applied researchers who treat a statistically meaningful estimate as proof that the effect always occurs. The estimate is about the distribution. The distribution is not the single draw.

When Georges St-Pierre was 170 pounds of organized violence in his prime, his aggregate probability of winning any given welterweight fight was very high. But any single fight carried:

  • Opponent stylistic advantages that a model underweights
  • Chance of injury at any moment
  • Variance in execution (did the game plan hold? did the cuts go as expected?)
  • Pure stochastic noise — scrambles in grappling are notoriously non-deterministic

A single punch, a well-timed submission, or a referee stoppage can override any statistical edge a fighter might carry on paper. A scramble on the mat can convert a dominant position into a finished fight in one second.

If you ran that version of GSP against a solid but beatable opponent ten times, you’d expect him to win eight or nine. In one fight, you’d expect him to win — but “expect” means something probabilistic, not certain.

The reference-class problem is the real fight

Choosing the right reference class is where frequentist analysis actually gets difficult in MMA, and where most naive applications go wrong.

Holmes, McHale, and Żychaluk (2023) proposed a Markov chain simulation approach that explicitly estimates fighters’ skills in key aspects of the sport — striking, grappling, clinch work, etc. — and then simulates the fight rather than simply predicting a binary outcome. This matters because the outcome of a fight is not a single draw from a bivariate distribution. It is the aggregate result of hundreds of micro-decisions, each of which has its own stochastic component.

When you model it that way — as they did, comparing favorably to bookmaker odds — you are treating the fight as a sequence of probabilistic events from a well-defined class. The frequentist framework is doing real work. It is just applied at the level of techniques and positions rather than whole fights.

That is a defensible and scientifically useful construction. It is also, notably, the same move statisticians make in clinical medicine: you cannot run a randomized trial on a specific patient, but you can estimate their probability of response from a well-defined reference population and then apply that estimate to guide the decision.

What this means for how you talk about fight outcomes

The mistake most analysts — and most fans — make is binary: the favorite “should” win, the underdog “shouldn’t,” and an upset is a surprise requiring explanation.

The better frame: a fight between a 70% favorite and a 30% underdog is not a fight with a predetermined outcome. It is a draw from a distribution with that shape. Three out of ten times, the underdog wins. That is not an upset requiring special explanation. That is the distribution working correctly.

An upset does warrant explanation if it happens at a rate higher than the odds imply — i.e., if the market is miscalibrated. The Fightful analysis of heavy favorites suggests the market is reasonably calibrated at the extremes, noisier in the middle.

The key methodological point: a single fight is a sample of size one from a fighter’s underlying skill distribution. Skilled fighters have higher-mean distributions with less variance; fighters with fight-ending power but defensive liabilities have wide distributions. The outcome of one draw tells you something, but not much.

This is why it is statistically incoherent to say, after a single upset, that the underdog was “always better.” It is equally incoherent to say, after a dominant win, that the favorite “always wins this.” One draw is one draw.

The Bayesian objection, and why it doesn’t fully resolve things

Bayesians handle this better in some respects. They assign a prior belief about the outcome, update on evidence, and end up with a posterior. For singular events, this is the most coherent framework — a posterior probability genuinely represents a degree of belief about this specific contest, not a reference class.

But the Bayesian approach faces its own problem in combat sports: you need a prior, and priors in MMA are hard to specify. Two fighters may never have faced similar opponents, may have competed in different promotions, and may have trained specifically for this contest in ways the market cannot observe.

In practice, the most useful approach is not doctrinal but instrumental: build the best reference class you can using comparable fights and fighters, fit a model, and interpret the output as a probability statement about the type of contest this represents, while acknowledging that any single fight is an unreliable single observation of the underlying process.

That is frequentism in practice — not a claim that probabilities don’t apply to single fights, but a demand that you ground the probability in something empirically defined.

The upshot

MMA is genuinely less predictable than most sports. The upset rate is substantially higher than in major team sports, and the mechanisms are clear: it is a sport defined by rapid phase transitions (standup, clinch, ground), each carrying its own variance.

That unpredictability is not a bug in the analysis. It is a feature of the process. A skilled fighter’s edge is real and measurable in the aggregate — the betting market shows this plainly at heavy price differentials — but it is not deterministic in a single contest.

The fighter who “should” win is the one with the higher probability under the best available model. Whether they win this time is a separate question, and anyone who collapses the two has made the same mistake researchers make when they treat a significant p-value as proof that the effect always occurs.

One fight is one trial. The distribution is real. The single draw tells you much less than you think.

See also: - Semantic and Cognitive Tools to Aid Statistical Science — the case for replacing “significant” with “compatible” and describing what the data are more and less consistent with - [TODO: link to another relevant post on inference under uncertainty]

References

Back to top ]]> statistics https://lesslikely.com/statistics/mma-single-event.html Tue, 23 Jun 2026 04:00:00 GMT Multi-Language Code Examples Zad Rafi https://lesslikely.com/statistics/code-examples.html

Introduction

This post demonstrates how to include Stata, Julia, and pseudocode in your statistical posts on Less Likely.

Stata Examples

Stata code can be included for syntax highlighting:

* ============================================
* Example: Linear Regression Analysis
* ============================================

* Load example dataset
sysuse auto, clear

* Descriptive statistics
summarize price mpg weight foreign

* Create a scatter plot
scatter price mpg, title("Price vs. MPG")

* Run linear regression
regress price mpg weight foreign

* Store estimates
estimates store model1

* Display results with confidence intervals
regress price mpg weight foreign, level(95)

* Test joint significance
test mpg weight

* Predict fitted values
predict price_hat

* Calculate residuals
predict residuals, residuals

* Plot residuals
scatter residuals price_hat, ///
  yline(0) ///
  title("Residual Plot")

Advanced Stata: Panel Data Analysis

* ============================================
* Panel Data Example
* ============================================

* Load panel dataset
webuse nlswork, clear

* Declare panel structure
xtset idcode year

* Summary statistics by panel
xtsum ln_wage age

* Fixed effects regression
xtreg ln_wage age tenure, fe

* Random effects regression
xtreg ln_wage age tenure, re

* Hausman test
hausman fe re

Julia Examples

Julia code for statistical computing:

# ============================================
# Example: Linear Regression in Julia
# ============================================

using DataFrames, Statistics, GLM, Plots

# Create sample data
n = 100
x = randn(n)
y = 2.0 .+ 3.0 .* x .+ randn(n) .* 0.5

# Create DataFrame
df = DataFrame(x = x, y = y)

# Fit linear model
model = lm(@formula(y ~ x), df)

# Display results
println(model)

# Plot data and fitted line
scatter(x, y, label="Data", alpha=0.6)
plot!(x, predict(model), label="Fitted Line", linewidth=2)
xlabel!("X")
ylabel!("Y")
title!("Linear Regression")

Advanced Julia: Bootstrap Confidence Intervals

# ============================================
# Bootstrap Example
# ============================================

using Distributions, StatsBase

# Define a function to bootstrap
function bootstrap_mean(data, n_bootstrap=1000)
    n = length(data)
    bootstrap_means = zeros(n_bootstrap)

    for i in 1:n_bootstrap
        # Resample with replacement
        sample = rand(data, n)
        bootstrap_means[i] = mean(sample)
    end

    return bootstrap_means
end

# Generate sample data
data = randn(100) .+ 5.0

# Perform bootstrap
bootstrap_samples = bootstrap_mean(data)

# Calculate confidence interval
ci_lower = quantile(bootstrap_samples, 0.025)
ci_upper = quantile(bootstrap_samples, 0.975)

println("95% Bootstrap CI: [", ci_lower, ", ", ci_upper, "]")

# Plot bootstrap distribution
histogram(bootstrap_samples,
         bins=30,
         label="Bootstrap Distribution",
         xlabel="Sample Mean",
         ylabel="Frequency")
vline!([ci_lower, ci_upper],
       label="95% CI",
       linewidth=2,
       color=:red)

Pseudocode Examples

Example 1: Algorithm Block

Algorithm 1 Bootstrap Confidence Interval

Input: Data , statistic , confidence level

Output: Confidence interval

  1. for to do
    • Draw bootstrap sample by sampling observations from with replacement
    • Calculate
  2. end for
  3. Sort in ascending order
  4. Set
  5. Set
  6. return

Example 2: Simple Pseudocode Block

ALGORITHM: Linear Regression via Gradient Descent INPUT: X (n × p design matrix), y (n × 1 response vector), learning rate α OUTPUT: β (p × 1 coefficient vector) 1. Initialize β = 0, iteration = 0, maxiter = 1000 2. REPEAT until convergence OR iteration > maxiter: a. Compute predictions: ŷ = Xβ b. Compute residuals: r = y - ŷ c. Compute gradient: ∇L = -2X'r/n d. Update parameters: β = β - α∇L e. iteration = iteration + 1 3. RETURN β

Example 3: Detailed Algorithm with Comments

Algorithm 2 Maximum Likelihood Estimation

Input: Data , log-likelihood function , initial value

Output:

  1. Set ,
  2. while not converged do
    • Compute score function:
    • Compute Hessian:
    • Update:
    • Check convergence: if then break
    • if then break
  3. end while
  4. Compute standard errors:
  5. return

Comparing Approaches

Here’s how the same analysis looks across languages:

Task: Calculate Mean and 95% CI

R:

data <- rnorm(100, mean = 5, sd = 2)
mean_val <- mean(data)
ci <- t.test(data)$conf.int

Julia:

using Statistics, HypothesisTests
data = randn(100) .* 2 .+ 5
mean_val = mean(data)
ci = confint(OneSampleTTest(data))

Stata:

clear
set obs 100
generate data = rnormal(5, 2)
mean data

Python:

import numpy as np
from scipy import stats
data = np.random.normal(5, 2, 100)
mean_val = np.mean(data)
ci = stats.t.interval(0.95, len(data)-1,
                      loc=mean_val,
                      scale=stats.sem(data))

Best Practices

  1. Use syntax highlighting for display-only code - Most readers just need to see the code
  2. Execute code sparingly - Only execute when output is essential
  3. Document your setup - Note required packages and versions
  4. Test before publishing - Ensure code runs without errors
  5. Provide context - Explain what the code does and why

Resources

Conclusion

With support for multiple languages, you can now share statistical methods and algorithms in the language that best suits your analysis and audience.

Back to top ]]> statistics programming tutorial https://lesslikely.com/statistics/code-examples.html Tue, 06 Jan 2026 05:00:00 GMT Confidence, Posteriors, and the Bootstrap Zad Rafi Sir Panda https://lesslikely.com/statistics/bootstrap.html

The Power of the Bootstrap

The bootstrap1, 2 is an incredibly powerful method of approximation. Indeed, frequentists are even able to approximate posterior distributions using it, In an excerpt from the paper below, Efron describes how these procedures are similar and gives examples using the parametric bootstrap.

The point of this brief note is to say that in some situations the bootstrap, in particular the parametric bootstrap, offers an easier path toward the calculation of Bayes posterior distributions. An important but somewhat under-appreciated article by Newton and Raftery (1994) made the same point using nonparametric bootstrapping. By “going parametric,” we can illustrate more explicitly the bootstrap/MCMC connection. The arguments here will be made mainly in terms of a simple example, with no attempt at the mathematical justifications seen in Newton and Raftery.

It is not really surprising that the bootstrap and MCMC share some overlapping territory. Both are general-purpose computer-based algorithmic methods for assessing statistical accuracy, and both enable the statistician to deal effectively with nuisance parameters, obtaining inferences for the interesting part of the problem. On the less salubrious side, both share the tendency of general-purpose algorithms toward overuse.

Of course the two methodologies operate in competing inferential realms: frequentist for the bootstrap, Bayesian for MCMC. Here, as in Newton and Raftery, we leap over that divide, bringing bootstrap calculations to bear on the Bayesian world. The working assumption is that we have a Bayesian prior in mind and the only question is how to compute its posterior distribution. Arguments about the merits of Bayesian versus frequentist analysis will not be taken up here, except for our main point that the two camps share some common ground.

However, they can not only approximate posterior distributions, but confidence distributions too. Confidence distributions are an incredibly dense and technical topic and is practically an umbrella term to refer to multiple concepts, for example, they can refer to confidence curves, confidence densities and deviance functions. For a definitive source to learn more about what they are, see the following3, 4

In a paper prepared for Efron’s Fisher lecture, he writes that confidence distributions may provide the key to syncretism between frequentist and bayesian motivations:

Fiducial distributions, or confidence densities, of- fer a way to finesse this difficulty. A good argument can be made that the confidence density is the posterior density for the parameter of interest, after all of the nuisance parameters have been integrated out in an objective way. If this argument turns out to be valid, then our progress in constructing approximate confidence intervals, and approximate confidence densities, could lead to an easier use of Bayesian thinking in practical problems. This is all quite speculative, but here is a safe prediction for the 21st century: statisticians will be asked to solve bigger and more complicated prob- lems. I believe that there is a good chance that objective Bayes methods will be developed for such problems, and that something like fiducial infer- ence will play an important role in this develop- ment. Maybe Fisher’s biggest blunder will become a big hit in the 21st century!

Indeed, some authors have shown that the bootstrap distribution is equal to the confidence distribution because it meets the definition of a confidence distribution.5–7. Some have even referred to posteriors as a quick and dirty approximation to confidence distributions.8

Properties of Confidence Distributions

Xie and Singh, as summarized by Fraser define these properties as:

The first is the classical definition CL and defines the confidence distribution function as the distribution function version of the confidence quantile function ̃, where is a level confidence interval.

The second definition (2.1) identifies a confidence distribution function as a distribution function in θ for each given y and as a pivot with a uniform distribution for eachθ. But this also closely identifies with what Fisher (1930) offered: fiducial but with Fisher’s promotion replaced by a claim that the argument is pure frequentist. Of course it is pure frequentist just as Fisher (1930) was pure frequentist, except for Fisher’s accompanying claim to having purer probabilities, which was then a direct confrontation to the Bayes aficionados of the time.

The third definition (2.2) essentially gives just the quantile equivalent say ) of the distribution function where u is Uniform (0,1). Aren’t confidence quantile and confidence upper bound just different labelling for the same object? Confidence distributions can have many properties: the distribution function should of course be , but also it should inherit continuity when present in the model, should use all available information, and should generally be sensible. These properties aren’t really addressed in the authors’ proposal. A promotion of confidence distributions should acknowledge these inherent issues and also mention marginal and conditional conflicts as discussed in the literature.

The bootstrap distribution and the asymptotic consonance distribution would be defined as:

Efron writes,

As the authors point out, all of this has something to do with the bootstrap. Let represent B bootstrap replications ofθˆ, an estimator of parameter θ(possibly in the presence of nuisance parameters). The -th empirical quantile of the θˆ∗’s is then theupper endpoint of a first-order accurate-levelαconfidence interval. In this sense, the bootstrap distribution is an approximate confidence distribution. The BCa density, Efron & Tibshirani (1998), improves the confidence accuracy by reweighting the values. Let Gˆbe this empirical cdf, and and a be the bias correction and acceleration constants in my 1987 paper. Rather than equal weights 1/B, the BCa density puts weight proportional to . The reweighted bootstrap distribution then becomes a second-order accurate confidence distribution. Efron (2012) discusses this construction in the context of objective Bayes inference.

Fraser, however, argues that the connection is nothing special

The bootstrap.The bootstrap as in Section 2.3 provides an approximation to the distributions described by a model, and in doing this the bootstrap can also eliminate the influence of nuisance parameters. It can be applied to statistics or to pivots, with faster effect using suitable pivots. It can be used with least squares, or with maximum like lihood statistics, or with statistical quantities, or anywhere where distributions are wanted. Ofcourse confidence calculations are just one such use but there are many others including of course testing. So there is no particular attachment of the bootstrap to confidence distribution functions other than providing an approximate means of calculation for such.

Certain bootstrap methods such as the BCa method and t-bootstrap method also yield second order accuracy of consonance distributions.

Here, I demonstrate how to use these particular bootstrap methods to arrive at consonance curves and densities.

We’ll use the Iris dataset and construct a function that’ll yield a correlation coefficient and we will try to estimate the confidence intervals for them.

The Nonparametric Bootstrap

iris <- datasets::iris
foo <- function(data, indices) {
  dt <- data[indices, ]
  c(cor(dt[, 1], dt[, 2], method = "p"))
}

We can now use the curve_boot() method to construct a function. The default method used for this function is the “Bca” method provided by the bcaboot package (also written by Efron).

I will suppress the output of the function because it is unnecessarily long. But we’ve placed all the estimates into a list object called y.

The first item in the list will be the consonance distribution constructed by typical means, while the third item will be the bootstrap approximation to the consonance distribution.

ggplot2::qplot(boot::boot(iris, foo, 1000)$t, geom = "histogram") +
  theme_light()

ggcurve(data = y[[2]], type = "cd", nullvalue = TRUE, fill= "red") +
  theme_light()

We can also print out a table for TeX documents

ztable(gg <- curve_table(data = y[[1]]))
Lower Limit Upper Limit Interval Width Interval Level (%) CDF P-value S-value (bits)
2500 -0.14 -0.10 0.05 25.0 0.62 0.75 0.42
5000 -0.17 -0.07 0.10 50.0 0.75 0.50 1.00
7500 -0.20 -0.03 0.17 75.0 0.88 0.25 2.00
8000 -0.21 -0.02 0.19 80.0 0.90 0.20 2.32
8500 -0.22 -0.01 0.21 85.0 0.92 0.15 2.74
9000 -0.24 0.00 0.24 90.0 0.95 0.10 3.32
9500 -0.26 0.03 0.29 95.0 0.98 0.05 4.32
9750 -0.28 0.05 0.33 97.5 0.99 0.03 5.32
9900 -0.30 0.07 0.38 99.0 1.00 0.01 6.64

More bootstrap replications will lead to a smoother function. But for now, we can compare these two functions to see how similar they are.

result <- curve_from_se(estimate = 2.5, se = 0.8, measure = "mean")
ggcurve(result[[1]], type = "c", nullvalue = TRUE)


# Odds ratio with SE (on log scale internally)
result <- curve_from_se(estimate = 1.5, se = 0.2, measure = "ratio")
ggcurve(result[[1]], type = "c", nullvalue = TRUE)

If we wanted to look at the bootstrap standard errors, we could do so by loading the fifth item in the list

where in the top row, theta is the point estimate, and sdboot is the bootstrap estimate of the standard error, sdjack is the jacknife estimate of the standard error. z0 is the bias correction value and a is the acceleration constant.

The values in the second row are essentially the internal standard errors of the estimates in the top row.

One can also construct the confidence density, here I provide an example from the pvaluefunction R package The consonance curve and density are nearly identical. With more bootstrap replications, they are very likely to converge. [aadland2015ajcn]

library("pvaluefunctions")
result <- bcaboot::bcajack(
  x = iris, B = 10000, func = foo,
  alpha = c((1:99) / 100), verbose = FALSE
)

bca <- data.frame(result[["lims"]][, 1])

ggplot2::qplot(bca$result...lims......1., geom = "density") +
  theme_less()

The Parametric Bootstrap

For the examples above, we mainly used nonparametric bootstrap methods. Here I show an example using the parametric Bca bootstrap and the results it yields.

data(diabetes, package = "bcaboot")
X <- diabetes$x
y <- scale(diabetes$y, center = TRUE, scale = FALSE)
lm.model <- lm(y ~ X - 1)
mu.hat <- lm.model$fitted.values
sigma.hat <- stats::sd(lm.model$residuals)
t0 <- summary(lm.model)$adj.r.squared
y.star <- sapply(mu.hat, rnorm, n = 1000, sd = sigma.hat)
tt <- apply(y.star, 1, function(y) summary(lm(y ~ X - 1))$adj.r.squared)
b.star <- y.star %*% X

Now, we’ll use the same function, but set the method to bcapar for the parametric method.

bcadf <- (bcaboot::bcapar(
  t0 = t0, alpha = c((1:99) / 100),
  tt = tt, bb = b.star, cd = 1
)[["lims"]][, 1])

Now we can look at our outputs.

ggplot2::qplot((unlist(bcadf)), geom = "density") +
  theme_less()

That concludes our demonstration of the bootstrap method to approximate consonance functions.

See also: What Makes a Sensitivity Analysis? — the hub piece tying this material to the broader cluster on assumptions, robustness, and what happens when models bend.

References

Back to top

References

1. Efron B. (1979). “Bootstrap methods: Another look at the jackknife.” The Annals of Statistics. 7:1–26. doi: 10.1214/aos/1176344552. https://projecteuclid.org/journals/annals-of-statistics/volume-7/issue-1/Bootstrap-Methods-Another-Look-at-the-Jackknife/10.1214/aos/1176344552.full.
2. Cox DR, Efron B. (2017). “Statistical thinking for 21st century scientists.” Science Advances. 3:e1700768. doi: 10.1126/sciadv.1700768.
3. Cole SR, Edwards JK, Greenland S. (2020). “Surprise!” American Journal of Epidemiology. doi: 10/gg63md.
4. Brown HK, Hussain-Shamsy N, Lunsky Y, Dennis C-LE, Vigod SN. (2017). “The association between antenatal exposure to selective serotonin reuptake inhibitors and autism: A systematic review and meta-analysis.” The Journal of Clinical Psychiatry. 78:e48–e58. doi: 10.4088/JCP.15r10194.
5. Efron B. (1998). “R. A. Fisher in the 21st century (Invited paper presented at the 1996 R. A. Fisher Lecture).” Statistical Science. 13:95–122. doi: 10/cxg354.
6. Efron B, Narasimhan B. (2018). “The automatic construction of bootstrap confidence intervals.” :17.
7. Xie M, Singh K. (2013). “Confidence Distribution, the Frequentist Distribution Estimator of a Parameter: A Review.” International Statistical Review. 81:3–39. doi: 10.1111/insr.12000.
8. Fraser DaS. (2011). “Is Bayes Posterior just Quick and Dirty Confidence?” Statistical Science. 26:299–316. doi: 10/df4n7n.
]]> statistics https://lesslikely.com/statistics/bootstrap.html Sun, 12 Jan 2025 05:00:00 GMT Computing Confidence Interval Functions with cifunction Zad Rafi Zad R https://lesslikely.com/statistics/cifunction.html

1 Overview

Traditional confidence intervals present a single level of uncertainty (typically 95%), but this dichotomous approach discards valuable information about the data. The confidence interval function (CI-function) provides a more complete picture by displaying all possible confidence intervals simultaneously.

NoteAlternative Names

The CI-function has been described in the literature under various names:

  • Confidence curve1
  • P-value function2, 3
  • Consonance interval4
  • Confidence density5

2 Installation

Install cifunction from SSC (Statistical Software Components):

3 Syntax

{stata, echo=TRUE, collectcode=TRUE, statapath="/Applications/StataNow/StataMP.app/Contents/MacOS/StataMP"}cifunction #b, se(#) [options]

3.1 Required Arguments

Required parameters for cifunction
Argument Description
#b Point estimate (coefficient or exponentiated value like OR, HR, IRR)
se(#) Standard error of the estimate

3.2 Optional Arguments

Optional parameters for cifunction
Option Description
df(#) Degrees of freedom (for t-distributed estimates)
eform Indicates coefficient is exponentiated (OR, RR, IRR)
figure[(options)] Produces CI-function plot with optional twoway options
saving(filename, replace) Saves results to file

4 Understanding CI-Functions

A CI-function graphically represents the relationship between confidence levels and confidence intervals:

  • Peak of curve: Indicates the point estimate
  • Concentration around peak: Reflects precision of estimate
  • Narrow curve: High precision (large sample size)
  • Broad curve: Low precision (small sample size)

4.1 Advantages Over Single-Level CIs

  1. Complete information: Shows compatibility of all values with data
  2. Precision visualization: Curve width directly shows estimate precision
  3. Multiple comparisons: Reveals whether data provide reassurance about null effects
  4. Nuanced interpretation: Moves beyond binary “significant/non-significant” thinking
TipInterpretation Guide

Values toward the peak are more compatible with the observed data (assuming the statistical model is correct). The curve explicitly shows the gradual nature of statistical evidence rather than suggesting arbitrary cutoffs.

5 Practical Examples

5.1 Example 1: Basic Risk Ratio

Reproducing Figure 1 from Sullivan and Foster (1990):

* Get standard error for RR=2.0 with p=0.05
getregstats 2.0, p(0.05) mod(rr)

* Create CI-function with log-scaled x-axis
cifunction 2.0, se(.707306) eform ///
    fig(xscale(log) xlab(0.5 1 2 5 10) ///
    xtitle("Risk Ratio (Log Scale)"))
## file ()print() not found
## r(601);
## 
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |          2    .707306    -1.96   0.050            1           4
## ------------------------------------------------------------------------------

5.2 Example 2: Age-Stratified Rate Ratios

Using Doll-Hill smoking data, stratified by age group:

* Load data and fit models
webuse dollhill3, clear

* Fit Poisson models for each age category
forvalues i = 1/4 {
    poisson deaths smokes if agecat==`i', exposure(pyears) irr
}

* Plot all four age-stratified CI-functions
cifunction 5.736638 2.138812 1.46824 1.35606, ///
    se(4.181258 .6520701 .295728 .2748845) eform ///
    fig(xscale(log) xlabel(.3 .5 1 2 5 10 20 50 100) ///
    legend(label(1 "35-44") label(2 "45-54") ///
           label(3 "55-64") label(4 "65-74") ///
           title("Age Group", size(small))) ///
    xtitle("Rate Ratio (Log Scale)"))
## file ()print() not found
## r(601);
## 
## 
## (Doll and Hill (1966))
## 
## 
## Iteration 0:  Log likelihood = -4.0705386  
## Iteration 1:  Log likelihood = -3.9613823  
## Iteration 2:  Log likelihood = -3.9612634  
## Iteration 3:  Log likelihood = -3.9612634  
## 
## Poisson regression                                      Number of obs =      2
##                                                         LR chi2(1)    =   9.73
##                                                         Prob > chi2   = 0.0018
## Log likelihood = -3.9612634                             Pseudo R2     = 0.5511
## 
## ------------------------------------------------------------------------------
##       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
## -------------+----------------------------------------------------------------
##       smokes |   5.736638   4.181258     2.40   0.017     1.374811    23.93712
##        _cons |   .0001064   .0000753   -12.94   0.000     .0000266    .0004256
##   ln(pyears) |          1  (exposure)
## ------------------------------------------------------------------------------
## Note: _cons estimates baseline incidence rate.
## 
## Iteration 0:  Log likelihood = -5.4793565  
## Iteration 1:  Log likelihood = -5.4104407  
## Iteration 2:  Log likelihood =   -5.41027  
## Iteration 3:  Log likelihood =   -5.41027  
## 
## Poisson regression                                      Number of obs =      2
##                                                         LR chi2(1)    =   7.59
##                                                         Prob > chi2   = 0.0059
## Log likelihood = -5.41027                               Pseudo R2     = 0.4123
## 
## ------------------------------------------------------------------------------
##       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
## -------------+----------------------------------------------------------------
##       smokes |   2.138812   .6520701     2.49   0.013     1.176691    3.887609
##        _cons |   .0011243   .0003246   -23.52   0.000     .0006385    .0019798
##   ln(pyears) |          1  (exposure)
## ------------------------------------------------------------------------------
## Note: _cons estimates baseline incidence rate.
## 
## Iteration 0:  Log likelihood = -6.2456963  
## Iteration 1:  Log likelihood = -6.1713863  
## Iteration 2:  Log likelihood =  -6.171298  
## Iteration 3:  Log likelihood =  -6.171298  
## 
## Poisson regression                                      Number of obs =      2
##                                                         LR chi2(1)    =   4.01
##                                                         Prob > chi2   = 0.0453
## Log likelihood = -6.171298                              Pseudo R2     = 0.2450
## 
## ------------------------------------------------------------------------------
##       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
## -------------+----------------------------------------------------------------
##       smokes |    1.46824    .295728     1.91   0.057     .9893522     2.17893
##        _cons |   .0049037   .0009267   -28.14   0.000     .0033858    .0071021
##   ln(pyears) |          1  (exposure)
## ------------------------------------------------------------------------------
## Note: _cons estimates baseline incidence rate.
## 
## Iteration 0:  Log likelihood = -6.1640322  
## Iteration 1:  Log likelihood = -6.1203079  
## Iteration 2:  Log likelihood = -6.1202768  
## Iteration 3:  Log likelihood = -6.1202768  
## 
## Poisson regression                                      Number of obs =      2
##                                                         LR chi2(1)    =   2.43
##                                                         Prob > chi2   = 0.1189
## Log likelihood = -6.1202768                             Pseudo R2     = 0.1658
## 
## ------------------------------------------------------------------------------
##       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
## -------------+----------------------------------------------------------------
##       smokes |    1.35606   .2748845     1.50   0.133     .9114509    2.017551
##        _cons |   .0108317    .002047   -23.95   0.000     .0074789    .0156877
##   ln(pyears) |          1  (exposure)
## ------------------------------------------------------------------------------
## Note: _cons estimates baseline incidence rate.

This example demonstrates how CI-functions can reveal patterns across subgroups—in this case, showing how the smoking-mortality relationship varies by age.

5.3 Example 3: Crude vs Adjusted Estimates

Comparing crude and pooled odds ratios from Mann et al. (1976):

* Case-control analysis
cci 10 36 5 40, woolf
cci 18 78 7 86, woolf

* Get standard errors
getregstats 2.835165, ucl(7.151794) mod(or)
getregstats 2.222222, ucl(7.118025) mod(or)

* Compare crude vs pooled estimates
cifunction 2.222222 2.835165, se(1.319891 1.33843) eform ///
    fig(xscale(log) xlabel(.2 .5 1 2 5 10 20) ///
    legend(label(1 Crude) label(2 Pooled)) ///
    xtitle("Odds Ratio (Log Scale)"))
## file ()print() not found
## r(601);
## 
## 
## 
##                                                          Proportion
##                  |   Exposed   Unexposed  |      Total      exposed
## -----------------+------------------------+------------------------
##            Cases |        10          36  |         46       0.2174
##         Controls |         5          40  |         45       0.1111
## -----------------+------------------------+------------------------
##            Total |        15          76  |         91       0.1648
##                  |                        |
##                  |      Point estimate    |    [95% conf. interval]
##                  |------------------------+------------------------
##       Odds ratio |         2.222222       |      .69377    7.118025 (Woolf)
##  Attr. frac. ex. |              .55       |      -.4414    .8595116 (Woolf)
##  Attr. frac. pop |         .1195652       |
##                  +-------------------------------------------------
##                                chi2(1) =     1.87  Pr>chi2 = 0.1719
## 
## 
##                                                          Proportion
##                  |   Exposed   Unexposed  |      Total      exposed
## -----------------+------------------------+------------------------
##            Cases |        18          78  |         96       0.1875
##         Controls |         7          86  |         93       0.0753
## -----------------+------------------------+------------------------
##            Total |        25         164  |        189       0.1323
##                  |                        |
##                  |      Point estimate    |    [95% conf. interval]
##                  |------------------------+------------------------
##       Odds ratio |         2.835165       |    1.123936    7.151794 (Woolf)
##  Attr. frac. ex. |         .6472868       |    .1102698    .8601749 (Woolf)
##  Attr. frac. pop |         .1213663       |
##                  +-------------------------------------------------
##                                chi2(1) =     5.18  Pr>chi2 = 0.0228
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |   2.835165   1.338429     2.21   0.027     1.123936    7.151794
## ------------------------------------------------------------------------------
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |   2.222222   1.319891     1.34   0.179     .6937698    7.118025
## ------------------------------------------------------------------------------

5.4 Example 4: Multi-Study Comparison

Comparing results across four geographic regions (Gufferman et al. 1984):

* Get standard errors from confidence limits
getregstats 0.7, l(.2) mod(rr)
getregstats 1.2, l(.8) mod(rr)
getregstats 1.3, l(.5) mod(rr)
getregstats 3.6, l(1.3) mod(rr)

* Plot all studies
cifunction .7 1.2 1.3 3.6, ///
    se(.4474236 .2482485 .6337692 1.870876) eform ///
    fig(xscale(log) xlabel(.05 .1 .2 .5 1 2 5 10 30) ///
    legend(label(1 "Finland") label(2 "NO/LA") ///
           label(3 "Denmark") label(4 "Upstate NY")) ///
    xtitle("Relative Risk (Log Scale)"))
## file ()print() not found
## r(601);
## 
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |         .7   .4474236    -0.56   0.577           .2        2.45
## ------------------------------------------------------------------------------
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |        1.2   .2482485     0.88   0.378           .8         1.8
## ------------------------------------------------------------------------------
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |        1.3   .6337692     0.54   0.590           .5        3.38
## ------------------------------------------------------------------------------
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |        3.6   1.870876     2.46   0.014          1.3    9.969231
## ------------------------------------------------------------------------------

This visualization makes it immediately apparent which studies had greater precision and how their point estimates compare.

5.5 Example 5: Meta-Analysis Context

From Rothman et al. (2008), Figure 10.5:

* Get standard errors
getregstats 0.87, l(0.70) model(or)

* Compare individual study vs meta-analysis
cifunction 0.87 0.88, se(.0965065 .0658232) eform ///
    fig(xlab(.5(.1)1.3) xtitle("Odds Ratio") ///
    legend(label(1 "Eisenberger et al (1998)") ///
           label(2 "Summary of 10 studies") ///
           ring(0) position(11))) ///
    sav(figdata, replace)
## file ()print() not found
## r(601);
## 
## 
## 
## 
## ------------------------------------------------------------------------------
##              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
## -------------+----------------------------------------------------------------
##    Estimates |        .87   .0965065    -1.26   0.209           .7    1.081286
## ------------------------------------------------------------------------------

6 Output Variables

When using saving(), cifunction stores these variables:

Variables saved in output dataset
Variable Description
cilev Confidence levels (0 to 99.99)
plev P-values (0.0001 to 1.0)
sval S-values (Shannon information, see6)
lcl# Lower confidence limits for estimate #
ucl# Upper confidence limits for estimate #
NoteS-Values

S-values represent the information content against a null hypothesis, measured in bits. An S-value of 4.32 corresponds to p=0.05 (roughly 4.3 bits of information against the null). See6 for details.

7 Best Practices

  1. Always use log scale for ratio measures (OR, RR, HR, IRR)
  2. Label curves clearly when plotting multiple estimates
  3. Save output data for further analysis or custom plotting
  4. Consider degrees of freedom when using regression-based estimates
  5. Interpret curves holistically rather than focusing on single cutoffs

8 Theoretical Foundation

CI-functions rest on the principle that statistical evidence is continuous rather than dichotomous. As2 argues:

“The P-value function provides a convenient way to present the information in the data concerning the magnitude of an effect… [It] avoids the arbitrariness of any single significance level.”

The approach aligns with modern calls for moving beyond p-value thresholds7 by presenting the full gradient of statistical evidence.

9 Comparison with Other Approaches

Comparison of uncertainty visualization methods
Approach Information Provided Computational Complexity
Single CI One interval at one level Low
Multiple CIs Discrete intervals Low
CI-function Continuous curve of all intervals Low
Bootstrap distribution Empirical sampling distribution High
Likelihood profile Likelihood across parameter space Medium-High

10 Limitations

  • Assumes the statistical model is correctly specified
  • Does not account for unmeasured confounding or bias
  • Cannot compensate for poor study design or execution
  • Interpretation requires understanding of continuous evidence
WarningModel Assumptions

CI-functions, like all frequentist intervals, are valid only to the extent that the underlying statistical model and assumptions are appropriate for the data.

11 Citation

cifunction is free research software. Please cite it as:

Linden, A. (2019). CIFUNCTION: Stata module for computing and graphically displaying all possible confidence intervals around a point estimate. Statistical Software Components, Boston College Department of Economics.

12 Author

Ariel Linden President, Linden Consulting Group, LLC Email: alinden@lindenconsulting.org

13 See Also

  • getregstats (SSC): Helper for obtaining standard errors
  • niceloglabels (SSC): Improved logarithmic axis labels
  • concurve (R package): Similar functionality in R

14 References

1. Birnbaum A. (1961). “Confidence curves: An omnibus technique for estimation and testing statistical hypotheses.” Journal of the American Statistical Association. 56:246–249.
2. Poole C. (1987). “Beyond the confidence interval.” American Journal of Public Health. 77:195–199.
3. Miettinen OS. (1985). “Theoretical epidemiology: Principles of occurrence research in medicine.” New York: John Wiley & Sons.
4. Folks JL. (1981). “Ideas of statistics.” New York: John Wiley & Sons.
5. Xie M, Singh K. (2013). “Confidence Distribution, the Frequentist Distribution Estimator of a Parameter: A Review.” International Statistical Review. 81:3–39. doi: 10.1111/insr.12000.
6. Greenland S. (2019). “Valid p-values behave exactly as they should: Some misleading criticisms of p-values and their resolution with s-values.” The American Statistician. 73:106–114. doi: 10.1080/00031305.2018.1529625.
7. Wasserstein RL, Schirm AL, Lazar NA. (2019). “Moving to a world beyond "p < 0.05".” The American Statistician. 73:1–19. doi: 10.1080/00031305.2019.1583913.
Back to top

15 Appendix: Technical Details

15.1 Mathematical Formulation

For a normally distributed estimator with standard error , the confidence limits at level are:

The CI-function plots these limits across all values of .

For t-distributed estimators (when degrees of freedom are specified):

15.2 Exponentiated Coefficients

When eform is specified, computations occur on the log scale:

  1. Transform: with
  2. Compute CI limits on log scale
  3. Back-transform:

This ensures symmetric intervals on the log scale and proper geometric interpretation.

Last updated: 2026-09-13 Source code: Available on request License: Free for academic and non-commercial use

Citation

BibTeX citation:
@article{rafi2024,
  author = {Rafi, Zad and R, Zad},
  title = {Computing {Confidence} {Interval} {Functions} with
    Cifunction},
  journal = {Less Likely},
  date = {2024-03-23},
  url = {https://lesslikely.com/statistics/cifunction},
  langid = {en-US},
  abstract = {The cifunction package for Stata computes and graphically
    displays all possible confidence intervals around a point estimate,
    creating what’s known as a confidence interval function, confidence
    curve, p-value function, or consonance interval. This approach
    conveys more information than traditional single-level confidence
    intervals by showing the full range of values compatible with the
    data at all confidence levels.}
}
]]> statistics stata confidence intervals data visualization statistical methods https://lesslikely.com/statistics/cifunction.html Sat, 23 Mar 2024 04:00:00 GMT Your Models Are Neither Useful Nor Approximate Zad Rafi Sir Panda https://lesslikely.com/statistics/assumptions.html

If the assumptions of a model are not derived from theory, and if predictions are not tested against reality, then deductions from the model must be quite shaky. However, without the model, the data cannot be used to answer the research question. Indeed, the research hypothesis may not really be translatable into an empirical claim except as a statement about nominal significance levels of coefficients in a model. - David Freedman, 1991

Statistical Models as Approximations

Scientific inference is a highly complex process often accompanied by numerical outputs from statistical models. To some, such outputs are not convincing as they believe that anyone can lie with numbers, but the reality is that it is a lot easier to lie without them, making them crucial because if deployed correctly, they may be close approximations to reality. The presence of a statistical model can allow others to view an answer to a complex question from a numerical standpoint that many may be able to verify.

Unfortunately, there are rampant issues in the building, reporting, and interpretation of such statistical models, and such issues can lead to devastating policies. We have elsewhere reviewed proposals to reform terminology and improve interpretations of conventional statistics by emphasizing logical and information concepts over probability concepts (1–3), as the latter has continued to confuse even well-credentialed statisticians familiar with the misinterpretations.

What Goes into a Model?

We here focus on how common descriptions of statistical outputs can be technically accurate yet still mislead when there is considerable uncertainty about what are known as background or auxiliary assumptions. Such uncertainty is typical in many contexts, where assumptions may include both mathematical and methodological ones:

For example, in the popular regression book by Gelman, the following assumptions are given for regression models in decreasing order of importance:

  • validity
  • additivity and linearity
  • independence of errors
  • equal variance of errors
  • normality of errors

This is typical to see in many regression modeling books, but important methodological assumptions underlying the model are rarely discussed such as:

  • patient outcomes are independent
  • interactions and trends follow the regression model used for analysis (e.g., linear or logistic)
  • no uncontrolled confounding
  • subject-selection bias
  • measurement error
  • sparse-data bias
  • no selective reporting of results* based on their P-values, interval estimates, or any other output – or if there is such selection, any bias it could produce has been adjusted for (or blocked) in the computation and presentation of the final results.

P-value assumptions

Conditional versus unconditional interpretations of P-values, S-values, and compatibility intervals (CIs). (A) Conditional interpretation, in which background model assumptions, such as no systematic error, are assumed to be correct; thus,the information provided by the P-value and S-value is targeted towards the test hypothesis. (B)Unconditional interpretation, in which no aspect of the statistical model is assumed to be correct; thus,the information provided by the P-value and S-value is targeted toward the entire test model.

In addition to these general statistical and methodological assumptions, many assume that the statistical model being fit is answering a proper scientific question that has been translated into a statistical one, with assumptions that match or approximate reality. When there is a disconnect between the scientific question and the statistical one being answered by the model, the analyst is committing a type-III error, and will produce statistical outputs with of little use, unless the assumption violations of the chosen and reported are interrogated/varied and considered in the overall analysis.

Many are not aware of what a type-III error is due to the focus on type-I and type-II errors. I paste the following definition from the APA dictionary, which I found myself agreeing with:

Type-III Error: the error that occurs when there is a discrepancy between the research focus and the hypothesis actually tested. For example, a Type III error would have happened if a researcher collected data on individual differences within a sample and determined the causes of variation but the question of interest concerned differences between populations. In other words, a Type III error involves providing the right answer for the wrong question. Also called Type 0 error.

Violation of Assumptions

Misleading interpretations arise because the descriptions of such statistical outputs are conditional on such assumptions (i.e., they treat them as given/being true), and so do not factor uncertainty regarding the assumptions into their assessments. While various risk-assessment methods can incorporate additional sources in quantified uncertainties, such methods demand considerably more skilled user input than do conventional regression methods (5–9) and still do not account for unquantified sources of uncertainty. We thus present a more direct and nontechnical approach based on logic and associated language to honor assumption uncertainty, called deconditioning.

For example, suppose that a model assumes independence, a common assumption in many statistical models, but this violated due to the presence of a correlation, this will lead to P-values that are invalid, demonstrable by their failure to achieve asymptotic uniformity.

For example, a model that is approximately correct will generate P-values that are asymptotically uniform (a characteristic used to assess the validity of P-values)

#' @title Simulation of valid P-values where test hypothesis is true
#' @param X The first variable we are simulating
#' @param Y The second variable we are simulating
#' @param n.sim # The number of simulations
#' @param t The object storing the t-test results
#' @param t.sim # Empty numeric vector to contain values
#' @param n.samp # Sample size in each group
#' @NOTE The null hypothesis does not have to be 0, it can be any value.

n.sim <- 10000
t.sim <- numeric(n.sim)
n.samp <- 1000

for (i in 1:n.sim) {
  X <- rnorm(n.samp, mean = 0, sd = 1)
  Y <- rnorm(n.samp, mean = 0, sd = 1)
  df <- data.frame(X, Y)
  t <- t.test(X, Y,
    mu = 0, paired = FALSE,
    var.equal = TRUE, data = df)
  t.sim[i] <- t[[3]]}

#' @title Simulation of invalid P-values where test hypothesis is false
#' @param n_simulations # The number of simulations
#' @param p_values The object storing the t-test results
#' @param p_value_data # Empty numeric vector to contain values
#' @param n_subjects # Sample size in each group
#' @param correlation # correlation to violate independence

library(MASS)
n_simulations <- 1000
n_subjects <- 50
correlation <- 0.5

generate_correlated_data <- function(n_subjects, correlation) {
    Sigma <- matrix(correlation, n_subjects, n_subjects) +
        diag(1 - correlation, n_subjects)
    data <- mvrnorm(n = 2, mu = rep(0, n_subjects),
        Sigma = Sigma)
    return(data)
}
run_simulation <- function(n_subjects, correlation) {
    data <- generate_correlated_data(n_subjects, correlation)
    t_test_result <- t.test(data[1, ], data[2, ])
    t_test_p_value <- t_test_result$p.value
    return(t_test_p_value)
}
p_values <- replicate(n_simulations, run_simulation(n_subjects,
    correlation))
p_value_data <- data.frame(PValue = p_values)

Deconditioning shifts the logical status of assumptions in the description of results from what is assumed as given (conditioned on) to an integral part of what is being checked by common statistics. This shift makes clear that the results might be explained by violations of background assumptions rather than, or in addition to, failure of a targeted hypothesis. Such deconditioning does not change calculations; hence it does not change any P-value, interval estimate, or posterior distribution. Instead, it provides a different interpretation of these statistics, one arising naturally in thorough contextual discussions, but which deserves more emphasis in basic statistical practice, including teaching and presentations.

In philosophical accounts of theory testing, it is well recognized that an apparent violations of a hypothesis may instead be due to violation of background or auxiliary assumptions (10–13). Among these assumptions are researcher competence and integrity in the conduct and reporting of statistical analyses which have come under scrutiny in the “replication crisis”) as well as common statistical assumptions such as linearity, outcome independence, and effect homogeneity, and methodological assumptions such as no uncontrolled confounding or selection bias. Perhaps because of this generality, we have found that our recommendation to decondition inferences (present inferences using an unconditional interpretation) (1,3) is often difficult for statistically sophisticated readers to comprehend or accept, and is even resisted, misunderstood, and misrepresented by some reviewers

See also: What Makes a Sensitivity Analysis? — the hub piece tying this material to the broader cluster on assumptions, robustness, and what happens when models bend.

Back to top

Citation

BibTeX citation:
@misc{rafi2024,
  author = {Rafi, Zad and Panda, Sir},
  title = {Your {Models} {Are} {Neither} {Useful} {Nor} {Approximate}},
  date = {2024-01-01},
  url = {https://lesslikely.com/statistics/assumptions},
  langid = {en-US},
  abstract = {A discussion about models and the assumptions that
    underlie them.}
}
]]> statistics https://lesslikely.com/statistics/assumptions.html Mon, 01 Jan 2024 05:00:00 GMT Using Stata: Producing Consonance Functions Zad Rafi Sir Panda https://lesslikely.com/statistics/stata.html

Although concurve was originally designed to be used in R, it is possible to achieve very similar results in Stata. We can use some datasets that are built into Stata to show how to achieve this. I’ll use the Statamarkdown R package so that I can obtain Stata outputs using RMarkdown via my Stata 16 package.

First, let’s load the auto2 dataset which contains data about cars and their characteristics.

sysuse auto2
#> (1978 automobile data)

Browse the data set in your data browser to get more familiar with some of the variables. Let’s say we’re interested in the relationship between miles per gallon and price. We could fit a very simple linear model to assess that relationship.

First, let’s visualize the data with a scatter plot.

sysuse auto2
scatter price mpg, mcolor(dkorange) scale(0.70)
graph export "scatter.svg", replace
#> (1978 automobile data)
#> 
#> 
#> file scatter.svg saved as SVG format

scatter

scatter

That’s what our data looks like. Clearly there seems to be an inverse relationship between miles per gallon and price.

Now we could fit a very simple linear model with miles per gallon being the predictor and price being the outcome and get some estimates of the relationship.

sysuse auto2
regress price mpg
#> (1978 automobile data)
#> 
#> 
#>       Source |       SS           df       MS      Number of obs   =        74
#> -------------+----------------------------------   F(1, 72)        =     20.26
#>        Model |   139449474         1   139449474   Prob > F        =    0.0000
#>     Residual |   495615923        72  6883554.48   R-squared       =    0.2196
#> -------------+----------------------------------   Adj R-squared   =    0.2087
#>        Total |   635065396        73  8699525.97   Root MSE        =    2623.7
#> 
#> ------------------------------------------------------------------------------
#>        price | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>          mpg |  -238.8943   53.07669    -4.50   0.000    -344.7008   -133.0879
#>        _cons |   11253.06   1170.813     9.61   0.000     8919.088    13587.03
#> ------------------------------------------------------------------------------

That’s what our output looks like.

Our output also gives us 95% consonance (confidence) intervals by default. But suppose we wished to fit a fractional polynomial model and graph it and get the confidence bands, here’s what we would do.

sysuse auto2
mfp: glm price mpg
twoway (fpfitci price mpg, estcmd(glm) fcolor(dkorange%20) alcolor(%40))  || scatter price mpg, mcolor(dkorange) scale(0.75)
graph export "mfp.svg", replace
#> (1978 automobile data)
#> 
#> 
#> Deviance for model with all terms untransformed = 1373.079, 74 observations
#> 
#> Variable     Model (vs.)   Deviance  Dev diff.   P      Powers   (vs.)
#> ----------------------------------------------------------------------
#> mpg          Lin.   FP2    1373.079    19.565  0.000+   1         -2 -2
#>              FP1           1356.927     3.413  0.182    -2        
#>              Final         1356.927                     -2
#> 
#> 
#> Transformations of covariates:
#> 
#> -> gen double Impg__1 = X^-2-.2204707671 if e(sample) 
#>    (where: X = mpg/10)
#> 
#> Final multivariable fractional polynomial model for price
#> --------------------------------------------------------------------
#>     Variable |    -----Initial-----          -----Final-----
#>              |   df     Select   Alpha    Status    df    Powers
#> -------------+------------------------------------------------------
#>          mpg |    4     1.0000   0.0500     in      2     -2
#> --------------------------------------------------------------------
#> 
#> Generalized linear models                         Number of obs   =         74
#> Optimization     : ML                             Residual df     =         72
#>                                                   Scale parameter =    5533697
#> Deviance         =  398426217.4                   (1/df) Deviance =    5533697
#> Pearson          =  398426217.4                   (1/df) Pearson  =    5533697
#> 
#> Variance function: V(u) = 1                       [Gaussian]
#> Link function    : g(u) = u                       [Identity]
#> 
#>                                                   AIC             =    18.3909
#> Log likelihood   = -678.4632599                   BIC             =   3.98e+08
#> 
#> ------------------------------------------------------------------------------
#>              |                 OIM
#>        price | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>      Impg__1 |   13163.85   2013.016     6.54   0.000      9218.41    17109.29
#>        _cons |   5538.395   289.7737    19.11   0.000     4970.449    6106.341
#> ------------------------------------------------------------------------------
#> Deviance = 1356.927.
#> 
#> 
#> file mfp.svg saved as SVG format

That’s what our model looks graphed.

fractional polynomial model

fractional polynomial model

Now suppose we got a single estimate (point or interval) for a parameter, and we wanted all the intervals for it at every level.

Here’s the code that we’ll be using to achieve that in Stata.

#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> r(4);

That’s a lot and may seem intimidating at first, but I’ll explain it line by line.

postfile topost level pvalue svalue lointerval upinterval using my_new_data, replace
#> no; dataset in memory has changed since last saved
#> r(4);

“postfile” is the command that will be responsible for pasting the data from our overall loop into a new dataset. Here, we are telling Stata that the internal Stata memory used to hold these results (the post) will be named “topost” and that it will have five variables, “level”, “pvalue”, “svalue”, “lointerval”, and “upinterval.”

  • “level” will contain the consonance level that corresponds to the limits of the interval, with “lointerval” being the lower bound of the interval and “upinterval” being the upper bound.

  • “pvalue”is computed by taking 1 - “level”, which is alpha.

  • “svalue”is computed by taking the of the computed P-value, and this column will be used to plot the surprisal function.

  • “my_new_data” is the filename that we’ve assigned to our new dataset.

  • “replace” indicates that if there is an existing filename that already exists, we’re willing to relace it.

Here are the next few major lines

#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> post topost not found
#> r(111);
#> 
#> r(111);

The command “forvalues” is responsible for taking a set of numbers that we provide it, and running the contents within the braces through those numbers. So here, we’ve set the local macro “i” to contain numbers between 10 and 99.99 for our consonance levels. Why 10? Stata cannot compute consonance intervals lower than 10%.

Our next line contains the actual contents of what we want to do. Here, it says that we will run a simple linear regression where mpg is the predictor and where price is the outcome, and that the outputs for each loop will be suppressed, hence the “quiet.”

Then, we have the command “level” with the local macro “i” inside of it. As you may already know, “level” dictates the consonance level that Stata provides us. By default, this is set to 95%, but here, we’ve set it “i”, which we established via “forvalues” as being set to numbers between 10 and 99.

The next line two lines

#> no; dataset in memory has changed since last saved
#> r(4);

indicate that we will take variables of a certain class r(), (this class contains the interval bounds we need) and place them within a matrix called E. Then we will list the contents of this matrix.

post topost (`i') (1-`i'/100) ( ln(1-`i'/100)/ln(2) * -1) (E[5,1]) (E[6,1])
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> post topost not found
#> r(111);
#> 
#> r(111);

From the contents of this matrix list, we will take the estimates from the fifth and sixth rows (look at the last two paranthesis of this line of code above and then the image below) in the first column which contain our consonance limits, with the fifth row containing the lower bound of the interval and the sixth containing the upper bound.

Trace plot of imputed datasets.

Trace plot of imputed datasets.

We will place the contents from the fifth row into the second variable we set originally for our new dataset, which was “lointerval.” The contents of the sixth row will be placed into “upinterval.”

All potential values of “i” (10-99) will be placed into the first variable that we set, “level”. From this first variable, we can compute the second variable we set up, which was “Pvalue” and we’ve done that here by subtracting “level” from 1 and then dividing the whole equation by 100, so that our P-value can be on the proper scale. Our third variable, which is the longest, computes the “Svalue” by using the previous variable, the “Pvalue” and taking the of it.

The relationships between the variables on this line and the variables we set up in the very first line are dictated by the order of the commands we have set, and therefore they correspond to the same order.

“post topost” is writing the results from each loop as new observations in this data structure.

With that, our loop has concluded, and we can now tell Stata that “post” is no longer needed

postclose topost
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> post topost not found
#> r(111);
#> 
#> r(111);

We then tell Stata to clear its memory to make room for the new dataset we just created and we can list the contents of this new dataset.

use my_new_data, clear
list
#> no; dataset in memory has changed since last saved
#> r(4);

Now we have an actual dataset with all the consonance intervals at all the levels we wanted, ranging from 10% all the way up to 99%.

In order to get a function, we’ll need to be able to graph these results, and that can be tricky since for each observation we have one y value (the consonance level), and two x values, the lower bound of the interval and the upper bound of the interval.

So a typical scatterplot will not work, since Stata will only accept one x value. To bypass this, we’ll have to use a paired-coordinate scatterplot which will allow us to plot two different y variables and two different x variables.

Of course, we don’t need two y variables, so we can set both options to the variable “level”, and then we can set our first x variable to “lointerval” and the second x variable to “upinterval.”

This can all be done with the following commands, which will also allow us to set the title and subtitle of the graph, along with the titles of the axes.

twoway (pcscatter level lointerval level upinterval), ///
ytitle(Consonance Level (%)) xtitle(Consonance Limits) ///
title(Consonance Curve) ///
subtitle(A function comprised of several consonance intervals at various levels.)
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> variable level not found
#> r(111);
#> 
#> r(111);

However, I would recommend using the menu to customize the plots as much as possible. Simply go to the Graphics menu and select Twoway Graphs. Then create a new plot definition, and select the Advanced plots and choose a paired coordinate scatterplot and fill in the y variables, both of which will be “levels” and the x variables, which will be “lointerval” and “upinterval”.

So now, here’s what our confidence/consonance function looks like.

clear
sysuse auto2
postfile topost level pvalue svalue lointerval upinterval using my_new_data, replace

forvalues i = 10/99.9 {
      quietly regress price weight, level(`i')
      matrix E = r(table)
      quietly matrix list E
      post topost (`i') (1-`i'/100) ( ln(1-`i'/100)/ln(2) * -1) (E[5,1]) (E[6,1])
    }

postclose topost
use my_new_data, clear

twoway (pcscatter pvalue lointerval pvalue upinterval, mcolor(maroon)), ytitle(Consonance Level (%)) xtitle(Consonance Limits) scale(0.75) ///
title(Consonance Curve) subtitle(A function comprised of several consonance intervals at various levels.)
graph export "confidence.svg", replace
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> (1978 automobile data)
#> 
#> 
#> 
#> 
#> 
#> 
#> file confidence.svg saved as SVG format

Pretty neat, eh? And below is what our surprisal function looks like, which is simply the (p) transformation of the observed P-value. For a more comprehensive discussion on surprisals, see this page and check out some of the references at the bottom.

clear
sysuse auto2
postfile topost level pvalue svalue lointerval upinterval using my_new_data, replace

forvalues i = 10/99.9 {
      quietly regress price weight, level(`i')
      matrix E = r(table)
      quietly matrix list E
      post topost (`i') (1-`i'/100) ( ln(1-`i'/100)/ln(2) * -1) (E[5,1]) (E[6,1])
    }

postclose topost
use my_new_data, clear

twoway (pcscatter svalue lointerval svalue upinterval, mcolor(maroon)), ytitle(Consonance Level (%)) xtitle(Consonance Limits)  scale( 0.75) ///
title(Surprisal Curve) subtitle(A function comprised of several consonance intervals at various levels.)
graph export "surprisal.svg", replace
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> (1978 automobile data)
#> 
#> 
#> 
#> 
#> 
#> 
#> file surprisal.svg saved as SVG format

Surprisal Function

Surprisal Function

It’s clear that in both plots, we’re missing values of intervals with a confidence/consonance level of less than 10%, but unfortunately, this is the best Stata can do, and what we’ll have to work with. It may not look as pretty as an output from R, but it’s far more useful than blankly staring at a 95% interval and thinking that it is the only piece of information we have regarding compatibility of different effect estimates.

The code that I have pasted above can be used for most commands in Stata that have an option to calculate a consonance level. Thus, if there’s an option for “level”, then the commands above will work to produce a data set of several consonance intervals. Though I am seriously hoping that a Stata expert will see this post and point out how I am wrong.

Now, suppose we wished to fit a generalized linear model, here’s what our code would look like.

#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> (1978 automobile data)
#> 
#> 
#> 
#> 
#> 
#> 
#>      +---------------------------------------------------+
#>      | level   pvalue     svalue   lointer~l   upinter~l |
#>      |---------------------------------------------------|
#>   1. |    10       .9   .1520031    -245.564   -232.2247 |
#>   2. |    11      .89   .1681228   -246.2351   -231.5536 |
#>   3. |    12      .88   .1844246   -246.9073   -230.8814 |
#>   4. |    13      .87   .2009127   -247.5808   -230.2079 |
#>   5. |    14      .86   .2175914   -248.2557    -229.533 |
#>      |---------------------------------------------------|
#>   6. |    15      .85   .2344653   -248.9321   -228.8566 |
#>   7. |    16      .84   .2515388   -249.6102   -228.1785 |
#>   8. |    17      .83   .2688168     -250.29   -227.4987 |
#>   9. |    18      .82   .2863042   -250.9717    -226.817 |
#>  10. |    19      .81   .3040062   -251.6554   -226.1333 |
#>      |---------------------------------------------------|
#>  11. |    20       .8   .3219281   -252.3412   -225.4475 |
#>  12. |    21      .79   .3400754   -253.0292   -224.7595 |
#>  13. |    22      .78    .358454   -253.7197    -224.069 |
#>  14. |    23      .77   .3770697   -254.4126   -223.3761 |
#>  15. |    24      .76   .3959287   -255.1083   -222.6804 |
#>      |---------------------------------------------------|
#>  16. |    25      .75   .4150375   -255.8067    -221.982 |
#>  17. |    26      .74   .4344028    -256.508   -221.2807 |
#>  18. |    27      .73   .4540316   -257.2125   -220.5762 |
#>  19. |    28      .72   .4739312   -257.9202   -219.8685 |
#>  20. |    29      .71   .4941091   -258.6312   -219.1575 |
#>      |---------------------------------------------------|
#>  21. |    30       .7   .5145732   -259.3459   -218.4428 |
#>  22. |    31      .69   .5353317   -260.0642   -217.7244 |
#>  23. |    32      .68   .5563933   -260.7865   -217.0022 |
#>  24. |    33      .67    .577767   -261.5129   -216.2758 |
#>  25. |    34      .66   .5994621   -262.2435   -215.5452 |
#>      |---------------------------------------------------|
#>  26. |    35      .65   .6214884   -262.9785   -214.8102 |
#>  27. |    36      .64   .6438562   -263.7183   -214.0704 |
#>  28. |    37      .63   .6665763   -264.4628   -213.3259 |
#>  29. |    38      .62   .6896599   -265.2124   -212.5762 |
#>  30. |    39      .61   .7131189   -265.9673   -211.8213 |
#>      |---------------------------------------------------|
#>  31. |    40       .6   .7369656   -266.7278   -211.0609 |
#>  32. |    41      .59   .7612131    -267.494   -210.2947 |
#>  33. |    42      .58   .7858752   -268.2662   -209.5225 |
#>  34. |    43      .57   .8109662   -269.0446    -208.744 |
#>  35. |    44      .56   .8365012   -269.8297    -207.959 |
#>      |---------------------------------------------------|
#>  36. |    45      .55   .8624965   -270.6215   -207.1672 |
#>  37. |    46      .54   .8889687   -271.4204   -206.3683 |
#>  38. |    47      .53   .9159358   -272.2268   -205.5619 |
#>  39. |    48      .52   .9434165    -273.041   -204.7477 |
#>  40. |    49      .51   .9714308   -273.8633   -203.9254 |
#>      |---------------------------------------------------|
#>  41. |    50       .5          1    -274.694   -203.0947 |
#>  42. |    51      .49   1.029146   -275.5337    -202.255 |
#>  43. |    52      .48   1.058894   -276.3825   -201.4061 |
#>  44. |    53      .47   1.089267   -277.2411   -200.5475 |
#>  45. |    54      .46   1.120294   -278.1099   -199.6788 |
#>      |---------------------------------------------------|
#>  46. |    55      .45   1.152003   -278.9893   -198.7994 |
#>  47. |    56      .44   1.184425   -279.8798   -197.9089 |
#>  48. |    57      .43   1.217591    -280.782   -197.0067 |
#>  49. |    58      .42   1.251539   -281.6965   -196.0922 |
#>  50. |    59      .41   1.286304   -282.6239   -195.1648 |
#>      |---------------------------------------------------|
#>  51. |    60       .4   1.321928   -283.5648   -194.2239 |
#>  52. |    61      .39   1.358454     -284.52   -193.2687 |
#>  53. |    62      .38   1.395929   -285.4902   -192.2985 |
#>  54. |    63      .37   1.434403   -286.4762   -191.3125 |
#>  55. |    64      .36   1.473931   -287.4789   -190.3098 |
#>      |---------------------------------------------------|
#>  56. |    65      .35   1.514573   -288.4992   -189.2894 |
#>  57. |    66      .34   1.556393   -289.5383   -188.2504 |
#>  58. |    67      .33   1.599462   -290.5971   -187.1916 |
#>  59. |    68      .32   1.643856   -291.6769   -186.1118 |
#>  60. |    69      .31    1.68966    -292.779   -185.0097 |
#>      |---------------------------------------------------|
#>  61. |    70       .3   1.736966   -293.9048   -183.8839 |
#>  62. |    71      .29   1.785875   -295.0559   -182.7328 |
#>  63. |    72      .28   1.836501   -296.2341   -181.5546 |
#>  64. |    73      .27   1.888969   -297.4413   -180.3474 |
#>  65. |    74      .26   1.943416   -298.6794   -179.1092 |
#>      |---------------------------------------------------|
#>  66. |    75      .25          2   -299.9511   -177.8376 |
#>  67. |    76      .24   2.058894   -301.2588   -176.5299 |
#>  68. |    77      .23   2.120294   -302.6054   -175.1833 |
#>  69. |    78      .22   2.184425   -303.9944   -173.7943 |
#>  70. |    79      .21   2.251539   -305.4294   -172.3592 |
#>      |---------------------------------------------------|
#>  71. |    80       .2   2.321928   -306.9149   -170.8738 |
#>  72. |    81      .19   2.395929   -308.4555   -169.3331 |
#>  73. |    82      .18   2.473931   -310.0572   -167.7315 |
#>  74. |    83      .17   2.556393   -311.7264   -166.0623 |
#>  75. |    84      .16   2.643856   -313.4709   -164.3178 |
#>      |---------------------------------------------------|
#>  76. |    85      .15   2.736966   -315.2999   -162.4888 |
#>  77. |    86      .14   2.836501   -317.2245   -160.5642 |
#>  78. |    87      .13   2.943416   -319.2578   -158.5308 |
#>  79. |    88      .12   3.058894   -321.4166   -156.3721 |
#>  80. |    89      .11   3.184425   -323.7211   -154.0676 |
#>      |---------------------------------------------------|
#>  81. |    90       .1   3.321928   -326.1977    -151.591 |
#>  82. |    91      .09   3.473931   -328.8804   -148.9082 |
#>  83. |    92      .08   3.643856    -331.815   -145.9737 |
#>  84. |    93      .07   3.836501   -335.0646   -142.7241 |
#>  85. |    94      .06   4.058894   -338.7206   -139.0681 |
#>      |---------------------------------------------------|
#>  86. |    95      .05   4.321928   -342.9227    -134.866 |
#>  87. |    96      .04   4.643856   -347.9005   -129.8882 |
#>  88. |    97      .03   5.058894   -354.0756   -123.7131 |
#>  89. |    98      .02   5.643856   -362.3692   -115.4195 |
#>  90. |    99      .01   6.643856   -375.6108   -102.1779 |
#>      +---------------------------------------------------+
#> 
#> 
#> command ytitle is unrecognized
#> r(199);
#> 
#> r(199);

We simply replace the first line within the loop with our intended command, just as I’ve replaced

regress price mpg
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#>       Source |       SS           df       MS      Number of obs   =        74
#> -------------+----------------------------------   F(1, 72)        =     20.26
#>        Model |   139449474         1   139449474   Prob > F        =    0.0000
#>     Residual |   495615923        72  6883554.48   R-squared       =    0.2196
#> -------------+----------------------------------   Adj R-squared   =    0.2087
#>        Total |   635065396        73  8699525.97   Root MSE        =    2623.7
#> 
#> ------------------------------------------------------------------------------
#>        price | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>          mpg |  -238.8943   53.07669    -4.50   0.000    -344.7008   -133.0879
#>        _cons |   11253.06   1170.813     9.61   0.000     8919.088    13587.03
#> ------------------------------------------------------------------------------

with

glm price mpg
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> Iteration 0:  Log likelihood = -686.53958  
#> 
#> Generalized linear models                         Number of obs   =         74
#> Optimization     : ML                             Residual df     =         72
#>                                                   Scale parameter =    6883554
#> Deviance         =  495615922.6                   (1/df) Deviance =    6883554
#> Pearson          =  495615922.6                   (1/df) Pearson  =    6883554
#> 
#> Variance function: V(u) = 1                       [Gaussian]
#> Link function    : g(u) = u                       [Identity]
#> 
#>                                                   AIC             =   18.60918
#> Log likelihood   = -686.5395809                   BIC             =   4.96e+08
#> 
#> ------------------------------------------------------------------------------
#>              |                 OIM
#>        price | Coefficient  std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>          mpg |  -238.8943   53.07669    -4.50   0.000    -342.9227    -134.866
#>        _cons |   11253.06   1170.813     9.61   0.000      8958.31    13547.81
#> ------------------------------------------------------------------------------

If we wanted fit something more complex, like a multilevel mixed model that used restricted maximum likelihood, here’s what our code would look like:

#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> (1978 automobile data)
#> 
#> 
#> variable outcome not found
#> r(111);
#> 
#> r(111);

Basically, our code doesn’t really change that much and with only a few lines of it, we are able to produce graphical tools that can better help us interpret the wide range of effect sizes that are compatible with the model and its assumptions.

Using the cifunction Command for Confidence Interval Functions

An alternative approach to producing confidence interval functions in Stata is to use the cifunction command, which computes and graphically displays all possible confidence intervals around a point estimate.

The cifunction command is an immediate command that allows you to specify a point estimate and standard error, and it will produce a confidence interval function (also called a confidence curve, P-value function, or consonance interval).

Syntax

cifunction #b , se(#) [ df(#) eform figure[(twoway_options)] saving(filename, replace) ]

Where #b can be specified as a coefficient or exponentiated value (e.g., OR, HR, IRR).

Options

  • se(#): specifies the standard error of the estimate (required)
  • df(#): specify the degrees of freedom if #b is t-distributed
  • eform: indicate that the coefficient is exponentiated (e.g., OR, RR, IRR)
  • figure[(twoway_options)]: produces a confidence interval function plot
  • saving(filename, replace): save results to filename

Example 1: Simple Risk Ratio

Reproduce a confidence interval function for a risk ratio of 2.0 with p-value of 0.05. First, we use getregstats to get the standard error:

getregstats 2.0, p(0.05) mod(rr)
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> 
#> ------------------------------------------------------------------------------
#>              | Risk Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
#> -------------+----------------------------------------------------------------
#>    Estimates |          2    .707306    -1.96   0.050            1           4
#> ------------------------------------------------------------------------------

Then use cifunction specifying the point estimate and standard error. We modify the figure to improve presentation:

cifunction 2.0, se(.707306) eform fig(xscale(log) xlab(0.5 1 2 5 10) xtitle("Risk Ratio (Log Scale)"))
#> no; dataset in memory has changed since last saved
#> r(4);

Example 2: Multiple Rate Ratios by Age Group

For multiple estimates across age groups from the dollhill3 dataset:

webuse dollhill3, clear
poisson deaths smokes if agecat==1, exposure(pyears) irr
poisson deaths smokes if agecat==2, exposure(pyears) irr
poisson deaths smokes if agecat==3, exposure(pyears) irr
poisson deaths smokes if agecat==4, exposure(pyears) irr
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> (Doll and Hill (1966))
#> 
#> 
#> Iteration 0:  Log likelihood = -4.0705386  
#> Iteration 1:  Log likelihood = -3.9613823  
#> Iteration 2:  Log likelihood = -3.9612634  
#> Iteration 3:  Log likelihood = -3.9612634  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   9.73
#>                                                         Prob > chi2   = 0.0018
#> Log likelihood = -3.9612634                             Pseudo R2     = 0.5511
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |   5.736638   4.181258     2.40   0.017     1.374811    23.93712
#>        _cons |   .0001064   .0000753   -12.94   0.000     .0000266    .0004256
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.
#> 
#> 
#> Iteration 0:  Log likelihood = -5.4793565  
#> Iteration 1:  Log likelihood = -5.4104407  
#> Iteration 2:  Log likelihood =   -5.41027  
#> Iteration 3:  Log likelihood =   -5.41027  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   7.59
#>                                                         Prob > chi2   = 0.0059
#> Log likelihood = -5.41027                               Pseudo R2     = 0.4123
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |   2.138812   .6520701     2.49   0.013     1.176691    3.887609
#>        _cons |   .0011243   .0003246   -23.52   0.000     .0006385    .0019798
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.
#> 
#> 
#> Iteration 0:  Log likelihood = -6.2456963  
#> Iteration 1:  Log likelihood = -6.1713863  
#> Iteration 2:  Log likelihood =  -6.171298  
#> Iteration 3:  Log likelihood =  -6.171298  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   4.01
#>                                                         Prob > chi2   = 0.0453
#> Log likelihood = -6.171298                              Pseudo R2     = 0.2450
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |    1.46824    .295728     1.91   0.057     .9893522     2.17893
#>        _cons |   .0049037   .0009267   -28.14   0.000     .0033858    .0071021
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.
#> 
#> 
#> Iteration 0:  Log likelihood = -6.1640322  
#> Iteration 1:  Log likelihood = -6.1203079  
#> Iteration 2:  Log likelihood = -6.1202768  
#> Iteration 3:  Log likelihood = -6.1202768  
#> 
#> Poisson regression                                      Number of obs =      2
#>                                                         LR chi2(1)    =   2.43
#>                                                         Prob > chi2   = 0.1189
#> Log likelihood = -6.1202768                             Pseudo R2     = 0.1658
#> 
#> ------------------------------------------------------------------------------
#>       deaths |        IRR   Std. err.      z    P>|z|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>       smokes |    1.35606   .2748845     1.50   0.133     .9114509    2.017551
#>        _cons |   .0108317    .002047   -23.95   0.000     .0074789    .0156877
#>   ln(pyears) |          1  (exposure)
#> ------------------------------------------------------------------------------
#> Note: _cons estimates baseline incidence rate.

Then we can use cifunction with multiple estimates:

cifunction 5.736638 2.138812 1.46824 1.35606, ///
  se(4.181258 .6520701 .295728 .2748845) eform ///
  fig(xscale(log) xlabel(.3 .5 1 2 5 10 20 50 100) ///
      legend(label(1 "35-44") label(2 "45-54") label(3 "55-64") ///
             label(4 "65-74") title("Age Group", size(small))) ///
      xtitle("Rate Ratio (Log Scale)"))
#> no; dataset in memory has changed since last saved
#> r(4);

Example 3: Comparing Crude and Pooled Estimates

Using case-control data to compare crude and pooled odds ratios:

#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#>                                                          Proportion
#>                  |   Exposed   Unexposed  |      Total      exposed
#> -----------------+------------------------+------------------------
#>            Cases |        10          36  |         46       0.2174
#>         Controls |         5          40  |         45       0.1111
#> -----------------+------------------------+------------------------
#>            Total |        15          76  |         91       0.1648
#>                  |                        |
#>                  |      Point estimate    |    [95% conf. interval]
#>                  |------------------------+------------------------
#>       Odds ratio |         2.222222       |      .69377    7.118025 (Woolf)
#>  Attr. frac. ex. |              .55       |      -.4414    .8595116 (Woolf)
#>  Attr. frac. pop |         .1195652       |
#>                  +-------------------------------------------------
#>                                chi2(1) =     1.87  Pr>chi2 = 0.1719
#> 
#> 
#>                                                          Proportion
#>                  |   Exposed   Unexposed  |      Total      exposed
#> -----------------+------------------------+------------------------
#>            Cases |        18          78  |         96       0.1875
#>         Controls |         7          86  |         93       0.0753
#> -----------------+------------------------+------------------------
#>            Total |        25         164  |        189       0.1323
#>                  |                        |
#>                  |      Point estimate    |    [95% conf. interval]
#>                  |------------------------+------------------------
#>       Odds ratio |         2.835165       |    1.123936    7.151794 (Woolf)
#>  Attr. frac. ex. |         .6472868       |    .1102698    .8601749 (Woolf)
#>  Attr. frac. pop |         .1213663       |
#>                  +-------------------------------------------------
#>                                chi2(1) =     5.18  Pr>chi2 = 0.0228
#> 
#> 
#> 
#> ------------------------------------------------------------------------------
#>              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
#> -------------+----------------------------------------------------------------
#>    Estimates |   2.835165   1.338429     2.21   0.027     1.123936    7.151794
#> ------------------------------------------------------------------------------
#> 
#> 
#> 
#> ------------------------------------------------------------------------------
#>              | Odds Ratio   Std. Err.      z    P>|z|     [95% Conf. Interval]
#> -------------+----------------------------------------------------------------
#>    Estimates |   2.222222   1.319891     1.34   0.179     .6937698    7.118025
#> ------------------------------------------------------------------------------

Then compare them with cifunction:

#> no; dataset in memory has changed since last saved
#> r(4);

Output from cifunction

The cifunction command produces and stores several variables when the saving() option is specified:

  • cilev: sequencing range of CIs (from 0 to 99.99)
  • plev: sequencing range of P-values (from 0.0001 to 1.0)
  • sval: S-values computed for respective plev values
  • lcl(#): computed lower confidence limits
  • ucl(#): computed upper confidence limits

References for cifunction

  • Birnbaum, A. 1961. Confidence curves: An omnibus technique for estimation and testing statistical hypotheses. Journal of the American Statistical Association 56: 246-249.
  • Folks, J. F. 1981. Ideas of Statistics. New York: John Wiley & Sons.
  • Greenland, S. 2019. Valid P-Values Behave Exactly as They Should: Some Misleading Criticisms of P-Values and Their Resolution With S-Values. The American Statistician 73(sup1): 106-114.
  • Sullivan, K. M., and Foster, D. 1990. Confidence curves versus confidence intervals. American Journal of Public Health 80(4): 452-453.

Cite R Packages

about
#> no; dataset in memory has changed since last saved
#> r(4);
#> 
#> 
#> 
#> StataNow/MP 19.5 for Mac (Apple Silicon)
#> Revision 12 Aug 2026
#> Copyright 1985-2025 StataCorp LLC
#> 
#> Total physical memory: 48.01 GB
#> 
#> Stata license: Single-user 2-core , expiring  6 Feb 2027
#> Serial number: 501909358563
#>   Licensed to: Zad Rafi
#>

Session info

si <- sessionInfo()
print(si, RNG = TRUE, locale = TRUE)
#> R version 4.6.1 (2026-06-24)
#> Platform: aarch64-apple-darwin25.4.0
#> Running under: macOS Golden Gate 27.0
#> 
#> Matrix products: default
#> BLAS:   /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib 
#> LAPACK: /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib;  LAPACK version 3.12.1
#> 
#> Random number generation:
#>  RNG:     Mersenne-Twister 
#>  Normal:  Inversion 
#>  Sample:  Rejection 
#>  
#> locale:
#> [1] C.UTF-8/C.UTF-8/C.UTF-8/C/C.UTF-8/C.UTF-8
#> 
#> time zone: America/New_York
#> tzcode source: internal
#> 
#> attached base packages:
#>  [1] splines   grid      stats4    parallel  stats     graphics  grDevices utils     datasets  methods   base     
#> 
#> other attached packages:
#>  [1] cli_3.6.6             texPreview_2.1.0      tinytex_0.60          rmarkdown_2.31        brms_2.23.0          
#>  [6] bootImpute_1.3.0      knitr_1.51            boot_1.3-32           gtsummary_2.5.1       reshape2_1.4.5       
#> [11] ProfileLikelihood_1.3 ImputeRobust_1.3-1    gamlss_5.5-0          gamlss.dist_6.1-1     gamlss.data_6.0-7    
#> [16] mvtnorm_1.4-2         performance_0.17.1    summarytools_1.1.5    tidybayes_3.0.7       htmltools_0.5.9      
#> [21] Statamarkdown_0.9.7   car_3.1-5             carData_3.0-6         qqplotr_0.0.7         ggcorrplot_0.3.0     
#> [26] mitml_0.4-5           pbmcapply_1.5.1       Amelia_1.8.3          Rcpp_1.1.2            blogdown_1.24        
#> [31] doParallel_1.0.17     iterators_1.0.14      foreach_1.5.2         lattice_0.23-1        bayesplot_1.15.0     
#> [36] wesanderson_0.3.7     VIM_7.0.0             colorspace_2.1-3      here_1.0.2            progress_1.2.3       
#> [41] loo_2.10.1            mi_1.3.1              Matrix_1.7-6          broom_1.0.13          yardstick_1.4.0      
#> [46] svglite_2.2.2         Cairo_1.7-0           cowplot_1.2.0         mgcv_1.9-4            nlme_3.1-170         
#> [51] xfun_0.60             broom.mixed_0.2.9.7   reticulate_1.46.0     kableExtra_1.4.1      posterior_1.7.0      
#> [56] checkmate_2.3.4       parallelly_1.48.0     miceFast_0.9.1        randomForest_4.7-1.2  missForest_1.6.1     
#> [61] miceadds_3.20-10      quantreg_6.1          SparseM_1.84-2        MCMCpack_1.7-1        MASS_7.3-66          
#> [66] coda_0.19-4.1         latex2exp_0.9.8       rstan_2.32.7          StanHeaders_2.32.10   lubridate_1.9.5      
#> [71] forcats_1.0.1         stringr_1.6.0         dplyr_1.2.1           purrr_1.2.2           readr_2.2.0          
#> [76] tibble_3.3.1          ggplot2_4.0.3         tidyverse_2.0.0       ggtext_0.1.2          concurve_3.0.0       
#> [81] showtext_0.9-8        showtextdb_3.0        sysfonts_0.8.9        future.apply_1.20.2   future_1.75.0        
#> [86] tidyr_1.3.2           magrittr_2.0.5        mice_3.19.0           rms_8.1-1             Hmisc_5.2-6          
#> 
#> loaded via a namespace (and not attached):
#>   [1] dichromat_2.0-1       nnet_7.3-21           TH.data_1.1-5         vctrs_0.7.3           digest_0.6.39        
#>   [6] png_0.1-9             shape_1.4.6.1         proxy_0.4-29          magick_2.9.1          withr_3.0.3          
#>  [11] survival_3.8-9        doRNG_1.8.6.3         MatrixModels_0.5-4    emmeans_2.0.4         systemfonts_1.3.2    
#>  [16] zoo_1.9-0             V8_8.2.0              ggdist_3.3.3          DEoptimR_1.2-0        Formula_1.2-6        
#>  [21] prettyunits_1.2.0     rematch2_2.1.2        httr_1.4.8            otel_0.2.0            globals_0.19.1       
#>  [26] rstudioapi_0.19.0     extremevalues_2.4.1   pan_2.0               generics_0.1.4        processx_3.9.0       
#>  [31] base64enc_0.1-6       curl_7.1.0            mitools_2.4           lgr_0.5.2             xtable_1.8-8         
#>  [36] svUnit_1.0.8          desc_1.4.3            pracma_2.4.6          evaluate_1.0.5        hms_1.1.4            
#>  [41] glmnet_5.0            rcartocolor_2.1.2     lmtest_0.9-40         palmerpenguins_0.1.1  robustbase_0.99-7    
#>  [46] matrixStats_1.5.0     svgPanZoom_0.3.4      class_7.3-24          pillar_1.11.1         caTools_1.18.4       
#>  [51] compiler_4.6.1        stringi_1.8.9         paradox_1.0.1         jomo_2.7-6            minqa_1.2.8          
#>  [56] plyr_1.8.9            crayon_1.5.3          abind_1.4-8           sp_2.2-3              rapportools_1.2      
#>  [61] twosamples_2.0.1      sandwich_3.1-3        whisker_0.4.1         codetools_0.2-20      multcomp_1.4-31      
#>  [66] textshaping_1.0.5     QuickJSR_1.10.0       e1071_1.7-17          gridtext_0.1.6        lme4_2.0-6           
#>  [71] fs_2.1.0              itertools_0.1-3       listenv_1.0.0         Rdpack_2.6.6          pkgbuild_1.4.8       
#>  [76] estimability_2.0.0    callr_3.8.0           tzdb_0.5.0            pkgconfig_2.0.3       tools_4.6.1          
#>  [81] rbibutils_2.4.1       viridisLite_0.4.3     DBI_1.3.0             fastmap_1.2.0         scales_1.4.0         
#>  [86] opdisDownsampling_1.6 insight_1.5.2         rpart_4.1.27          farver_2.1.2          moocore_0.3.2        
#>  [91] reformulas_0.4.4      yaml_2.3.12           foreign_0.8-91        lifecycle_1.0.5       bbotk_1.12.0         
#>  [96] backports_1.5.1       mlr3tuning_1.6.1      Brobdingnag_1.2-9     timechange_0.4.0      gtable_0.3.6         
#>  [ reached 'max' / getOption("max.print") -- omitted 42 entries ]
Back to top

Citation

BibTeX citation:
@misc{rafi2024,
  author = {Rafi, Zad and Panda, Sir},
  title = {Using {Stata:} {Producing} {Consonance} {Functions}},
  date = {2024-01-01},
  url = {https://lesslikely.com/statistics/stata},
  langid = {en-US},
  abstract = {A simple guide on how to produce consonance functions in
    Stata.}
}
]]> statistics https://lesslikely.com/statistics/stata.html Mon, 01 Jan 2024 05:00:00 GMT Simulation of a Two-Group Parallel-Arm RCT with Interim Analyses Zad Rafi https://lesslikely.com/statistics/interim_analyses.html

Recently Andrew Althouse informed me that he was going to simulate a two-group parallel-arm randomized trial with interim analyses using the rpact R package, so I offered to also help in constructing the R code to do so. He already has a number of R scripts on his GitHub repo for doing similar simulations, which can be viewed here and a number of tweets explaining these simulations. For this example, his goal was to simulate a trial where the outcome was binary and the probability of death for each group could be tuned in addition to:

  • the total number of participants
  • the number of interim analyses
  • the schedule of the interim analyses
  • the group-sequential design used

along with the usual trial analysis parameters such as:

  • the -level
  • the type of test (1-sided vs. 2-sided).

The goal was to be able to produce a table of various statistics such as:

  • the odds ratio
  • the confidence limits
  • the -value
  • the number of successes

for each of the interim analyses specified.

The function below is a reflection of our efforts to do so, and also returns several plots from the rpact package for the design that is chosen along with a plot comparing the design to other designs. In order to get similar results, you will need to load the R function first, and then simply enter the proper inputs. While there may be more efficient ways to write the code, for example using lapply() instead of for loops, we have chosen not to do so, and we have also tried to minimize the number of R packages necessary for the function to work but the following will be required:

You can quickly install and load both using:

Setting up the Function

#' @title Simulation of a Two-Group Parallel-Arm Trial With Interim Analyses
#' @docType Custom function for simulation from the rpact package
#' @author Andrew Althouse with edits by Zad Rafi
#' NOTE: If you want to confirm "type 1 error" under different stopping rules,
#' make death = in the two treatment arms (e.g. no treatment effect)
#' NOTE: I have set this one up to test the power for a treatment that would reduce mortality
#' from 40% in control group (1) to 30% in treatment group (2)
#' NOTE: Trial Design Parameters - Part 1
#' Here we will specify the basics: total N patients to enroll, and death rate for each treatment arm
#' NOTE: Trial Design Parameters - Part 2
#' Here we will define the interim analysis strategy and stopping rules
#' For this trial we will include provisions for efficacy stopping only (no pre-specified futility stopping)
#' We will use the rpact package to compute the stopping/success thresholds at the interim and final analysis
#' NOTE: Required packages: rpact and stringr
#' @param nSims # The number of simulations, the default is 1000
#' @param nPatients # here is where you specify the planned max number of patients you want included in each RCT
#' @param death1 # here is where you specify the event rate for patients receiving 'treatment 1' in these trial
#' @param death2 # here is where you specify the event rate for patients receiving 'treatment 2' in these trials
#' @param nLooks # here is where you put the number of looks that will take place (INCLUDING the final analysis)
#' @param analyses_scheduled # schedule of interim analyses
#' @param sided # Whether the test is 1-sided or 2-sided
#' @param alpha # Specified alpha level, the default is 0.05
#' @param informationRates #
#' @param trials # The total number of trials you wish to load in the table results.
#' @param typeOfDesign # The type of design.
#' @param seed # Argument to set the seed for the simulations
#' @return list of dataframes and plots
#' @example See below this code block
interim_sim <- function(nPatients = 1000, death1 = 0.4, death2 = 0.3,
                        nLooks = 4, analyses_scheduled = c(0.25, 0.50, 0.75, 1),
                        sided = 1, alpha = 0.05,
                        informationRates = analyses_scheduled,
                        typeOfDesign = "asOF", nSims = 1000, trials = 10,
                        seed = 1031) {

efficacy_thresholds <- numeric(nLooks)
design <- getDesignGroupSequential(
  sided = sided, alpha = alpha,
  informationRates = analyses_scheduled,
  typeOfDesign = typeOfDesign)
design_2 <- getDesignGroupSequential(typeOfDesign = "P") # Pocock
design_3 <- getDesignGroupSequential(typeOfDesign = "asP") # Alpha-spending Pocock
design_4 <- getDesignGroupSequential(typeOfDesign = "OF") # O'Brien-Fleming
designSet <- getDesignSet(designs = c(design, design_2,
                                      design_3, design_4),
                          variedParameters = "typeOfDesign")

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed(seed)
for (j in 1:nLooks) {
  efficacy_thresholds[j] <- design$stageLevels[j]
}

analyses_nPatients <- analyses_scheduled * nPatients
efficacy_thresholds
pb <- txtProgressBar(min = 0, max = nSims, initial = 0, style = 3)
trialnum <- numeric(nSims)
or <- data.frame(matrix(ncol = nLooks, nrow = nSims))
lcl <- data.frame(matrix(ncol = nLooks, nrow = nSims))
ucl <- data.frame(matrix(ncol = nLooks, nrow = nSims))
pval <- data.frame(matrix(ncol = nLooks, nrow = nSims))
success <- data.frame(matrix(ncol = nLooks, nrow = nSims))

strings <- c("OR_%d", "LCL_%d", "UCL_%d",
             "Pval_%d", "Success_%d")

colnames(or) <- sprintf("OR_%d", (1:nLooks))
colnames(lcl) <- sprintf("LCL_%d", (1:nLooks))
colnames(ucl) <- sprintf("UCL_%d", (1:nLooks))
colnames(pval) <- sprintf("Pval_%d", (1:nLooks))
colnames(success) <- sprintf("Success_%d", (1:nLooks))
overall_success <- numeric(nSims)
df <- data.frame(trialnum, or, lcl, ucl,
                 pval, success, overall_success)

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed(seed)
time <- system.time(
for (i in 1:nSims) {
  trialnum[i] <- i
  pid <- seq(1, nPatients, by = 1)
  treatment <- rep(1:2, nPatients / 2)
  deathprob <- numeric(nPatients)
  deathprob[treatment == 1] <- death1
  deathprob[treatment == 2] <- death2
  death <- rbinom(nPatients, 1, deathprob)
  trialdata <- data.frame(cbind(pid, treatment, death))

  for (j in 1:nLooks) {
    analysisdata <- subset(trialdata, pid <= analyses_nPatients[j])
    model <- glm(death ~ treatment,
                 family = binomial(link = "logit"),
                 data = analysisdata)
    or[i, j] <- exp(summary(model)$coefficients[2])
    lcl[i, j] <- exp(confint.default((model))[2, 1])
    ucl[i, j] <- exp(confint.default((model))[2, 2])
    pval[i, j] <- summary(model)$coefficients[8]
    success[i, j] <- ifelse(or[i, j] < 1 & pval[i, j] < efficacy_thresholds[j], 1, 0)
  }
  overall_success[i] <- 0
  for (j in 1:nLooks) {
    if (success[i, j] == 1) {
      overall_success[i] <- 1
    }
  }
    setTxtProgressBar(pb, i)
})

df <- data.frame(trialnum, or, lcl, ucl, pval,
                 success, overall_success)
simulation_results <- data.frame(matrix(vector(),
  nrow = nPatients, ncol = (length(df))))
colnames(simulation_results) <- c("trialnum", (do.call(
  rbind,
  lapply(1:length(strings),
    FUN = function(j) {
      (do.call(rbind, lapply(1:nLooks,
        FUN = function(i) ((sprintf(strings, i)))
      )[]))[, j]
    }
  )
)), "overall_success")
simulation_results[intersect(names(df), names(simulation_results))] <- (df[intersect(
  names(df),
  names(simulation_results)
)])
simulation_results <- as.data.frame(simulation_results)

outputs <- simulation_results[, c(-1, -length(simulation_results))]

cols <- as.character(1:nLooks)
rows <- as.character(1:length(strings))
interim <- matrix(
  nrow = length(strings), ncol = nLooks,
  dimnames = list((rows), (cols)),
  data = rep(0, (length(strings) * nLooks)))

for (i in cols) {
  interim[, i] <- str_subset(colnames(outputs), i)
}

colnames(interim) <- sprintf("Interim_Look_%d", (1:nLooks))
colnames(simulation_results)[1] <- c("TrialNum")
colnames(simulation_results)[length(simulation_results)] <- c("Overall_Success")
simulation_results_trials <- head(simulation_results, trials)
results <- list(
  summary(design),
  summary(time),
  plot(design, 1) +
    theme_light() +
    theme(
      plot.title = element_text(size = 14, face = "bold"),
      axis.title = element_text(size = 12, face = "bold")
    ),
  plot(designSet, type = 1) +
    theme_light() +
    theme(
      plot.title = element_text(size = 14, face = "bold"),
      axis.title = element_text(size = 12, face = "bold")
    ),
  table(overall_success),
  simulation_results_trials)

names(results) <- c(
  "Design Summary", "Time to complete simulation",
  "Main Design Plot", "Plot of Various Designs",
  "Table of Overall Success", "Simulation Results")

return(results)
}

A Simulated Example

results <- interim_sim(nPatients = 1000, death1 = 0.4, death2 = 0.3,
                       nLooks = 4, analyses_scheduled = c(0.25, 0.50, 0.75, 1),
                       sided = 1, alpha = 0.025, typeOfDesign = "asOF",
                       informationRates =  c(0.25, 0.50, 0.75, 1),
                       nSims = 1000, trials = 20, seed = 1031)

Examining the Results

results[1:5]
#> $`Design Summary`
#> 
#> $`Time to complete simulation`
#>    user  system elapsed 
#>    6.57    1.47    7.45 
#> 
#> $`Main Design Plot`
#> NULL
#> 
#> $`Plot of Various Designs`
#> NULL
#> 
#> $`Table of Overall Success`
#> overall_success
#>   0   1 
#> 134 866

table_results <- results[[6]]

table_results[ , 2:21] <- round((table_results[ , 2:21]), 3)

table_results$TrialNum <- row.names(table_results)
row.names(table_results) <- NULL

for (i in colnames(table_results)){
  table_results[,i] <- cell_spec(table_results[,i], color = "#777")
}

table_results <- table_results[c(colnames(table_results))]

To examine the results, I used the kableExtra package, though this is not necessary, and simply using the following script will suffice

table_results <- results[[6]]
View(table_results)
Interim_Look_1
Interim_Look_2
Interim_Look_3
Interim_Look_4
TrialNum OR_1 LCL_1 UCL_1 Pval_1 Success_1 OR_2 LCL_2 UCL_2 Pval_2 Success_2 OR_3 LCL_3 UCL_3 Pval_3 Success_3 OR_4 LCL_4 UCL_4 Pval_4 Success_4 Overall_Success
1 0.936 0.565 1.55 0.797 0 0.868 0.601 1.255 0.453 0 1.036 0.767 1.398 0.818 0 0.891 0.687 1.157 0.387 0 0
2 0.87 0.519 1.459 0.598 0 0.757 0.525 1.092 0.136 0 0.745 0.551 1.007 0.056 0 0.721 0.555 0.936 0.014 1 1
3 0.59 0.35 0.996 0.048 0 0.555 0.382 0.806 0.002 0 0.676 0.5 0.916 0.011 0 0.686 0.529 0.89 0.005 1 1
4 0.652 0.385 1.103 0.111 0 0.671 0.461 0.976 0.037 0 0.725 0.533 0.988 0.042 0 0.75 0.576 0.976 0.032 0 0
5 0.674 0.398 1.142 0.143 0 0.763 0.526 1.108 0.155 0 0.835 0.617 1.132 0.246 0 0.764 0.587 0.994 0.045 0 0
6 0.525 0.309 0.892 0.017 0 0.622 0.427 0.907 0.014 0 0.668 0.491 0.909 0.01 0 0.625 0.479 0.815 0.001 1 1
7 0.742 0.433 1.269 0.275 0 0.769 0.525 1.125 0.175 0 0.76 0.558 1.037 0.083 0 0.719 0.551 0.938 0.015 1 1
8 0.661 0.394 1.108 0.116 0 0.437 0.301 0.632 0 1 0.535 0.397 0.721 0 1 0.551 0.425 0.714 0 1 1
9 1.288 0.76 2.183 0.348 0 1.113 0.769 1.612 0.571 0 0.87 0.645 1.173 0.361 0 0.771 0.596 0.999 0.049 0 0
10 0.783 0.466 1.316 0.356 0 0.701 0.484 1.015 0.06 0 0.655 0.482 0.89 0.007 1 0.614 0.472 0.798 0 1 1
11 0.577 0.344 0.968 0.037 0 0.639 0.444 0.921 0.016 0 0.56 0.415 0.754 0 1 0.577 0.446 0.748 0 1 1
12 0.538 0.315 0.919 0.023 0 0.629 0.434 0.913 0.015 0 0.576 0.426 0.78 0 1 0.561 0.431 0.731 0 1 1
13 0.79 0.474 1.315 0.364 0 0.763 0.531 1.095 0.142 0 0.789 0.588 1.06 0.115 0 0.791 0.611 1.025 0.076 0 0
14 0.535 0.318 0.902 0.019 0 0.69 0.477 0.999 0.049 0 0.583 0.43 0.792 0.001 1 0.621 0.477 0.809 0 1 1
15 0.695 0.419 1.152 0.158 0 0.744 0.516 1.073 0.114 0 0.73 0.541 0.985 0.04 0 0.64 0.493 0.83 0.001 1 1
16 1.116 0.657 1.896 0.685 0 0.763 0.526 1.108 0.155 0 0.69 0.509 0.935 0.017 0 0.69 0.532 0.896 0.005 1 1
17 0.533 0.32 0.886 0.015 0 0.585 0.407 0.839 0.004 0 0.679 0.504 0.913 0.011 0 0.622 0.48 0.806 0 1 1
18 0.613 0.364 1.033 0.066 0 0.692 0.478 1 0.05 0 0.78 0.577 1.055 0.107 0 0.705 0.543 0.914 0.008 1 1
19 0.841 0.502 1.409 0.511 0 0.768 0.531 1.11 0.16 0 0.685 0.506 0.927 0.014 0 0.665 0.511 0.864 0.002 1 1
20 0.275 0.159 0.476 0 1 0.449 0.31 0.651 0 1 0.498 0.369 0.672 0 1 0.498 0.384 0.646 0 1 1
Abbreviations:
TrialNum: Trial Number | OR: Odds Ratio | LCL: Lower Confidence Level | UCL: Upper Confidence Level | Pval: P-value

Statistical Environment

The analyses were run on:

#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.6.1 (2026-06-24)
#>  os       macOS Golden Gate 27.0
#>  system   aarch64, darwin25.4.0
#>  ui       unknown
#>  language (EN)
#>  collate  en_US
#>  ctype    C.UTF-8
#>  tz       America/New_York
#>  date     2026-09-13
#>  pandoc   3.11 @ /opt/homebrew/bin/ (via rmarkdown)
#>  quarto   1.10.18 @ /usr/local/bin/quarto
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package           * version  date (UTC) lib source
#>  abind               1.4-8    2024-09-12 [1] CRAN (R 4.6.1)
#>  Amelia            * 1.8.3    2024-11-08 [1] CRAN (R 4.6.1)
#>  arm                 1.15-3   2026-04-15 [1] CRAN (R 4.6.1)
#>  arrayhelpers        1.1-2    2026-07-24 [1] CRAN (R 4.6.1)
#>  backports           1.5.1    2026-04-03 [1] CRAN (R 4.6.1)
#>  base              * 4.6.1    2026-06-24 [2] local
#>  base64enc           0.1-6    2026-02-02 [1] CRAN (R 4.6.1)
#>  bayesplot         * 1.16.0   2026-08-25 [1] CRAN (R 4.6.1)
#>  bitops              1.1-0    2026-07-30 [1] CRAN (R 4.6.1)
#>  blogdown          * 1.24     2026-06-19 [1] CRAN (R 4.6.1)
#>  boot              * 1.3-32   2025-08-29 [1] CRAN (R 4.6.1)
#>  bootImpute        * 1.3.0    2025-12-15 [1] CRAN (R 4.6.1)
#>  bridgesampling      1.2-1    2025-11-19 [1] CRAN (R 4.6.1)
#>  brms              * 2.23.0   2025-09-09 [1] CRAN (R 4.6.1)
#>  Brobdingnag         1.2-9    2022-10-19 [1] CRAN (R 4.6.1)
#>  broom             * 1.0.13   2026-05-14 [1] CRAN (R 4.6.1)
#>  broom.mixed       * 0.2.9.7  2026-02-17 [1] CRAN (R 4.6.1)
#>  Cairo             * 1.7-0    2025-10-29 [1] CRAN (R 4.6.1)
#>  car               * 3.1-5    2026-02-03 [1] CRAN (R 4.6.1)
#>  carData           * 3.0-6    2026-01-30 [1] CRAN (R 4.6.1)
#>  caTools             1.18.4   2026-07-20 [1] CRAN (R 4.6.1)
#>  checkmate         * 2.3.4    2026-02-03 [1] CRAN (R 4.6.1)
#>  class               7.3-24   2026-08-03 [1] CRAN (R 4.6.1)
#>  cli               * 3.6.6    2026-04-09 [1] CRAN (R 4.6.1)
#>  clipr               0.8.1    2026-05-25 [1] CRAN (R 4.6.1)
#>  cluster             2.1.8.3  2026-07-30 [1] CRAN (R 4.6.1)
#>  coda              * 0.19-4.1 2024-01-31 [1] CRAN (R 4.6.1)
#>  codetools           0.2-20   2024-03-31 [1] CRAN (R 4.6.0)
#>  colorspace        * 2.1-3    2026-07-12 [1] CRAN (R 4.6.1)
#>  compiler            4.6.1    2026-06-24 [2] local
#>  concurve          * 3.0.5    2026-09-13 [1] local
#>  cowplot           * 1.2.0    2025-07-07 [1] CRAN (R 4.6.1)
#>  crayon              1.5.3    2024-06-20 [1] CRAN (R 4.6.1)
#>  curl                8.0.0    2026-08-25 [1] CRAN (R 4.6.1)
#>  data.table          1.18.6.1 2026-08-24 [1] CRAN (R 4.6.1)
#>  datasets          * 4.6.1    2026-06-24 [2] local
#>  DBI                 1.3.0    2026-02-25 [1] CRAN (R 4.6.1)
#>  DEoptimR            1.2-1    2026-08-20 [1] CRAN (R 4.6.1)
#>  desc                1.4.3    2023-12-10 [1] CRAN (R 4.6.1)
#>  details             0.4.0    2025-02-09 [1] CRAN (R 4.6.0)
#>  dichromat           2.0-1    2026-07-22 [1] CRAN (R 4.6.1)
#>  digest              0.6.39   2025-11-19 [1] CRAN (R 4.6.1)
#>  distributional      0.9.0    2026-09-08 [1] CRAN (R 4.6.1)
#>  doParallel        * 1.0.17   2022-02-07 [1] CRAN (R 4.6.1)
#>  doRNG               1.8.6.3  2026-02-05 [1] CRAN (R 4.6.1)
#>  dplyr             * 1.2.1    2026-04-03 [1] CRAN (R 4.6.1)
#>  e1071               1.7-17   2025-12-18 [1] CRAN (R 4.6.1)
#>  emmeans             2.0.4    2026-07-15 [1] CRAN (R 4.6.1)
#>  estimability        2.0.0    2026-06-26 [1] CRAN (R 4.6.1)
#>  evaluate            1.0.5    2025-08-27 [1] CRAN (R 4.6.1)
#>  extremevalues       2.4.1    2024-12-17 [1] CRAN (R 4.6.1)
#>  farver              2.1.2    2024-05-13 [1] CRAN (R 4.6.1)
#>  fastmap             1.2.0    2024-05-15 [1] CRAN (R 4.6.1)
#>  forcats           * 1.0.1    2025-09-25 [1] CRAN (R 4.6.1)
#>  foreach           * 1.5.2    2022-02-02 [1] CRAN (R 4.6.1)
#>  foreign             0.8-91   2026-01-29 [1] CRAN (R 4.6.1)
#>  Formula             1.2-6    2026-08-03 [1] CRAN (R 4.6.1)
#>  fs                  2.1.0    2026-04-18 [1] CRAN (R 4.6.1)
#>  furrr               0.4.0    2026-03-31 [1] CRAN (R 4.6.1)
#>  future            * 1.75.0   2026-07-20 [1] CRAN (R 4.6.1)
#>  future.apply      * 1.20.2   2026-02-20 [1] CRAN (R 4.6.1)
#>  gamlss            * 5.5-0    2025-08-19 [1] CRAN (R 4.6.1)
#>  gamlss.data       * 6.0-7    2025-09-04 [1] CRAN (R 4.6.1)
#>  gamlss.dist       * 6.1-11   2026-09-10 [1] CRAN (R 4.6.1)
#>  generics            0.1.4    2025-05-09 [1] CRAN (R 4.6.1)
#>  ggcorrplot        * 0.3.0    2026-07-24 [1] CRAN (R 4.6.1)
#>  ggdist              3.3.3    2025-04-23 [1] CRAN (R 4.6.1)
#>  ggplot2           * 4.0.3    2026-04-22 [1] CRAN (R 4.6.1)
#>  ggtext            * 0.2.0    2026-08-28 [1] CRAN (R 4.6.1)
#>  glmnet              5.0      2026-05-04 [1] CRAN (R 4.6.1)
#>  globals             0.19.1   2026-03-13 [1] CRAN (R 4.6.1)
#>  glue                1.8.1    2026-04-17 [1] CRAN (R 4.6.1)
#>  graphics          * 4.6.1    2026-06-24 [2] local
#>  grDevices         * 4.6.1    2026-06-24 [2] local
#>  grid              * 4.6.1    2026-06-24 [2] local
#>  gridExtra           2.3.1    2026-06-25 [1] CRAN (R 4.6.1)
#>  gridtext            0.1.6    2026-02-19 [1] CRAN (R 4.6.0)
#>  gtable              0.3.6    2024-10-25 [1] CRAN (R 4.6.1)
#>  gtsummary         * 2.6.1    2026-09-04 [1] CRAN (R 4.6.1)
#>  here              * 1.0.2    2025-09-15 [1] CRAN (R 4.6.1)
#>  Hmisc             * 5.3-0    2026-09-06 [1] CRAN (R 4.6.1)
#>  hms                 1.1.4    2025-10-17 [1] CRAN (R 4.6.1)
#>  htmlTable           2.5.0    2026-04-22 [1] CRAN (R 4.6.1)
#>  htmltools         * 0.5.9    2025-12-04 [1] CRAN (R 4.6.1)
#>  htmlwidgets         1.6.4    2023-12-06 [1] CRAN (R 4.6.1)
#>  httr                1.4.9    2026-09-01 [1] CRAN (R 4.6.1)
#>  ImputeRobust      * 1.3-1    2018-11-30 [1] CRAN (R 4.6.1)
#>  inline              0.3.21   2025-01-09 [1] CRAN (R 4.6.1)
#>  insight             1.5.4    2026-09-05 [1] CRAN (R 4.6.1)
#>  iterators         * 1.0.14   2022-02-05 [1] CRAN (R 4.6.1)
#>  itertools           0.1-3    2014-03-12 [1] CRAN (R 4.6.1)
#>  jomo                2.7-6    2023-04-15 [1] CRAN (R 4.6.1)
#>  jsonlite            2.0.0    2025-03-27 [1] CRAN (R 4.6.1)
#>  kableExtra        * 1.4.1    2026-07-08 [1] CRAN (R 4.6.1)
#>  knitr             * 1.52     2026-09-06 [1] CRAN (R 4.6.1)
#>  labeling            0.4.3    2023-08-29 [1] CRAN (R 4.6.1)
#>  laeken              0.5.3    2024-01-25 [1] CRAN (R 4.6.1)
#>  latex2exp         * 0.9.8    2026-01-09 [1] CRAN (R 4.6.1)
#>  lattice           * 0.23-1   2026-08-12 [1] CRAN (R 4.6.1)
#>  lifecycle           1.0.5    2026-01-08 [1] CRAN (R 4.6.1)
#>  listenv             1.0.0    2026-06-22 [1] CRAN (R 4.6.1)
#>  lme4                2.0-6    2026-07-16 [1] CRAN (R 4.6.1)
#>  lmtest              0.9-40   2022-03-21 [1] CRAN (R 4.6.1)
#>  loo               * 2.10.1   2026-07-24 [1] CRAN (R 4.6.1)
#>  lubridate         * 1.9.5    2026-02-04 [1] CRAN (R 4.6.1)
#>  magick              2.9.1    2026-02-28 [1] CRAN (R 4.6.1)
#>  magrittr          * 2.0.5    2026-04-04 [1] CRAN (R 4.6.1)
#>  MASS              * 7.3-66   2026-07-15 [1] CRAN (R 4.6.1)
#>  Matrix            * 1.7-6    2026-07-25 [1] CRAN (R 4.6.1)
#>  MatrixModels        0.5-4    2025-03-26 [1] CRAN (R 4.6.1)
#>  matrixStats         1.5.0    2025-01-07 [1] CRAN (R 4.6.1)
#>  mcmc                0.9-8    2023-11-16 [1] CRAN (R 4.6.1)
#>  MCMCpack          * 1.7-1    2024-08-27 [1] CRAN (R 4.6.1)
#>  methods           * 4.6.1    2026-06-24 [2] local
#>  mgcv              * 1.9-4    2025-11-07 [1] CRAN (R 4.6.0)
#>  mi                * 1.3.1    2026-07-28 [1] CRAN (R 4.6.1)
#>  mice              * 3.19.0   2025-12-10 [1] CRAN (R 4.6.1)
#>  miceadds          * 3.20-10  2026-05-28 [1] CRAN (R 4.6.1)
#>  miceFast          * 0.9.1    2026-02-26 [1] CRAN (R 4.6.1)
#>  minqa               1.2.8    2024-08-17 [1] CRAN (R 4.6.1)
#>  missForest        * 1.6.1    2025-10-26 [1] CRAN (R 4.6.1)
#>  mitml             * 0.4-5    2023-03-08 [1] CRAN (R 4.6.1)
#>  mitools             2.7      2026-08-31 [1] CRAN (R 4.6.1)
#>  multcomp            1.4-32   2026-08-21 [1] CRAN (R 4.6.1)
#>  mvtnorm           * 1.4-2    2026-07-12 [1] CRAN (R 4.6.1)
#>  nlme              * 3.1-171  2026-09-01 [1] CRAN (R 4.6.1)
#>  nloptr              2.2.1    2025-03-17 [1] CRAN (R 4.6.1)
#>  nnet                7.3-21   2026-08-03 [1] CRAN (R 4.6.1)
#>  opdisDownsampling   1.7      2026-09-04 [1] CRAN (R 4.6.1)
#>  otel                0.2.0    2025-08-29 [1] CRAN (R 4.6.1)
#>  pan                 2.0      2026-06-30 [1] CRAN (R 4.6.1)
#>  pander              0.6.6    2025-03-01 [1] CRAN (R 4.6.1)
#>  parallel          * 4.6.1    2026-06-24 [2] local
#>  parallelly        * 1.48.0   2026-06-29 [1] CRAN (R 4.6.1)
#>  pbmcapply         * 1.5.1    2022-04-28 [1] CRAN (R 4.6.1)
#>  performance       * 0.18.2   2026-09-10 [1] CRAN (R 4.6.1)
#>  pillar              1.11.1   2025-09-17 [1] CRAN (R 4.6.1)
#>  pkgbuild            1.4.8    2025-05-26 [1] CRAN (R 4.6.1)
#>  pkgconfig           2.0.3    2019-09-22 [1] CRAN (R 4.6.1)
#>  plyr                1.8.9    2023-10-02 [1] CRAN (R 4.6.1)
#>  png                 0.1-9    2026-03-15 [1] CRAN (R 4.6.0)
#>  polspline           1.1.25   2024-05-10 [1] CRAN (R 4.6.1)
#>  posterior         * 1.7.0    2026-04-01 [1] CRAN (R 4.6.1)
#>  pracma              2.4.6    2025-10-22 [1] CRAN (R 4.6.1)
#>  prettyunits         1.2.0    2023-09-24 [1] CRAN (R 4.6.1)
#>  ProfileLikelihood * 1.3      2023-08-25 [1] CRAN (R 4.6.1)
#>  progress          * 1.2.3    2023-12-06 [1] CRAN (R 4.6.1)
#>  proxy               0.4-29   2025-12-29 [1] CRAN (R 4.6.1)
#>  purrr             * 1.2.2    2026-04-10 [1] CRAN (R 4.6.1)
#>  qqconf              1.3.2    2023-04-14 [1] CRAN (R 4.6.1)
#>  qqplotr           * 0.0.7    2025-09-05 [1] CRAN (R 4.6.1)
#>  quantreg          * 6.1      2025-03-10 [1] CRAN (R 4.6.1)
#>  QuickJSR            1.11.0   2026-08-21 [1] CRAN (R 4.6.1)
#>  R6                  2.6.1    2025-02-15 [1] CRAN (R 4.6.1)
#>  randomForest      * 4.7-1.2  2024-09-22 [1] CRAN (R 4.6.1)
#>  ranger              0.18.0   2026-01-16 [1] CRAN (R 4.6.1)
#>  rappdirs            0.3.4    2026-01-17 [1] CRAN (R 4.6.1)
#>  rapportools         1.2      2025-02-28 [1] CRAN (R 4.6.1)
#>  rbibutils           2.4.1    2026-01-21 [1] CRAN (R 4.6.1)
#>  RColorBrewer        1.1-3    2022-04-03 [1] CRAN (R 4.6.1)
#>  Rcpp              * 1.1.2    2026-07-05 [1] CRAN (R 4.6.1)
#>  RcppParallel        6.2.1    2026-08-27 [1] CRAN (R 4.6.1)
#>  Rdpack              2.6.6    2026-02-08 [1] CRAN (R 4.6.1)
#>  readr             * 2.2.0    2026-02-19 [1] CRAN (R 4.6.1)
#>  reformulas          0.4.4    2026-02-02 [1] CRAN (R 4.6.1)
#>  rematch2            2.1.2    2020-05-01 [1] CRAN (R 4.6.1)
#>  reshape2          * 1.4.5    2025-11-12 [1] CRAN (R 4.6.1)
#>  reticulate        * 1.47.0   2026-09-03 [1] CRAN (R 4.6.1)
#>  rlang               1.3.0    2026-07-05 [1] CRAN (R 4.6.1)
#>  rmarkdown         * 2.32     2026-09-01 [1] CRAN (R 4.6.1)
#>  rms               * 8.1-1    2026-02-18 [1] CRAN (R 4.6.1)
#>  rngtools            1.5.2    2021-09-20 [1] CRAN (R 4.6.1)
#>  robustbase          0.99-7   2026-02-05 [1] CRAN (R 4.6.1)
#>  rpact             * 4.4.0    2026-03-04 [1] CRAN (R 4.6.1)
#>  rpart               4.1.27   2026-03-27 [1] CRAN (R 4.6.0)
#>  rprojroot           2.1.1    2025-08-26 [1] CRAN (R 4.6.1)
#>  rstan             * 2.32.7   2025-03-10 [1] CRAN (R 4.6.1)
#>  rstantools          2.7.1    2026-09-04 [1] Github (stan-dev/rstantools@4d84659)
#>  rstudioapi          0.19.0   2026-06-11 [1] CRAN (R 4.6.1)
#>  S7                  0.2.2    2026-04-22 [1] CRAN (R 4.6.1)
#>  sandwich            3.1-3    2026-08-03 [1] CRAN (R 4.6.1)
#>  scales              1.4.0    2025-04-24 [1] CRAN (R 4.6.1)
#>  sessioninfo         1.2.4    2026-06-04 [1] CRAN (R 4.6.1)
#>  shape               1.4.6.1  2024-02-23 [1] CRAN (R 4.6.1)
#>  showtext          * 0.9-8    2026-03-21 [1] CRAN (R 4.6.1)
#>  showtextdb        * 3.0      2020-06-04 [1] CRAN (R 4.6.1)
#>  sp                  2.2-3    2026-07-19 [1] CRAN (R 4.6.1)
#>  SparseM           * 1.84-2   2024-07-17 [1] CRAN (R 4.6.1)
#>  splines           * 4.6.1    2026-06-24 [2] local
#>  StanHeaders       * 2.39.1   2026-09-02 [1] CRAN (R 4.6.1)
#>  Statamarkdown     * 1.0.0    2026-08-21 [1] CRAN (R 4.6.1)
#>  stats             * 4.6.1    2026-06-24 [2] local
#>  stats4            * 4.6.1    2026-06-24 [2] local
#>  stringi             1.8.9    2026-08-04 [1] CRAN (R 4.6.1)
#>  stringr           * 1.6.0    2025-11-04 [1] CRAN (R 4.6.1)
#>  summarytools      * 1.1.5    2026-06-25 [1] Github (dcomtois/summarytools@2b613cf)
#>  survival            3.8-12   2026-09-09 [1] CRAN (R 4.6.1)
#>  svglite           * 2.2.2    2025-10-21 [1] CRAN (R 4.6.1)
#>  svgPanZoom          0.3.4    2020-02-15 [1] CRAN (R 4.6.1)
#>  svUnit              1.0.8    2025-08-26 [1] CRAN (R 4.6.1)
#>  sysfonts          * 0.8.9    2024-03-02 [1] CRAN (R 4.6.1)
#>  systemfonts         1.3.2    2026-03-05 [1] CRAN (R 4.6.1)
#>  tcltk               4.6.1    2026-06-24 [2] local
#>  tensorA             0.36.2.1 2023-12-13 [1] CRAN (R 4.6.0)
#>  texPreview        * 2.1.0    2024-01-24 [1] CRAN (R 4.6.0)
#>  textshaping         1.0.5    2026-03-06 [1] CRAN (R 4.6.0)
#>  TH.data             1.1-5    2025-11-17 [1] CRAN (R 4.6.0)
#>  tibble            * 3.3.1    2026-01-11 [1] CRAN (R 4.6.1)
#>  tidybayes         * 3.0.7    2024-09-15 [1] CRAN (R 4.6.1)
#>  tidyr             * 1.3.2    2025-12-19 [1] CRAN (R 4.6.1)
#>  tidyselect          1.2.1    2024-03-11 [1] CRAN (R 4.6.1)
#>  tidyverse         * 2.0.0    2023-02-22 [1] CRAN (R 4.6.1)
#>  timechange          0.4.0    2026-01-29 [1] CRAN (R 4.6.1)
#>  tinytex           * 0.60     2026-06-16 [1] CRAN (R 4.6.1)
#>  tools               4.6.1    2026-06-24 [2] local
#>  twosamples          2.0.1    2023-06-23 [1] CRAN (R 4.6.1)
#>  tzdb                0.5.0    2025-03-15 [1] CRAN (R 4.6.1)
#>  utils             * 4.6.1    2026-06-24 [2] local
#>  V8                  8.2.0    2026-04-21 [1] CRAN (R 4.6.0)
#>  vcd                 1.4-14   2026-07-29 [1] CRAN (R 4.6.1)
#>  vctrs               0.7.3    2026-04-11 [1] CRAN (R 4.6.1)
#>  VIM               * 7.3.1    2026-09-11 [1] CRAN (R 4.6.1)
#>  viridisLite         0.4.3    2026-02-04 [1] CRAN (R 4.6.1)
#>  wesanderson       * 0.3.7    2023-10-31 [1] CRAN (R 4.6.1)
#>  whisker             0.4.1    2022-12-05 [1] CRAN (R 4.6.1)
#>  withr               3.0.3    2026-06-19 [1] CRAN (R 4.6.1)
#>  xfun              * 0.60     2026-07-09 [1] CRAN (R 4.6.1)
#>  xgboost             3.2.1.1  2026-03-18 [1] CRAN (R 4.6.1)
#>  xml2                1.6.0    2026-06-22 [1] CRAN (R 4.6.1)
#>  xtable              1.8-8    2026-02-22 [1] CRAN (R 4.6.1)
#>  yaml                2.3.12   2025-12-10 [1] CRAN (R 4.6.1)
#>  yardstick         * 1.4.0    2026-04-07 [1] CRAN (R 4.6.1)
#>  zoo                 1.9-0    2026-07-31 [1] CRAN (R 4.6.1)
#> 
#>  [1] /opt/homebrew/lib/R/4.6/site-library
#>  [2] /opt/homebrew/Cellar/r/4.6.1/lib/R/library
#>  * ── Packages attached to the search path.
#> 
#> ─ External software ──────────────────────────────────────────────────────────
#>  setting        value
#>  cairo          1.18.4
#>  cairoFT        2.14.3/2.18.1
#>  pango
#>  png
#>  jpeg
#>  tiff
#>  tcl            9.0
#>  curl           8.7.1
#>  zlib           1.2.12
#>  bzlib          1.0.8, 13-Jul-2019
#>  xz             5.8.3
#>  deflate
#>  zstd           1.5.7
#>  PCRE           10.48 2026-08-31
#>  ICU            78.1
#>  TRE            TRE 0.8.0 R_fixes (BSD)
#>  iconv          Apple or GNU libiconv 1.11 /usr/lib/libiconv.2.dylib
#>  readline       8.3
#>  BLAS           /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib
#>  lapack         /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib
#>  lapack_version 3.12.1
#> 
#> ─ Python configuration ───────────────────────────────────────────────────────
#>  Python is not available
#> 
#> ──────────────────────────────────────────────────────────────────────────────
#> $RNGkind
#> [1] "L'Ecuyer-CMRG" "Inversion"     "Rejection"

Article Citation

1

Back to top

References

1. Wassmer G, Pahlke F. (2018). “Rpact: Confirmatory adaptive clinical trial design and analysis.” doi: 10.32614/CRAN.package.rpact. https://doi.org/10.32614/CRAN.package.rpact.

Citation

BibTeX citation:
@misc{rafi2021,
  author = {Rafi, Zad and Rafi, Zad},
  title = {Simulation of a {Two-Group} {Parallel-Arm} {RCT} with
    {Interim} {Analyses}},
  date = {2021-02-01},
  url = {https://lesslikely.com/statistics/interim_analyses},
  langid = {en-US},
  abstract = {A simulation of a two-group parallel-arm randomized trial
    with interim analysis using the rpact package.}
}
]]> statistics https://lesslikely.com/statistics/interim_analyses.html Mon, 01 Feb 2021 05:00:00 GMT Tables, Graphs, and Computations from Rafi & Greenland (2020) Zad Rafi https://lesslikely.com/statistics/RG2020BMC.html

The following post provides some of the code that was used to construct the figures and tables from Rafi & Greenland, 20201. An enhanced PDF version of the paper can be found here. For further discussion of the computations, see the appendix of the main paper, along with our technical supplement.2

Disclaimer: I am responsible for all the code and mistakes below, and none of them can be attributed to my coauthors or my fellow package developers.

In order to recreate the functions, I would recommend installing the latest version of concurve from CRAN, as it has patched some issues with graphing when the outcome is binary. Use the script below to get the latest version and load the R package. A number of other R packages are also used in this post, which are listed below.

install.packages("concurve")
library("concurve")

Valid -values Are Uniform Under the Null Model

Here we show that valid -values have specific properties, when the null model is true. We first generate two variables (, ) that come from the same normal distribution with a of 0 and of 1, each with a total of 1000 observations. We assume that there is no relationship between these two variables. We run a simple t-test between and and iterate this 100000 times and compute 100000 -values to see the overall distribution of the -values, which we then plot using a histogram./

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
n.sim <- 100
t.sim <- numeric(n.sim)
n.samp <- 1000

for (i in 1:n.sim) {
    X <- rnorm(n.samp, mean = 0, sd = 1)
    Y <- rnorm(n.samp, mean = 0, sd = 1)
    df <- data.frame(X, Y)
    t <- t.test(X, Y, data = df)
    t.sim[i] <- t[[3]]
}

ggplot(NULL, aes(x = t.sim)) + geom_histogram(bins = 30, col = "black",
    fill = "#99c7c7", alpha = 0.25) + labs(title = "Distribution of P-values Under the Null",
    x = "P-value") + scale_x_continuous(breaks = seq(0, 1, 0.1)) +
    theme_bw()

This can also be shown using the TeachingDemos R package, which has a function dedicated to showing this phenomenon.

library("TeachingDemos")
RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
obs_p <- Pvalue.norm.sim(n = 1000, mu = 0, mu0 = 0, sigma = 1,
    sigma0 = 1, test = "t", alternative = "two.sided", alpha = 0.05,
    B = 1e+05)

ggplot(NULL, aes(x = obs_p)) + geom_histogram(bins = 30, col = "black",
    fill = "#99c7c7", alpha = 0.25) + labs(title = "Distribution of P-values Under the Null",
    x = "P-value") + scale_x_continuous(breaks = seq(0, 1, 0.1)) +
    theme_bw()

As you can see, when the null model is true, the distribution of -values is uniform. Valid -values are uniform under the null hypothesis and their corresponding -values are exponentially distributed. We run the same simulation as before, but then convert the obtained -values into -values, to see how they are distributed.

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
n.sim <- 100
t.sim <- numeric(n.sim)
n.samp <- 1000

for (i in 1:n.sim) {
    X <- rnorm(n.samp, mean = 0, sd = 1)
    Y <- rnorm(n.samp, mean = 0, sd = 1)
    df <- data.frame(X, Y)
    t <- t.test(X, Y, data = df)
    t.sim[i] <- t[[3]]
}

ggplot(NULL, aes(x = -log2(t.sim))) + geom_histogram(bins = 30,
    col = "black", fill = "#d46c5b", alpha = 0.5) + labs(title = "Distribution of S-values Under the Null",
    x = "S-value (Bits of Information)") + theme_bw()

Posterior Predictive -values

Despite the name, posterior predictive -values are not considered valid frequentist -values because they do not meet the uniformity criterion, instead they are pulled towards the parameter value 0.5. For further discussion, see our technical supplement along with Greenland (2019) for comprehensive theoretical discussions.2, 3

A quick excerpt from our main paper and the technical supplement explains why this is not the case,

As discussed in the Supplement, in Bayesian settings one may see certain -values that are not valid frequentist -values, the primary example being the posterior predictive -value4;5; unfortunately, the negative logs of such invalid -values do not measure surprisal at the statistic given the model, and so are not valid -values.

The decision rule “reject H if ” will reject H 100% of the time under sampling from a model M obeying H (i.e., the Type-1 error rate of the test will be ) provided the random variable corresponding to is valid (uniform under the model M used to compute it), but not necessarily otherwise6. This is one reason why frequentist writers reject invalid -values (such as posterior predictive -values, which highly concentrate around 0.50) and devote considerable technical coverage to uniform -values6;5;4. A valid -value (“U-value”) translates into an exponentially distributed -value with a mean of 1 nat or bits where is the base of the natural logs.

Uniformity is also central to the “refutational information” interpretation of the -value used here, for it is necessary to ensure that the -value from which is derived is in fact the percentile of the observed value of the test statistic in the distribution of the statistic under M, thus making small surprising under M and making the corresponding degree of surprise. Because posterior predictive -values do not translate into sampling percentiles of the statistic under the hypothesis (in fact, they are pulled toward 0.5 from the correct percentiles)5;4, the resulting negative log does not measure surprisal at the statistic given M, and so is not a valid -value in our terms.

And indeed, we can show this phenomenon below with simulations. Here we fit a simple Bayesian regression model with a weakly informative prior normal(0, 10) using rstanarm, where both the predictor and response variable come from the same distribution and have the same location and scale parameters (, . We then calculate the observed test statistics, along with their distributions, and convert them into posterior predictive -values and plot them using Bayesplot functions. Then, we iterate this 1000 times to examine the distribution of posterior predictive -values and compare them to standard -values that are known to be uniform.

But first, we’ll generate the distribution of the test statistic from one model.

library("bayesplot")
library("rstan")
library("rstanarm")
rstan_options(auto_write = TRUE)
options(mc.cores = 4)
teal <- c("#99c7c7")
color_scheme_set("teal")
RNGkind(kind = "L'Ecuyer-CMRG")
n.samp <- 100
X <- rnorm(n.samp, mean = 0, sd = 1)
Y <- rnorm(n.samp, mean = 0, sd = 1)
df <- data.frame(X, Y)
mod1 <- stan_glm(Y ~ X, data = df, chains = 1, cores = 4, refresh = 0,
    prior = normal(0, 10))
#> Error in `if (!prior_dist_name %in% unlist(ok_dists)) ...`:
#> ! argument is of length zero
yrep <- posterior_predict(mod1)
#> Error:
#> ! object 'mod1' not found

h <- ppc_stat(Y, yrep, stat = "median", freq = TRUE, binwidth = 0.01) +
    labs(title = "Distribution of Posterior Test Statistic") +
    theme_bw() + yaxis_text(on = TRUE)
#> Error:
#> ! object 'yrep' not found

values_all <- h[["plot_env"]][["T_yrep"]]
#> Error:
#> ! object 'h' not found
prob_to_find <- h[["plot_env"]][["T_y"]]
#> Error:
#> ! object 'h' not found
quantInv <- function(distr, value) ecdf(distr)(value)
posterior_p_value <- quantInv(values_all, prob_to_find)
#> Error:
#> ! object 'values_all' not found

l <- ppc_stat_2d(Y, yrep) + theme_bw() + labs(title = "Distribution of Posterior Test Statistic")
#> Error:
#> ! object 'yrep' not found

plot(h)
#> Error in `h()`:
#> ! error in evaluating the argument 'x' in selecting a method for function 'plot': object 'h' not found
plot(l)
#> Error in `h()`:
#> ! error in evaluating the argument 'x' in selecting a method for function 'plot': object 'l' not found
print(prob_to_find)
#> Error in `h()`:
#> ! error in evaluating the argument 'x' in selecting a method for function 'print': object 'prob_to_find' not found

The above is the posterior predictive test statistic for one model, along with it’s distribution. Now let’s iterate this process, convert them into posterior predictive -values, and generate their distribution.

rstan_options(auto_write = TRUE)
options(mc.cores = 4)
RNGkind(kind = "L'Ecuyer-CMRG")
set.seed <- 1031
n.sim <- 100
ppp <- numeric(n.sim)
n.samp <- 1000

quantInv <- function(distr, value) ecdf(distr)(value)

for (i in 1:length(ppp)) {
    X <- rnorm(n.samp, mean = 0, sd = 1)
    Y <- rnorm(n.samp, mean = 0, sd = 1)
    df <- data.frame(X, Y)
    mod1 <- stan_glm(Y ~ X, data = df, chains = 1, cores = 4,
        iter = 1000, refresh = 0, prior = normal(0, 10))
    yrep <- posterior_predict(mod1)
    h <- ppc_stat(Y, yrep, stat = "median")
    values_all <- h[["plot_env"]][["T_yrep"]]
    prob_to_find <- h[["plot_env"]][["T_y"]]
    posterior_p_value <- quantInv(values_all, prob_to_find)
    ppp[i] <- posterior_p_value
}
#> Error in `if (!prior_dist_name %in% unlist(ok_dists)) ...`:
#> ! argument is of length zero

ggplot(NULL, aes(x = ppp)) + geom_histogram(bins = 30, col = "black",
    fill = "#99c7c7", alpha = 0.25) + labs(title = "Distribution of Posterior Predictive P-values",
    subtitle = "From Simulation of 1000 Simple Linear Regressions",
    x = "Posterior Predictive P-value") + scale_x_continuous(breaks = seq(0,
    1, 0.1)) + theme_bw()

We can also show this in Stata 16.1, using their new bayesstats ppvalues command which generates posterior predictive -values.

clear
set seed 1031
set obs 100

generate x = rnormal(0, 1)
generate y = rnormal(0, 1)

* Bayesian linear regression of y on x, mirroring the rstanarm model.
* Stata parameterizes the normal() prior by VARIANCE, so rstanarm's
* normal(0, sd = 10) is normal(0, 100) here. {y:} puts the prior on all
* regression coefficients including the intercept -- bayesmh requires a
* prior for every parameter. The variance gets a proper inverse-gamma
* prior; a normal prior is invalid for {var} (its support includes
* negative values).
bayesmh y x, likelihood(normal({var}))  ///
    prior({y:}, normal(0, 100))         ///
    prior({var}, igamma(0.01, 0.01))    ///
    rseed(1031) mcmcsize(1000)          ///
    saving(bmh_sim, replace)

* Simulate replicated outcomes from the posterior predictive
* distribution. This must be saved to a DIFFERENT file than the
* bayesmh simulation dataset above.
bayespredict {_ysim}, rseed(1031) saving(bmh_pred, replace)

* Posterior predictive p-values for test statistics of the replicated
* outcome. (The R analog uses the median; @median is not a built-in
* bayespredict/bayesstats function, so we use the mean and variance,
* matching the worked example in [BAYES] bayesstats ppvalues.)
bayesstats ppvalues (tmean:@mean({_ysim})) (tvar:@variance({_ysim})) ///
    using bmh_pred
#> could not find feasible values
#> r(491);
#> 
#> 
#> 
#> 
#> Number of observations (_N) was 0, now 100.
#> 
#> 
#> 
#> 
#> Burn-in ...
#> Simulation ...
#> 
#> Model summary
#> -------------------------------------------------------------------------
#> > -----
#> Likelihood: 
#>   y ~ normal(xb_y,{var})
#> 
#> Priors: 
#>   {y:x _cons} ~
#>                 normal(0,100)                                            
#>                 >   (1)
#>         {var} ~ igamma(0.01,0.01)
#> -------------------------------------------------------------------------
#> > -----
#> (1) Parameters are elements of the linear form xb_y.
#> 
#> Bayesian normal regression                       MCMC iterations  =      
#> > 3,500
#> Random-walk Metropolis–Hastings sampling         Burn-in          =      
#> > 2,500
#>                                                  MCMC sample size =      
#> > 1,000
#>                                                  Number of obs    =      
#> >   100
#>                                                  Acceptance rate  =      
#> > .2428
#>                                                  Efficiency:  min =    .0
#> > 01044
#>                                                               avg =     .
#> > 04252
#> Log marginal-likelihood = -153.90987                          max =     .
#> > 08305
#> 
#> -------------------------------------------------------------------------
#> > -----
#>              |                                                Equal-taile
#> > d
#>              |      Mean   Std. dev.     MCSE     Median  [95% cred. inte
#> > rval]
#> -------------+-----------------------------------------------------------
#> > -----
#> y            |
#>            x |  .0115139   .1551555   .017026   .0145751  -.2995666   .34
#> > 70796
#>        _cons |  .1022613    .137549   .020865   .1348998  -.2513679   .34
#> > 55865
#> -------------+-----------------------------------------------------------
#> > -----
#>          var |  .9643251   .1503727    .14715   .9896135   .7582507   1.3
#> > 60154
#> -------------------------------------------------------------------------
#> > -----
#> 
#> file bmh_sim.dta saved.
#> 
#> 
#> Computing predictions ...
#> 
#> file bmh_pred.dta saved.
#> file bmh_pred.ster saved.
#> 
#> 
#> Posterior predictive summary   MCMC sample size =     1,000
#> 
#> -----------------------------------------------------------
#>            T |      Mean   Std. dev.  E(T_obs)  P(T>=T_obs)
#> -------------+---------------------------------------------
#>        tmean |  .0550687   .1146021   .0206952         .632
#>         tvar |  .9896307   .2164376   .9710317         .484
#> -----------------------------------------------------------
#> Note: P(T>=T_obs) close to 0 or 1 indicates lack of fit.
#> 
#> OMP: Warning #96: Cannot form a team with 2 threads, using 1 instead.
#> OMP: Hint Consider unsetting KMP_DEVICE_THREAD_LIMIT (KMP_ALL_THREADS), KMP_TEAMS_THREAD_LIMIT, and OMP_THREAD_LIMIT (if any are set).

As we can see from the R plot above, the test statistics are are generally concentrated around 0.5 and the distribution (as shown above via the R code) hardly resembles the uniform distribution of -value under the null model. Further, the base-2 log transformation of posterior predictive -values is not exponentially distributed, making them invalid -values in our terms.

ggplot(NULL, aes(x = (-log2(ppp)))) + geom_histogram(bins = 30,
    col = "black", fill = "#d46c5b", alpha = 0.25) + labs(title = "Distribution of -log2(Posterior Predictive P-values)",
    subtitle = "S-values Produced From Posterior Predictive P-values",
    x = "-log2(Posterior Predictive P-value)") + theme_bw()
#> Error in `geom_histogram()`:
#> ! Problem while computing stat.
#> ℹ Error occurred in the 1st layer.
#> Caused by error in `seq_len()`:
#> ! argument must be coercible to non-negative integer

Table 1: Some -values, -values, Maximum Likelihood Ratios, and Likelihood Ratio Statistics

Here, we look at some common -values, and how they correspond to other information statistics and likelihood measures.

pvalue <- c(0.99, 0.9, 0.5, 0.25, 0.1, 0.05, 0.025, 0.01, 0.005,
    1e-04, paste("5 sigma (~ 2.9 in 10 million)"), paste("1 in 100 million (GWAS)"),
    paste("6 sigma (~ 1 in a billion)"))

svalue <- round(c(-log2(c(0.99, 0.9, 0.5, 0.25, 0.1, 0.05, 0.025,
    0.01, 0.005, 1e-04)), 21.7, 26.6, 29.9), 2)

pvalue[1:10] <- formatC(pvalue[1:10], format = "e", digits = 2)

mlr <- round(c(1, 1.01, 1.26, 1.94, 3.87, 6.83, 12.3, 27.6, 51.4,
    1935, (5.2 * (10^5)), (1.4 * (10^7)), (1.3 * (10^8))), 2)

# likelihood ratio statistic

lr <- round(c(0.00016, 0.016, 0.45, 1.32, 2.71, 3.84, 5.02, 6.63,
    7.88, 15.1, 26.3, 32.8, 37.4), 2)

table1 <- data.frame(pvalue, svalue, mlr, lr)

colnames(table1) <- c("P-value (compatibility)", "S-value (bits)",
    "Maximum Likelihood Ratio", "Deviance Statistic 2ln(MLR)")

kbl(table1, format = "html", padding = 45, longtable = TRUE,
    color = "#777") %>%
    row_spec(row = 0, color = "#777", bold = T) %>%
    column_spec(column = 1:4, color = "#777", bold = F) %>%
    kable_classic(full_width = TRUE, html_font = "Cambria", lightable_options = "basic",
        font_size = 17, fixed_thead = list(enabled = F)) %>%
    footnote(general_title = "Abbreviations: ", title_format = "underline",
        general = c("Table 1 $P$-values                  and binary $S$-values, with corresponding maximum-likelihood ratios (MLR) and deviance (likelihood-ratio) statistics for a simple test hypothesis H under background assumptions A"))
P-value (compatibility) S-value (bits) Maximum Likelihood Ratio Deviance Statistic 2ln(MLR)
0.99 0.01 1.00e+00 0.00
0.9 0.15 1.01e+00 0.02
0.5 1.00 1.26e+00 0.45
0.25 2.00 1.94e+00 1.32
0.1 3.32 3.87e+00 2.71
0.05 4.32 6.83e+00 3.84
0.025 5.32 1.23e+01 5.02
0.01 6.64 2.76e+01 6.63
0.005 7.64 5.14e+01 7.88
1e-04 13.29 1.94e+03 15.10
5 sigma (~ 2.9 in 10 million) 21.70 5.20e+05 26.30
1 in 100 million (GWAS) 26.60 1.40e+07 32.80
6 sigma (~ 1 in a billion) 29.90 1.30e+08 37.40
Abbreviations:
Table 1 $P$-values and binary $S$-values, with corresponding maximum-likelihood ratios (MLR) and deviance (likelihood-ratio) statistics for a simple test hypothesis H under background assumptions A

For further discussion of 5 and 6 sigma cutoffs, see7.

Frequentist Interval Estimate Percentiles

Refer to Long-Run Coverage

Here we simulate a study where one group with 100 participants has an of 100 with a of 20 and the second group has the same number of participants but an average of 80 and a standard deviation of 20. We compare them using a Welch’s -test and generate 95% compatibility intervals several times, specifically 100 times, and then plot them. Since we know the mean difference is 20, we wish to see how often the interval estimates cover this true parameter value.

This shows that the intervals tend to vary simply as a result of randomness, and that the proper interpretation of the attached percentile is about long run coverage of the true parameter. However, this does not preclude interpretation of a single interval estimate from a study.8 As we write in the Replace unrealistic “confidence” claims with compatibility measures section of our paper.1

The fact that “confidence” refers to the procedure behavior, not the reported interval, seems to be lost on most researchers. Remarking on this subtlety, when Jerzy Neyman discussed his confidence concept in 1934 at a meeting of the Royal Statistical Society, Arthur Bowley replied, “I am not at all sure that the ‘confidence’ is not a confidence trick.”9. And indeed, 40 years later, Cox and Hinkley warned, “interval estimates cannot be taken as probability statements about parameters, and foremost is the interpretation ‘such and such parameter values are consistent with the data.’”8. Unfortunately, the word “consistency” is used for several other concepts in statistics, while in logic it refers to an absolute condition (of noncontradiction); thus, its use in place of “confidence” would risk further confusion.

To address the problems above, we exploit the fact that a 95% CI summarizes the results of varying the test hypothesis H over a range of parameter values, displaying all values for which > 0.0510 and hence < 4.32 bits3;11. Thus the CI contains a range of parameter values that are more compatible with the data than are values outside the interval, under the background assumptions3;12. Unconditionally (and thus even if the background assumptions are uncertain), the interval shows the values of the parameter which, when combined with the background assumptions, produce a test model that is “highly compatible” with the data in the sense of having less than 4.32 bits of information against it. We thus refer to CI as compatibility intervals rather than confidence intervals13;3;11; their abbreviation remains “CI.” We reject calling these intervals “uncertainty intervals,” because they do not capture uncertainty about background assumptions13.

Indeed, this also has to do with our paper on deemphasizing the model assumptions that are often behind common statistical outputs, and why it is necessary to treat them as uncertain, rather than given.14 For example, a 95% interval estimate is assumed to be free of bias in its construction, however, it’s coverage claims are no longer so, once there are systematic errors in play. Many of these common, classical statistical procedures are designed to deal with random variation, and less so with bias (a relatively new field, with shiny new methods).

Brown et al. (2017) Reanalysis

Taken from the Brown et al. data.15, 16

Here we take the reported statistics from the Brown et al. studies in order to run statistical tests of different alternative test hypotheses and use those results to construct various functions.

We calculate the standard errors from the point estimate and the confidence (compatibility) limits.

se <- log(2.59/0.997)/3.92

logUL <- log(2.59)
logLL <- log(0.997)

logpoint <- log(1.61)

logpoint + (1.96 * se)
#> [1] 0.954
logpoint - (1.96 * se)
#> [1] -0.0011

Table 2: -values, -values, and Likelihoods for Targeted Hypotheses About Hazard Ratios for Brown et al.

testhypothesis <- c("Halving of hazard", "No effect (null)",
    "Point estimate", "Doubling of hazard", "Tripling of hazard",
    "Quintupling of hazard")

hazardratios <- c(0.5, 1, 1.61, 2, 3, 5)
pvals <- c(1.6e-06, 0.05, 1, 0.37, 0.01, 3.2e-06)
svals <- round(-log2(pvals), 2)
mlr2 <- c((1 * 10^5), 6.77, 1, 1.49, 26.2, (5 * 10^4))
lr <- round(c(exp((((log(0.5/1.61))/se)^2)/(2)), exp((((log(1/1.61))/se)^2)/(2)),
    exp((((log(1.61/1.61))/se)^2)/(2)), exp((((log(2/1.61))/se)^2)/(2)),
    exp((((log(3/1.61))/se)^2)/(2)), exp((((log(5/1.61))/se)^2)/(2))),
    3)

LR <- formatC(lr, format = "e", digits = 2)

table2 <- data.frame(testhypothesis, hazardratios, pvals, svals,
    mlr2, LR)

colnames(table2) <- c("Test Hypothesis", "HR", "P-values", "S-values",
    "MLR", "LR")

kbl(table2, format = "html", padding = 45, longtable = TRUE,
    color = "#777") %>%
    row_spec(row = 0, color = "#777", bold = T) %>%
    column_spec(column = 1:6, color = "#777", bold = F) %>%
    kable_classic(full_width = TRUE, html_font = "Cambria", lightable_options = "basic",
        font_size = 15, fixed_thead = list(enabled = F))
Test Hypothesis HR P-values S-values MLR LR
Halving of hazard 0.50 0.00 19.25 1.00e+05 1.02e+05
No effect (null) 1.00 0.05 4.32 6.77e+00 6.77e+00
Point estimate 1.61 1.00 0.00 1.00e+00 1.00e+00
Doubling of hazard 2.00 0.37 1.43 1.49e+00 1.49e+00
Tripling of hazard 3.00 0.01 6.64 2.62e+01 2.62e+01
Quintupling of hazard 5.00 0.00 18.25 5.00e+04 5.03e+04

Plot the point estimate and 95% compatibility interval

label <- c("Brown et al. (2017)\n JAMA", "Brown et al. (2017)\n J Clin Psychiatry")

point <- c(1.61, 1.7)
lower <- c(0.997, 1.1)
upper <- c(2.59, 2.6)

df <- data.frame(label, point, lower, upper)

Here we plot the 95% compatibility interval estimate reported from the high-dimensional propensity score analysis.

ggplot(data = df, mapping = aes(x = label, y = point, ymin = lower,
    ymax = upper, group = label)) + geom_pointrange(color = "#007C7C",
    size = 1.5, alpha = 0.4) + geom_hline(yintercept = 1, lty = 3,
    color = "#d46c5b", alpha = 0.5) + coord_flip() + scale_y_log10(breaks = scales::pretty_breaks(n = 10),
    limits = c(0.8, 4)) + labs(title = "Association Between Serotonergic Antidepressant Exposure
               \nDuring Pregnancy and Child Autism Spectrum Disorder",
    subtitle = "Reported 95% Compatibility Intervals From Primary Results",
    x = "Study", y = "Hazard Ratio (JAMA) +
            Adjusted Pooled Odds Ratio (J Clin Psychiatry)") +
    annotate(geom = "text", x = 1.5, y = 1, size = 5, label = "Null parameter value of 1",
        color = "#d46c5b", alpha = 0.75) + annotate(geom = "text",
    x = 2.2, y = 1.8, size = 5, label = "Point Estimate = 1.61,
           Lower Limit = 0.997, Upper Limit = 2.59",
    color = "#000000", alpha = 0.75) + annotate(geom = "text",
    x = 1.2, y = 1.8, size = 5, label = "Point Estimate = 1.7,
           Lower Limit = 1.1, Upper Limit = 2.6",
    color = "#000000", alpha = 0.75) + theme_light() + theme(axis.text.y = element_text(angle = 90,
    hjust = 1, size = 13)) + theme(axis.text.x = element_text(size = 13)) +
    theme(title = element_text(size = 13))

Once again, the authors reported that15

In the 2837 pregnancies (7.9%) exposed to antidepressants, 2.0% (95% CI, 1.6%-2.6%) of children were diagnosed with autism spectrum disorder. Risk of autism spectrum disorder was significantly higher with serotonergic antidepressant exposure (4.51 exposed vs 2.03 unexposed per 1000 person-years; between-group difference, 2.48 95% CI, 2.33-2.62 per 1000 person-years) in crude (HR, 2.16 95% CI, 1.64-2.86) and multivariable-adjusted analyses (HR, 1.59 95% CI, 1.17-2.17) (Table 2). After inverse probability of treatment weighting based on the HDPS, the association was not significant (HR, 1.61 95% CI, 0.997-2.59) (Table 2).

and concluded with

In children born to mothers receiving public drug coverage in Ontario, Canada, in utero serotonergic antidepressant exposure compared with no exposure was not associated with autism spectrum disorder in the child. Although a causal relationship cannot be ruled out, the previously observed association may be explained by other factors.

We will show using the graphical and tabular summaries below, why a more nuanced summary such as,

“After HDPS adjustment for confounding, a 61% hazard elevation remained; however, under the same model, every hypothesis from no elevation up to a 160% hazard increase had > 0.05; Thus, while quite imprecise, these results are consistent with previous observations of a positive association between serotonergic antidepressant prescriptions and subsequent ASD. Because the association may be partially or wholly due to uncontrolled biases, further evidence will be needed for evaluating what, if any, proportion of it can be attributed to causal effects of prenatal serotonergic antidepressant use on ASD incidence.”

would have been far more appropriate, as we discuss in our paper.1

-value and -value Functions

In order to use this information to construct a -value function, we can use the concurve functions.

library("concurve")

We enter the reported point estimates and compatibility limits and produce all possible intervals + -values + -values. This is calculated assuming normal approximations with the curve_rev() function.

curve1 <- curve_rev(point = 1.61, LL = 0.997, UL = 2.59, measure = "ratio")

It is stored in the object curve1.

We can generate a table of the relevant statistics from this reanalysis, similar to the table from above, but this time using the concurve function, curve_table(), but this time it will give us interval estimates of various percentiles (%, %, %, %, etc.). These are in turn used to construct the entire confidence (compatibility) distribution or -value function.

kbl(curve_table(curve1[[1]]), format = "html", padding = 45,
    longtable = TRUE, color = "#777", row.names = F, col.names = colnames(curve_table(curve1[[1]]))) %>%
    row_spec(row = 0, color = "#777", bold = T) %>%
    column_spec(column = 1:7, color = "#777", bold = F) %>%
    kable_classic(full_width = TRUE, html_font = "Cambria", lightable_options = "basic",
        font_size = 17, fixed_thead = list(enabled = F)) %>%
    footnote(general_title = c("Table of Statistics for Various Interval Estimate Percentiles"),
        general = " ")
Lower Limit Upper Limit Interval Width Interval Level (%) CDF P-value S-value (bits)
1.490 1.74 0.250 25.0 0.625 0.750 0.415
1.366 1.90 0.531 50.0 0.750 0.500 1.000
1.217 2.13 0.914 75.0 0.875 0.250 2.000
1.178 2.20 1.021 80.0 0.900 0.200 2.322
1.134 2.29 1.152 85.0 0.925 0.150 2.737
1.079 2.40 1.325 90.0 0.950 0.100 3.322
0.999 2.60 1.596 95.0 0.975 0.050 4.322
0.933 2.78 1.846 97.5 0.988 0.025 5.322
0.860 3.02 2.155 99.0 0.995 0.010 6.644
Table of Statistics for Various Interval Estimate Percentiles

Figure 3: -value (Compatibility) Function

Plot the -value (Compatibility) function of the Brown et al. data

Now we plot the -value function from the data stored in curve1 using the ggcurve() function from concurve.

ggcurve(curve1[[1]], type = "c", measure = "ratio", nullvalue = c(1),
    levels = c(0.5, 0.75, 0.95)) + labs(title = expression(paste(italic(P),
    "-value (Compatibility) Function")), subtitle = expression(paste(italic(P),
    "-values for a range of hazard ratios (HR)")), x = "Hazard Ratio (HR)",
    y = expression(paste(italic(P), "-value"))) + geom_vline(xintercept = 1.61,
    lty = 1, color = "gray", alpha = 0.2) + geom_vline(xintercept = 2.59,
    lty = 1, color = "gray", alpha = 0.2) + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11))

It is practically the same as the published version from Rafi & Greenland, 20201.

curve1 <- curve_rev(point = 1.61, LL = 0.997, UL = 2.59, measure = "ratio")

curve2 <- curve_rev(point = 1.7, LL = 1.1, UL = 2.6, measure = "ratio")

Cumulative Confidence (Compatibility Distribution)

Although we do not cover this figure in our paper, it is easy to construct using concurve's ggcurve() function by specifying type as “cdf” to see the median estimate within the confidence distribution. The horizontal line that meets at the curve, is approximately the same as the maximum at the -value function/confidence (compatibility) curve.

ggcurve(curve1[[2]], type = "cdf", measure = "ratio", nullvalue = c(1)) +
    labs(title = "P-values for a range of hazard ratios (HR)",
        subtitle = "Association Between Serotonergic Antidepressant Exposure
       \nDuring Pregnancy and Child Autism Spectrum Disorder",
        x = "Hazard Ratio (HR)", y = "Cumulative Compatibility") +
    theme(plot.title = element_text(size = 12), plot.subtitle = element_text(size = 11),
        axis.title.x = element_text(size = 12), axis.title.y = element_text(size = 12),
        text = element_text(size = 11))

We will also calculate the likelihoods so that we can generate the corresponding likelihood functions.

lik2 <- curve_rev(point = 1.7, LL = 1.1, UL = 2.6, type = "l",
    measure = "ratio")

lik1 <- curve_rev(point = 1.61, LL = 0.997, UL = 2.59, type = "l",
    measure = "ratio")

We can also see how consistent these results are with previous studies conducted by the same research group, given the overlap of the functions, which can be compared using the plot_compare() function. Let’s compare the relative likelihood functions from both studies from this research group to see how consistent the results are.

plot_compare(data1 = lik1[[1]], data2 = lik2[[1]], type = "l1",
    measure = "ratio", nullvalue = TRUE, title = "Brown et al. 2017. J Clin Psychiatry. vs.
              \nBrown et al. 2017. JAMA.",
    subtitle = "J Clin Psychiatry: OR = 1.7, 1/6.83 LI: LL = 1.1, UL = 2.6
                         \nJAMA: HR = 1.61, 1/6.83 LI: LL = 0.997, UL = 2.59",
    xaxis = "Hazard Ratio / Odds Ratio") + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11))

and the -value functions.

plot_compare(data1 = curve1[[1]], data2 = curve2[[1]], type = "c",
    measure = "ratio", nullvalue = TRUE, title = "Brown et al. 2017. J Clin Psychiatry. vs.
              \nBrown et al. 2017. JAMA.",
    subtitle = "J Clin Psychiatry: OR = 1.7, 1/6.83 LI: LL = 1.1, UL = 2.6
              \nJAMA: HR = 1.61, 1/6.83 LI: LL = 0.997, UL = 2.59",
    xaxis = "Hazard Ratio / Odds Ratio") + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11))

Figure 4: -value (Suprisal) Function

Plot the -value (Surprisal) function of the Brown et al. data with ggcurve()

ggcurve(data = curve1[[1]], type = "s", measure = "ratio", nullvalue = c(1),
    title = "S-value (Surprisal) Function", subtitle = "S-Values (surprisals) for a range of hazard ratios (HR)",
    xaxis = "Hazard Ratio", yaxis1 = "S-value (bits of information)") +
    theme(plot.title = element_text(size = 12), plot.subtitle = element_text(size = 11),
        axis.title.x = element_text(size = 12), axis.title.y = element_text(size = 12),
        text = element_text(size = 11))

which is quite close to the figure from our paper.

Likelihood (Support) Functions

Here we provide the code to show how we constructed the likelihood functions from our paper, which are the supplementary figures found here (S1) and here (S2).

Calculate and Plot Likelihood (Support) Intervals.

Here we use the reported estimates to construct the -scores and standard errors, which are in turn used to compute the likelihoods.

hrvalues <- seq(from = 0.65, to = 3.98, by = 0.01)

se <- log(2.59/0.997)/3.92

zscore <- sapply(hrvalues, function(i) (log(i/1.61)/se))

Figure S1: Relative Likelihood Function

We then standardize all the likelihoods by their maximum at the likelihood function,17 to produce relative likelihoods, likelihood intervals, and their corresponding relative likelihood function.18

support <- exp((-zscore^2)/2)

likfunction <- data.frame(hrvalues, zscore, support)

ggplot(data = likfunction, mapping = aes(x = hrvalues, y = support)) +
    geom_line() + geom_vline(xintercept = 1, lty = 3, color = "#d46c5b",
    alpha = 0.75) + geom_hline(yintercept = (1/6.83), lty = 1,
    color = "#333333", alpha = 0.05) + geom_ribbon(aes(x = hrvalues,
    ymin = min(support), ymax = support), fill = "#d46c5b", alpha = 0.1) +
    labs(title = "Relative Likelihood Function", subtitle = "Relative likelihoods for a range of hazard ratios",
        x = "Hazard Ratio (HR)", y = "Relative Likelihood \n1/MLR") +
    annotate(geom = "text", x = 1.65, y = 0.2, label = "1/6.83 LI = 0.997, 2.59",
        size = 4, color = "#000000") + theme_bw() + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11)) +
    scale_x_log10(breaks = scales::pretty_breaks(n = 10)) + scale_y_continuous(expand = expansion(mult = c(0.01,
    0.0125)), breaks = seq(0, 1, 0.1))

The likelihood interval corresponds to the 95% compatibility interval, as shown by the figure from our paper.

Below we use the calculated -scores to construct the log-likelihood function which is the upward-concave parabola = 

support <- (zscore^2)/2

likfunction <- data.frame(hrvalues, zscore, support)

ggplot(data = likfunction, mapping = aes(x = hrvalues, y = support)) +
    geom_line() + geom_vline(xintercept = 1, lty = 3, color = "#d46c5b",
    alpha = 0.75) + geom_ribbon(aes(x = hrvalues, ymin = support,
    ymax = max(support)), fill = "#d46c5b", alpha = 0.1) + labs(title = "Log-Likelihood Function",
    subtitle = "Log-likelihoods for a range of hazard ratios",
    x = "Hazard Ratio (HR)", y = "ln(MLR)") + theme_bw() + theme(axis.title.x = element_text(size = 13),
    axis.title.y = element_text(size = 13)) + scale_x_log10(breaks = scales::pretty_breaks(n = 10)) +
    scale_y_continuous(expand = expansion(mult = c(0.01, 0.0125)),
        breaks = seq(0, 7, 1)) + theme(text = element_text(size = 15)) +
    theme(plot.title = element_text(size = 12), plot.subtitle = element_text(size = 12))

Figure S2: Deviance Function

Known as the deviance function, it is twice the log-likelihood, which maps to the -value function.

support <- (zscore^2)

likfunction <- data.frame(hrvalues, zscore, support)

ggplot(data = likfunction, mapping = aes(x = hrvalues, y = support)) +
    geom_line() + geom_vline(xintercept = 1, lty = 3, color = "#d46c5b",
    alpha = 0.75) + geom_hline(yintercept = 3.84, lty = 1, color = "#333333",
    alpha = 0.05) + geom_ribbon(aes(x = hrvalues, ymin = support,
    ymax = max(support)), fill = "#d46c5b", alpha = 0.1) + annotate(geom = "text",
    x = 1.65, y = 4.4, label = "1/6.83 LI = 0.997, 2.59", size = 4,
    color = "#000000") + theme_bw() + theme(plot.title = element_text(size = 12),
    plot.subtitle = element_text(size = 11), axis.title.x = element_text(size = 12),
    axis.title.y = element_text(size = 12), text = element_text(size = 11)) +
    scale_x_log10(breaks = scales::pretty_breaks(n = 10)) + scale_y_continuous(expand = expansion(mult = c(0.01,
    0.0125)), breaks = seq(0, 14, 2)) + labs(title = "Deviance Function",
    subtitle = "Deviance statistics for a range of hazard ratios",
    x = "Hazard Ratio (HR)", y = " Deviance Statistic \n2ln(MLR)")

And the version from our paper can be found here.

It is important to note that although we have calculated these likelihood functions manually here, they can also be generated easily using the curve_lik() function which takes inputs from the ProfileLikelihood R package. To see a further discussion, please see the following article, which gives several examples for a wide variety of models.

Further, we urge some caution. Although we endorse the construction and presentation of likelihood functions along with -value and -value functions, along with the tabulations, the use of pure likelihood methods has been highly controversial among some statisticians, with some going as far as to say that likelihood is blind although not all statisticians believe this, and have responded to such criticisms in kind, see discussants.19 Thus, we support providing both -value/-value and likelihood-based functions for a complete picture.

Indeed, we can see how they all easily map to one another when plotted side by side, with the following script.

library("cowplot")
plot_grid(confcurve, surprisalcurve, relsupportfunction, deviancefunction,
    ncol = 2, nrow = 2)

A clearer plot comparing the four functions is seen below (click on the image to view in full).

Additional adjustments that were made to the figures from Rafi & Greenland, 20201 were done using Adobe Illustrator and Photoshop.

All errors are ours, and we welcome critical feedback and reporting of errors. To report possible errors in our analyses, please post it as a comment below or as a bug here. We will compile them onto a public errata, if any errors or flaws are reported to us.

Statistical Package Citations

Please remember to cite the R packages if you use any of the R scripts from above. The citation for our1 can be found below in the References section or it can be downloaded here for a reference manager.

citation("concurve")
citation("TeachingDemos")
citation("ProfileLikelihood")
citation("rstan")
citation("rstanarm")
citation("bayesplot")
citation("knitr")
citation("kableExtra")
citation("ggplot2")
citation("cowplot")
citation("Statamarkdown")

Environment

The analyses were run on:

si <- sessionInfo()
print(si, RNG = TRUE, locale = TRUE)
#> R version 4.6.1 (2026-06-24)
#> Platform: aarch64-apple-darwin25.4.0
#> Running under: macOS Golden Gate 27.0
#> 
#> Matrix products: default
#> BLAS:   /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib 
#> LAPACK: /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib;  LAPACK version 3.12.1
#> 
#> Random number generation:
#>  RNG:     L'Ecuyer-CMRG 
#>  Normal:  Inversion 
#>  Sample:  Rejection 
#>  
#> locale:
#> [1] en_US/C.UTF-8/en_US/C/en_US/en_US
#> 
#> time zone: America/New_York
#> tzcode source: internal
#> 
#> attached base packages:
#>  [1] splines   grid      stats4    parallel  stats     graphics  grDevices
#>  [8] utils     datasets  methods   base     
#> 
#> other attached packages:
#>   [1] plotly_4.12.1         TSstudio_0.1.7        tfautograph_0.3.2    
#>   [4] tfdatasets_2.18.0     keras_2.16.1          tensorflow_2.20.0    
#>   [7] timetk_2.9.1          modeltime_1.3.5       workflowsets_1.1.1   
#>  [10] workflows_1.3.0       tune_2.1.0            tailor_0.1.0         
#>  [13] rsample_1.3.2         recipes_1.4.0         parsnip_1.6.0        
#>  [16] modeldata_1.6.0       infer_1.1.0           dials_1.4.4          
#>  [19] scales_1.4.0          tidymodels_1.5.0      xgboost_3.2.1.1      
#>  [22] prophet_1.1.7         rlang_1.3.0           astsa_2.5            
#>  [25] bayesforecast_1.0.5   smooth_4.5.2          greybox_2.0.8        
#>  [28] glmnet_5.0            forecast_9.0.2        fable_0.5.0          
#>  [31] fabletools_0.8.0      tsibbledata_0.4.1     tsibble_1.2.0        
#>  [34] rstanarm_2.32.2       texPreview_2.1.0      tinytex_0.60         
#>  [37] rmarkdown_2.32        brms_2.23.0           bootImpute_1.3.0     
#>  [40] knitr_1.52            boot_1.3-32           JuliaCall_0.17.6     
#>  [43] reshape2_1.4.5        ProfileLikelihood_1.3 ImputeRobust_1.3-1   
#>  [46] gamlss_5.5-0          gamlss.dist_6.1-11    gamlss.data_6.0-7    
#>  [49] mvtnorm_1.4-2         performance_0.18.2    summarytools_1.1.5   
#>  [52] tidybayes_3.0.7       htmltools_0.5.9       Statamarkdown_1.0.0  
#>  [55] car_3.1-5             carData_3.0-6         qqplotr_0.0.7        
#>  [58] ggcorrplot_0.3.0      mitml_0.4-5           pbmcapply_1.5.1      
#>  [61] Amelia_1.8.3          Rcpp_1.1.2            blogdown_1.24        
#>  [64] doParallel_1.0.17     iterators_1.0.14      foreach_1.5.2        
#>  [67] lattice_0.23-1        bayesplot_1.16.0      wesanderson_0.3.7    
#>  [70] VIM_7.3.1             colorspace_2.1-3      here_1.0.2           
#>  [73] progress_1.2.3        loo_2.10.1            mi_1.3.1             
#>  [76] Matrix_1.7-6          broom_1.0.13          yardstick_1.4.0      
#>  [79] svglite_2.2.2         Cairo_1.7-0           cowplot_1.2.0        
#>  [82] mgcv_1.9-4            nlme_3.1-171          xfun_0.60            
#>  [85] broom.mixed_0.2.9.7   reticulate_1.47.0     kableExtra_1.4.1     
#>  [88] posterior_1.7.0       checkmate_2.3.4       parallelly_1.48.0    
#>  [91] miceFast_0.9.1        randomForest_4.7-1.2  missForest_1.6.1     
#>  [94] miceadds_3.20-10      quantreg_6.1          SparseM_1.84-2       
#>  [97] MCMCpack_1.7-1        MASS_7.3-66           coda_0.19-4.1        
#> [100] latex2exp_0.9.8       rstan_2.32.7          StanHeaders_2.39.1   
#> [103] lubridate_1.9.5       forcats_1.0.1         stringr_1.6.0        
#> [106] dplyr_1.2.1           purrr_1.2.2           readr_2.2.0          
#> [109] tibble_3.3.1          ggplot2_4.0.3         tidyverse_2.0.0      
#> [112] ggtext_0.2.0          concurve_3.0.5        showtext_0.9-8       
#> [115] showtextdb_3.0        sysfonts_0.8.9        future.apply_1.20.2  
#> [118] future_1.75.0         tidyr_1.3.2           magrittr_2.0.5       
#> [121] mice_3.19.0           rms_8.1-1             Hmisc_5.3-0          
#> 
#> loaded via a namespace (and not attached):
#>   [1] igraph_2.3.3          Formula_1.2-6         rematch2_2.1.2       
#>   [4] tidyselect_1.2.1      bridgesampling_1.2-1  rngtools_1.5.2       
#>   [7] dichromat_2.0-1       png_0.1-9             cli_3.6.6            
#>  [10] arrayhelpers_1.1-2    textshaping_1.0.5     curl_8.0.0           
#>  [13] mime_0.13             evaluate_1.0.5        V8_8.2.0             
#>  [16] stringi_1.8.9         backports_1.5.1       desc_1.4.3           
#>  [19] qqconf_1.3.2          httpuv_1.6.17         rappdirs_0.3.4       
#>  [22] details_0.4.0         prodlim_2026.03.11    doRNG_1.8.6.3        
#>  [25] DT_0.34.0             DBI_1.3.0             withr_3.0.3          
#>  [28] reformulas_0.4.4      class_7.3-24          systemfonts_1.3.2    
#>  [31] tfruns_1.5.4          rprojroot_2.1.1       lmtest_0.9-40        
#>  [34] formatR_1.14          colourpicker_1.3.0    htmlwidgets_1.6.4    
#>  [37] fs_2.1.0              labeling_0.4.3        ranger_0.18.0        
#>  [40] DEoptimR_1.2-1        zoo_1.9-0             itertools_0.1-3      
#>  [43] svUnit_1.0.8          timechange_0.4.0      caTools_1.18.4       
#>  [46] extremevalues_2.4.1   data.table_1.18.6.1   timeDate_4052.112    
#>  [49] pan_2.0               clipr_0.8.1           yaml_2.3.12          
#>  [52] survival_3.8-12       crayon_1.5.3          tensorA_0.36.2.1     
#>  [55] RColorBrewer_1.1-3    later_1.4.8           codetools_0.2-20     
#>  [58] base64enc_0.1-6       shape_1.4.6.1         estimability_2.0.0   
#>  [61] foreign_0.8-91        DiceDesign_1.10       pkgconfig_2.0.3      
#>  [64] xml2_1.6.0            viridisLite_0.4.3     xtable_1.8-8         
#>  [67] plyr_1.8.9            httr_1.4.9            rbibutils_2.4.1      
#>  [70] tools_4.6.1           globals_0.19.1        hardhat_1.4.3        
#>  [73] pkgbuild_1.4.8        htmlTable_2.5.0       shinyjs_2.1.1        
#>  [76] crosstalk_1.2.2       twosamples_2.0.1      MatrixModels_0.5-4   
#>  [79] lme4_2.0-6            digest_0.6.39         furrr_0.4.0          
#>  [82] farver_2.1.2          tzdb_0.5.0            rapportools_1.2      
#>  [85] rpart_4.1.27          glue_1.8.1            mixtime_0.3.0        
#>  [88] fracdiff_1.5-4        generics_0.1.4        ggtime_1.0.0         
#>  [91] opdisDownsampling_1.7 statmod_1.5.2         arm_1.15-3           
#>  [94] minqa_1.2.8           mcmc_0.9-8            tcltk_4.6.1          
#>  [97] texreg_1.40           gower_1.0.2           mitools_2.7          
#> [100] gtools_3.9.5          gridExtra_2.3.1       shiny_1.14.0         
#> [103] lava_1.9.3            pander_0.6.6          threejs_0.3.4        
#> [106] rstudioapi_0.19.0     cluster_2.1.8.3       QuickJSR_1.11.0      
#> [109] whisker_0.4.1         rstantools_2.7.1      hms_1.1.4            
#> [112] anytime_0.3.13        jomo_2.7-6            xts_0.14.3           
#> [115] shinythemes_1.2.0     ipred_0.9-16          laeken_0.5.3         
#> [118] e1071_1.7-17          urca_1.3-4            TH.data_1.1-5        
#> [121] matrixStats_1.5.0     emmeans_2.0.4         abind_1.4-8          
#> [124] bitops_1.1-0          Rdpack_2.6.6          promises_1.5.0       
#> [127] inline_0.3.21         sandwich_3.1-3        proxy_0.4-29         
#> [130] compiler_4.6.1        prettyunits_1.2.0     distributional_0.9.0 
#> [133] listenv_1.0.0         insight_1.5.4         gridtext_0.1.6       
#> [136] R6_2.6.1              fastmap_1.2.0         multcomp_1.4-32      
#> [139] vecvec_1.3.0          vcd_1.4-14            ggdist_3.3.3         
#> [142] nnet_7.3-21           gtable_0.3.6          shinystan_2.7.0      
#> [145] miniUI_0.1.2          RcppParallel_6.2.1    polspline_1.1.25     
#> [148] lifecycle_1.0.5       S7_0.2.2              Brobdingnag_1.2-9    
#> [151] nloptr_2.2.1          dygraphs_1.1.1.6      vctrs_0.7.3          
#> [154] zeallot_0.2.0         robustbase_0.99-7     sp_2.2-3             
#> [157] pracma_2.4.6          pillar_1.11.1         magick_2.9.1         
#> [160] otel_0.2.0            jsonlite_2.0.0        svgPanZoom_0.3.4     
#> [163] markdown_2.0

Stata

about
version
#> could not find feasible values
#> r(491);
#> 
#> 
#> 
#> StataNow/MP 19.5 for Mac (Apple Silicon)
#> Revision 12 Aug 2026
#> Copyright 1985-2025 StataCorp LLC
#> 
#> Total physical memory: 48.01 GB
#> 
#> Stata license: Single-user 2-core , expiring  6 Feb 2027
#> Serial number: 501909358563
#>   Licensed to: Zad Rafi
#>                
#> 
#> version 19.5
#> 
#> OMP: Warning #96: Cannot form a team with 2 threads, using 1 instead.
#> OMP: Hint Consider unsetting KMP_DEVICE_THREAD_LIMIT (KMP_ALL_THREADS), KMP_TEAMS_THREAD_LIMIT, and OMP_THREAD_LIMIT (if any are set).

See also: What Makes a Sensitivity Analysis? — the hub piece tying this material to the broader cluster on assumptions, robustness, and what happens when models bend.

References

Back to top

References

1. Rafi Z, Greenland S. (2020). “Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise.” BMC Medical Research Methodology. 20:244. doi: 10.1186/s12874-020-01105-9.
2. Greenland S, Rafi Z. (2020). “Technical Issues in the Interpretation of S-values and Their Relation to Other Information Measures.” arXiv:200812991 [statME]. https://arxiv.org/abs/2008.12991.
3. Greenland S. (2019). “Valid P-values behave exactly as they should: Some misleading criticisms of P-values and their resolution with S-values.” The American Statistician. 73:106–114. doi: 10.1080/00031305.2018.1529625.
4. Bayarri MJ, Berger JO. (2000). “P Values for Composite Null Models.” Journal of the American Statistical Association. 95:1127–1142. doi: 10/dpvq8c.
5. Robins JM, van der Vaart A, Ventura V. (2000). “Asymptotic Distribution of P Values in Composite Null Models.” Journal of the American Statistical Association. 95:1143–1156. doi: 10/gg7krv.
6. Kuffner TA, Walker SG. (2019). “Why are P-values controversial?” The American Statistician. 73:1–3. doi: 10.1080/00031305.2016.1277161.
7. Cousins RD. (2017). “The Jeffreys paradox and discovery criteria in high energy physics.” Synthese. 194:395–432. doi: 10.1007/s11229-014-0525-z.
8. Cox DR, Hinkley DV. (1974). “Chapter 7, Interval estimation.” In: Theoretical Statistics. Chapman and Hall/CRC. p. 207–249. doi: 10.1201/b14832.
9. Bowley AL. (1934). “Discussion on Dr. Neyman’s Paper. P. 607 in: Neyman J. On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection (with discussion).” Journal of the Royal Statistical Society. 4:558–625. doi: 10.2307/2342192.
10. Cox DR. (2006). “Principles of Statistical Inference.” Cambridge University Press.
11. Amrhein V, Trafimow D, Greenland S. (2019). “Inferential statistics as descriptive statistics: There is no replication crisis if we don’t expect replication.” The American Statistician. 73:262–270. doi: 10.1080/00031305.2018.1543137.
12. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. (2016). “Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations.” European Journal of Epidemiology. 31:337–350. doi: 10.1007/s10654-016-0149-3.
13. Greenland S. (2019). “Are confidence intervals better termed ‘uncertainty intervals’? No: Call them compatibility intervals.” BMJ. 366. doi: 10.1136/bmj.l5381.
14. Greenland S, Rafi Z. (2020). “To Aid Scientific Inference, Emphasize Unconditional Descriptions of Statistics.” arXiv:190908583 [statME]. https://arxiv.org/abs/1909.08583.
15. Brown HK, Ray JG, Wilton AS, Lunsky Y, Gomes T, Vigod SN. (2017). “Association between serotonergic antidepressant use during pregnancy and autism spectrum disorder in children.” Journal of the American Medical Association. 317:1544–1552. doi: 10.1001/jama.2017.3415.
16. Brown HK, Hussain-Shamsy N, Lunsky Y, Dennis C-LE, Vigod SN. (2017). “The association between antenatal exposure to selective serotonin reuptake inhibitors and autism: A systematic review and meta-analysis.” The Journal of Clinical Psychiatry. 78:e48–e58. doi: 10.4088/JCP.15r10194.
17. Jewell NP. (2003). “Statistics for Epidemiology.” CRC Press.
18. Royall R. (1997). “Statistical Evidence: A Likelihood Paradigm.” CRC Press.
19. Davies PL. (2008). “Approximating data [with discussion and a rejoinder].” Journal of the Korean Statistical Society. 37:191–211. doi: 10/bn8cgk.
]]> https://lesslikely.com/statistics/RG2020BMC.html Sat, 12 Dec 2020 05:00:00 GMT What Makes a Sensitivity Analysis? Zad Rafi Sir Panda https://lesslikely.com/statistics/sensitivity.html

Cargo-Cult Uncertainty

Sensitivity analyses are an important part of statistical science and many other disciplines when conducted in a principle and systematic manner. However, in the published sensitivity analyses literature, there are many inconsistencies, misconceptions, and highly misleading findings from these analyses. A giant obstacle that prevents these issues from recurring is that they (sensitivity analysis techniques) are quite difficult to learn and often advanced statistical methods that even many statisticians have difficulty with.

Yet, the appearance of difficulty does not dissuade certain researchers away from adopting them for their own work, (so that they may give their colleagues and their stakeholders that they report to, the impression of rigor and methodological expertise), and when this is done mindlessly, researchers will often skip over learning the details and theory, and like many statistical procedures, they will rely on default settings built in the statistical software.

While there are many suites/commands/functions/libraries available to conduct such analyses, much of these procedures masquerade as meticulous sensitivity analyses and to the users, are often a formality to appease stakeholders and give researchers a false sense of confidence about what they are doing. And yet, their users have little to no idea what they’re actually doing. Thus, like many statistical procedures new and old, they too will inevitably be abused as they become more popular and as they are required in research reports.

As Stark & Saltelli1, along with many others such as Gigerenzer2 and Greenland3 have written in the past. Below is an excerpt where Stark describes that much of statistics is simply people masquerading as data analysts and being rewarded as experts despite not having a single clue what they’re doing.

In our experience, many applications of statistics are cargo-cult statistics: practitioners go through the motions of fitting models, computing p-values or confidence intervals, or simulating posterior distributions. They invoke statistical terms and procedures as incantations, with scant understanding of the assumptions or relevance of the calculations, or even the meaning of the terminology. This demotes statistics from a way of thinking about evidence and avoiding self-deception to a formal “blessing” of claims. The effectiveness of cargo-cult statistics is predictably uneven. But it is effective at getting weak work published - and is even required by some journals…

Here, Stark takes on the issue of widely accessible statistical software for academics, students, and analysts.

Statistical software does not help you know what to compute, nor how to interpret the result. It does not offer to explain the assumptions behind methods, nor does it flag delicate or dubious assumptions. It does not warn you about multiplicity or p-hacking. It does not check whether you picked the hypothesis or analysis after looking at the data, nor track the number of analyses you tried before arriving at the one you sought to publish - another form of multiplicity. The more “powerful” and “user-friendly” the software is, the more it invites cargo-cult statistics.1

Greenland3 focuses on how traditional statistical education is inadequate to deal with the messy and chaotic nature of the world and the missing data that is produced by it. Greenland writes that traditional statistical practice opens the room for several cognitive biases, making many reports and conclusions highly misleading. He provides some hope however, by pointing to the utility of sensitivity analyses for modeling bias and uncertainty.

I argue that current training in statistics and analytical methods is inadequate for addressing major sources of inference distortion, and that it should be expanded to cover the biased perceptual and thinking processes (cognitive biases) that plague research reports. As commonly misused, null-hypothesis significance testing (NHST) combines several cognitive problems to create highly distorted interpretations of study results. Interval estimation has proven highly vulnerable to the same problems. Sensitivity and bias analyses address model uncertainties by varying and relaxing assumptions, but (like Bayesian analyses) they are difficult to perform with proper accounting for prior information and are easily manipulated because they depend on specification of many models and parameters.

However, he then follows up to discuss how difficult sensitivity and bias analyses can be due to the plethora of decisions that the analyst must make when specifying parameters in the analysis. These judgements will differ from researcher to researcher, even in the same group, and such analyses have not been studied as widely as conventional statistical practice.

Even with realistic choices, the sensitivity of sensitivity and bias analyses must be evaluated (51). The plausibility of an estimated bias function is determined by intuitions, prejudices, and understanding of the applied context; those can vary dramatically across researchers, in turn leading to very different specifications and inferences even if they are anchored to the same conventional analysis. Adding to this problem, sensitivity and bias analyses are more difficult to perform correctly and more easily massaged toward preferred conclusions, because they require specification of many more equations and their parameters.

And unlike NHST, abuse of sensitivity and bias analysis is as yet barely studied because the pool of such analyses remains small and highly selective. It thus seems implausible that these analyses will increase replicability of inferences, although they can reveal how assumptions affect those inferences. (Here “replicability” is used according to recommendations of the American Statistical Association (52) to denote independent checks of reported results with new data; “reproducibility” then denotes checks of reported results using the original data and computer code.)

Indeed, this is one drawback of (relatively) newly adopted methods/procedures, they have not been studied long enough as say somethng such as a P-value, which has been studied for centuries. Even something as useful as exploring assumptions and uncertainty such as a principled sensitivity analysis is not free from this.

it seems Greenland’s predictions were accurate; in a large systematic review of published sensitivity analyses led by the mathematician Andrea Saltelli and his group, they found that highly cited papers in top impact factor journals rarely contained sensitivity analyses, and that many of these were poorly done and one-dimensional, in the sense of only varying one parameter at a time, which is hardly informative as a multi-dimensional analysis in which multiple parameters are varied at the same time.

Here, we will explore one example of how mindless sensitivity analyses are conducted within the context of clinical trials, although we will not explore multi-dimensional sensitivity analyses here. I make this distinction because the environment to conduct sensitivity/bias analyses differs substantially in trials where regulators are involved versus the environment in which many observational studies are analyzed and presented.

In medicine, sensitivity analyses (SA) are often conducted in trials and observational studies, but again, little thought is given to what they should entail.1 None of this is surprising given that they are not usually taught in traditional settings, although they have been discussed extensively within the statistical and epidemiological literature4–10 Yet, the other barrier that remains is that they are incredibly technical and will require the assistance of an analyst familiar with both the theory and the applications. It also does not help that the topic is vast and there are multiple new statistical methods being published every day.

Indeed, the Handbook of Missing Data Methodology,11 one of the most authoritative and comprehensive books to date on statistical methods to handle missing data in clinical trials, has nearly six chapters devoted to the topic of sensitivity analyses and different approaches to use. Unfortunately, at the rate that this field is advancing, this book may already be outdated, despite only coming out six years ago.

This article will focus on what a sensitivity analysis is often assumed to be based on practices in the literature, some differing points of view regarding sensitivity analyses (with an example of a highly promoted measure for conducting sensitivity analyses that has been very controversial, and a practical example of conducting principled(by principled, I mean non ad-hoc techniques, as James Carpenter would say) sensitivity analyses in the context of missing data in a randomized clinical trial.

Everything is Sensitive

Before we move forward, we must consider what a sensitivity analysis actually entails. The first word of the phrase is an obvious tell, it suggests an analysis that examines how sensitive or robust a result is. But a natural question is, sensitive specifically to what? What sort of perturbations and how relevant are they? Can I vary anything in the analysis to test the sensitivity of the result or assumption? For example, if I obtained an odds ratio of 1.7 from the primary analysis in my experiment and I conducted an extra analysis to see whether the odds ratio would change by a resampling method, is that a sensitivity analysis?

What if I did a completely different type of analysis, for example, suppose the primary analysis was a classical statistical test with an adjustment for multiple comparisons and I decided to run a hierarchical Bayesian regression with a spiked prior on the null, would that be a sensitivity analysis, given they are different frameworks and procedures (although being used for similar goals)? The possibilities of what can be varied are endless when a loose definition of sensitivity analysis is assumed. This is unique to every discipline and their culture. In epidemiology, sensitivity analyses and bias analyses which attempt to extensively explore assumptions and the robustness of the result are used synonymously, yet in the clinical trial world, this is not the case.

Frequent Misconceptions

Take for example in clinical trials and observational studies, in particular, it is common to see primary analyses often being intent-to-treat (ITT) analyses and sensitivity analyses being per-protocol analyses (PP).

I will not define these terms here and would encourage readers to consult other sources.

The following is a similar example from a high-profile trial published in the The Lancet in 2002,12

The Multicentre Aneurysm Screening Study group randomised 67,800 men to receive an invitation to an abdominal ultrasound scan or not [6]. Of those invited to receive an abdominal scan, 20% did not accept. The primary analysis was by intention to treat, thus estimating the effect of being randomised to abdominal ultrasound. Another analysis investigated the complier average causal effect, which considers what the (average) effect of treatment was in patients who would have adhered to protocol however they were randomised [7].

To many, this may seem perfectly fine, even great. The authors used the question they were primarily interested in as the main analysis, and then conducted an additional analysis to see if these results are consistent. Unfortunately, this is highly problematic as Morris et al. (2014)13 describes

These questions are different, and observing different results should not shake our confidence in either. The CACE analysis was a secondary analysis, not a sensitivity analysis.

It is common for authors to compare the results of intention-to-treat with per-protocol analysis; see for example [8, 9]. While it is hard to pin down the precise question of per-protocol analysis [10], this is clearly different to the question intention-to-treat addresses. Per-protocol analysis should not therefore be considered as a sensitivity analysis for intention-to-treat but as a secondary analysis, if at all.

Randomness & Uncertainty

Portable Sensitivity Analyses

These sorts of misunderstandings are so common and so prevalent throughout the literature, so it should come as no surprise that sensitivity analyses are rarely done, or they are done incorrectly. Although not in clinical trials, one particular controversial example has been the promotion and use of the -value (the ‘E’ apparently stands for ‘evidence’, but I am not sure.) within epidemiology to assess the amount of confounding necessary within an observational study result to practically reduce the effect estimate to something that is practically null.

The method and value has been adopted with open arms by many epidemiologists and health researchers given that it has simplified a task that is often arduous and requires extensive and careful thought in comparison to the traditional classical analysis. Yet, others have also been highly critical of it for a number of reasons, and although they see the value in promoting sensitivity analyses to more researchers around the world, many are concerned that this measure will also eventually go down the road of P-values, but for observational research.

Regulators & Sensitivity Analyses

As mentioned above, there are many possible ways to conduct a sensitivity analysis, especially if the phrase is used in a loose/vague way. Whether or not these are valid and principled approaches to conducting sensitivity analyses is another question. Luckily, many of us do not have to ponder day and night about the semantics about this because both The Handbook of Missing Data Methodology and the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) have given this topic much thought, and for the latter it is reflected by the fact that they recently created an entire new addition to their E9 guidance document (Statistical Principles for Clinical Trials) which has served as a reference for clinical trial statisticians for decades. In the addendum, titled Addendum on Estimands and Sensitivity Analysis In Clinical Trials, which has now been legally adopted by regulatory agencies around the world including the FDA and EMA, they elaborate on the concept of estimands and the role sensitivity analyses play.

To be clear, estimands are not new and have been discussed in the statistical literature since Tukey14 but the ICH working group’s addendum formalized its adoption in the clinical trial world and the importance of sensitivity analyses.15–17

Estimands & Sensitivity

A.5.2.1. Role of Sensitivity Analysis

Inferences based on a particular estimand should be robust to limitations in the data and deviations from the assumptions used in the statisticral model for the main estimator. This robustness is evaluated through a sensitivity analysis. Sensitivity analysis should be planned for the main estimators of all estimands that will be important for regulatory decision making and labelling in the product information. This can be a topic for discussion and agreement between sponsor and regulator.

The statistical assumptions that underpin the main estimator should be documented. One or more analyses, focused on the same estimand, should then be pre-specified to investigate these assumptions with the objective of verifying whether or not the estimate derived from the main estimator is robust to departures from its assumptions. This might be characterised as the extent of departures from assumptions that change the interpretation of the results in terms of their statistical or clinical significance (e.g. tipping point analysis).

Distinct from sensitivity analysis, where investigations are conducted with the intent of exploring robustness of departures from assumptions, other analyses that are conducted in order to more fully investigate and understand the trial data can be termed “supplementary analysis” (see Glossary; A.5.3.). Where the primary estimand(s) of interest is agreed between sponsor and regulator, the main estimator is pre-specified unambiguously, and the sensitivity analysis verifies that the estimate derived is reliable for interpretation, supplementary analyses should generally be given lower priority in assessment.

A.5.2.2. Choice of Sensitivity Analysis

When planning and conducting a sensitivity analysis, altering multiple aspects of the main analysis simultaneously can make it challenging to identify which assumptions, if any, are responsible for any potential differences seen. It is therefore desirable to adopt a structured approach, specifying the changes in assumptions that underlie the alternative analyses, rather than simply comparing the results of different analyses based on different sets of assumptions. The need for analyses varying multiple assumptions simultaneously should then be considered on a case by case basis. A distinction between testable and untestable assumptions may be useful when assessing the interpretation and relevance of different analyses.

ICH E9 (R1) Guideline

The need for sensitivity analysis in respect of missing data is established and retains its importance in this framework. Missing data should be defined and considered in respect of a particular estimand (see A.4.). The distinction between data that are missing in respect of a specific estimand and data that are not directly relevant to a specific estimand gives rise to separate sets of assumptions to be examined in sensitivity analysis.

they explicitly define a sensitivity analysis as being an analysis which:

  • realistically varies the assumptions from the primary analysis
  • still targets the same estimand
  • examines the robustness of the results to
  • assumption violations or departures
  • can possibly change the results/conclusions drawn

They contrast this with more extensive analyses that investigate these violations/departures of assumptions, and characterize those analyse as being supplementary rather than sensitivity. The latter is typically what is seen in rigorous epidemiological studies that employ quantitative bias analyses (a more specialized form of sensitivity analysis).

The ICH E9 addendum contains similar views that are echoed by the National Research Council’s advice on clinical trials (National Research Council 2010)18 regarding estimands and sensitivity

Recommendation 15: Sensitivity analysis should be part of the primary reporting of findings from clinical trials. Examining sensitivity to the assumptions about the missing data mechanism should be a mandatory component of reporting.

Now, back to the NEJM paper. The reason why an ITT analysis and a PP analysis cannot serve as primary and sensitivity analyses, respectively, is because they are targeting entirely different estimands.13 Thus, because they are answering completely different questions, they’re just two different analyses. And the additional PP analysis, although important to certain stakeholders, can be classified as a supplementary analysis or a non-sensitivity analysis.

Indeed, the following flowchart from Morris et al. (2014)13 is particularly useful as a guide to differentiate sensitivity analyses from non-sensitivity analyses.

Adapted sensitivity-analysis flowchart.

Adapted sensitivity-analysis flowchart.

Adapted flowchart from Morris et al. (2014).

Utility of Subject-Matter Experts

So what does a sensitivity analysis actually look like within the context of a clinical trial or experiment? Below, I walk through the analysis of a trial to give some examples.

Sensitivity in Clinical Trials

I’ll use a sample clinical trial dataset, which is described here:19

“This dataset is typical of diastolic blood pressure data measured in small clinical trials in hypertension from the mid-to-late 1960s and for approximately a decade thereafter.

During this time, hypertension was more severe, the number of effective treatments was relatively small, and the definition (DBP > 95 mmHg) of essential hypertension was not as stringent as it is now (DBP > 80 mmHg) - as seen in the 1967 report from the Veterans Administration Cooperative Study Group on Antihypertensive Agents VA Study Group (1967).

In Table 3.1, diastolic blood pressure (DBP) was measured (mmHg) in the supine position at baseline (i.e., DBP) before randomization… Patients’ age and sex were recorded at baseline and represent potential covariates. The primary objective in this chapter in the analysis of this dataset is to test whether treatment A (new drug) may be effective in lowering DBP as compared to B (placebo) and to describe changes in DBP across the times at which it was measured.”

So to recap, the primary investigators of this trial were interested to see whether treatment A was more effective in lowering DBP when compared to placebo.

Although we will not be using some of the scripts and project setup that I discussed in my statistical workflow post for the sake of efficiency and ease, I would urge and recommend others do so that it remains easy to catch errors. In particular, I would especially recommend using the following packages:

For the analyses in this post, see the R packages in the session info section at the end of this post, these will need to be loaded in order for the scripts below to work.

req_packs <- c("rms", "mice", "magrittr", "tidyr", "future",
    "showtext", "concurve", "ggtext", "tidyverse", "ggplot2",
    "parallel", "kableExtra", "cowplot", "Cairo", "svglite",
    "yardstick", "broom", "mi", "Hmisc", "here", "VIM", "wesanderson",
    "bayesplot", "lattice", "doParallel", "blogdown", "Amelia",
    "bootImpute", "brms", "rmarkdown", "tinytex", "texPreview",
    "ggcorrplot", "qqplotr", "car", "Statamarkdown", "rstan",
    "missForest", "htmltools", "tidybayes", "performance", "mvtnorm",
    "ImputeRobust", "gamlss", "ProfileLikelihood", "reshape2",
    "boot", "knitr", "MASS", "miceMNAR")

# Load all packages at once

lapply(req_packs, library, character.only = TRUE)

I will now simulate some of the data from this clinical trial.

# Simulate Trial Data

set.seed(seed = 1031,
         kind = "L'Ecuyer-CMRG",
         normal.kind = "Inversion",
         sample.kind = "Rejection")

BaselineDBP <- rnorm(500, 117, 2)
Group <- rbinom(500, 1, 0.5)
Age <- rnorm(500, 50, 7)
R1 <- rnorm(500, 0, 1)
R2 <- rnorm(500, 1, 0.5)

errors <- mvtnorm::rmvnorm(500,
  mean = c(0, 0),
  sigma = matrix(c(1, 0.5, 0.5, 1),
    nrow = 2, byrow = TRUE
  )
)

PostDBP <- BaselineDBP + (Age * 0.33) +
  (Group * 9.7) + errors[, 1]

I have made some minor adjustments to this dataset and also generated a new variable (Z) based on the existing ones in the dataset that is strongly correlated with the outcome of interest (PostDBP).

# Explore Data

str(BP_full)
#> 'data.frame':    500 obs. of  7 variables:
#>  $ PostDBP    : num  142 140 129 148 135 ...
#>  $ BaselineDBP: num  119 116 115 119 119 ...
#>  $ Group      : Factor w/ 2 levels "Drug X","Drug Y": 1 1 2 1 2 2 1 2 1 1 ...
#>  $ Age        : num  41.8 44.4 41.1 56.6 50.2 ...
#>  $ Z          : num  0.0141 -0.0629 -0.1101 0.0313 0.0495 ...
#>  $ R1         : num  0.523 1.021 -0.839 0.413 -0.175 ...
#>  $ R2         : num  0.451 0.35 0.897 1.227 1.166 ...
Patient Characteristics
Variable N Overall
N = 5001		 Drug X
N = 2581		 Drug Y
N = 2421
PostDBP 500 138.5 (133.4, 142.8) 142.5 (140.5, 144.6) 133.4 (131.3, 135.3)
BaselineDBP 500 116.88 (115.56, 118.22) 116.90 (115.53, 118.21) 116.81 (115.59, 118.23)
Age 500 49 (45, 55) 49 (45, 55) 50 (46, 55)
Z 500 0.00 (-0.04, 0.05) 0.00 (-0.04, 0.04) 0.00 (-0.05, 0.05)
R1 500 0.00 (-0.73, 0.56) 0.05 (-0.73, 0.60) -0.10 (-0.74, 0.52)
R2 500 0.96 (0.64, 1.28) 0.96 (0.59, 1.30) 0.97 (0.67, 1.28)
1 Median (Q1, Q3) 
summary(BP_full)
#>     PostDBP     BaselineDBP     Group          Age             Z                   R1                  R2        
#>  Min.   :124   Min.   :110   Drug X:258   Min.   :27.7   Min.   :-1.85e-01   Min.   :-3.205036   Min.   :-0.646  
#>  1st Qu.:133   1st Qu.:116   Drug Y:242   1st Qu.:45.3   1st Qu.:-4.44e-02   1st Qu.:-0.732386   1st Qu.: 0.645  
#>  Median :138   Median :117                Median :49.4   Median : 9.45e-05   Median :-0.000058   Median : 0.960  
#>  Mean   :138   Mean   :117                Mean   :49.7   Mean   : 0.00e+00   Mean   :-0.042800   Mean   : 0.962  
#>  3rd Qu.:143   3rd Qu.:118                3rd Qu.:54.7   3rd Qu.: 4.56e-02   3rd Qu.: 0.552206   3rd Qu.: 1.283  
#>  Max.   :151   Max.   :123                Max.   :68.4   Max.   : 2.02e-01   Max.   : 3.377526   Max.   : 2.480
PostDBP BaselineDBP Group Age Z R1 R2
142 119 Drug X 41.8 0.01 0.52 0.45
140 116 Drug X 44.4 -0.06 1.02 0.35
129 115 Drug Y 41.1 -0.11 -0.84 0.90
148 119 Drug X 56.6 0.03 0.41 1.23
135 119 Drug Y 50.2 0.05 -0.17 1.17
135 118 Drug Y 55.3 0.01 -0.89 0.77
143 117 Drug X 54.3 0.00 1.07 0.34
133 116 Drug Y 55.8 -0.05 -1.91 0.96
144 118 Drug X 46.9 0.08 1.03 1.75
150 119 Drug X 62.0 0.01 -0.94 2.48

Now, we can quickly explore the other characteristics of the dataset, and first we look at the rank correlations between the predictors and the response variable.

# Data Attributes

study_formula <- as.formula(PostDBP ~ .)

plot(spearman2(study_formula, BP_full),
  cex = 1,
  pch = 18, col = alpha("darkred", 0.75)
)

abline(v = 0, col = alpha(zred, 0.75), lty = 3)

Along with a correlation matrix between the variables of interest.

We will not be looking at the estimates from a fitted model yet. We are mainly looking at a few characteristics of this dataset before we move on to the next step.

The Threat of Missing Data

Now that we have familiarized ourselves with this dataset, we suddenly “notice” that there are missing data (which I actually generated below using a custom function) and suppose we didn’t actually know how the missing data were generated, we might suspect or start with the assumption that the data are missing at random (MAR).

We will explore a number of different methods used to handle missing data, including multiple imputation, selection models, and a number of ad-hoc methods commonly used by researchers handle missing data. However, before we move on to discussing some of these methods, we will quickly review concepts and terms regarding missing data mechanisms.

In the figure below, the three missing data mechanisms (MCAR, MAR, and MNAR) and ignorability (whether we need to model the mechanism of missing data) in relation to observed data (Y obs), missing data (Y mis), the missingness matrix (R), and their relationships (q; parameters that explain missingness, i.e., mechanism). The solid arrows, dotted arrows, and arrows with crosses represent “connection,” “possible connection,” and “no connection,” respectively.

The lines connecting ignorability and missingness group the three mechanisms into the two ignorability categories. Also no pure forms of MCAR, MAR, and MNAR exist, and all missingness can be considered as a form of MAR missingness; this is represented by the shaded continuum bar on the left. - Modified from Nakagawa and Freckleton (2011)

Missing-data mechanism flowchart.

Missing-data mechanism flowchart.

Flowchart from Ecological Statistics: Contemporary Theory and Applications.

Of course, here, we do have an idea of what the missing data mechanism is, but we shall assume the role of a data analyst who has just been given a dataset with missing values in the outcome from their clinical colleague.

# Generate Missing Data

set.seed(1031,
  kind = "L'Ecuyer-CMRG",
  normal.kind = "Inversion",
  sample.kind = "Rejection"
)
Ry <- ifelse(lesslikely() > 0, 1, 0)
PostDBP[Ry == 0] <- NA
BP_miss <- data.frame(
  PostDBP, BaselineDBP,
  Group, Age, Z, R1, R2
)
str(BP_miss)
#> 'data.frame':    500 obs. of  7 variables:
#>  $ PostDBP    : num  142 140 129 148 135 ...
#>  $ BaselineDBP: num  119 116 115 119 119 ...
#>  $ Group      : Factor w/ 2 levels "Drug X","Drug Y": 1 1 2 1 2 2 1 2 1 1 ...
#>  $ Age        : num  41.8 44.4 41.1 56.6 50.2 ...
#>  $ Z          : num  0.0141 -0.0629 -0.1101 0.0313 0.0495 ...
#>  $ R1         : num  0.523 1.021 -0.839 0.413 -0.175 ...
#>  $ R2         : num  0.451 0.35 0.897 1.227 1.166 ...

sum(is.na(BP_miss))
#> [1] 245

# Export data for Stata chunks: Group must be numeric and NAs must be blank,
# otherwise `import delimited` reads those columns as strings. Code Drug Y as 1
# so Stata's `i.group` matches R's factor contrast (base = Drug X).
dir.create("static/datasets", recursive = TRUE, showWarnings = FALSE)
BP_stata <- BP_miss
BP_stata$Group <- as.integer(BP_stata$Group == "Drug Y")
write.csv(BP_stata, "static/datasets/temp.csv", row.names = FALSE, na = "")

We will now attempt to visualize our dataset and examine what proportion of these data are missing.

missing_col <- c(zblue, zred)
aggr(BP_miss,
  col = alpha(missing_col, 0.60),
  plot = TRUE, prop = F, numbers = T
)

So now, we must examine this missing dataset and its characteristics and look for any systematic differences. We also examine the proportion of missingness and the influx and outflux patterns. This is important to assess how well connectected the missing data are to the observed data, explained by van Buuren here

The influx of a variable quantifies how well its missing data connect to the observed data on other variables. The outflux of a variable quantifies how well its observed data connect to the missing data on other variables. In general, higher influx and outflux values are preferred…

Influx and outflux are summaries of the missing data pattern intended to aid in the construction of imputation models. Keeping everything else constant, variables with high influx and outflux are preferred. Realize that outflux indicates the potential (and not actual) contribution to impute other variables. A variable with high may turn out to be useless for imputation if it is unrelated to the incomplete variables. On the other hand, the usefulness of a highly predictive variable is severely limited by a low .

More refined measures of usefulness are conceivable, e.g., multiplying by the average proportion of explained variance. Also, we could specialize to one or a few key variables to impute. Alternatively, analogous measures for could be useful. The further development of diagnostic summaries for the missing data pattern is a promising area for further investigation.

# Examine Flux Patterns

round(fico(BP_miss), 1)[2:6]
#> BaselineDBP       Group         Age           Z          R1 
#>         0.5         0.5         0.5         0.5         0.5
POBS Influx Outflux AINB AOUT FICO
PostDBP 0.51 0.45 0 1 0.00 0.00
BaselineDBP 1.00 0.00 1 0 0.08 0.49
Group 1.00 0.00 1 0 0.08 0.49
Age 1.00 0.00 1 0 0.08 0.49
Z 1.00 0.00 1 0 0.08 0.49
R1 1.00 0.00 1 0 0.08 0.49
R2 1.00 0.00 1 0 0.08 0.49

Next, We start off doing some preliminary analyses to eyeball differences between the observed data and the missing data.

#> [1] 245
pobs influx outflux ainb aout fico
PostDBP 0.51 0.45 0 1 0.00 0.00
BaselineDBP 1.00 0.00 1 0 0.08 0.49
Group 1.00 0.00 1 0 0.08 0.49
Age 1.00 0.00 1 0 0.08 0.49
Z 1.00 0.00 1 0 0.08 0.49
R1 1.00 0.00 1 0 0.08 0.49
R2 1.00 0.00 1 0 0.08 0.49

Having explored these data, we fit a preliminary model quickly to these missing data using the rlm() function from the MASS package and glance at what our estimates look like.

# Exploratory Model Estimates

eda_1 <- rlm(
  PostDBP ~ factor(Group) + BaselineDBP +
    Age + Z + R1 + R2,
  data = BP_miss,
  method = "MM", psi = psi.huber
)
Value Std. Error t value
(Intercept) 11.68 5.77 2.02
factor(Group)Drug Y -9.58 0.13 -71.57
BaselineDBP 0.98 0.05 19.86
Age 0.34 0.01 34.33
Z -0.93 1.48 -0.62
R1 -0.03 0.07 -0.43
R2 -0.13 0.13 -0.94
AIC AICc BIC RMSE Sigma
749 750 778 1.02 1.03

Because the MASS::rlm() function will automatically delete missing observations and do a complete-case analysis, we can gauge what our estimates would look like if we assume that the data are missing completely at random (MCAR), which is almost always unrealistic.

It is almost always more realistic to assume that the data are missing at random (MAR), . This can often be a reasonable assumption to start off with for the primary analysis.20

Multiple Imputation Workflow

For our main analysis, we will build an imputation model to essentially ‘predict’ the missing values in our dataset. This is because multiple imputation is a well-studied set of algorithms that are efficient and robust to deviances from normality.

Modern missing data methodologies include maximum-likelihood estimation (MLE) methods such as expectation–maximisation (EM) and multiple imputation (MI), both recommended for data which is MAR [3]. MI has been shown to be robust under departures from normality, in cases of low sample size, and when the proportion of missing data is high [2]. With complete outcome variables, MI is typically less computationally expensive than MLE, and MLE tends to be problem-specific with a different model being required for each analysis [8].

  • Plumpton CO, Morris T, Hughes DA, White IR. Multiple imputation of multiple multi-item scales when a full imputation model is infeasible. BMC Res Notes. 2016 Jan 26;9:45. doi: 10.1186/s13104-016-1853-5.

How exactly does multiple imputation work?

The standard multiple imputation procedure (Rubin, 1987; Schafer, 1997) replaces missing covariate data with values drawn from a set of specified imputation models based on the observed relationships between the covariates and outcome, typically under a missing at random (MAR) assumption (Little and Rubin, 1987). A key feature of the procedure is that it outputs a number, say M, of imputed datasets. A prediction model is then fitted to each imputed dataset to produce imputation-specific regression coefficients, which can be averaged using “Rubin’s rules” (Rubin, 1987) to provide pooled regression coefficients.

To extract predictions from such a prediction model and multiply imputed data, either the sets of imputation-specific regression coefficients or the pooled regression coefficients could be used, and we must decide how to handle individuals with multiply imputed covariates. Thus there are numerous combinations for constructing a set of predictions, including imputation-specific predictions (M predictions for each individual) and pooled predictions (an averaged prediction over imputed datasets for each individual). The differences between them and their advantages and drawbacks are unclear.

Wood, A.M., Royston, P. and White, I.R. (2015), The estimation and use of predictions for the assessment of model performance using large samples with multiply imputed data. Biom. J., 57: 614-632. https://doi.org/10.1002/bimj.201400004

We include all possible information in our imputation model, for example, that might mean entail specifying all the variables in our dataset as covariates, and impute the missing data using a set of reliable and well-studied imputation algorithms, and then analyze each dataset and pool the results. Our basic workflow will look like this:

Missingness workflow flowchart.

Missingness workflow flowchart.

Flowchart from Ecological Statistics: Contemporary Theory and Applications.

Concepts in Multiple Imputation

It can also be written using the following notation, which will be explained below.

Nagakawa writes:

There are three main steps in MI: imputation, analysis, and pooling (figure 4.5C). In the imputation step, you create copies of completed data set by using data imputation methods such as the EM/EMB algorithms or the MCMC procedure. In the analysis step, you run separate statistical analyses on each of data sets. Finally, in the pooling step, you aggregate sets of results to produce unbiased parameter and uncertainty estimates.

This aggregation process is done by the following equations (which are automatically calculated in R): where is the mean of (e.g., regression coefficients), which is a parameter estimated from the ith data set (), is the within-imputation variance calculated from the standard error associated with , is the between-imputation variance estimates, and is the total variance ( is the overall standard error for ).

This set of equations for combining estimates from sets of results is often referred to as Rubin’s rules, as it was developed by Rubin (1987). Statistical significance and confidence intervals (CIs) of pooled parameters are obtained by, where is the number of degrees of freedom used for t-tests or to obtain values and CI calculations, and is the significance level (e.g., 95% CI, = 0.05).:

Nagakawa then discusses the similarities between imputation algorithms and MCMC methods,

MCMC procedures (and also Gibbs sampling) are often called Bayesian methods (chapter 1) because their goal is to create the posterior distributions of parameters, but methods using MCMC have much wider applications than Bayesian statistics). The MCMC procedure, is akin to the EM algorithm (Schafer 1997) in that it uses a two-step iterative algorithm to find and .

The imputation step (I-step) uses stochastic regression with observed data. Next, the posterior step (P-step) uses this filled-in data set to construct the posterior distributions of and . Then, it uses a Monte Carlo method to sample a new set of and from these distributions. These new parameter estimates are used for the subsequent I-step. Iterations of the two steps create the Markov chain, which even actually converges into fully fledged posterior distributions of and (figure 4.6B). These distributions are, in turn, used for multiple imputation.

Now that we’ve quickly reviewed some of the concepts of multipe imputation and missing data, we can now begin to build our imputation model. This is generally done by building a model with all relevant covariates, however, this should be compared a reduced model with just the treatment group. This is because we want both the adjusted and unadjusted estimates. We will then compare these models using the likelihood-ratio test21 to see whether there are any substantial differences between the fully-adjusted and reduced model.

The imputation model for a particular parameter will generally resemble the estimation model being used.

When developing your imputation model, it is important to assess if your imputation model is “congenial” or consistent with your analytic model. Consistency means that your imputation model includes (at the very least) the same variables that are in your analytic or estimation model. This includes any transformations to variables that will be needed to assess your hypothesis of interest. This can include log transformations, interaction terms, or recodes of a continuous variable into a categorical form, if that is how it will be used in later analysis. The reason for this relates back to the earlier comments about the purpose of multiple imputation.

Since we are trying to reproduce the proper variance/covariance matrix for estimation, all relationships between our analytic variables should be represented and estimated simultaneously. Otherwise, you are imputing values assuming they have a correlation of zero with the variables you did not include in your imputation model. This would result in underestimating the association between parameters of interest in your analysis and a loss of power to detect properties of your data that may be of interest such as non-linearities and statistical interactions. - From the UCLA Statistical Consulting Website

Monte Carlo Error Analysis

However, before we begin to build the model, we must also have an idea of how many imputations we need to compute. To many, this may seem like a trivial and even nonsensical task when any individual can impute a large number of datasets. Indeed, it may seem that more imputations would give a more reliable and precise estimate, however, nothing in life is free, and running a imputation algorithm that is computationally demanding (and which can potentially take hours or days) is not feasible for everyone, hence the reason for “sample-size” calculations for imputation algorithms.

Multiple imputation is a stochastic procedure. Each time we reimpute our data, we get different sets of imputations because of the randomness of the imputation step, and therefore we get different multiple-imputation estimates. However, we want to be able to reproduce MI results. Of course, we can always set the random-number seed to ensure reproducibility by obtaining the same imputed values. However, what if we use a different seed? Would we not want our results to be similar regardless of what seed we use? This leads us to a notion we call statistical reproducibility—we want results to be similar across repeated uses of the same imputation procedure; that is, we want to minimize the simulation error associated with our results.

To assess the level of simulation error, White, Royston, and Wood (2011) propose to use a Monte Carlo error of the MI results, defined as the standard deviation of the results across repeated runs of the same imputation procedure using the same data. The authors suggest evaluating Monte Carlo error estimates not only for parameter estimates but also for other statistics, including p-values and confidence intervals, as well as MI statistics including RVI and FMI.

The issue with this is that without giving careful thought to how many imputations may be needed for a particular study, one may run the risk of imputing too few datasets and losing information, or they may waste resources imputing a nonsensical amount that may not be necessary. Indeed, the latter seems especially trivial, however, it may often yield no advantages whatsoever, and only incur costs as many authors have argued.

The argument is that “the additional resources that would be required to create and store more than a few imputations would not be well spent” (Schafer 1997, 107), and “in most situations there is simply little advantage to producing and analyzing more than a few imputed datasets” (Schafer and Olsen 1998, 549).

Therefore, we will examine certain characteristics of our dataset with missing values and use them in an analysis to determine how many datasets we should impute to efficiently achieve our goals. We will base this off a number of characteristics such as the fraction of missing information, the proportion of missing observations, and the losses that we are willing to incur.

We first start by running a “dry” or naive imputation of our data to quickly examine some of these characteristics. This imputed dataset will not be used for our primary analysis, it is simply serving to guide us in our main imputation model. This is also why we have set the number of iterations to 0.

Once again, we will include all the variables in our dataset, and use the predictive mean matching method with approximately 5 donors.

# Dry Imputation

form <- list(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2)
pred <- make.predictorMatrix(BP_miss)

init_imp <- mice::mice(
  BP_miss,
  method = "pmm",
  predictorMatrix = pred,
  m = 20,
  maxit = 0,
  seed = 1031,
  formulas = form,
  donors = 5,
  remove.collinear = TRUE,
  remove.constant = TRUE,
  printFlag = FALSE,
  visitSequence = "monotone"
)

head(init_imp$loggedEvents, 10)
#> NULL

Now that we have imputed our dataset, we may examine it for any issues before moving onto the next step of analyzing it.

Although it is only a dry run, there seems to be no issues with it. We now pool our estimates using Rubin’s rules.

init_res <- mice::pool(with(
  init_imp,
  rlm(
    PostDBP ~ factor(Group) +
      BaselineDBP + Age +
      Z + R1 + R2,
    method = "MM", psi = psi.huber
  )
))

init_res_sum <- summary(init_res)

colnames(init_res_sum) <- c(
  "Term", "Estimate", "SE",
  "Statistic", "df", "P-val"
)
ztable(init_res_sum)
Term Estimate SE Statistic df P-val
(Intercept) 41.77 17.18 2.43 NA NA
factor(Group)Drug Y -8.22 0.43 -19.26 NA NA
BaselineDBP 0.74 0.15 5.00 NA NA
Age 0.28 0.03 10.23 NA NA
Z 1.57 4.20 0.37 NA NA
R1 0.03 0.23 0.14 NA NA
R2 -0.08 0.36 -0.21 NA NA

Now that we have our vector, we can quickly examine certain characteristics such as the fraction of missing information (FIMO), the between-imputation variance and the within-imputation variance. The reason that we focus on these particular attributes is because they are crucial for several tasks in any statistical analysis plan, such as having enough power/precision, minimum long-run coverage, etc.

We use the FIMO from the dry imputation and a number of other criteria to determine how many imputations we need:

  • Some have suggested taking the FIMO and multiplying by 100 to obtain the number of imputed datasets needed

  • For our goals, we wish to ensure that the monte-carlo standard error from the imputations are less than 10% of the between-imputation standard error

  • We wish to minimize the monte-carlo error that is derived from dividing the chosen number of datasets to impute from the fraction of missing information so that the monte-carlo standard error is less than 0.01

  • We also wish to minimize the total variance so that the square-root of it is no greater than the ideal variance and the corresponding confidence interval width.

  • Furthermore, it is commonly advocated by missing data experts to impute the same or similar number of datasets as fractions of missing observations

We start by extracting the FMIs from our dry imputation and calculating all these numbers.

# Calculate Fraction of Missing Information
# `mice::pool()`/`summary.mipo()` have changed their returned columns across
# versions. Prefer the explicit `fmi` column when present; otherwise fall back
# to `lambda`, and finally compute it from Rubin's variance components.
extract_fmi <- function(model) {
  if (is(model, "mira")) {
    model <- mice::pool(model)
  }
  if (!is(model, "mipo")) {
    stop("Model must be multiply imputed.")
  }

  pooled <- model$pooled
  finite_max <- function(x) {
    x <- suppressWarnings(as.numeric(x))
    x <- x[is.finite(x)]
    if (length(x)) max(x) else NA_real_
  }

  fmi_value <- if ("fmi" %in% names(pooled)) finite_max(pooled$fmi) else NA_real_
  if (!is.na(fmi_value)) return(min(max(fmi_value, 0), 0.999))

  lambda_value <- if ("lambda" %in% names(pooled)) finite_max(pooled$lambda) else NA_real_
  if (!is.na(lambda_value)) return(min(max(lambda_value, 0), 0.999))

  if (all(c("b", "ubar") %in% names(pooled))) {
    m_count <- if (!is.null(model$m)) model$m else finite_max(pooled$m)
    riv <- (1 + 1 / m_count) * pooled$b / pooled$ubar
    lambda <- riv / (1 + riv)
    if ("df" %in% names(pooled)) {
      fmi_calc <- lambda + (2 / (pooled$df + 3))
    } else {
      fmi_calc <- lambda
    }
    fmi_value <- finite_max(fmi_calc)
    if (!is.na(fmi_value)) return(min(max(fmi_value, 0), 0.999))
  }

  stop("Could not extract or compute fraction of missing information from pooled model.")
}

fmi <- extract_fmi(init_res)

m <- fmi / 0.01 # FMI Imputations

mcse <- sqrt(max(init_res$pooled$b, na.rm = TRUE) / m) # MCSE

se <- max((init_res$pooled$ubar) / sqrt(500), na.rm = TRUE) # SE

What we find is that the fraction of missing information is nearly 50% which is very high and that in order to get a monte-carlo error of less than 0.01, we would need a minimum of at least 50 imputed datasets. We also calculate both the monte-carlo standard error and the standard error from the dry run and find that they are practically equivalent.

However, we must also explore other possibilities in terms of number of imputed datasets to reduce the monte-carlo errors so we run a quick function to do that, checking the effects of a number of imputed datasets on the width of the confidence interval, which is directly tied to the monte-carlo error.

Indeed, Von Hippel (2018) proposed a relationship between the fraction of missing information and the number of imputations needed to achieve a targeted CI length or monte-carlo error using a quadratic formula,

where and are the coefficient of variation (CV), summarizing the imputation variation in the SE estimates. We can graphically construct a function to depict this relationship and calculate how many imputations we will need based on some desired error and further display the distribution of this relationship by varying these parameters.

In the function below, adopted from von Hippel, and van Buuren,20 I vary both the coefficient of variation and the -level.

#' @title How Many Imputations?
#' @description Implements two-stage "how_many_imputations" from von Hippel (2018)
#' @param model Either a `mira` object (created by running a model on a data set which was imputed via "mice")
#' or a `mipo` object (creating by runing `pool()` on a `mira` object).
#' @param cv Desired precision of standard errors. Default to .05. (I.e., if the data were re-imputed, the
#' estimated standard errors would differ by no more than this amount.)
#' @param alpha Significance level for choice of "conservative" FMI.
#' @return The number of required imputations to obtain the `cv` level of precision.
#' @export
#' @importFrom methods is
#' @importFrom stats plogis qlogis qnorm
#' @importFrom mice pool
#' @references von Hippel, Paul T. (2018)
#' \sQuote{How Many Imputations Do You Need? A Two-stage Calculation Using a Quadratic Rule.},
#' \emph{Sociological Methods & Research} p.0049124117747303.
#'
powerimp <- function(model, cv = .05, alpha = .05) {
  if (is(model, "mira")) {
    model <- mice::pool(model)
  }
  if (!is(model, "mipo")) {
    stop("Model must be multiply imputed.")
  }
  fmi <- extract_fmi(model)
  z <- qnorm(1 - alpha / 2)
  fmiu <- plogis(qlogis(fmi) + z * sqrt(2 / model$m))
  ceiling(1 + 1 / 2 * (fmiu / cv)^2)
}

ci_width <- function() {
  cv_i <- seq(0.01, 0.20, 0.01)
  results1 <- lapply(
    (seq_along(cv_i) / 100),
    function(i) {
      powerimp(
        model = init_res,
        cv = i, alpha = 0.01
      )
    }
  )
  results2 <- lapply(
    (seq_along(cv_i) / 100),
    function(i) {
      powerimp(
        model = init_res,
        cv = i, alpha = 0.05
      )
    }
  )
  results3 <- lapply(
    (seq_along(cv_i) / 100),
    function(i) {
      powerimp(
        model = init_res,
        cv = i, alpha = 0.1
      )
    }
  )
  a_0.01 <- do.call(rbind, results1)
  df <- data.frame(cv_i, a_0.01)
  df$a_0.05 <- do.call(rbind, results2)
  df$a_0.1 <- do.call(rbind, results3)
  return(df)
}

some <- ci_width()

With this function, we can now graphically examine the effects of different parameter values on the monte-carlo errors from the imputations.

We can see above that as we aim for smaller monte-carlo errors, we need many more imputations to achieve our desired target.

Now that I have constructed and graphed three differing relationships between monte-carlo errors and number of imputations needed to achieve those errors, and varied the -levels, we can now move onto choosing a specific number of imputations to construct.

If I choose an level of 5%, which the maximum tolerable type-I error rate, and a coefficient of variation of 5%

# Estimate Number of Imputations

powerimp(model = init_res, cv = 0.10, alpha = 0.05)
#> [1] 24

powerimp(model = init_res, cv = 0.05, alpha = 0.05)
#> [1] 92

powerimp(model = init_res, cv = 0.01, alpha = 0.05)
#> [1] 2257

I need to construct approximately 124 imputations to achieve this coefficient of variation, which is what I will use for all the analyses from here on, and it is clear that it is far more reasonable than trying to impute 3000 datasets to achieve a tiny standard error.

However, I want to verify my calculations by running a similar estimation command that Stata provides. The documentation says the following regarding RVI and FMI:

Returning to the output, average RVI reports the average relative increase (averaged over all coefficients) in variance of the estimates because of the missing bmi values. A relative variance increase is an increase in the variance of the estimate because of the loss of information about the parameter due to nonresponse relative to the variance of the estimate with no information lost. The closer this number is to zero, the less effect missing data have on the variance of the estimate. Note that the reported RVI will be zero if you use mi estimate with the complete data or with missing data that have not been imputed. In our case, average RVI is small: 0.0312.

Largest FMI reports the largest of all the FMI about coefficient estimates due to nonresponse. This number can be used to get an idea of whether the specified number of imputations is sufficient for the analysis. A rule of thumb is that M ≥ 100 × FMI provides an adequate level of reproducibility of MI analysis. In our example, the largest FMI is 0.14 and the number of imputations, 20, exceeds the required number of imputations: 14

Although I cannot get the number of imputations directly from a command, I can run the mi estimate command, and look at the overall fraction of missing information as the number of imputations increases. If my previous calculations are consistent, I should look to increase the number of imputations until I achieve an FMI of 1.24 or less: 124/100

set linesize 255

clear all

set seed 1031

quietly import delimited "static/datasets/temp.csv", numericcols(2 3 5 6 7 8)

quietly mi set mlong

quietly mi register imputed baselinedbp postdbp age z group r1 r2

quietly mi impute chained (pmm, knn(5)) postdbp = baselinedbp age z r1 r2 group, burnin(5) add(120) rseed (1031) nomonotone

quietly mi estimate, saving(miest, replace): rreg postdbp i.group baselinedbp age z r1 r2

mi estimate using miest, mcerror level(95)

mi estimate, saving(miest, replace) vartable mcerror level(95) nocitable: rreg postdbp i.group baselinedbp age z r1 r2

mi estimate using miest, mcerror level(95)
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> Multiple-imputation estimates                   Imputations       =        120
#> Robust regression                               Number of obs     =        500
#>                                                 Average RVI       =     0.7202
#>                                                 Largest FMI       =     0.4949
#>                                                 Complete DF       =        493
#> DF adjustment:   Small sample                   DF:     min       =     165.96
#>                                                         avg       =     202.37
#>                                                         max       =     274.66
#> Model F test:       Equal FMI                   F(   6,  431.4)   =    1183.09
#>                                                 Prob > F          =     0.0000
#> 
#> ------------------------------------------------------------------------------
#>      postdbp | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>      1.group |  -9.523045   .1168215   -81.52   0.000    -9.753025   -9.293066
#>              |   .0057905   .0025234     1.76   0.000     .0084985    .0067603
#>              |
#>  baselinedbp |    .980327   .0453156    21.63   0.000       .89096    1.069694
#>              |   .0026919   .0011629     0.57   0.000     .0038643    .0032359
#>              |
#>          age |   .3297468   .0088947    37.07   0.000      .312222    .3472716
#>              |   .0004879   .0002219     0.92   0.000     .0006042    .0007118
#>              |
#>            z |  -1.062159   1.509542    -0.70   0.483    -4.041587    1.917269
#>              |   .0945004   .0410673     0.07   0.041     .1227202    .1282877
#>              |
#>           r1 |   -.018845   .0725934    -0.26   0.795    -.1621706    .1244805
#>              |   .0046239   .0026182     0.06   0.049     .0075822    .0063906
#>              |
#>           r2 |  -.1529839   .1325282    -1.15   0.250    -.4145243    .1085565
#>              |   .0082342   .0037686     0.07   0.027     .0120192    .0102984
#>              |
#>        _cons |    11.6797   5.307458     2.20   0.029     1.213107     22.1463
#>              |   .3147167   .1363694     0.07   0.005     .3681237    .4611054
#> ------------------------------------------------------------------------------
#> Note: Values displayed beneath estimates are Monte Carlo error estimates.
#> 
#> 
#> Multiple-imputation estimates                   Imputations       =        120
#> Robust regression
#> 
#> Variance information
#> ------------------------------------------------------------------------------
#>              |        Imputation variance                             Relative
#>              |    Within   Between     Total       RVI       FMI    efficiency
#> -------------+----------------------------------------------------------------
#>      1.group |    .00959   .004024   .013647   .423052   .299522        .99751
#>              |   .000042   .000576   .000588   .060302   .030372       .000254
#>              |                                                  
#>  baselinedbp |   .001177    .00087   .002054   .745092   .430409       .996426
#>              |   5.1e-06   .000104   .000105   .088822   .029698       .000248
#>              |                                                  
#>          age |    .00005   .000029   .000079   .572671   .367007       .996951
#>              |   2.2e-07   3.9e-06   3.9e-06   .077358   .031978       .000267
#>              |                                                  
#>            z |   1.19815   1.07164   2.27872   .901867   .478054       .996032
#>              |   .005198   .122213   .123705   .102669   .028885       .000241
#>              |                                                  
#>           r1 |   .002683   .002566    .00527     .9643   .494902       .995893
#>              |   .000012   .000373   .000379   .139641   .037113       .000309
#>              |                                                  
#>           r2 |    .00936   .008136   .017564   .876541   .470898       .996091
#>              |   .000041    .00098   .000996    .10503   .030376       .000253
#>              |                                                  
#>        _cons |   16.1845   11.8856   28.1691   .740502   .428884       .996439
#>              |   .070211   1.42688    1.4443   .088798   .029842       .000249
#> ------------------------------------------------------------------------------
#> Note: Values displayed beneath estimates are Monte Carlo error estimates.
#> 
#> 
#> 
#> Multiple-imputation estimates                   Imputations       =        120
#> Robust regression                               Number of obs     =        500
#>                                                 Average RVI       =     0.7202
#>                                                 Largest FMI       =     0.4949
#>                                                 Complete DF       =        493
#> DF adjustment:   Small sample                   DF:     min       =     165.96
#>                                                         avg       =     202.37
#>                                                         max       =     274.66
#> Model F test:       Equal FMI                   F(   6,  431.4)   =    1183.09
#>                                                 Prob > F          =     0.0000
#> 
#> ------------------------------------------------------------------------------
#>      postdbp | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>      1.group |  -9.523045   .1168215   -81.52   0.000    -9.753025   -9.293066
#>              |   .0057905   .0025234     1.76   0.000     .0084985    .0067603
#>              |
#>  baselinedbp |    .980327   .0453156    21.63   0.000       .89096    1.069694
#>              |   .0026919   .0011629     0.57   0.000     .0038643    .0032359
#>              |
#>          age |   .3297468   .0088947    37.07   0.000      .312222    .3472716
#>              |   .0004879   .0002219     0.92   0.000     .0006042    .0007118
#>              |
#>            z |  -1.062159   1.509542    -0.70   0.483    -4.041587    1.917269
#>              |   .0945004   .0410673     0.07   0.041     .1227202    .1282877
#>              |
#>           r1 |   -.018845   .0725934    -0.26   0.795    -.1621706    .1244805
#>              |   .0046239   .0026182     0.06   0.049     .0075822    .0063906
#>              |
#>           r2 |  -.1529839   .1325282    -1.15   0.250    -.4145243    .1085565
#>              |   .0082342   .0037686     0.07   0.027     .0120192    .0102984
#>              |
#>        _cons |    11.6797   5.307458     2.20   0.029     1.213107     22.1463
#>              |   .3147167   .1363694     0.07   0.005     .3681237    .4611054
#> ------------------------------------------------------------------------------
#> Note: Values displayed beneath estimates are Monte Carlo error estimates.

The Primary Analysis

For the primary analysis, I will use the predictive-mean matching method and specify 5 donors (based on a number of simulation studies that have examined the effect of various donors specified)22, and fit a robust regression that includes all the variables within the dataset. Our imputation model will be

So we will now impute our datasets.

# Impute Missing Data

form <- list(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2)
pred <- make.predictorMatrix(BP_miss)

imp1 <- mice::mice(
  BP_miss,
  method = "pmm",
  predictorMatrix = pred,
  m = 120,
  maxit = 5,
  seed = 1031,
  formulas = form,
  ridge = 0,
  donors = 5,
  remove.collinear = TRUE,
  remove.constant = TRUE, allow.na = TRUE,
  printFlag = FALSE, visitSequence = "monotone"
)

head(imp1$loggedEvents, 10)
#> NULL

Now that we have imputed our datasets, we may examine them for any issues before moving onto the next step of analyzing it.

Our imputed datasets seem to look fine so far. We now fit our models to each of these datasets and combine them using Rubin’s rules.

# Primary Analysis

analysis1 <- with(imp1, rlm(
  PostDBP ~ (Group) +
    BaselineDBP + Age + Z + R1 + R2,
  method = "MM", psi = psi.huber
))
analysis2 <- with(imp1, rlm(PostDBP ~ (Group),
  method = "MM", psi = psi.huber
))

anova(analysis1, analysis2,
  method = "D3", use = "LR"
)
#>    test statistic df1 df2 dfcom p.value riv
#>  1 ~~ 2         0   5 600    NA       1 217
result1 <- mice::pool(analysis1)

results1_sum <- summary(result1)

colnames(results1_sum) <- c(
  "Term", "Estimate", "SE",
  "Statistic", "df", "P-val"
)
Term Estimate SE Statistic df P-val
(Intercept) 11.25 5.29 2.13 NA NA
GroupDrug Y -9.51 0.13 -75.89 NA NA
BaselineDBP 0.98 0.05 21.82 NA NA
Age 0.33 0.01 34.81 NA NA
Z -1.33 1.41 -0.94 NA NA
R1 -0.02 0.07 -0.32 NA NA
R2 -0.14 0.13 -1.09 NA NA

While we have our primary analysis estimates, we also continue to examine our imputations for any anomalies.

We’ve done our primary analysis, in which we imputed the missing data under the assumption of MAR, which is often a reasonable assumption to start with. Unfortunately, we cannot verify whether or not this assumption is true, but we can vary this missing data assumption and assume that the data are missing not at random (MNAR) , and that in the individuals with missing data, we can assume different numbers than those with complete data. Our likelihood-ratio test also suggests that there’s very little difference between the full model and the reduced model, so we generally will go with the full model.

SA I: Controlled Imputations

Now suppose we wish to handle these missing data under the assumption of MNAR, we would now be conducting a sensitivity analysis because we are still targeting the same estimand, but only varying the assumptions. The question / target remain the same. To handle missing data that we assume are MNAR, there are a number of different and complex approaches.

We start off with the -adjustment, controlled multiple imputation method,23 in which we assume that the group with missing observations differs systematically from the group with complete observations by a certain quantity. We produce a range of these quantities and add or subtract them from the imputed values and then conduct our analysis on this adjusted, imputed dataset. Here I will make the assumption that the average PostDBP estimate from the missing value group differs from at least 0 mmHg to at most 20 mmHg from the group with no missing observations.

# Controlled MI

ini <- mice(BP_miss,
  method = "pmm", maxit = 0,
  predictorMatrix = pred
)

ini$method["BaselineDBP"] <- ""
ini$method["PostDBP"] <- "pmm"
ini$method["Age"] <- ""
ini$method["R1"] <- ""
ini$method["Group"] <- ""
ini$method["R2"] <- ""
ini$method["Z"] <- ""
meth <- ini$method

delta <- seq(-20, 20, 5)

imp.all <- vector("list", length(delta))
post <- ini$post
for (i in 1:length(delta)) {
  d <- delta[i]
  cmd <- paste("imp[[j]][, i] <- imp[[j]][, i] +", d)
  post["PostDBP"] <- cmd
  imp <- mice::mice(
    BP_miss,
    method = meth,
    predictorMatrix = pred,
    m = 5,
    maxit = 1,
    donors = 5,
    ridge = 0,
    formulas = form,
    remove.collinear = TRUE,
    remove.constant = TRUE,
    allow.na = TRUE, post = post,
    printFlag = FALSE
  )
  imp.all[[i]] <- imp
}
for (i in 1:length(delta)) {
  imp.all[[i]][["data"]][["Group"]] <- as.factor(imp.all[[i]][["data"]][["Group"]])
}

Now that we have our imputed datasets that are modified by -adjustments, we can fit our models to each imputed dataset and combine the estimates using Rubin’s rules.

# Sensitivity Analysis 1

output <- sapply(imp.all,
  function(x) {
    mice::pool(with(
      x,
      rlm(
        PostDBP ~ factor(Group) +
          BaselineDBP + Age + Z + R1 + R2,
        method = "MM", psi = psi.huber
      )
    ))
  },
  simplify = "array", USE.NAMES = TRUE
)

a <- (as.data.frame(t(output)))$pooled
r <- array(rep(NA, length(delta) * 7),
  dim = c(length(delta), 7),
  dimnames = list(
    c(delta),
    c(
      "Intercept", "Group",
      "BaselineDBP", "Age",
      "Z", "R1", "R2"
    )
  )
)
for (i in 1:length(delta)) {
  r[i, ] <- cbind(a[[i]][["estimate"]])
}

r <- as.data.frame(r)
r <- data.frame(Delta = row.names(r), r, row.names = NULL)
Delta Intercept Group BaselineDBP Age Z R1 R2
-20 -112.03 -11.90 1.99 0.30 -3.18 0.09 -1.19
-15 -49.65 -10.64 1.46 0.31 -2.10 0.06 -0.60
-10 -25.15 -10.18 1.26 0.32 -1.17 0.06 -0.41
-5 -2.75 -9.80 1.09 0.32 -0.93 0.00 -0.25
0 13.07 -9.54 0.97 0.33 -0.59 -0.01 -0.13
5 32.79 -9.18 0.82 0.33 0.00 -0.07 -0.04
10 51.86 -8.84 0.67 0.33 -0.18 -0.09 0.05
15 80.67 -8.37 0.44 0.34 0.75 -0.11 0.18
20 138.66 -7.17 -0.05 0.35 1.00 -0.19 0.78

The table above gives the various ’s (the first column with values ranging from -30 - 30 in increments of 5) that were added to the multiply imputed datasets, and the corresponding estimates from the models fit to those datasets. So now we’ve conducted our first sensitivity analysis using the -adjustment, controlled multiple imputation method. This form of sensitivity analysis is also typically what is used and preferred by regulators.23

Regulators prefer simple methods that impute the missing outcomes under MAR, and then add an adjustment δ to the imputes, while varying δ over a plausible range and independently for each treatment group (Permutt 2016).

The most interesting scenarios will be those where the difference between the δ’s correspond to the size of the treatment effect in the completers. Contours of the p-values may be plotted on a graph as a function of the δ’s to assist in a tipping-point analysis (Liublinska and Rubin 2014).

And as expected, lower values gave us results that were consistent with the primary analysis, while the upper end of the extreme values gave us estimates that were substantially off the mark.

Next, we quickly examine the state of a random sample of these imputations to make sure they’re reliable.

Now we move onto our next and final sensitivity analysis.

SA II: Selection Models

To further explore the MNAR assumption, we could also use a Heckman selection model , as described by van Buuren here20

The selection model multiplies the marginal distribution in the population with the response weights . Both and are unknown, and must be specified by the user. The model where is normal and where is a probit model is known as the Heckman model. This model is widely used in economics to correct for selection bias.

We can implement this in R using the miceMNAR package, which is an extension to the mice R package.

# Selection Models
has_miceMNAR <- requireNamespace("miceMNAR", quietly = TRUE)
has_mnar_api <- FALSE

if (has_miceMNAR) {
  miceMNAR_exports <- getNamespaceExports("miceMNAR")
  has_mnar_api <- all(c("generate_JointModelEq", "MNARargument") %in% miceMNAR_exports)
}

if (!has_mnar_api) {
  message(
    "Skipping SA II (selection model): 'miceMNAR' with generate_JointModelEq/MNARargument is unavailable in this environment."
  )
  imp2 <- NULL
} else {
  JointModelEq <- miceMNAR::generate_JointModelEq(
    data = BP_miss,
    varMNAR = "PostDBP"
  )

  JointModelEq[, "PostDBP_var_sel"] <- c(0, 1, 1, 1, 1, 1, 1)
  JointModelEq[, "PostDBP_var_out"] <- c(0, 1, 1, 1, 0, 1, 1)

  arg <- miceMNAR::MNARargument(
    data = BP_miss,
    varMNAR = "PostDBP",
    JointModelEq = JointModelEq
  )

  imp2 <- mice(
    data = arg$data_mod,
    method = "pmm", seed = 1031,
    predictorMatrix = arg$predictorMatrix,
    JointModelEq = arg$JointModelEq,
    control = arg$control,
    remove.collinear = TRUE,
    remove.constant = FALSE, allow.na = FALSE,
    printFlag = FALSE, visitSequence = "monotone"
  )

  imp2$data$Group <- as.factor(imp2$data$Group)
}

Now we fit our models to these new datasets and pool them using Rubin’s rules.

# Sensitivity Analysis 2
if (is.null(imp2)) {
  analysis3 <- NULL
  result3 <- NULL
  results3_sum <- data.frame(
    Term = character(),
    Estimate = numeric(),
    SE = numeric(),
    Statistic = numeric(),
    df = numeric(),
    `P-val` = numeric()
  )
} else {
  analysis3 <- with(imp2, rlm(
    PostDBP ~ factor(Group) +
      BaselineDBP + Age + Z + R1 + R2,
    method = "MM", psi = psi.huber
  ))
  result3 <- mice::pool(analysis3)

  results3_sum <- summary(result3)
  colnames(results3_sum) <- c(
    "Term", "Estimate", "SE",
    "Statistic", "df", "P-val"
  )
}
Term Estimate SE Statistic df P-val
(Intercept) 10.12 6.14 1.65 NA NA
factor(Group)Drug Y -9.50 0.20 -47.69 NA NA
BaselineDBP 0.99 0.06 18.01 NA NA
Age 0.33 0.01 27.99 NA NA
Z -1.09 1.85 -0.59 NA NA
R1 -0.02 0.08 -0.30 NA NA
R2 -0.15 0.16 -0.96 NA NA

Now we examine our imputed values.

so now we have completed our second sensitivity analysis.

The results from both the primary analysis, the first sensitivity analysis, and the second seem somewhat consistent. However, with the first sensitivity analysis where we applied -adjustments, as the adjustments became larger, the coefficients changed drastically. However, such large -adjustments are not realistic and those that were smaller led to coefficients that were closer to the other two analyses.

Supplementary Analyses

The following analyses are what I mostly consider to be supplementary analyses that may further investigate some violations of assumptions but often go far beyond what would be necessary for a principled sensitivity analysis that would typically accompany a primary analysis.

First, I like to check that the results from my analyses are consistent across different statistical software So I may check to see that the results from my primary analysis from R are also similar to the results from another statistical software suite like Stata.

So in Stata, I would likely run something similar to the following in order to mimic the primary analysis in R.

set linesize 255

clear all

set seed 1031

import delimited "static/datasets/temp.csv", numericcols(2 3 5 6 7 8)

summarize

mi set mlong

mi register imputed baselinedbp postdbp age z group r1 r2

mi impute chained (pmm, knn(5)) postdbp = baselinedbp age z r1 r2 group, burnin(20) add(100) rseed (1031) savetrace(static/datasets/trace1, replace) nomonotone

mi estimate, mcerror: rreg postdbp i.group baselinedbp age z r1 r2

quietly mi passive: generate byte imputed = _mi_miss
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> (encoding automatically selected: ISO-8859-2)
#> (7 vars, 500 obs)
#> 
#> 
#>     Variable |        Obs        Mean    Std. dev.       Min        Max
#> -------------+---------------------------------------------------------
#>      postdbp |        255    138.3033    5.413468   123.8279   150.0574
#>  baselinedbp |        500    116.8569    1.997616   109.8634   123.0737
#>        group |        500        .484    .5002444          0          1
#>          age |        500    49.73008    6.938333   27.65241    68.4464
#>            z |        500    2.78e-13    .0626851  -.1845036   .2024892
#> -------------+---------------------------------------------------------
#>           r1 |        500   -.0428001    .9472443  -3.205036   3.377526
#>           r2 |        500    .9620217    .5061543  -.6455948   2.480457
#> 
#> 
#> (245 m=0 obs now marked as incomplete)
#> 
#> note: missing-value pattern is monotone.
#> 
#> Conditional models:
#>            postdbp: pmm postdbp baselinedbp age z r1 r2 group , knn(5)
#> 
#> Performing chained iterations ...
#> 
#> Multivariate imputation                     Imputations =      100
#> Chained equations                                 added =      100
#> Imputed: m=1 through m=100                      updated =        0
#> 
#> Initialization: monotone                     Iterations =     2000
#>                                                 burn-in =       20
#> 
#>            postdbp: predictive mean matching
#> 
#> ------------------------------------------------------------------
#>                    |               Observations per m             
#>                    |----------------------------------------------
#>           Variable |   Complete   Incomplete   Imputed |     Total
#> -------------------+-----------------------------------+----------
#>            postdbp |        255          245       245 |       500
#> ------------------------------------------------------------------
#> (Complete + Incomplete = Total; Imputed is the minimum across m
#>  of the number of filled-in observations.)
#> 
#> 
#> Multiple-imputation estimates                   Imputations       =        100
#> Robust regression                               Number of obs     =        500
#>                                                 Average RVI       =     0.7136
#>                                                 Largest FMI       =     0.5035
#>                                                 Complete DF       =        493
#> DF adjustment:   Small sample                   DF:     min       =     151.95
#>                                                         avg       =     188.67
#>                                                         max       =     233.67
#> Model F test:       Equal FMI                   F(   6,  421.4)   =    1181.06
#>                                                 Prob > F          =     0.0000
#> 
#> ------------------------------------------------------------------------------
#>      postdbp | Coefficient  Std. err.      t    P>|t|     [95% conf. interval]
#> -------------+----------------------------------------------------------------
#>      1.group |  -9.520968   .1207769   -78.83   0.000    -9.758918   -9.283017
#>              |   .0070306   .0035352     2.31   0.000     .0118962    .0076126
#>              |
#>  baselinedbp |   .9811365   .0448784    21.86   0.000     .8926164    1.069657
#>              |   .0028784   .0010755     0.54   0.000     .0037961    .0033965
#>              |
#>          age |    .328745   .0089894    36.57   0.000     .3110251    .3464649
#>              |   .0005493   .0002679     1.12   0.000      .000839    .0006919
#>              |
#>            z |  -1.017329   1.500306    -0.68   0.499    -3.979496    1.944838
#>              |   .1020678    .050788     0.06   0.041     .1658641    .1196211
#>              |
#>           r1 |  -.0169674   .0731975    -0.23   0.817    -.1615836    .1276488
#>              |   .0051452   .0023361     0.07   0.056     .0063651    .0075475
#>              |
#>           r2 |  -.1393538   .1309672    -1.06   0.289    -.3978634    .1191559
#>              |   .0087814   .0046722     0.07   0.032     .0138722    .0117957
#>              |
#>        _cons |   11.61969   5.244316     2.22   0.028     1.276334    21.96305
#>              |   .3346446   .1264252     0.08   0.005     .3897239    .4489929
#> ------------------------------------------------------------------------------
#> Note: Values displayed beneath estimates are Monte Carlo error estimates.
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> (Summaries of imputed values from -mi impute chained-)
#> 
#> 
#> Contains data from static/datasets/trace1.dta
#>  Observations:         2,100                  Summaries of imputed values from -mi impute chained-
#>     Variables:             4                  13 Sep 2026 04:16
#> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#> Variable      Storage   Display    Value
#>     name         type    format    label      Variable label
#> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#> iter            byte    %12.0g                Iteration numbers
#> m               byte    %12.0g                Imputation numbers
#> postdbp_mean    float   %9.0g                 Mean of postdbp
#> postdbp_sd      float   %9.0g                 Std. dev. of postdbp
#> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#> Sorted by: 
#> 
#> 
#> 
#> reshaping m=0 data ...
#> (j = 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 8
#> > 7 88 89 90 91 92 93 94 95 96 97 98 99 100)
#> 
#> Data                               Long   ->   Wide
#> -----------------------------------------------------------------------------
#> Number of observations            2,100   ->   21          
#> Number of variables                   4   ->   201         
#> j variable (100 values)               m   ->   (dropped)
#> xij variables:
#>                            postdbp_mean   ->   postdbp_mean1 postdbp_mean2 ... postdbp_mean100
#>                              postdbp_sd   ->   postdbp_sd1 postdbp_sd2 ... postdbp_sd100
#> -----------------------------------------------------------------------------
#> 
#> assembling results ...
#> 
#> 
#> Time variable: iter, 0 to 20
#>         Delta: 1 unit
#> 
#> 
#> 
#> file ()print() not found
#> r(601);
#> 
#> r(601);

As you can see from above, I imported the missing data I generated in R into Stata, and then used it to multiply impute the datasets using the predictive mean matching method with 5 donors. I chose the exact same number of variables to include in the imputation model as I did with the primary and sensitivity analyses. Just like the primary analysis, I also used robust regression, rreg in Stata, to analyze the datasets. The numbers seem to be fairly consistent with those from the primary analysis and the some of the results from the sensitivity analyses.

I will now conduct another supplementary analysis, in which I fit a Bayesian regression model using a t-distribution so that it is analogous to the other robust regression models I have utilized so far, and I will specify a weakly informative prior. I will fit one model using the brm_multiple() function which will fit the model to each of the imputed datasets from before, while I fit another Bayesian model which will impute values during the model fitting process and compare the results.

# Quantile Regression  --------------------------

implist <- mitml::mids2mitml.list(imp1)

vcov.rq <- function(object, ...) {
  summary(object,
    se = "nid",
    covariance = TRUE
  )$cov
}

fit1 <- with(implist, rq(PostDBP ~ factor(Group) + BaselineDBP +
  Age + Z + R1 + R2, tau = 0.5))
mitml::testEstimates(fit1)
#> 
#> Call:
#> 
#> mitml::testEstimates(model = fit1)
#> 
#> Final parameter estimates and inferences obtained from 120 imputed data sets.
#> 
#>                      Estimate Std.Error   t.value        df   P(>|t|)       RIV       FMI 
#> (Intercept)            10.092     6.610     1.527   519.639     0.127     0.918     0.481 
#> factor(Group)Drug Y    -9.433     0.157   -59.956   622.364     0.000     0.777     0.439 
#> BaselineDBP             0.991     0.056    17.617   522.900     0.000     0.912     0.479 
#> Age                     0.336     0.011    31.319   690.038     0.000     0.710     0.417 
#> Z                      -1.397     1.755    -0.796   594.157     0.426     0.810     0.449 
#> R1                     -0.054     0.091    -0.600   403.495     0.549     1.189     0.545 
#> R2                     -0.137     0.160    -0.859   451.123     0.391     1.056     0.516 
#> 
#> Unadjusted hypothesis test as appropriate in larger samples.

confint(mitml::testEstimates(fit1))
#>                      2.5 % 97.5 %
#> (Intercept)         -2.894 23.078
#> factor(Group)Drug Y -9.742 -9.124
#> BaselineDBP          0.880  1.101
#> Age                  0.315  0.357
#> Z                   -4.843  2.049
#> R1                  -0.233  0.124
#> R2                  -0.452  0.177
# Bayesian Regression  --------------------------

BP_miss$Group <- as.factor(BP_miss$Group)

brm_form2 <- bf(
  PostDBP | mi() ~ factor(Group) + BaselineDBP +
    Age + Z + R1 + R2, quantile = 0.50
)

prior1 <- get_prior(brm_form2, data = BP_miss, knots = 4,
                    family = asym_laplace(link = 'identity'))

b_2 <- brm(formula = brm_form2, data = BP_miss, prior = prior1,
           family = asym_laplace(link = 'identity'), sample_prior = 'yes',
           control = list(max_treedepth = 10, adapt_delta = 0.80),
           iter = 2000, refresh = 0, backend = "rstan", chains = 4,
           algorithm = "sampling", knots = 4, future = FALSE, cores = 8L,
           warmup = 500, seed = 1031)
# `post` is already the mice post-processing vector above, so use a distinct name
bayes_post <- if (exists("b_2")) summary(b_2) else NULL

if (is.null(bayes_post)) {
  cat("Bayesian model summary skipped: `b_2` was not fitted in this environment.\n")
} else {
  bayes_post
}
#> Bayesian model summary skipped: `b_2` was not fitted in this environment.

We can also compare the results we get from brms/Stan with MCMCpack.

posterior_95 <- MCMCquantreg(
  PostDBP ~ factor(Group) + BaselineDBP +
    Age + Z + R1 + R2,
  data = BP_miss,
  tau = 0.95, mcmc = 5000, chains = 4,
  thin = 10, seed = 1031
)
summary(posterior_95)
#> 
#> Iterations = 1001:5991
#> Thinning interval = 10 
#> Number of chains = 1 
#> Sample size per chain = 500 
#> 
#> 1. Empirical mean and standard deviation for each variable,
#>    plus standard error of the mean:
#> 
#>                        Mean      SD Naive SE Time-series SE
#> (Intercept)         10.2441 19.8021  0.88558        1.00205
#> factor(Group)Drug Y -9.6451  0.4329  0.01936        0.02600
#> BaselineDBP          1.0197  0.1688  0.00755        0.00855
#> Age                  0.3108  0.0324  0.00145        0.00156
#> Z                   -3.0856  5.6527  0.25279        0.27623
#> R1                  -0.0277  0.2696  0.01206        0.01383
#> R2                  -0.4040  0.3937  0.01761        0.01950
#> 
#> 2. Quantiles for each variable:
#> 
#>                        2.5%    25%     50%    75%  97.5%
#> (Intercept)         -27.035 -2.560  9.2822 22.020 49.797
#> factor(Group)Drug Y -10.572 -9.916 -9.6297 -9.350 -8.846
#> BaselineDBP           0.679  0.918  1.0263  1.130  1.338
#> Age                   0.253  0.290  0.3094  0.332  0.375
#> Z                   -14.438 -6.933 -3.3466  1.069  7.636
#> R1                   -0.550 -0.214 -0.0152  0.146  0.507
#> R2                   -1.189 -0.646 -0.3728 -0.158  0.312

Next, we examine the results from the Bayesian model in which values were imputed during model fitting.

posterior_summary(b_2)[1:8]
(post <- posterior_samples(b_2))

They seem to be similar to the results we’ve been seeing so far.

Indeed, it is no surprise that the Bayesian model with a weakly dispersed prior would give similar results to the base rlm() function. We now look at robust methods and resampling and the results they give.

Although this is currently under the supplementary analysis section, I would very much consider these next two analyses to be sensitivity analyses. The first method utilized here is known as the random indicator method, and it is an experimental method that essentially combines a selection model and a pattern-mixture model and is designed to handle data that are missing not at random. As van Buuren writes20

The random indicator method (Jolani 2012)24 is an experimental iterative method that redraws the missing data indicator under a selection model, and imputes the missing data under a pattern-mixture model, with the objective of estimating δ from the data under relaxed assumptions. Initial simulation results look promising. The algorithm is available as the ri method in mice.

The script below utilizes this method (random indicator), but bootstraps the missing dataset before the imputation procedure, and then after multiply imputing the datasets, obtains estimates using maximum likelihood multiple (MLMI) imputation, an alternative and more efficient method to the more popular posterior draw multiple imputation (PDMI). More about MLMI can be found here.25

# Bootstrap Inference

bootMice(BP_miss, method = "ri", maxit = 5,
         predictorMatrix = pred, nBoot = 96, nImp = 2,
         seed = 1031, formulas = form, nCores = 8L,
         ridge = 0, remove.collinear = TRUE,
         remove.constant = TRUE, allow.na = TRUE,
         printFlag = FALSE, visitSequence = "monotone") -> imps

analyseImp <- function(inputData) {
  return(coef(rlm(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2,
    data = inputData, method = "MM", psi = psi.huber
  )))
}

analyseImp1 <- function(inputData) {
  return(coef(glm(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2,
    data = inputData, family = gaussian()
  )))
}

ests <- bootImputeAnalyse(imps, analyseImp, quiet = TRUE)
ests1 <- bootImputeAnalyse(imps, analyseImp1, quiet = TRUE)

ests <- as.data.frame(ests)

colnames(ests) <- c("Estimate", "Var",
                    "CI.ll", "CI.ul", "df")

rownames(ests) <- c("Intercept", "Group", "BaselineDBP",
                    "Age", "Z", "R1", "R2")
ests1 <- as.data.frame(ests1)

colnames(ests1) <- c("Estimate", "Var",
                     "CI.ll", "CI.ul", "df")

rownames(ests1) <- c("Intercept", "Group", "BaselineDBP",
                     "Age", "Z", "R1", "R2")
Estimate Var CI.ll CI.ul df
Intercept 10.72 16.51 2.19 19.25 18.3
Group -9.62 0.02 -9.89 -9.34 48.0
BaselineDBP 0.99 0.00 0.91 1.06 19.6
Age 0.34 0.00 0.32 0.36 55.4
Z -1.00 2.32 -4.06 2.05 51.2
R1 -0.02 0.01 -0.19 0.15 64.3
R2 -0.14 0.02 -0.39 0.11 44.8
Estimate Var CI.ll CI.ul df
Intercept 10.37 16.67 1.84 18.89 19.7
Group -9.63 0.02 -9.89 -9.36 47.5
BaselineDBP 0.99 0.00 0.92 1.06 20.7
Age 0.33 0.00 0.31 0.35 57.3
Z -1.02 2.21 -4.00 1.96 50.1
R1 -0.02 0.01 -0.18 0.15 62.6
R2 -0.15 0.01 -0.39 0.09 43.3

For our second supplementary analysis, We will utilize the GAMLSS method, which is an extension of the generalized linear model and the generalized additive model. For the vast majority of the imputations we have conducted, we have drawn values from an assumed normal distribution. Thus, GAMLSS allows us to be far more flexible with our assumptions. However, it is also worth noting that general normal imputations are robust against violations of the assumption of normality

In general, normal imputations appear to be robust against violations of normality. Demirtas, Freels, and Yucel (2008) found that flatness of the density, heavy tails, non-zero peakedness, skewness and multimodality do not appear to hamper the good performance of multiple imputation for the mean structure in samples n > 400, even for high percentages (75%) of missing data in one variable. The variance parameter is more critical though, and could be off-target in smaller samples.

Regardless, we will be using the GAMLSS method and also imputing from a t-distribution, which is much more robust to outliers.

gamt <- mice(BP_miss, method = "gamlssTF",
             predictorMatrix = pred,
             m = 5, maxit = 1, seed = 1031,
             cores = cores, formulas = form, ridge = 0,
             remove.collinear = TRUE, remove.constant = TRUE,
             allow.na = TRUE, printFlag = FALSE)

gt1 <- with(gamt, rlm(PostDBP ~ factor(Group) +
  BaselineDBP + Age +
  Z + R1 + R2))

resultsgt <- mice::pool(gt1)
ztable(summary(resultsgt))
term estimate std.error statistic df p.value
(Intercept) 13.23 4.95 2.67 NA NA
factor(Group)Drug Y -9.52 0.13 -76.02 NA NA
BaselineDBP 0.97 0.04 23.48 NA NA
Age 0.32 0.01 35.44 NA NA
Z -0.62 1.39 -0.44 NA NA
R1 -0.02 0.07 -0.34 NA NA
R2 -0.14 0.10 -1.35 NA NA

Having done that, I shall try one more supplementary analysis which piqued my curiosity. I recently came across a few papers that concluded that random forests outperformed other imputation algorithms in recovering the true parameter values.

Generally, imputation algorithms are evaluated based on a general simulation design which often resembles the following:

  1. Sampling mechanism only. The basic simulation steps are: choose , take samples , fit the complete-data model, estimate and and calculate the outcomes aggregated over .
  1. Sampling and missing data mechanisms combined. The basic simulation steps are: choose , take samples , generate incomplete data , impute, estimate and and calculate outcomes aggregated over and .
  1. Missing data mechanism only. The basic simulation steps are: choose , generate incomplete data , impute, estimate and and calculate outcomes aggregated over .

These designs generally lead to evaluating a number of metrics which include the following:

  • Raw Bias: where is bias and is the difference between the estimate and the truth , which may also be expressed as a percent
  • Coverage Rate: which is simply the proportion of times the intervals actually contain the true parameter value
  • Average Width: the width of the interval estimates, with longer widths indicating less information
  • Root mean squared error (RMSE)

The authors primarily based their conclusions on the root mean squared error (RMSE). However, this is problematic for a number of reasons; imputation is not the same as prediction, and so the same loss functions are not as useful; indeed, the RMSE is highly misleading as a measure of performance.

van Buuren explains here:

It is well known that the minimum RMSE is attained by predicting the missing by the linear model with the regression weights set to their least squares estimates. According to this reasoning the “best” method replaces each missing value by its most likely value under the model. However, this will find the same values over and over, and is single imputation.

This method ignores the inherent uncertainty of the missing values (and acts as if they were known after all), resulting in biased estimates and invalid statistical inferences. Hence, the method yielding the lowest RMSE is bad for imputation. More generally, measures based on similarity between the true and imputed values do not separate valid from invalid imputation methods.

Having set up our simulation design, we can now test the performances between regression imputation norm.predict and and regular regression imputation norm and we will be examining the metrics above to evaluate performance. Regression imputation generally gives very biased results given that it uses its predictions to impute the missing values. Thus, I will show here how RMSE is highly misleading as a metric to evaluate the performance of imputation algorithms.

# Evaluate MI Algorithms

create.data <- function(beta = 1, sigma2 = 1, n = 500,
                        run = 1) {
  set.seed(seed = run)
  x <- rnorm(n)
  y <- beta * x + rnorm(n, sd = sqrt(sigma2))
  cbind(x = x, y = y)
}

make.missing <- function(data, p = 0.5) {
  rx <- rbinom(nrow(data), 1, p)
  data[rx == 0, "x"] <- NA
  data
}

test.impute <- function(data, m = 5, method = "norm", ...) {
  imp <- mice(data,
    method = method, m = m,
    print = FALSE, cores = cores, ...
  )
  fit <- with(imp, lm(y ~ x))
  tab <- summary(mice::pool(fit), "all", conf.int = TRUE)
  as.numeric(tab["x", c("estimate", "2.5 %", "97.5 %")])
}

rmse <- function(truedata, imp, v = "x") {
  mx <- is.na(mice::complete(imp, 0))[, v]
  mse <- rep(NA, imp$m)
  for (k in seq_len(imp$m)) {
    filled <- mice::complete(imp, k)[mx, v]
    true <- truedata[mx, v]
    mse[k] <- mean((filled - true)^2)
  }
  sqrt(mean(mse))
}

mae <- function(truedata, imp, v = "x") {
  mx <- is.na(mice::complete(imp, 0))[, v]
  mse <- rep(NA, imp$m)
  for (k in seq_len(imp$m)) {
    filled <- mice::complete(imp, k)[mx, v]
    true <- truedata[mx, v]
    mse[k] <- mean((filled - true))
  }
  abs(mean(mse))
}

simulate1 <- function(runs = 10) {
  res <- array(NA, dim = c(2, runs, 1))
  dimnames(res) <- list(
    c("norm.predict", "norm"),
    as.character(1:runs),
    "MAE"
  )
  for (run in 1:runs) {
    truedata <- create.data(run = run)
    data <- make.missing(truedata)
    imp <- mice(data,
      method = "norm.predict",
      m = 1, cores = cores,
      print = FALSE
    )
    res[1, run, ] <- mae(truedata, imp)
    imp <- mice(data, method = "norm", print = FALSE)
    res[2, run, ] <- mae(truedata, imp)
  }
  res
}

simulate2 <- function(runs = 10) {
  res <- array(NA, dim = c(2, runs, 1))
  dimnames(res) <- list(
    c("norm.predict", "norm"),
    as.character(1:runs),
    "RMSE"
  )
  for (run in 1:runs) {
    truedata <- create.data(run = run)
    data <- make.missing(truedata)
    imp <- mice(data,
      method = "norm.predict",
      m = 1, cores = cores,
      print = FALSE
    )
    res[1, run, ] <- rmse(truedata, imp)
    imp <- mice(data, method = "norm", print = FALSE)
    res[2, run, ] <- rmse(truedata, imp)
  }
  res
}

Our simulation above will generate a dataset, delete values from the dataset and then impute the missing values for that dataset using regression imputation and stochastic regression imputation. We will use two metrics to evaluate the performance of these two algorithms, RMSE and MAE. Once again, it is worth keeping in mind that regression imputation generally performs very poorly when it comes to imputation and that stochastic regression imputation performs better.

set.seed(1031,
  kind = "L'Ecuyer-CMRG",
  normal.kind = "Inversion",
  sample.kind = "Rejection"
)
res1 <- simulate1(1000)
res_1 <- apply(res1, c(1, 3),
  mean,
  na.rm = TRUE
)
res2 <- simulate2(1000)
res_2 <- apply(res2, c(1, 3),
  mean,
  na.rm = TRUE
)
ztable(cbind(res_1, res_2))

What this simulation clearly shows is that despite regression imputation norm.predict giving highly biased results as usual relative to the truth, it still has a much lower RMSE than stochastic regression imputation, which both had a higher RMSE and a closer distance to the correct parameter values. Once again, this is not surprising given that imputation is not the same thing as prediction and therefore the loss functions cannot be used synonymously. However, when it came to raw bias and mean absolute error, stochastic regression imputation outperformed regression imputation which had a much larger mean absolute error and raw bias.

This is why the most informative metrics to focus on when evaluating an imputation algorithm include information/width, sufficient long-run coverage, and less misleading metrics such as mean absolute error and mean absolute percentage error.

The authors specifically used the missForest package. I tried using the package, but encountered too many difficulties so I simply specified the rf method in mice, and worked from there.

# MI Using Random Forest

form <- list(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2)

rf <- mice::mice(
  BP_miss,
  method = "rf",
  ntree = 20,
  predictorMatrix = pred,
  m = 10,
  maxit = 5,
  seed = 1031,
  formulas = form,
  ridge = 0,
  remove.collinear = TRUE,
  remove.constant = TRUE,
  allow.na = TRUE,
  printFlag = FALSE,
  visitSequence = "monotone"
)

rf1 <- with(rf, rlm(PostDBP ~ factor(Group) +
  BaselineDBP + Age +
  Z + R1 + R2,
  method = "MM", psi = psi.huber
))

results5 <- mice::pool(rf1)

results5_sum <- summary(results5)

colnames(results5_sum) <- c(
  "Term", "Estimate", "SE",
  "Statistic", "df", "P-val"
)
ztable(results5_sum)
Term Estimate SE Statistic df P-val
(Intercept) 33.94 9.94 3.42 NA NA
factor(Group)Drug Y -9.45 0.18 -53.97 NA NA
BaselineDBP 0.80 0.09 9.38 NA NA
Age 0.29 0.02 17.77 NA NA
Z 0.02 2.58 0.01 NA NA
R1 0.02 0.09 0.22 NA NA
R2 -0.02 0.18 -0.10 NA NA

Full Analysis Set

After we run all those analyses, we can finally check our numbers from the full dataset with no missing observations to see how our approaches to handle the missing data have performed. We first look at the fully adjusted model fit with rlm().

# Full Analysis Set

final_full_mod <- rlm(
  PostDBP ~ factor(Group) + BaselineDBP +
    Age + Z + R1 + R2,
  data = BP_full,
  method = "MM", psi = psi.huber
)
Value Std. Error t value
(Intercept) 9.19 3.72 2.47
factor(Group)Drug Y -9.66 0.09 -106.85
BaselineDBP 1.00 0.03 31.68
Age 0.33 0.01 50.17
Z -2.02 1.01 -2.00
R1 0.01 0.05 0.21
R2 -0.01 0.09 -0.09
Term VIF VIF_CI_low VIF_CI_high SE_factor Tolerance Tolerance_CI_low Tolerance_CI_high
factor(Group) 1.01 1.00 489.74 1.00 0.99 0.00 1.00
BaselineDBP 1.97 1.75 2.26 1.40 0.51 0.44 0.57
Age 1.02 1.00 4.09 1.01 0.98 0.24 1.00
Z 1.98 1.75 2.27 1.41 0.51 0.44 0.57
R1 1.01 1.00 33.65 1.01 0.99 0.03 1.00
R2 1.01 1.00 1603.17 1.00 0.99 0.00 1.00

We can also graphically inspect our assumptions.

Then we look at the unadjusted model, where the difference is the group that the participants were randomized to.

# Unadjusted Model

final_reduced_mod <- rlm(PostDBP ~ factor(Group),
  data = BP_full,
  method = "MM", psi = psi.huber
)

final_reduced_mod_sum <- summary(final_reduced_mod)[["coefficients"]]

colnames(final_reduced_mod_sum) <- c(
  "Value",
  "Std. Error",
  "t-value"
)
Value Std. Error t-value
(Intercept) 142.6 0.20 728.8
factor(Group)Drug Y -9.2 0.28 -32.7

We can also compare the two to see if there are any substantial differences.

Name Model AIC AIC_wt AICc AICc_wt BIC BIC_wt RMSE Sigma
final_full_mod rlm 1420 1 1420 1 1453 1 0.98 0.99
final_reduced_mod rlm 2556 0 2557 0 2569 0 3.10 3.11

While the results have mostly been consistent, I also wanted to run a few additional analyses of my own to see whether these effect estimates were consistent I first started by constructing the confidence distribution using the [concurve] package to see the range of parameter values consistent with the data given the background model assumptions.26, 27

# Compatibility of Values

fit1 <- glm(
  PostDBP ~ factor(Group) + BaselineDBP +
    Age + Z + R1 + R2,
  data = BP_full, family = gaussian()
)

summary(fit1)
#> 
#> Call:
#> glm(formula = PostDBP ~ factor(Group) + BaselineDBP + Age + Z + 
#>     R1 + R2, family = gaussian(), data = BP_full)
#> 
#> Coefficients:
#>                     Estimate Std. Error t value Pr(>|t|)    
#> (Intercept)          9.02736    3.65879    2.47    0.014 *  
#> factor(Group)Drug Y -9.64556    0.08906 -108.30   <2e-16 ***
#> BaselineDBP          1.00567    0.03120   32.23   <2e-16 ***
#> Age                  0.32696    0.00645   50.69   <2e-16 ***
#> Z                   -1.97391    0.99551   -1.98    0.048 *  
#> R1                   0.01071    0.04711    0.23    0.820    
#> R2                  -0.02493    0.08799   -0.28    0.777    
#> ---
#> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#> 
#> (Dispersion parameter for gaussian family taken to be 0.983)
#> 
#>     Null deviance: 15519.32  on 499  degrees of freedom
#> Residual deviance:   484.49  on 493  degrees of freedom
#> AIC: 1419
#> 
#> Number of Fisher Scoring iterations: 2

fit2 <- glm(PostDBP ~ factor(Group),
  data = BP_full, family = gaussian()
)

summary(fit2)
#> 
#> Call:
#> glm(formula = PostDBP ~ factor(Group), family = gaussian(), data = BP_full)
#> 
#> Coefficients:
#>                     Estimate Std. Error t value Pr(>|t|)    
#> (Intercept)          142.596      0.193   737.3   <2e-16 ***
#> factor(Group)Drug Y   -9.263      0.278   -33.3   <2e-16 ***
#> ---
#> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#> 
#> (Dispersion parameter for gaussian family taken to be 9.65)
#> 
#>     Null deviance: 15519.3  on 499  degrees of freedom
#> Residual deviance:  4805.8  on 498  degrees of freedom
#> AIC: 2556
#> 
#> Number of Fisher Scoring iterations: 2

curve1 <- curve_gen(fit1,
  var = "factor(Group)Drug Y", method = "lm",
  log = FALSE, steps = 1000, table = TRUE
)

Lower Limit Upper Limit Interval Width Interval Level (%) CDF P-value S-value (bits)
-9.67 -9.62 0.057 25.0 0.625 0.750 0.415
-9.71 -9.59 0.120 50.0 0.750 0.500 1.000
-9.75 -9.54 0.205 75.0 0.875 0.250 2.000
-9.76 -9.53 0.228 80.0 0.900 0.200 2.322
-9.77 -9.52 0.256 85.0 0.925 0.150 2.737
-9.79 -9.50 0.293 90.0 0.950 0.100 3.322
-9.82 -9.47 0.349 95.0 0.975 0.050 4.322
-9.85 -9.45 0.399 97.5 0.988 0.025 5.322
-9.88 -9.42 0.459 99.0 0.995 0.010 6.644
Table of Statistics for Various Interval Estimate Percentiles

Indeed, our confidence distribution ends up being very similar to our Bayesian posterior distribution for the treatment effect. Under certain situations, the maximum likelihood estimate ends up also being equivalent to the median unbiased estimate. In this case, it happens to be very similar to the maximum a posteriori estimate.

#> Posterior density plot skipped: `b_2` was not fitted in this environment.

We can also check to see that our confidence distribution and posterior distribution are consistent with the profile-likelihood function, which is free of nuisance parameters. The advantage of profiling is allowing us to recover more accurate parameter estimates when there are several nuisance parameters or messy data. Below are implementations in both R and Stata, except this time we do not specify errors from the t-distribution.

analyseImp3 <- function(inputData, i) {
  dt <- inputData[i, ]
  (coef(glm(PostDBP ~ factor(Group) + BaselineDBP + Age + Z + R1 + R2,
    data = dt, family = gaussian()
  ))[2])
}

bootdist <- boot::boot(
  data = BP_full, statistic = analyseImp3,
  R = 4000, sim = "ordinary", stype = "i",
  parallel = "multicore", ncpus = 8
)

sd <- boot::boot.ci(bootdist, conf = c((1:99) / 100), type = "bca")[4]

A <- sd$bca

qplot((c(A[, 4], A[, 5])), geom = "density", fill = I("#3f8f9b44")) +
  theme_less() +
  labs(
    title = "bCA Bootstrap Density Plot",
    x = "'Group' Parameter Value (Treatment Effect mmHg)",
    y = "Density"
  )

We now focus on profiling on the treatment effect and calculating the profile deviance function. We then multiply our deviance by -0.5 to obtain the profile log-likelihood function.

# Profile Likelihood

prof <- profile(fit1)
disp <- attr(prof, "summary")$dispersion
mindev <- attr(prof, "original.fit")$deviance

dev1 <- prof[[1]]$tau^2
dev2 <- dev1 * disp + mindev

tmpf <- function(x, n) {
  data.frame(
    par = n, tau = x$tau,
    deviance = x$tau^2 * disp + mindev,
    x$par.vals, check.names = FALSE
  )
}

pp <- do.call(rbind, mapply(tmpf, prof, names(prof),
  SIMPLIFY = FALSE
))

pp2 <- melt(pp, id.var = 1:3)
pp3 <- subset(pp2, par == variable, select = -variable)
pp4 <- pp3[-c(1:13), ]

pp4$loglik <- pp4$deviance * -0.5

set linesize 255
clear all
set seed 1031
import delimited "static/datasets/temp.csv", numericcols(2 3 5 6 7 8)
capture which pllf
if _rc == 0 {
foreach var of varlist  group baselinedbp age z r1 r2 {
grstyle init
grstyle color background white, opacity(.1)
quietly pllf glm postdbp group baselinedbp age z r1 r2, profile(`var')
graph copy Graph `var'
}
graph combine  group baselinedbp age z r1 r2
graph export "loglikfunction.svg", replace
}
else {
display as text "Skipping profile-likelihood plots: the -pllf- command is not installed."
}
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> (encoding automatically selected: ISO-8859-2)
#> (7 vars, 500 obs)
#> 
#> 
#> 
#> Skipping profile-likelihood plots: the -pllf- command is not installed.
Lower Limit Upper Limit CI Width Level (%) CDF P-value S-value
-9.67 -9.62 0.057 25.0 0.625 0.750 0.415
-9.71 -9.59 0.120 50.0 0.750 0.500 1.000
-9.75 -9.54 0.205 75.0 0.875 0.250 2.000
-9.76 -9.53 0.228 80.0 0.900 0.200 2.322
-9.77 -9.52 0.256 85.0 0.925 0.150 2.737
-9.79 -9.50 0.293 90.0 0.950 0.100 3.322
-9.82 -9.47 0.349 95.0 0.975 0.050 4.322
-9.85 -9.45 0.399 97.5 0.988 0.025 5.322
-9.88 -9.42 0.459 99.0 0.995 0.010 6.644
Table of Statistics for Various Interval Estimate Percentiles

I then compared the results of the full analysis set to the results from the primary analysis and sensitivity analyses.

#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 11.359786 -9.5274391 .98342353 .32837421 -1.3210281 -.02229379 -.12617403

[concurve]: R: #R Stata: #stata

Examining the Analyses

Evaluation of Imputation Algorithms
PMM Heckman Stata Bayes GAMLSS RI RF True
(Intercept) 11.25 10.12 11.36 NA 13.23 10.37 33.94 9.19
factor(Group)Drug Y -9.51 -9.50 -9.53 NA -9.52 -9.63 -9.45 -9.66
BaselineDBP 0.98 0.99 0.98 NA 0.97 0.99 0.80 1.00
Age 0.33 0.33 0.33 NA 0.32 0.33 0.29 0.33
Z -1.33 -1.09 -1.32 NA -0.62 -1.02 0.02 -2.02
R1 -0.02 -0.02 -0.02 NA -0.02 -0.02 0.02 0.01
R2 -0.14 -0.15 -0.13 NA -0.14 -0.15 -0.02 -0.01
MPE -203.93 -214.28 -171.96 NA -194.38 -224.45 -53.76 0.00
Huber Loss 0.26 0.13 0.28 NA 0.64 0.17 3.69 0.00
MAE 0.44 0.32 0.45 NA 0.83 0.34 3.89 0.00
MSD -0.39 -0.26 -0.41 NA -0.77 -0.29 -3.82 0.00
Metric Abbreviations:
MPE = Mean Percentage Error; MAE = Mean Absolute Error; MSD = Mean Signed Deviation
MI Abbreviations:
PMM = Predictive Mean Matching; Heckman = Heckman Selection Model; RI = Random Indicator; RF = Random Forest; Bayes = Imputation During Model Fitting

We can see that that the overall treatment effect is approximately 9.7-9.9, and for most of the principled methods we used to handle the missing data, we did not obtain similar estimates. For example, both the primary analysis and the controlled multiple imputation methods did not yield the correct treatment effect estimate in the adjusted model.

In fact, nearly all of the methods used except for the Heckman selection model, yielded overestimates of the treatment effect. Even though such analyses can be substantially improved with more specification and more information, the reality is that data that are assumed to be MNAR are difficult to handle and that no method can guarantee the correct answer. Indeed, we knew the missing data model / mechanism, via our custom function

# Missing Data Mechanism

lesslikely <- function() {
  set.seed(seed = 1031,
           kind = "L'Ecuyer-CMRG",
           normal.kind = "Inversion",
           sample.kind = "Rejection"
  )
  logistic <- function(x) exp(x) / (1 + exp(x))
  1 - rbinom(n, 1, logistic(errors[, 2]))
}

Ry <- ifelse(lesslikely() > 0, 1, 0)
PostDBP[Ry == 0] <- NA

for MNAR we set

Thus, we obtain the following model:

Even the random indicator method, which has shown much promise in simulation studies24 and offers the advantage of not having to specify a pattern-mixture or selection model could not recover the correct treatment effect estimate either, showing the difficulty of handling missing data and the limitations of automated approaches.

I hope it is clear from the examples above that it is quite easy to do any analyses with missing data and use a variety of ad-hoc approaches or even principled approaches to explore whether or not the results you’ve obtained are trustworthy or at least even give that impression, but doing a careful and principled one is quite difficult and requires much thought that will take into account real departures from assumptions and whether or not those will lead to substantial changes in results and conclusions/decisions.

Acknowledgements

I’d like to thank Dr. Tim Morris and Dr. Andrew Althouse for helpful comments on an earlier version of this piece. My acknowledgment of their help does not imply endorsement of my views by these colleagues or vice versa, and I remain solely responsible for the views expressed herein. All errors are my own. To point out such errors, please leave a comment below or use the contact form to get in touch.

Statistical Environments

R Environment

#> ─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.6.1 (2026-06-24)
#>  os       macOS Golden Gate 27.0
#>  system   aarch64, darwin25.4.0
#>  ui       unknown
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       America/New_York
#>  date     2026-09-13
#>  pandoc   3.11 @ /opt/homebrew/bin/ (via rmarkdown)
#>  quarto   1.10.18 @ /Applications/quarto/bin/quarto
#> 
#> ─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  package           * version  date (UTC) lib source
#>  Amelia            * 1.8.3    2024-11-08 [1] CRAN (R 4.6.1)
#>  bayesplot         * 1.16.0   2026-08-25 [1] CRAN (R 4.6.1)
#>  blogdown          * 1.24     2026-06-19 [1] CRAN (R 4.6.1)
#>  boot              * 1.3-32   2025-08-29 [1] CRAN (R 4.6.1)
#>  bootImpute        * 1.3.0    2025-12-15 [1] CRAN (R 4.6.1)
#>  brms              * 2.23.0   2025-09-09 [1] CRAN (R 4.6.1)
#>  broom             * 1.0.13   2026-05-14 [1] CRAN (R 4.6.1)
#>  broom.mixed       * 0.2.9.7  2026-02-17 [1] CRAN (R 4.6.1)
#>  Cairo             * 1.7-0    2025-10-29 [1] CRAN (R 4.6.1)
#>  car               * 3.1-5    2026-02-03 [1] CRAN (R 4.6.1)
#>  carData           * 3.0-6    2026-01-30 [1] CRAN (R 4.6.1)
#>  checkmate         * 2.3.4    2026-02-03 [1] CRAN (R 4.6.1)
#>  cli               * 3.6.6    2026-04-09 [1] CRAN (R 4.6.1)
#>  coda              * 0.19-4.1 2024-01-31 [1] CRAN (R 4.6.1)
#>  colorspace        * 2.1-3    2026-07-12 [1] CRAN (R 4.6.1)
#>  concurve          * 3.0.5    2026-09-13 [1] local
#>  cowplot           * 1.2.0    2025-07-07 [1] CRAN (R 4.6.1)
#>  doParallel        * 1.0.17   2022-02-07 [1] CRAN (R 4.6.1)
#>  dplyr             * 1.2.1    2026-04-03 [1] CRAN (R 4.6.1)
#>  forcats           * 1.0.1    2025-09-25 [1] CRAN (R 4.6.1)
#>  foreach           * 1.5.2    2022-02-02 [1] CRAN (R 4.6.1)
#>  futile.logger     * 1.4.9    2025-12-29 [1] CRAN (R 4.6.0)
#>  future            * 1.75.0   2026-07-20 [1] CRAN (R 4.6.1)
#>  future.apply      * 1.20.2   2026-02-20 [1] CRAN (R 4.6.1)
#>  gamlss            * 5.5-0    2025-08-19 [1] CRAN (R 4.6.1)
#>  gamlss.data       * 6.0-7    2025-09-04 [1] CRAN (R 4.6.1)
#>  gamlss.dist       * 6.1-11   2026-09-10 [1] CRAN (R 4.6.1)
#>  ggcorrplot        * 0.3.0    2026-07-24 [1] CRAN (R 4.6.1)
#>  ggplot2           * 4.0.3    2026-04-22 [1] CRAN (R 4.6.1)
#>  ggtext            * 0.2.0    2026-08-28 [1] CRAN (R 4.6.1)
#>  gtsummary         * 2.6.1    2026-09-04 [1] CRAN (R 4.6.1)
#>  here              * 1.0.2    2025-09-15 [1] CRAN (R 4.6.1)
#>  Hmisc             * 5.3-0    2026-09-06 [1] CRAN (R 4.6.1)
#>  htmltools         * 0.5.9    2025-12-04 [1] CRAN (R 4.6.1)
#>  ImputeRobust      * 1.3-1    2018-11-30 [1] CRAN (R 4.6.1)
#>  iterators         * 1.0.14   2022-02-05 [1] CRAN (R 4.6.1)
#>  kableExtra        * 1.4.1    2026-07-08 [1] CRAN (R 4.6.1)
#>  knitr             * 1.52     2026-09-06 [1] CRAN (R 4.6.1)
#>  latex2exp         * 0.9.8    2026-01-09 [1] CRAN (R 4.6.1)
#>  lattice           * 0.23-1   2026-08-12 [1] CRAN (R 4.6.1)
#>  loo               * 2.10.1   2026-07-24 [1] CRAN (R 4.6.1)
#>  lubridate         * 1.9.5    2026-02-04 [1] CRAN (R 4.6.1)
#>  magrittr          * 2.0.5    2026-04-04 [1] CRAN (R 4.6.1)
#>  MASS              * 7.3-66   2026-07-15 [1] CRAN (R 4.6.1)
#>  Matrix            * 1.7-6    2026-07-25 [1] CRAN (R 4.6.1)
#>  MCMCpack          * 1.7-1    2024-08-27 [1] CRAN (R 4.6.1)
#>  mgcv              * 1.9-4    2025-11-07 [1] CRAN (R 4.6.0)
#>  mi                * 1.3.1    2026-07-28 [1] CRAN (R 4.6.1)
#>  mice              * 3.19.0   2025-12-10 [1] CRAN (R 4.6.1)
#>  miceadds          * 3.20-10  2026-05-28 [1] CRAN (R 4.6.1)
#>  miceFast          * 0.9.1    2026-02-26 [1] CRAN (R 4.6.1)
#>  missForest        * 1.6.1    2025-10-26 [1] CRAN (R 4.6.1)
#>  mitml             * 0.4-5    2023-03-08 [1] CRAN (R 4.6.1)
#>  mvtnorm           * 1.4-2    2026-07-12 [1] CRAN (R 4.6.1)
#>  nlme              * 3.1-171  2026-09-01 [1] CRAN (R 4.6.1)
#>  parallelly        * 1.48.0   2026-06-29 [1] CRAN (R 4.6.1)
#>  pbmcapply         * 1.5.1    2022-04-28 [1] CRAN (R 4.6.1)
#>  performance       * 0.18.2   2026-09-10 [1] CRAN (R 4.6.1)
#>  posterior         * 1.7.0    2026-04-01 [1] CRAN (R 4.6.1)
#>  ProfileLikelihood * 1.3      2023-08-25 [1] CRAN (R 4.6.1)
#>  progress          * 1.2.3    2023-12-06 [1] CRAN (R 4.6.1)
#>  purrr             * 1.2.2    2026-04-10 [1] CRAN (R 4.6.1)
#>  qqplotr           * 0.0.7    2025-09-05 [1] CRAN (R 4.6.1)
#>  quantreg          * 6.1      2025-03-10 [1] CRAN (R 4.6.1)
#>  randomForest      * 4.7-1.2  2024-09-22 [1] CRAN (R 4.6.1)
#>  Rcpp              * 1.1.2    2026-07-05 [1] CRAN (R 4.6.1)
#>  readr             * 2.2.0    2026-02-19 [1] CRAN (R 4.6.1)
#>  reshape2          * 1.4.5    2025-11-12 [1] CRAN (R 4.6.1)
#>  reticulate        * 1.47.0   2026-09-03 [1] CRAN (R 4.6.1)
#>  rmarkdown         * 2.32     2026-09-01 [1] CRAN (R 4.6.1)
#>  rms               * 8.1-1    2026-02-18 [1] CRAN (R 4.6.1)
#>  rstan             * 2.32.7   2025-03-10 [1] CRAN (R 4.6.1)
#>  showtext          * 0.9-8    2026-03-21 [1] CRAN (R 4.6.1)
#>  showtextdb        * 3.0      2020-06-04 [1] CRAN (R 4.6.1)
#>  SparseM           * 1.84-2   2024-07-17 [1] CRAN (R 4.6.1)
#>  StanHeaders       * 2.39.1   2026-09-02 [1] CRAN (R 4.6.1)
#>  Statamarkdown     * 1.0.0    2026-08-21 [1] CRAN (R 4.6.1)
#>  stringr           * 1.6.0    2025-11-04 [1] CRAN (R 4.6.1)
#>  svglite           * 2.2.2    2025-10-21 [1] CRAN (R 4.6.1)
#>  sysfonts          * 0.8.9    2024-03-02 [1] CRAN (R 4.6.1)
#>  texPreview        * 2.1.0    2024-01-24 [1] CRAN (R 4.6.0)
#>  tibble            * 3.3.1    2026-01-11 [1] CRAN (R 4.6.1)
#>  tidybayes         * 3.0.7    2024-09-15 [1] CRAN (R 4.6.1)
#>  tidyr             * 1.3.2    2025-12-19 [1] CRAN (R 4.6.1)
#>  tidyverse         * 2.0.0    2023-02-22 [1] CRAN (R 4.6.1)
#>  tinytex           * 0.60     2026-06-16 [1] CRAN (R 4.6.1)
#>  tryCatchLog       * 1.3.3    2025-12-04 [1] CRAN (R 4.6.0)
#>  VIM               * 7.3.1    2026-09-11 [1] CRAN (R 4.6.1)
#>  wesanderson       * 0.3.7    2023-10-31 [1] CRAN (R 4.6.1)
#>  xfun              * 0.60     2026-07-09 [1] CRAN (R 4.6.1)
#>  xtable            * 1.8-8    2026-02-22 [1] CRAN (R 4.6.1)
#>  yardstick         * 1.4.0    2026-04-07 [1] CRAN (R 4.6.1)
#> 
#>  [1] /opt/homebrew/lib/R/4.6/site-library
#>  [2] /opt/homebrew/Cellar/r/4.6.1/lib/R/library
#>  * ── Packages attached to the search path.
#> 
#> ─ External software ────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting        value
#>  cairo          1.18.4
#>  cairoFT        2.14.3/2.18.1
#>  pango
#>  png
#>  jpeg
#>  tiff
#>  tcl            9.0
#>  curl           8.7.1
#>  zlib           1.2.12
#>  bzlib          1.0.8, 13-Jul-2019
#>  xz             5.8.3
#>  deflate
#>  zstd           1.5.7
#>  PCRE           10.48 2026-08-31
#>  ICU            78.1
#>  TRE            TRE 0.8.0 R_fixes (BSD)
#>  iconv          Apple or GNU libiconv 1.11 /usr/lib/libiconv.2.dylib
#>  readline       8.3
#>  BLAS           /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib
#>  lapack         /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib
#>  lapack_version 3.12.1
#> 
#> ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> Quarto: 1.10.18
#> pandoc: 3.11
#> knitr engines: awk, bash, coffee, gawk, groovy, haskell, lein, mysql, node, octave, perl, php, psql, Rscript, ruby, sas, scala, sed, sh, stata, zsh, asis, asy, block, block2, bslib, c, cat, cc, comment, css, ditaa, dot, embed, eviews, exec, fortran, fortran95, go, highlight, js, julia, python, R, Rcpp, sass, scss, sql, stan, targets, tikz, verbatim, ojs, mermaid, glue, glue_sql, gluesql, details, texpreview

Stata Environment

set linesize 255
display %tcMonth_dd,CCYY_hh:MM_am now()
about
ado dir
#> file ()print() not found
#> r(601);
#> 
#> 
#> 
#>  September 13,2026 4:18 am
#> 
#> 
#> StataNow/MP 19.5 for Mac (Apple Silicon)
#> Revision 12 Aug 2026
#> Copyright 1985-2025 StataCorp LLC
#> 
#> Total physical memory: 48.01 GB
#> 
#> Stata license: Single-user 2-core , expiring  6 Feb 2027
#> Serial number: 501909358563
#>   Licensed to: Zad Rafi
#>                
#> 
#> 
#> [1] package parallel from http://fmwww.bc.edu/repec/bocode/p
#>       'PARALLEL': module for Parallel Computing
#> 
#> [2] package svalue from http://fmwww.bc.edu/RePEc/bocode/s
#>       'SVALUE': module for computing and graphically displaying S-values against their respective P-values
#> 
#> [3] package cifunction from http://fmwww.bc.edu/RePEc/bocode/c
#>       'CIFUNCTION': module for computing and graphically displaying all possible confidence intervals around a point estimate
#> 
#> [4] package getregstats from http://fmwww.bc.edu/RePEc/bocode/g
#>       'GETREGSTATS': module for computing all values in a regression table when only the coefficient and one other statistic is available
#> 
#> [5] package gr0072_2 from http://www.stata-journal.com/software/sj25-4
#>       SJ25-4 gr0072_2. Update: Nice axis labels for ...
#> 
#> [6] package grstyle from http://fmwww.bc.edu/repec/bocode/g
#>       'GRSTYLE': module to customize the overall look of graphs
#> 
#> [7] package palettes from http://fmwww.bc.edu/repec/bocode/p
#>       'PALETTES': module to provide color palettes, symbol palettes, and line pattern palettes

References

Back to top

References

1. Stark PB, Saltelli A. (2018). “Cargo-cult statistics and scientific crisis.” Significance. 15:40–43. doi: 10.1111/j.1740-9713.2018.01174.x.
2. Gigerenzer G. (2018). “Statistical Rituals: The Replication Delusion and How We Got There.” Advances in Methods and Practices in Psychological Science. 1:198–218. doi: 10.1177/2515245918771329.
3. Greenland S. (2017). “Invited commentary: The need for cognitive science in methodology.” American Journal of Epidemiology. 186:639–645. doi: 10.1093/aje/kwx259.
4. Greenland S. (2005). “Multiple-bias modelling for analysis of observational data.” Journal of the Royal Statistical Society Series A (Statistics in Society). 168:267–306. doi: 10.1111/j.1467-985X.2004.00349.x.
5. Greenland S, Lash TL. (2008). “Bias analysis.” In: Rothman KJ, Greenland S, Lash TL, editors. Modern Epidemiology. 3rd edition. Lippincott Williams & Wilkins. p. 345–380.
6. Lash TL, Fox MP, Fink AK. (2009). “Applying Quantitative Bias Analysis to Epidemiologic Data.” Springer New York.
7. Lash TL, Fox MP, MacLehose RF, Maldonado G, McCandless LC, Greenland S. (2014). “Good practices for quantitative bias analysis.” International Journal of Epidemiology. 43:1969–1985. doi: 10.1093/ije/dyu149.
8. Lash TL, Ahern TP, Collin LJ, Fox MP, MacLehose RF. (2021). “Bias Analysis Gone Bad.” American Journal of Epidemiology. doi: 10.1093/aje/kwab072.
9. Gustafson P. (2021). “Invited Commentary: Toward Better Bias Analysis.” American Journal of Epidemiology. doi: 10.1093/aje/kwab068.
10. Greenland S. (2021). “Dealing with the Inevitable Deficiencies of Bias Analysis and All Analyses.” American Journal of Epidemiology. doi: 10.1093/aje/kwab069.
11. Molenberghs G, Fitzmaurice G, Kenward MG, Tsiatis A, Verbeke G. (2014). “Handbook of Missing Data Methodology.” CRC Press.
12. Scott RaP. (2002). “The Multicentre Aneurysm Screening Study (MASS) into the effect of abdominal aortic aneurysm screening on mortality in men: A randomised controlled trial.” The Lancet. 360:1531–1539. doi: 10.1016/S0140-6736(02)11522-4.
13. Morris TP, Kahan BC, White IR. (2014). “Choosing sensitivity analyses for randomised trials: principles.” BMC Medical Research Methodology. 14:11. doi: 10.1186/1471-2288-14-11.
14. Mosteller F, Tukey JW. (1987). “Data analysis including statistics.” In: The Collected Works of John W. Tukey: Philosophy and Principles of Data Analysis 1965-1986. CRC Press.
15. Akacha M, Bretz F, Ohlssen D, Rosenkranz G, Schmidli H. (2017). “Estimands and Their Role in Clinical Trials.” Statistics in Biopharmaceutical Research. 9:268–271. doi: 10.1080/19466315.2017.1302358.
16. Permutt T. (2020). “Do Covariates Change the Estimand?” Statistics in Biopharmaceutical Research. 12:45–53. doi: 10.1080/19466315.2019.1647874.
17. Mitroiu M, Oude Rengerink K, Teerenstra S, Pétavy F, Roes KCB. (2020). “A narrative review of estimands in drug development and regulatory evaluation: Old wine in new barrels?” Trials. 21:671. doi: 10.1186/s13063-020-04546-1.
18. Little RJ, D’Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, et al. (2012). “The Prevention and Treatment of Missing Data in Clinical Trials.” New England Journal of Medicine. 367:1355–1360. doi: 10.1056/NEJMsr1203730.
19. Chen D-G(Din), Peace KE, Zhang P. (2017). “Clinical trial data analysis using r and SAS.” 2nd edition. Chapman; Hall/CRC. doi: 10.1201/9781315155104.
20. Buuren S van. (2018). “Flexible Imputation of Missing Data, Second Edition.” CRC Press.
21. Meng X-L, Rubin DB. (1992). “Performing Likelihood Ratio Tests with Multiply-Imputed Data Sets.” Biometrika. 79:103–111. doi: 10.2307/2337151.
22. Schenker N, Taylor J. (1996). “Partially parametric techniques for multiple imputation.” Computational Statistics & Data Analysis. 22:425–446. doi: 10.1016/0167-9473(95)00057-7.
23. Permutt T. (2016). “Sensitivity analysis for missing data in regulatory submissions.” Statistics in Medicine. 35:2876–2879. doi: 10.1002/sim.6753.
24. Jolani S. (2012). “Dual Imputation Strategies for Analyzing Incomplete Data.”
25. von Hippel PT, Bartlett J. (2019). “Maximum likelihood multiple imputation: Faster imputations and consistent standard errors without posterior draws.” arXiv:12100870 [stat]. https://arxiv.org/abs/1210.0870.
26. Rafi Z, Greenland S. (2020). “Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise.” BMC Medical Research Methodology. 20:244. doi: 10.1186/s12874-020-01105-9.
27. Greenland S, Rafi Z. (2020). “To Aid Scientific Inference, Emphasize Unconditional Descriptions of Statistics.” arXiv:190908583 [statME]. https://arxiv.org/abs/1909.08583.

Citation

BibTeX citation:
@online{rafi2020,
  author = {Rafi, Zad and Panda, Sir},
  title = {What {Makes} a {Sensitivity} {Analysis?}},
  date = {2020-12-12},
  url = {https://lesslikely.com/statistics/sensitivity.html},
  langid = {en-US}
}
]]> statistics https://lesslikely.com/statistics/sensitivity.html Sat, 12 Dec 2020 05:00:00 GMT Medicine Is Being Treated with Snake-Oil Statistics Zad Rafi Andrew Gelman Aleksi Reito Sander Greenland https://lesslikely.com/statistics/snakeoilstats.html Suggested Running Head: Snake Oil Statistics

Statistics has helped medicine move away from an eminence-based framework, where subject-matter experts decided what worked and what didn’t, towards an evidence-based one. Among the techniques deployed, null-hypothesis statistical testing (NHST) is the most common, where the word “null” is invariably taken by users to mean that the only hypotheses tested are those of “no association” or “no effect” (sometimes labeled “nil hypotheses”). Despite objections to them extending throughout the past century1, these tests are supposed to serve as a safeguard against researchers fooling themselves and others, as well as serving as a central component of experimental design and analysis.

Unfortunately, these tests are easily subverted into playing the reverse role of providing a badge of approval, allowing researchers to make stronger claims than warranted by valid statistical analyses. Confidence intervals have been extensively promoted to address this problem, but they too have been subverted by being treated as if they are only testing null hypotheses. The consequences for evidence-based medicine have been dire.

There are three commonly stated principles (shown in Figure 1) of evidence-based medicine2:

  1. reliance on statistically significant results (and thus NHST) from randomized controlled trials,
  2. balancing of costs, benefits, and uncertainties in decision making, and
  3. combining clinical expertise with external evidence to tailor treatments for individuals.

Figure 1

Unfortunately, the use of NHST can get in the way of the movement toward an evidence-based framework. This may sound paradoxical, given that one of the foundations of evidence-based medicine is hypothesis testing based on randomized controlled trials, deemed by many to be the most reliable forms of evidence2. One problem is that reliance on statistical significance (principle 1) may conflict with the other principles: In conflict with balancing costs and benefits under uncertainty (principle 2), statistical significance or non-significance is typically used to replace uncertainty with certainty3, 4 — indeed, researchers are encouraged to do this and it is even forced by some journals5.

Reliance on statistical significance can interact badly with expertise and background evidence (principle 3), often leading to incoherent attempts at resolution. For example, researchers may do separate analyses for men and women, often with no biological basis for expecting other than small differences in effects (if any). In doing so they often find that one group (usually the larger one, which is usually men) show a “significant” effect while the other does not. They then misreport this difference in “significance” as if it represented a significant difference in effect between groups when in fact it only reflects a difference in the group sizes, plus random differences in group-specific P-values. Such misinterpretations fool researchers into thinking that the data support targeting treatments at the group showing “significance” and mislead clinicians into tailoring treatment plans for individuals based on nothing more than random variation.

The unfortunate reality is that estimating effects for individuals or population subsets requires far more sophistication than basic testing procedures6. It can take several times more patients to estimate variation in effects (“interactions”) than average effects7. Given that few studies are large enough to estimate main effects of interest, it will typically be impossible to obtain reliable estimates of effect variation even if the study is otherwise flawless. That problem should be dealt with under principle (2) by recognizing that subgroups will have very imprecise estimates, leading to enormous uncertainties about who if anyone should be targeted for or excluded from treatment. Unfortunately, the prevalent misunderstandings of classical statistics have trained most researchers, editors, and reviewers to demand statistical significance and certainty as a prerequisite for publication8 and decision making. Applying those demands within subgroups all but guarantees that distorted impressions of patient-specific effects will follow.

Through neglect of basic education, the statistics profession is partly responsible for these issues, but there is nothing new about medicine’s desire for certainty. That desire is part of human nature and thus existed long before the adoption of statistical methods. The resulting demands for certainty in reported results have provided ample opportunities for overconfident researchers to rise to prominence. Physicians have long capitalized on several opportunities with their sophisticated knowledge of physiology and anatomy, and used their medical authority to argue what treatments worked, shape public policy, and design clinical guidelines, maintaining a form of medicine characterized as “eminence based.”

The landscape changed when advances in quantitative methods eventually reached medicine9, leading to a newfound demand for scientific rigor and making many individuals wary of the claims of subject-matter experts10. These demands may be some of the biggest contributors to medicine’s adoption of statistical methods and its movement towards an evidence-based framework. Unfortunately, in its pursuit for objectivity, medicine was sold snake oil statistics and as a result, its desire for objectivity and rigor backfired.

Figure 2: Snake oil statistics. Methodological interventions that have been oversold by early adopters as cure alls despite no supporting evidence of utility in a particular application.

To see the reach of these problems, one simply needs to look wherever a new drug has been considered effective only if it has been shown statistically significantly better than a control in one or more randomized clinical trials (a standard that applies to FDA new-drug approvals, though not to medical devices, which can be cleared through other pathways). The largest enforcers of these methods have been regulatory agencies that wish to minimize treatments that do not work (false positives) and minimize adverse events from treatments. One should ask: what specifically convinced these agencies that statistical significance and in particular NHST was the best analysis criterion to achieve these goals?

In a nonexistent ideal world, the regulatory agencies looked at several statistical methods, tested each of them in many settings to see how well they identified true benefits and harms while avoiding false conclusions, and from that determined NHST performed best out of all options – with periodic updates as new methods appeared. Of course, history shows something else entirely, even indicating that adoption of NHST was mainly a result of political desperation11, 12. To understand the harsh reality, we must go back to the mid-20th century, when pharmaceutical companies submitted new drug applications to the FDA that often lacked any protocols and statistical analysis plans, making the entire drug approval process chaotic. By the end of the 1960s, the FDA had become desperate to standardize the drug-approval process and make it scientifically rigorous, especially with impending pressures from the Drug Amendments of 1962, which demanded rigorous evidence for drug approval.

At the same time, applied researchers were looking for rigorous ways to summarize experiments with numerical quantities. Their search struck gold with the works of the prominent statisticians, Ronald A. Fisher13–15, Jerzy Neyman16, and Egon Pearson17, who created and popularized powerful statistical tools for researchers that lacked statistical training. Eventually, researchers across many disciplines adopted NHST and the now-infamous 0.05 cutoff. Soon, the FDA followed and incorporated these methods into its regulatory process, without any formal debate about the methods’ utility or evidence12.

Figure 3: Austin Bradford Hill. The statistician and epidemiologist who conducted the first randomized controlled trial in medicine.

It thus appears that embracement of the NHST paradigm arose from expediently following the ascendant trend in research rather than a critical evaluation of various options emerging in the same period. Since its adoption into the regulatory process, this framework has been rigorously enforced by the FDA as the gold standard in the approval process.

We find it ironic that the gatekeepers of evidence-based medicine — regulatory agencies, journal editors and reviewers, and medical researchers – continue to insist on enforcement of a statistical framework that was never critically examined for its utility in medicine12. This failure makes it understandable why it can be credibly argued that conventional statistical methods have done more harm than good in medical science. These methods, including null-hypothesis tests and so-called “confidence” intervals are a set of decision-making tools designed for tightly controlled randomized experiments in which the treatment effects (if any) can be distinguished from all other causal effects, and can be distinguished from random error (noise) by increasing the study size in an affordable manner. Examples of such scenarios arise in agricultural research where the experimental units are plants or plots, and industrial quality control, where the experimental unit is a part or product. The tools were originally developed for these environments, in which (compared to clinical studies) experimenters have almost godlike control over the selection of units and their subsequent experiences, aided by having a rather short follow-up period and low cost per experimental unit. And then, the decisions to be made are relatively simple and easily monitored, e.g., change a fertilizer formulation or manufacturing tolerance17. Experimental psychology is similar in terms of control, cost, and even lower cost-benefit consequences.

In clinical environments, however, the cost per experimental unit (the patient) is far higher, creating severe limits on study size and thus noise reduction, and the costs and benefits of decisions can be enormous. At the same time, there is far less control of extraneous selection and confounding effects: Physicians and their patients can and do selectively refuse to participate or cease to adhere to assigned treatment protocols, and may drop out for unknown reasons. Meanwhile, direct physical control of the patient environment is extremely limited or nonexistent, especially when follow-up extends beyond hospital stay. And then, amplifying these limits, the required decisions are often complex and of highly uncertain consequence, yet may be pivotal to the experimental results and clinical decisions, e.g., when to withdraw treatment from patients apparently experiencing side effects or when to switch treatments for nonresponsive patients.

These vast differences haven’t stopped medical researchers from using conventional methods as if they were operating in a tightly controlled environment, treating ambiguous trial results as if they supported decisive conclusions despite obvious uncertainties and potentially devastating consequences. The usual depictions of this problem involves researchers trying to “game” statistical methods so that they show “significant” effects18, 19; while such “significance questing” is a real problem, warnings about it usually ignore or dismiss opposite behavior in which showing “nonsignificance” will facilitate publication in prestigious medical journals, especially where so-called “replication failure” has become a hot topic3. Adding to these distortions is the publication bias that results when researchers or editors deem results unworthy of submission or acceptance because they are just not interesting because they fail to report a “discovery” or only replicate what is “known.” This desire for novelty or publicity, along with demands for certainty, has distorted the medical literature with dubious, inflated effects20 and misleading claims of replication failure based on fundamentally ambiguous results21, 22.

Responses to the problems

In response to the ongoing abuse of statistical testing, the American Statistical Association released a statement in 2016 cautioning against the fixating on P-values and statistical significance23. Three years later, the organization published an issue titled “A World Beyond P < 0.05” with 43 commentaries from statistical experts on how to improve statistical inference, with or without P-values24. The issue was accompanied by a highly discussed commentary in Nature titled “Scientists Rise up Against Statistical Significance” that discouraged mindless automation and dichotomization of statistical results21, and was supported by signatures of some 800 applied researchers.

While these calls are impressive and a growing number of journals — including the New England Journal of Medicine and, in modified form, JAMA — have updated their statistical-reporting policies, most regulatory agencies and the bulk of the medical literature continue to fixate on statistical significance5, 25. Correspondingly, we can expect medical researchers to keep cutting corners to achieve or remove statistical significance18, 19 or misinterpret ambiguous results as if definitive, a practice that is often labeled as “spin”26.

We can interpret the persistence of null hypothesis significance testing (NHST) in two ways. One story is that the value of NHST is recognized by real-world decision makers, despite the carping of ivory-tower critics such as the authors of the present article. The other story is that the counterproductive nature of NHST has been denied by the medical establishment5, 25, and that better alternatives are available12. There are legitimate practical concerns behind both these perspectives. On one hand, active researchers and regulators have legitimate concerns about working on or approving treatments that do not work. On the other hand, many well-publicized examples have made it clear that NHST can easily lead to overconfidence and erroneous inferences, as seen in discussions that treat statistically significance as demonstrating presence of effects and nonsignificance as demonstrating absence of effects.

Methodologists propose new methods to address NHST in their field

Fear of false positives has dominated most discussions of scientific rigor18, 27, 28, often relegating false negatives to more technical discussions of power. Although false positives can be costly, so can missing clinically meaningful effects21. Take postmarketing surveillance for pharmaceuticals; in such situations, serious adverse events from drugs are often underreported and have in some cases been actively suppressed29, 30, as seen in Cochrane reviews of unpublished trial data, leading to a scarcity of data and resulting in studies that will never be able to show a “significant” effect because of the lack of resources and time. If a study comparing adverse events from those taking an approved drug and some control group is unable to show statistical significance, even when the estimated effect is important and plausible, the results will typically be confused and used for evidence of absence21, 31, and prescribing is unlikely to be curtailed, leading to continued harm.

The confusion of statistical nonsignificance with evidence of absence will remain a problem in fields where effects are often small and yet studies powered to detect them are infeasible. Most studies in surgical science rarely have more than a dozen participants. Again, this makes it incredibly difficult to achieve statistical significance, even when clinically important effects are likely. In such fields, adopting the NHST framework for analysis all but guarantees failure to detect those effects. As a result, researchers have looked to other methods — including Bayes factors, second-generation P-values, and post-hoc power calculations — to circumvent the bad hand that was dealt to them32–34. Unfortunately, such transitions are not always helpful: while reasonably defensible alternatives such as interval testing and posterior intervals do exist, several of the most-publicized proposals introduce their own errors, and in fact, some may actually result in more errors than before.

For example, a group of surgeons recently published several statistical recommendations on what to do if a surgical study result was nonsignificant35–37, providing hope to researchers who have had their inquiries halted by that result. But the recommendations are not only statistically invalid and clinically misleading, and thus have been discouraged by statisticians for nearly two decades38, 39. In a similar tale, two sports scientists published a statistical method40 which was supposed to improve the classification of individual treatment responses by reducing the influence of random error in small studies. Unfortunately, reducing the influence of random error in a valid manner requires improvement of study design, including increase in study size, so unsurprisingly the proposed method has unacceptably poor statistical properties41.

The efforts behind new methods and recommendations are a response to an arbitrary dichotomy that has been imposed upon researchers by the medical and scientific establishment: decide whether a result is “positive” or “negative”, with no allowance for what is usually the most reasonable interpretation – ambiguous or indecisive. Unfortunately, conventional statistics has aggravated this problem by offering methods like NHST which produce only dichotomous answers. These methods which swept research sciences in the mid-20th century and are now firmly rooted in tradition, as if that tradition is the best we can do. But it isn’t.

The problem with upending this tradition however is that there is no consensus about what to replace it with, a problem that only grows as alternatives continue to proliferate. A idealistic (and we think naïve) view would simply allow authors to choose alternatives of their liking. The problem with this anarchic approach is that many of the alternatives are themselves misleading and even defective or at best inferior to other methods in demonstrable ways. Yet seeing these problems requires not only sufficient technical expertise to evaluate methods, but also a willingness to see them – a willingness that should not be assumed for originators and adopters of the method. Unfortunately, mere publication of a method in a peer-reviewed journal is a faulty indicator of method validity. That is especially so if publication is not in a statistics journal, for in that case it means only that peer-review was by referees who may have had little of the special technical expertise needed for thorough evaluation.

Solutions

There are no simple solutions or universal guidelines that medical researchers can always use to improve scientific rigor within their area of work. We nonetheless offer some recommendations and resources that we believe may be useful to those who recognize problems in their field and wish for some sort of guidance:

  1. Collaborate with well-qualified statisticians to design and conduct studies that are rigorous, efficient, and cost effective42, 43. Look to these statistical collaborators for guidance about honestly reporting uncertainty, not certainty19.
  2. If possible, pool resources to run larger studies that are likely to be more precise and informative than individual studies, which may simply waste resources and offer little yield44. However, more data is not synonymous with more information: Increasing study size may be detrimental if it entails a reduction in data quality45.
  3. Aim to be more descriptive and less inferential22. Accept uncertainty and the anxiety that comes with it4, along with the idea that no one study warrants conclusions about the true nature of a phenomenon, a delusion that is not even true in particle physics! And if a phenomenon is subtle, even several studies may be insufficient for valid conclusions beyond “more research is needed.”
  4. When making real world decisions, use all information available to you, balancing costs, benefits, and uncertainties46, rather than basing decisions on whether a single numerical value is above or below an arbitrary cutoff.
  5. Be mindful of cognitive biases that may distort your conclusions throughout the study and the many cognitive biases that will afflict you, your colleagues, and your collaborators3, 47, 48.
  6. Use statistical methods that have been reviewed and validated by the statistical community beyond their developers and promoters. That validation is provided not only by publication in statistical journals, but also by applications that can be judged as having reached sound, well-cautioned conclusions in context. Especially, beware of any method that (like NHST) claims to offer firm conclusions based on purely numeric comparisons.
Back to top

References

1. Boring EG. (1919). “Mathematical vs. Scientific significance.” Psychological Bulletin. 16:335–338. doi: 10.1037/h0074554.
2. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. (1996). “Evidence based medicine: What it is and what it isn’t.” BMJ (Clinical research ed). 312:71–72. doi: 10.1136/bmj.312.7023.71.
3. Greenland S. (2017). “Invited commentary: The need for cognitive science in methodology.” American Journal of Epidemiology. 186:639–645. doi: 10.1093/aje/kwx259.
4. McShane BB, Gal D, Gelman A, Robert C, Tackett JL. (2019). “Abandon Statistical Significance.” The American Statistician. 73:235–245. doi: 10.1080/00031305.2018.1527253.
5. Bauchner H, Golub RM, Fontanarosa PB. (2019). “Reporting and Interpretation of Randomized Clinical Trials.” Journal of the American Medical Association. 322:732–735. doi: 10.1001/jama.2019.12056.
6. Senn S. (2018). “Statistical pitfalls of personalized medicine.” Nature. 563:619–621. doi: 10.1038/d41586-018-07535-2.
7. Greenland S. (1983). “Tests for interaction in epidemiologic studies: A review and a study of power.” Statistics in Medicine. 2:243–251. doi: 10.1002/sim.4780020219.
8. Rosenthal R. (1979). “The file drawer problem and tolerance for null results.” Psychol Bull. 86:638. doi: 10.1037/0033-2909.86.3.638.
9. Council MR. (1948). “Streptomycin Treatment of Pulmonary Tuberculosis.” British Medical Journal. 2:769–782.
10. Goodman SN. (2019). “Why is Getting Rid of P-Values So Hard? Musings on Science and Statistics.” The American Statistician. 73:26–30. doi: 10.1080/00031305.2018.1558111.
11. Kennedy-Shaffer L. (2017). “When the Alpha is the Omega: P-Values, ‘Substantial Evidence,’ and the 0.05 Standard at FDA.” Food Drug Law J. 72:595–635.
12. Ruberg SJ, Jr FEH, Gamalo-Siebers M, LaVange L, Lee JJ, Price K, et al. (2019). “Inference and Decision Making for 21st-Century Drug Development and Approval.” The American Statistician. 73:319–327. doi: 10.1080/00031305.2019.1566091.
13. Fisher R. (1955). “Statistical Methods and Scientific Induction.” Journal of the Royal Statistical Society Series B (Methodological). 17:69–78. doi: 10.1111/j.2517-6161.1955.tb00180.x.
14. Fisher RA. (1925). “Statistical Methods for Research Workers.” Oliver and Boyd: Edinburgh.
15. Fisher RA. (1935). “The Design of Experiments.” Oxford, England: Oliver & Boyd.
16. Neyman J, Pearson ES. (1933). “On the Problem of the Most Efficient Tests of Statistical Hypotheses.” Philosophical Transactions of the Royal Society of London Series A, Containing Papers of a Mathematical or Physical Character. 231:289–337. doi: 10.1098/rsta.1933.0009.
17. Pearson ES. (1933). “A Survey of the Uses of Statistical Method in the Control and Standardization of the Quality of Manufactured Products.” Journal of the Royal Statistical Society. 96:21–75. doi: 10.2307/2341869.
18. Simmons JP, Nelson LD, Simonsohn U. (2011). “False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.” Psychological Science. 22:1359–1366. doi: 10.1177/0956797611417632.
19. Wang MQ, Yan AF, Katz RV. (2018). “Researcher requests for inappropriate analysis and reporting: A U.S. Survey of consulting biostatisticians.” Annals of Internal Medicine. 169:554. doi: 10.7326/M18-1230.
20. Dickersin K, Berlin JA. (1992). “Meta-analysis: State-of-the-Science.” Epidemiologic Reviews. 14:154–176. doi: 10.1093/oxfordjournals.epirev.a036084.
21. Amrhein V, Greenland S, McShane B. (2019). “Scientists rise up against statistical significance.” Nature. 567:305. doi: 10.1038/d41586-019-00857-9.
22. Amrhein V, Trafimow D, Greenland S. (2019). “Inferential statistics as descriptive statistics: There is no replication crisis if we don’t expect replication.” The American Statistician. 73:262–270. doi: 10.1080/00031305.2018.1543137.
23. Wasserstein RL, Lazar NA. (2016). “The ASA Statement on p-Values: Context, Process, and Purpose.” The American Statistician. 70:129–133. doi: 10.1080/00031305.2016.1154108.
24. Wasserstein RL, Schirm AL, Lazar NA. (2019). “Moving to a world beyond ‘p 0.05’.” The American Statistician. 73:1–19. doi: 10.1080/00031305.2019.1583913.
25. Harrington D, D’Agostino RB, Gatsonis C, Hogan JW, Hunter DJ, Normand S-LT, et al. (2019). “New guidelines for statistical reporting in the journal.” New England Journal of Medicine. 381:285–286. doi: 10.1056/NEJMe1906559.
26. Khan MS, Lateef N, Siddiqi TJ, Rehman KA, Alnaimat S, Khan SU, et al. (2019). “Level and Prevalence of Spin in Published Cardiovascular Randomized Clinical Trial Reports With Statistically Nonsignificant Primary Outcomes: A Systematic Review.” JAMA Network Open. 2:e192622–e192622. doi: 10.1001/jamanetworkopen.2019.2622.
27. Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers E, Berk R, et al. (2017). “Redefine statistical significance.” Nature Human Behaviour. 2:6–10. doi: 10.1038/s41562-017-0189-z.
28. Ioannidis JPA. (2018). “The Proposal to Lower P Value Thresholds to .005.” JAMA. 319:1429–1430. doi: 10.1001/jama.2018.1536.
29. Vandenbroucke JP, Psaty BM. (2008). “Benefits and risks of drug treatments: How to combine the best evidence on benefits with the best data about adverse effects.” JAMA. 300:2417–2419. doi: 10.1001/jama.2008.723. https://doi.org/10.1001/jama.2008.723.
30. Doshi P, Jefferson T, Del Mar C. (2012). “The imperative to share clinical study reports: Recommendations from the Tamiflu experience.” PLoS Medicine. 9:e1001201. doi: 10.1371/journal.pmed.1001201. https://doi.org/10.1371/journal.pmed.1001201.
31. Altman DG, Bland JM. (1995). “Absence of evidence is not evidence of absence.” BMJ (Clinical research ed). 311:485. doi: 10.1136/bmj.311.7003.485.
32. Gigerenzer G, Marewski JN. (2015). “Surrogate Science: The Idol of a Universal Method for Scientific Inference.” Journal of Management. 41:421–440. doi: 10.1177/0149206314547522.
33. van Ravenzwaaij D, Monden R, Tendeiro JN, Ioannidis JPA. (2019). “Bayes factors for superiority, non-inferiority, and equivalence designs.” BMC Medical Research Methodology. 19:71. doi: 10.1186/s12874-019-0699-7.
34. Simonsohn U, Nelson LD, Simmons JP. (2014). “P-Curve and Effect Size: Correcting for Publication Bias Using Only Significant Results.” Perspect Psychol Sci. 9:666–681. doi: 10.1177/1745691614553988.
35. Bababekov YJ, Stapleton SM, Mueller JL, Fong ZV, Chang DC. (2018). “A Proposal to Mitigate the Consequences of Type 2 Error in Surgical Science.” Annals of Surgery. 267:621. doi: 10.1097/SLA.0000000000002547.
36. Bababekov YJ, Hung Y-C, Hsu Y-T, Udelsman BV, Mueller JL, Lin H-Y, et al. (2019). “Is the Power Threshold of 0.8 Applicable to Surgical Science? The Underpowered Study.” Journal of Surgical Research. 241:235–239. doi: 10.1016/j.jss.2019.03.062.
37. Bababekov YJ, Chang DC. (2019). “Post Hoc Power: A Surgeon’s First Assistant in Interpreting ‘Negative’ Studies.” Annals of Surgery. doi: 10.1097/SLA.0000000000002914.
38. Greenland S. (2012). “Nonsignificance plus high power does not imply support for the null over the alternative.” Annals of Epidemiology. 22:364–368. doi: 10.1016/j.annepidem.2012.02.007.
39. Hoenig JM, Heisey DM. (2001). “The Abuse of Power.” The American Statistician. 55:19–24. doi: 10.1198/000313001300339897.
40. Dankel SJ, Loenneke JP. (2019). “A Method to Stop Analyzing Random Error and Start Analyzing Differential Responders to Exercise.” Sports Medicine. doi: 10.1007/s40279-019-01147-0.
41. Tenan M, Vigotsky AD, Caldwell AR. (2019). “On the Statistical Properties of the Dankel-Loenneke Method.” SportRxiv. doi: 10.31236/osf.io/8ndhg.
42. Gelman A, Carlin J. (2014). “Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors.” Perspectives on Psychological Science. 9:641–651. doi: 10.1177/1745691614551642.
43. Rothman KJ, Greenland S. (2018). “Planning study size based on precision rather than power.” Epidemiology. 29:599–603. doi: 10.1097/EDE.0000000000000876.
44. Moshontz H, Campbell L, Ebersole CR, IJzerman H, Urry HL, Forscher PS, et al. (2018). “The Psychological Science Accelerator: Advancing Psychology Through a Distributed Collaborative Network:” Advances in Methods and Practices in Psychological Science. doi: 10.1177/2515245918797607.
45. Cox DR, Donnelly CA. (2011). “Principles of Applied Statistics.” Cambridge University Press.
46. Parmigiani G, Inoue L. (2009). “Decision Theory: Principles and Approaches.” John Wiley & Sons.
47. Gigerenzer G. (2004). “Mindless statistics.” The Journal of Socio-Economics. 33:587–606. doi: 10.1016/j.socec.2004.09.033.
48. Stark PB, Saltelli A. (2018). “Cargo-cult statistics and scientific crisis.” Significance. 15:40–43. doi: 10.1111/j.1740-9713.2018.01174.x.

Citation

BibTeX citation:
@online{rafi2020,
  author = {Rafi, Zad and Rafi, Zad and Gelman, Andrew and Reito, Aleksi
    and Greenland, Sander},
  title = {Medicine {Is} {Being} {Treated} with {Snake-Oil}
    {Statistics}},
  date = {2020-11-11},
  url = {https://lesslikely.com/statistics/snakeoilstats.html},
  langid = {en-US}
}
]]> statistics https://lesslikely.com/statistics/snakeoilstats.html Wed, 11 Nov 2020 05:00:00 GMT Quality Control in Statistical Analyses Zad Rafi https://lesslikely.com/statistics/statistical-quality-control.html

Experienced statisticians and data analysts are familiar with stories where a coding error has led to an entire conclusion changing, or even a retraction.[@aboumatarNoticeRetractionAboumatar2019] It’s the sort of stuff that keeps people up at night. Unfortunately, not many of us think about these sorts of scenarios until we realize it’s very possible that it could happen to any of us.

To me, it seems that many of these issues could be avoided by having a principled data management and statistical workflow, and making it as transparent, open, and reproducible as possible. I’d like to quickly go over a few things that I’ve found helpful over the years, and I’ll first start with data management and data entry and then move onto analysis workflows. I largely consider this to be a living document, and I’m sure many people who will read this will have far better suggestions, so please leave them down below in the comments!

Before I go on, I want to emphasize that backing up your data, scripts, and using version control is extremely important. There is no debate about this. It’s necessary so that other collaborators/colleagues can inspect your work and catch potential mistakes or see overall progress, but more importantly, it will prevent you from losing your data in a disaster, and it’ll help you catch your own mistakes, since you’ll be the most familiar with the data and scripts.

Data Entry & Management

A nice paper that I’d like to review is the one by Broman & Woo, 2018 on how to manage your data when working with spreadsheets.[@bromanDataOrganizationSpreadsheets2018] The sad reality is that even though spreadsheets like Microsoft Excel or Google Sheets are available everywhere, and easy to use, there are many risks when working with spreadsheets, just ask any statistician who works in genetics or any bioinformatician.[@ziemannGeneNameErrors2016]

One of the most fatal errors occurred recently when a group of researchers lost thousands of documented COVID cases because they entered data for each case as a column instead of a row, and Excel has a limit on how many columns and rows it can handle (1,048,576 rows and 16,384 columns, according to Microsoft), as a result, most of these cases were lost, resulting in an enormous waste of resources due to a careless and ignorant mistake, highlighting the dangers of recklessly inputting data and conducting statistical analyses. There is no doubt that reviewing principles of good data management and workflow are essential to any data analyst. I’d like to touch on some of the most important points of Broman & Woo, 2018 paper before moving onto some other “principles” I’d like to share:

General Principle: Be Consistent With Everything

  • When labeling missing values (NA, N/A, etc.):

    • Good: Choose one method (NA, N/A, etc.) and stick with it.

    • Bad: Leaving cells empty to indicate missingness.

  • When labeling response variables:

    • Good: If you label a response variable consistently, for example, fat-free-mass in every instance to refer to the same variable.

    • Bad: If you label a response variable fat-free-mass in one script/sheet, and ffm/fat_free_mass in another.

  • When formatting variable predictor names:

    • Good: predictor_1_week12, predictor_2_week12, response_variable_week12. This is consistent, with the order of the names, and dates/weeks, so it is easier to organize, inspect, and clean. Same thing applies to labeling response variables and pretty much all variables.

    • Bad: When you label one variable week12_predictor_1, the next predictor_2_week12, and the last, 12_week_response. Just no.

  • When formatting dates:

    • Good: Use one format, YYYY-MM-DD, consistently.

    • Bad: Everything else.

  • When filling out cells:

    • Good: One cell = one thing. No more, no less.

    • Bad: One cell = multiple entries or no entries at all.

  • When keeping track of the data:

    • Good: Create a comprehensive data dictionary so anyone can look at it and understand the spreadsheets/dataframes/variables.

    • Bad: Expecting yourself and others to figure it out based on the variable names that you thought were brilliant.

  • When using Excel or Google Sheet’s shiny features:

    • Good: Avoid using any of them, any formulas, highlighting, italicizing, bolding, etc.

    • Bad: If you’re the type of person to create charts in Excel.

  • Planning for disasters:

    • Good: Always, always, backup your files, save them in .txt files. Keep backups of those. And most importantly, use version control!

    • Bad: You don’t really plan for disasters and go with the flow.

  • Last but not least, when entering cases:

    • Good: Rows are for cases and observations, and columns are for variables and characteristics. Please do not switch these up!

    • Bad: Using them interchangeably.

Setting Up a Principled Statistical Workflow

Here’s what I’ve been doing for many years and what seems to work for me (none of these ideas are originally mine, and I actually picked them up over the years from others’ advice, which will be linked below).

Setting up the Project Folder

When setting up a folder specific for a project (if you’re not doing this, you absolutely need to), first, I will create a folder with the title of my project, and usually have an RStudio project set up in there.

Disclaimer: while much of this advice will be familiar and easy to understand for those who use R, I think the general principles are widely applicable, especially for those who use scripts in their statistical software).

This folder will contain many other folders (more on that below), so the structure will end up being a bit complex.

workflow

It may be a bit difficult to read the words within the circles, but clicking on the diagram should enlargen it and allow you to zoom in, giving you a sense of how I organize all these scripts and files.

If this ends up confusing you, you can just scroll all the way down to see an image of what the folder structure looks like.

Instead of constantly changing the working directory to each subfolder when I need to do something inside that subfolder for a particular project or analysis, I set the project folder as the working directory only once, and then run the following:

# Very seriously consider installing the `here` package

# https://cran.r-project.org/package=here

library("here")

# This and the scripts below take
# the directory used and set it as the top

set_here()

here()

This not only sets the working directory, but gives you far more control over how you can save your files from any place within the hierarchical folder structure. It will create a file called .here inside the Main Project folder, which will indicate that this is the top level of the hierarchy.

Then, I will usually have created a README.Rmd file in the Main Project folder with updates on what I’ve done and what I still need to work on, so I will remember and so my collaborators can see it too (when it has been pushed to GitHub or other version control repos).

Next, I’ll set up a Data folder inside the Main Project folder. This is where all my data/spreadsheets/.txt files and data dictionaries will go. The original data files will stay inside this folder, while I create two other subsubfolders inside this Data folder called Transformed and Models. In Transformed, I will typically save .rds files that were a result of cleaning and transforming the data, including imputing missing data. I will touch more on exactly how I do that later below. A Models subfolder will obviously store fitted models and any validation/sensitivity analyses of those models.[@morrisChoosingSensitivityAnalyses2014]

Now, back to the Main Project folder, I’ll set up another folder within it called R. This will be where all of my .R scripts/files live. I will number them sequentially along with a title for a specific purpose, like so

  • main.R

  • 01-functions.R

  • 02-cleaning.R

  • 03-inspection.R

  • 04-analysis.R

  • 04.5-validation.R

  • 05-sensitivity.R

  • 06-tables.R

Setting up the Functions Script

All potential R packages and custom functions that I need will belong in 01-functions.R. This .R file will only serve this purpose and nothing else. As you’ve guessed by now, the other .R files will be doing the same, they have very specific purposes and are organized to reflect this.

An example of the first script can be found below.

# Necessary Packages ------------------------------------

req_packs <- c("rms", "concurve", "mice",
               "tidyverse", "parallel",
               "bayesplot", "projpred",
               "Hmisc", "loo", "rstan", "here",
               "tryCatchLog", "futile.logger",
               "patchwork", "corrplot", "beepr",
               "summarytools", "broom", "wakefield",
               "boot", "mfp", "knitr", "flextable",
               "MASS", "lme4", "brms", "miceMNAR")

# Load all packages at once

lapply(req_packs, library,
       character.only = TRUE)

# Loggings messages
options(keep.source = TRUE)
options("tryCatchLog.write.error.dump.file" = TRUE)

# Set seed for random number generator

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed(1031) # My birthday

# RStan Settings  -------------------------------------

theme_set(theme_bw())
color_scheme_set("red")
rstan_options(auto_write = TRUE)
options(mc.cores = 4)

has_build_tools(debug = TRUE)

# Stan Settings  ---------------------------------------

dotR <- file.path(Sys.getenv("HOME"), ".R")
if (!file.exists(dotR)) dir.create(dotR)
M <- file.path(dotR, "Makevars")
if (!file.exists(M)) file.create(M)
cat("\nCXX14FLAGS=-O3 -march=native -mtune=native -fPIC",
  "CXX14=g++",
  file = M, sep = "\n", append = TRUE)

The script above loads all the necessary R packages each time it is called, along with the specified functions, so most of the other .R files will depend upon this one.

However, I will not be running each of these .R files/scripts individually, line by line, or by selecting all the lines in an .R file and running the script.

Instead, after I’ve created all these files (figuring out what I need to do carefully and writing it down and annotating it), every single .R file except for main.R will have the following script, which I will explain later below (or some iteration of this script to match the name of the file) at the beginning:

Setting up the Error Catching Script

library("here")

set_here()

source(here("R", "01-functions.R"))

library(futile.logger)
library(tryCatchLog)

# Keeps source code file name and line number tracking
options(keep.source = TRUE)
options("tryCatchLog.write.error.dump.file" = TRUE)

# Logs messages into a file
flog.appender(appender.file("my_app.log"))
flog.threshold(INFO)

# Function that catches any messages
tryCatchLog(source("script.R"))

You’ll notice several things. First, I’m once again calling the here package and telling it that I’m working within this folder (R), and then I’m calling the 01-functions.R file by using the source() function, but also notice how the source() function is followed by a here() function/argument. This here() function allows you to fully control files in a specific folder from anywhere else, without having to actually be in that folder. So, suppose I was in the Main Project -> Data -> Models folder and I was saving my work there (in the Models folder), which is pretty far from the R folder, I can still, using functions like source(), save(), etc., call or manipulate files from a totally different folder by specifying the hierarchy using here(). This can also be done in other ways, but those are far more cumbersome, and not flexible.

This is how I always call the necessary packages and functions I need from every .R script, simply by using the source() function and using here() to direct it to the 01-functions.R file in the R folder. Now let’s look at the next few lines.

flog.appender(appender.file("script.log"))
flog.threshold(INFO)

tryCatchLog(source("script.R"))

This script is designed to catch any warnings that occur inside the .R script and save them to a .log file in another subfolder within the R folder called Errors. So here’s an example of a full script for a mix of data generation and inspection, so that you can see it in action.

Example of Data Loading & Inspection

library("here")

set_here()

source(here("R", "01-functions.R")) # Loads all R functions

# First we need some data

RNGkind(kind = "L'Ecuyer-CMRG")
set.seed(1031) # My birthday

# Suppose you wanted to simulate a clinical trial
library("simstudy")

# Simulating a Clinical Trial --------------------------

def <- defData(varname = "male", dist = "binary",
               formula = .5 , id="cid")
def <- defData(def, varname = "over65",
               dist = "binary",
               formula = "-1.7 + .8 * male",
               link = "logit")
def <- defData(def,varname = "baseDBP",
               dist = "normal",
                 formula = 70, variance = 40)
def <- defData(def, varname = "age",
               dist = "normal",
               formula = 10,
               variance = 2)
def <- defData(def, varname = "visits",
               dist = "poisson",
    formula = "1.5 - 0.2 * age + 0.5 * male",
    link = "log")
def <- defData(def, varname = "weight",
               dist = "normal",
                 formula = 60, variance = 10)
dtstudy <- genData(500, def)
study <- trtAssign(dtstudy, nTrt = 2,
                   balanced = FALSE, grpName = "rxGrp")
study <-  as.data.frame(study)
study$iq <- iq <- wakefield::iq(500)
study$height <- height <- wakefield::height(500)
study$income <- income <- wakefield::income(500,
                              digits = 2, name = "Income")
study$SAT_score <- wakefield::sat(500, mean = 1500,
                 sd = 100, min = 0, max = 2400,
                 digits = 0, name = "SAT")
study$visits <- as.numeric(study$visits)
# Add noisy predictors
study[, 11:20] <- lapply(1:10, FUN = function(i)
                         (rnorm(n = 500,
                                mean = 1, sd = 2)))
varname <- sprintf("x%d", (11:20))
colnames(study)[11:20] <- varname

study1 <- trtAssign(dtstudy, n = 2, balanced = TRUE,
                    strata = c("male", "over65"),
    grpName = "rxGrp")

study1

# Inspecting Data  -------------------------------------

summarytools::view(dfSummary(study))

study$over65 <- as.factor(study$over65)
study$male <- as.factor(study$male)
study$rxGrp <- as.factor(study$rxGrp)

study_formula <- as.formula(baseDBP ~ .)

plot(spearman2(study_formula, study))
abline(v = 0)

x_1 <- model.matrix(~ ., data = study)[, -1]

# Initial Data Analysis  ------------------------------

(cormatrix <- (cor(x_1)))

col <- colorRampPalette(c("#BB4444", "#EE9988",
                          "#FFFFFF",
                          "#77AADD", "#4477AA"))

corrplot(cormatrix, method = "color",
         col = col(200),
         tl.col = "black", tl.cex = 0.40,
         addCoef.col = "black", number.cex = .35)


clus1 <- varclus(x_1, similarity = "h")
clus2 <- varclus(x_1, similarity = "s")
clus3 <- varclus(x_1, similarity = "p")

plot(clus1, ylab = "Hoeffing's D statistic",
     lwd = 2.5, lty = 1, cex = 0.75)
title("Hierarchical Cluster Analysis")


plot(clus2, lwd = 2.5, lty = 1, cex = 0.75)
plot(clus3, lwd = 2.5, lty = 1, cex = 0.75)

Pre <- names(study)
(fmla <- as.formula(paste("~",
                          paste(Pre,
                                collapse = "+"))))

# Redundancy analysis

redun <- redun(fmla,
  data = study,
  r2 = 0.90, type = "ordinary", allcat = FALSE,
  tlinear = FALSE, iterms = FALSE, pr = TRUE)

The above script simulated some fake data for a hypothetical clinical trial (usually we use this first script to load data from a .csv or .txt file) and then we used several functions to inspect the dataframe for the structure and distributions of the variables and missing data, etc. We also attempted to properly code the variables. Now, in order to capture any possible errors, we will have to run this entire script from another R script instead of running it line by line or selecting all the code and running it.

We will do this from a Main source script, which will allow us to use the tryCatchLog() function.

Example of the Main Source Script

library("here")
library("tryCatchLog")
library("futile.logger")

set_here() # Load and set here

here() # Call here

options(keep.source = TRUE)
options("tryCatchLog.write.error.dump.file" = TRUE)

# Script loads all necessary functions
flog.appender(appender.file(
  here("Errors", "01-functions.log")))
tryCatchLog(source(here("R", "01-functions.R")))

# Line script to clean the data
flog.appender(appender.file(
  here("Errors", "02-cleaning.log")))
tryCatchLog(source(here("R", "02-cleaning.R")))

# Line script below to inspect the data
flog.appender(appender.file(
  here("Errors", "03-inspection.log")))
tryCatchLog(source(here("R", "03-inspection.R")))

# Load saved objects from cleanings
flog.appender(appender.file(
  here("Errors", "03.5-load_data.log")))
tryCatchLog(source(here("R", "03.5-load_data.R")))

# Main analysis script
flog.appender(appender.file(
  here("Errors", "04-analysis.log")))
tryCatchLog(source(here("R", "04-analysis.R")))

# Below is the validation of main analysis results
flog.appender(appender.file(
  here("Errors", "04.5-validation.log")))
tryCatchLog(source(here("R", "04.5-validation.R")))

# Test the sensitivity of the results by
# varying assumptions/multiple parameters
flog.appender(appender.file(
  here("Errors", "05-sensitivity.log")))
tryCatchLog(source(here("R", "05-sensitivity.R")))

# Generate tables and figures of all your results
flog.appender(appender.file(
  here("Errors", "06-tables.log")))
tryCatchLog(source(here("R", "06-tables.R")))

# Export these into actual files that can be
# inserted into papers, reports, etc.
flog.appender(appender.file(
  here("Errors", "07-export.log")))
tryCatchLog(source(here("R", "07-export.R")))

If anything went wrong during the simulations and their inspections in the first script from above, it would have been captured by the error-catching scripts that we have set up, which would be placed within a .log file in the Errors folder, which is inside the R folder.

Setting the Seed

Also, note that we have set the seed using a specific random number generator algorithm. This is not that important, however, it is important to correctly set the seed for reproducing any random numbers generated. This requires even more careful thought when running a simulation study, in which the states of the simulation need to be saved for a particular repetition, and so that streams do not overlap.[@morrisUsingSimulationStudies2019]

I will admit that the error script is not perfect at capturing warnings, but for most things, if something wrong occurs when a script is run, the function tryCatchLog() and its .log file usually catches the warning, so I would know that something went wrong and what specifically.

However, some may be skeptical of this approach and ask why I need to catch these errors and save them if I ran the scripts? Why not just scroll up and look at the console to see the errors or warnings?

Many reasons:

  • Suppose the analyses take a very long time, and are computationally intensive, and R/RStudio crashes, you may not be able to figure out what went wrong.
  • Suppose the analysis completed but because there was so much occurring in the console (for example, sampling from a posterior) that you missed the warnings/errors and got results that you don’t know have issues.

Thus, having this system set up to catch errors in every script also helps us automate the entire workflow once we’ve carefully set it up. After each .R script has one of these message-catching scripts, we can run our entire analysis from start to finish from main.R with one command. This not only makes your workspace organized, but also makes it easier for others to reproduce your work if you share your files and folders via a repository.

So, now the Error folder will contain .txt files with any warning messages or errors.

Typically, when I’m setting up a new project, I will also add a few other folders such as an Outputs folder, where I will save tables and graphs, and a Report folder, where I might be working on a manuscript.

Now that I’ve mentioned my hierarchy and how I catch errors, I can mention some other things that I believe are absolutely heinous practices.

Many people often will run an entire analysis and have several objects/vectors stored in their Global Environment in R and RStudio. Suppose they’re taking a break, finished with the analysis, turning off the computer, or stepping away for whatever reason, what they’ll typically do is click the save button on the top right pane and “Save the workspace image”. Indeed, if they don’t do that themselves, the IDE RStudio will explicitly ask them if they wish to save their workspace image.

This is a horrific practice and you should never do this because it can lead to several problems such as:

  • having several saved objects and packages conflict with one another once they’re all loaded together at once

  • having giant workspace images, that will probably cause R, and especially RStudio to constantly crash

  • not allowing you to load very specific objects that you need at a time while leaving everything else

I would say that it is a good idea to never save the workspace image, ever. It is also quite easy to turn off. Simply go the RStudio menu and click Preferences, and in the General section, you will see the following options:

It is essential to uncheck everything under R Sessions, Workspace, and History, so that you do not set your self up for a future disaster.

Instead of using these convenient but highly problematic options, always use the saveRDS() base R function, where the first argument is the object in your environment that you want to save, and the second argument is the path where you want to save it, which of course should be using the here() function.

Now suppose I decided to use multiple imputation because I had a dataset with many missing values, and suppose I had a grasp of the missing data mechanism, and so I used something like the not-at-random fully conditional specification approach (NARFCS)[@tompsettUseNotatrandomFully2018] to attempt to handle data that I believe to be missing not at random (MNAR), and I wanted to save the imputed dataset, and only load the imputed dataset next time. This is how I would do it (in the vaguest way possible). First, I’ll start with the dataset that we generated above and generate missing values via different missing data mechanisms using the ampute() function from the mice package.[@buurenMiceMultivariateImputation2011]

Important: This is all assuming that one has carefully constructed an imputation model, and that multiple imputation is the most appropriate solution. As many others have pointed out, simply running a script like mice(data) in R without careful thought to the imputation model, is a very, very bad idea. Missing data is an extremely complex topic and I recommend everyone carefully read the works[@buurenFlexibleImputationMissing2018] of missing data researchers such as:

Example of a Multiple Imputation Script

source(here("R", "01-functions.R"))

library("beepr")

# Package that uses multiple imputation
# by chained equations

library("mice")


# Generate Missing Data  --------------------------------

study_mnar <- ampute(data = study[, 5:10],
                     prop = 0.65, mech = "MNAR")
mnar_patterns <- study_mnar$patterns
mnar_weight <- study_mnar$weight
study_mnar_rerun <- ampute(data = study[, 5:10],
                           prop = 0.5,
                           mech = "MNAR",
                           patterns = mnar_patterns,
                           weights = mnar_weight)
study_mar <- ampute(data = study[, 2:4], prop = 0.5,
                    mech = "MAR")
mar_patterns <- study_mar$patterns
mar_weight <- study_mar$weight
study_mar_rerun <- ampute(data = study[, 2:4],
                          prop = 0.5,
                          mech = "MAR",
                          patterns = mar_patterns,
                          weights = mar_weight)

study_mar_rerun[[11]]$over65 %>%
  as.factor()
study_mar_rerun[[11]]$male %>%
  as.factor()
study_mar_rerun[[11]]$rxGrp %>%
  as.factor()

df <- study_mnar_rerun$amp
df2 <- study_mar_rerun$amp

# Inspect Data

str(df) # Inspect data frame
summary(df) # High level summary of dataframe
sum(is.na(df)) # Look at missing values etc.
colSums(is.na(df))
summarytools::view(dfSummary(df))

str(df2) # Inspect data frame
summary(df2) # High level summary of dataframe
sum(is.na(df2)) # Look at missing values etc.
colSums(is.na(df2))
summarytools::view(dfSummary(df2))

# Prepare Data for Imputation -------------------------

temp <- df
temp2 <- df2

missing_plot(temp)

missing_plot(temp2)

# Set-up predictor matrix

predMatrix <- make.predictorMatrix(temp)
predMatrix2 <- make.predictorMatrix(temp2)

# Set-up predictor matrix for unidentifiable part:
predSens <- matrix(rep(0, 6), ncol = 6, nrow = 6)
colnames(predSens) <- paste(":", names(temp), sep = "")
rownames(predSens) <- names(temp)

# Set-up list with sensitivity parameter values

pSens <- rep(list(list("")), ncol(temp))
names(pSens) <- names(temp)
pSens[["rxGrp"]] <- list(-.4)
pSens[["baseDBP"]] <- list(-.4)
pSens[["male"]] <- list(-.4)
pSens[["over65"]] <- list(-.2)

# MICE RI Imputation  ----------------------------------

z1 <- parlmice(data = temp, method = "ri",
              predictorMatrix = predMatrix,
              m = 8, maxit = 1,
              cluster.seed = 1031, n.core = 4,
              n.imp.core = 2, cl.type = "FORK",
              ridge = 1e-04, remove.collinear = TRUE,
              remove.constant = FALSE, allow.na = TRUE)

z2 <- parlmice(data = temp2, method = "pmm",
              predictorMatrix = predMatrix2,
              m = 8, maxit = 1,
              cluster.seed = 1031, n.core = 4,
              ridge = 1e-04, remove.collinear = TRUE,
              n.imp.core = 2, cl.type = "FORK")

zfinal <- cbind(z1, z2)

# Heckman Imputation  ----------------------------------

study$baseDBP <- temp$baseDBP
JointModelEq<-generate_JointModelEq(data=temp,
                                    varMNAR="baseDBP")
JointModelEq[,"baseDBP_var_sel"] <- c(0,1,1,1,1,1)
JointModelEq[,"baseDBP_var_out"] <- c(0,1,1,1,1,1)

arg <- MNARargument(data = temp, varMNAR = "baseDBP",
                    JointModelEq = JointModelEq)

arg$method["age"] <- "ri"
arg$method["visits"] <- "ri"
arg$method["weight"] <- "ri"
arg$method["iq"] <- "ri"
arg$method["height"] <- "ri"

imp1 <- parlmice(data = arg$data_mod,
                  method = arg$method,
                  predictorMatrix = arg$predictorMatrix,
                  JointModelEq = arg$JointModelEq,
                  control = arg$control,
                  maxit = 1, m = 8,
                  cluster.seed = 1031, n.core = 4,
                  n.imp.core = 2, cl.type = "FORK")

analysis1 <- with(zfinal, glm(baseDBP ~ age + visits + weight + iq + height + rxGrp + male + over65))
result1 <- pool(analysis1)
summary(result1)

z2 <- parlmice(data = temp2, method = "pmm",
              predictorMatrix = predMatrix2,
              m = 8, maxit = 1,
              cluster.seed = 1031, n.core = 4,
              n.imp.core = 2, cl.type = "FORK")

heckman <- cbind(imp1, z2)

beep(3) # I will touch on this function below.

saveRDS(zfinal, here("Main Project", "Data",
                     "zfinal.rds"))

Here is the workflow truly showing its advantages, saveRDS() is saving a very specific object (zfinal, the imputed dataset) into the Data folder by using the hierarchical structure of the folder which can easily be navigated using here(). Notice how to save something and use here() to specify the location, the first argument is Main Project, followed by Data, and then comes the object name (zfinal.rds).

Now I have saved my imputed dataset as an object called zfinal.rds and it is saved in the Data folder even though I am hypothetically working from the R folder.

Also notice that I use the beep() function from the beepr R package, which can be quite handy in letting you know if a particular script or analysis is done.

Next time, if I cleared my entire global environment and wanted to only load the imputed dataset, I would simply have to just click on the zfinal.rds file in the Data folder and it would load into the environment or I could use the following command,

Saving and Reading Data Carefully

# The function readRDS() and saveRDS()
# are possibly the most important base R functions

zfinal <- readRDS("zfinal.rds",
             here("Main Project", "Data",
                  "zfinal.rds"))

This gives us full control over the environment, and it’s also why I encourage people to turn off the automatic prompt to save the workspace that RStudio gives you by default (I am certain even Hadley Wickham has said the same on multiple occasions).

So if I now wanted to include the imputed dataset into one of my .R scripts that I am trying to automate with one command, whether for reproducibility, or speed, etc., I could do it with absolute control.

This is also how I save my models and how I validate them, which is especially handy because running those can also take an excruciatingly long time.

Here’s an example of a script that would be in the 04-analysis.R script, I’m leaving out the warning-catching script now to avoid making this too long. I don’t need to load the brms package because I’m already calling it from the first .R script, 01-functions.R (but I’m just showing it for now to avoid confusion), and then I am saving it in the Models folder which is found in the Data folder. And again, the beep() function would let me know when the script is over, and scripts like this can often take very long, due to the computational intensity.

The script below is an analysis script whose results we plan on saving once completed. Due to the arguments I have set (iterations, warmups, chains, and using an imputed dataset), it can often take a long time before it is finished, and if I did happen to do other tasks while I was waiting on that, the output would automatically save to the Models folder via saveRDS(), and I would also receive a notification via the beep() function, preventing me from wasting any time.

Example Modeling Script

# Calling all packages, please report for duty

source(here("R", "01-functions.R"))

library("brms")

# This is a script I used previously
# where I used Bayes for regularization
# Although these are all generic options

# Prior Information  --------------------------------

sample_z <- complete(zfinal, 1)

n_1 <- nrow(sample_z) # of observations

k_1 <- (ncol(sample_z) - 1) # of predictors

p0_1 <- 2 # Prior for the number of relevant variables

tau0_1 <- p0_1 / (k_1 - p0_1) * 1 / sqrt(n_1) # tau scale

# Regularized horseshoe prior
hs_prior <- set_prior("horseshoe(scale_global = tau0_1,
                      scale_slab = 1)", class = "b")

library("future")
plan(multiprocess)

# Regularized Bayesian quantile
# regression using regularized horseshoe prior
brms::brm_multiple(
  bf(baseDBP ~ age + visits + weight + iq + height + rxGrp + male + over65,
     quantile = 0.50),
     data = zfinal, prior = hs_prior,
     family = asym_laplace(), sample_prior = TRUE,
     seed = 1031, future = F,
     iter = 1000, warmup = 500, chains = 4,
     cores = 4, thin = 1, combine = TRUE,
     control = list(max_treedepth = 10,
                    adapt_delta = 0.80)) -> pen_model_1

# Principled way of saving results
saveRDS(pen_model_1,
        here("Main Project", "Data", "Models",
             "pen_model_1.rds"))

beep(3)

So I’ve saved the full contents of the fitted model (pen_model_1), which are usually several gigabytes large in size, within the Models folder. Once again, notice how the here() function allows you to work from a subfolder like R, which is inside the Main Project folder, and allows you to specify the hierarchy, and where you are within it, and save to whichever folder you wish, in this case Models, which is inside of Data. Now, I will typically conduct model checks and make sure it is performing well, and not misspecified,

source(here("R", "01-functions.R"))

# Inspect RHat values ---------------------------

rhats_1 <- pen_model_1$rhats

rhats_1_df <- (rhats_1 <= 1.1)
((as.numeric(table(rhats_1_df)) /
    (nrow(rhats_1_df) * ncol(rhats_1_df)) * 100))

rhats_1_vec <- as.numeric(unlist(c(
  rhats_1[, 1:ncol(rhats_1)])))

pdf(here("Outputs", "Figures", "Diagnostics",
         "OUTCOME_COMP_rhat.pdf"))
mcmc_rhat_hist(rhats_1_vec) +
  ggtitle("Model 1 Chain Convergence")
dev.off()

# Inspect Effective Sample Size ----------------------

mcmc_neff_hist(rhats_1_vec) +
  ggtitle("Effective Sample Size")

# Inspect Divergences ---------------------------

pdf(here("Outputs", "Figures",
         "Diagnostics", "OUTCOME_1.pdf"))
mcmc_nuts_divergence(nuts_params(pen_model_1),
                     log_posterior(pen_model_1))
dev.off()

# Examine Residual Plots ----------------------------

df_resid_stan_1 <- data.frame(fitted(pen_model_1)[, 1],
                              residuals(pen_model_1)[, 1])

pdf(here("Outputs", "Figures",
         "Diagnostics", "OUTCOME_1.pdf"))
ggplot(df_resid_stan_1,
       aes(sample = residuals(pen_model_1)[, 1])) +
  geom_qq() +
  geom_qq_line() +
  ggtitle("Model 1 Residuals")
dev.off()

(sum1 <- (posterior_summary(pen_model_1))[2:8, ])

mcmc_intervals(pen_model_1, point_est = "median",
               prob = 0.95, prob_outer = 0,
               pars = parnames(pen_model_1)) +
  ggplot2::scale_y_discrete() +
  theme(axis.text=element_text(size=13),
        axis.title=element_text(size=13),
        plot.title = element_text(size=15,
                                  face = "bold")) +
  annotate("rect",
    xmin = -0.05, xmax = 0.05, ymin = 0, ymax = 7,
    fill = "darkred", alpha = 0.075) +
  annotate("segment",
    x = -0.05, xend = -0.05, y = 0, yend = 7,
    colour = "#990000", alpha = 0.4,
    size = .75, linetype = 3) +
  annotate("segment",
    x = 0.05, xend = 0.05, y = 0, yend = 7,
    colour = "#990000", alpha = 0.4,
    size = .75, linetype = 3) +
  ggtitle("Change in Outcome 1")
dev.off()

pdf(here("Outputs", "Figures", "OUTCOME_1.pdf"))
mcmc_hist(pen_model_1)

# Posterior Predictive Checks

pdf(here("Outputs", "Figures", "Diagnostics",
         "OUTCOME_1check.pdf"))
pp_check(pen_model_1, nsamples = 250) +
  ggtitle("Posterior Predictive Check: Model 1")
dev.off()

# Shows dens_overlay plot by default

bayes_rsq1 <- bayes_R2(pen_model_1)

bayes_rsq1 <- print(median(bayes_rsq1))

write.csv(sum1, here("Outputs", "Tables",
                     "OUTCOME_1.csv"))

Once I have fit my model and done some initial checks, I usually conduct some more thorough checks using k-fold cross-validation or nested cross-validation, or bootstrap optimism.

A script of this can be found below.

source(here("R", "01-functions.R"))

# 10-Fold Cross-Validations -------------------------

kfold_1 <- kfold(pen_model_1, K = 10,
                 save_fits = TRUE, cores = 4)
saveRDS(kfold_1, here("Data", "Models",
                      "Validation", "kfold_1.rds"))
kfp_1 <- kfold_predict(kfold_1)
saveRDS(kfp_1, here("Data", "Models",
                    "Validation", "kfp_1.rds"))
kfold_rmse(y = kfp_1$y,
           yrep = kfp_1$yrep,
           type = "rmse",
           reps = 5000,
cores = 4)[[2]][["t"]]) -> kfold_rmse_1

kfold_rmse_1 <- median(kfold_rmse_1)

saveRDS(kfold_rmse_1,
        here("Data", "Models",
             "Validation", "kfold_rmse_1.rds"))

Once I’ve written all my analysis and validation scripts, I’ll typically create a subfolder in the Outputs folder called Figures and Tables and save them appropriately with the help here().

Example Table Script

source(here("R", "01-functions.R"))

# Read in Validation Objects ------------------------

kfold_1 <- readRDS("kfold_1.rds",
                   here("Main Project", "Data",
                        "Models", "Validation",
                        "kfold_1.rds"))
kfold_2 <- readRDS("kfold_1.rds",
                   here("Main Project", "Data",
                        "Models", "Validation",
                        "kfold_2.rds"))
kfp_1 <- readRDS("kfp_1.rds",
                 here("Main Project", "Data",
                      "Models", "Validation",
                      "kfp_1.rds"))
kfp_2 <- readRDS("kfp_1.rds",
                 here("Main Project", "Data",
                      "Models","Validation",
                      "kfp_2.rds"))

# Extract Estimates from Objects ---------------------

KFOLDIC_1 <- c(kfold_1[["estimates"]][3],
                     kfold_2[["estimates"]][3])
KFOLDIC_SE_1 <- c(kfold_1[["estimates"]][3, 2],
                        kfold_2[["estimates"]][3, 2])
BAYES_R2 <- c(bayes_rsq1, bayes_rsq2), 2)
KFOLD_ELPD_1 <- c(kfold_1[["estimates"]][1],
                        kfold_2[["estimates"]][1])
KFOLD_ELPD_SE_1 <- c(kfold_1[["estimates"]][1, 2],
                           kfold_2[["estimates"]][1, 2])

Outcome <- c("OUTCOME_1", "OUTCOME_2")

# Construct Data Frames ------------------------------

table4 <- data.frame(Outcome, BAYES_R2, KFOLD_ELPD_1,
                     KFOLD_ELPD_SE_1, KFOLDIC_1,
                     KFOLDIC_SE_1)

colnames(table4) <- c("Outcome", "Bayes R^2",
                      "10-fold CV ELPD",
                      "10-fold CV ELPD SE",
                      "10-fold CV IC",
                      "10-fold CV IC SE")

# Export as Tables ----------------------------------

write.csv(table4, here("Main Project","Outputs",
                       "Tables", "Table4.csv"))

table5 <- data.frame(Outcome, KFOLD_ELPD_1,
                     KFOLD_ELPD_SE_1,
                     KFOLDIC_1, KFOLDIC_SE_1)
colnames(table5) <- c("Outcome", "10-fold CV ELPD",
                      "10-fold CV ELPD SE",
                      "10-fold CV IC",
                      "10-fold CV IC SE")
write.csv(table5,
          here("Main Project","Outputs", "Tables",
               "Table5.csv"))

# Save Tables From Sensitivity Analyses --------------

write.csv(sens_table_OUTCOME_1,
          here("Main Project","Outputs", "Tables",
               "Table7_OUTCOME_1.csv"))
write.csv(sens_table_OUTCOME_2,
          here("Main Project","Outputs", "Tables",
               "Table7_OUTCOME_2.csv"))

Then, I’ll go back to the README.Rmd file, make notes of the hierarchy of the entire folder and subfolders, and what’s been completed, and what still needs to be done.

Now, suppose I deleted all of my saved models, imputed data, graphs, and tables, (which would be a devastating blow!) and only had the original dataset and organized scripts. Anyone could go into my project and simply highlight the entire main.R file and run it, and they would get all the same outputs and models, assuming a seed was set in advance, which is essential. Although, it might take them a long time to get the same outputs (if computationally intensive)… they would likely be able to get all the exact same results.

Possible Drawbacks of this Approach

Every proposal tends to have drawbacks or limitations, and those who do not explicitly tell you about them are misleading you. In the approach I have outlined above, one major drawback is that it takes a somewhat long time to set up and therefore could deter many statisticians and data analysts. The other drawback is that by loading all your functions and libraries in the first 01_functions.R script, there is potential for conflict between functions, therefore you’ll have to be mindful of what packages you need and when. Indeed, I ran into this issue when loading the brms package, which had several functions that conflicted with the base R functions. So please be mindful of this.

Further, and most importantly, I have no evidence that this approach can truly prevent errors, it is simply a belief I have as a result of my own experiences and suggesting them to others, and hearing positive comments. So please, take that into account, although I would be grateful if you were able to share your positive or even negative experiences using this approach below.

Please, please annotate your scripts, as I have done so several times above to explain some of the functions. This is not only helpful for others who are trying to go through your code and understand it, but also for yourself. If you end up forgetting what function does what, and what steps you were taking, you will be in for a really frustrating time, and possibly prone to making serious mistakes.

Here’s what my project folders tend to look like (figure below). And here is a template GitHub repo to show what the structure typically looks like.

It should go without saying that sharing your data (if possible) and code will help you and your collaborators catch errors before it’s too late, but also, sharing them with others/the public after a project is done will also help catch possible errors that you/your collaborators/reviewers didn’t catch, which may mislead other researchers and also result in a much more stressful situation for you.

And last but not least, it is also essential to give extensive details on which version of R you used and which packages you used and their versions. This is because many scripts that once worked on previous versions of the software may no longer give the same results or may not even work at all. In order for someone to reproduce your results, they’ll need to know the environment on which you ran your analyses on. You can provide these details easily simply by running the script sessionInfo(), which should be at the end of every R script you run.

Helpful Resources

Environment

The analyses were run on:

R version 4.6.0 (2026-04-24)
Platform: aarch64-apple-darwin25.4.0
Running under: macOS Tahoe 26.5.1

Matrix products: default
BLAS:   /opt/homebrew/Cellar/openblas/0.3.33/lib/libopenblasp-r0.3.33.dylib 
LAPACK: /opt/homebrew/Cellar/r/4.6.0/lib/R/lib/libRlapack.dylib;  LAPACK version 3.12.1

Random number generation:
 RNG:     Mersenne-Twister 
 Normal:  Inversion 
 Sample:  Rejection 
 
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/New_York
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] htmlwidgets_1.6.4 compiler_4.6.0    fastmap_1.2.0     cli_3.6.6         htmltools_0.5.9   tools_4.6.0       parallel_4.6.0   
 [8] otel_0.2.0        yaml_2.3.12       rmarkdown_2.31    knitr_1.51        data.table_1.18.4 jsonlite_2.0.0    xfun_0.59        
[15] digest_0.6.39     rlang_1.2.0       evaluate_1.0.5

See also: What Makes a Sensitivity Analysis? — the hub piece tying this material to the broader cluster on assumptions, robustness, and what happens when models bend.

References

Back to top ]]> statistics https://lesslikely.com/statistics/statistical-quality-control.html Sat, 13 Jun 2020 04:00:00 GMT Book Review: Regression and Other Stories by Gelman, Hill, and Vehtari Zad Rafi https://lesslikely.com/statistics/regression-stories.html

Over a decade ago, Andrew Gelman and Jennifer Hill gave applied researchers a comprehensive book (Data Analysis Using Regression and Multilevel/Hierarchical Models) on fitting simple and complex statistical models in R both from a classical framework and a Bayesian one. Now they’re back with an updated version and a new author (Aki Vehtari).

Much has changed in applied statistics since 2006 (when the book was first released). The primary software used at the time and in the book to fit Bayesian models was BUGS (Bayesian inference Using Gibbs Sampling).

However, both BUGS and some of the R code in the first edition are now outdated. The new edition updates the R code and contains intuitive instructions on how to fit simple and complex models using the probabilistic programming language, Stan (also developed by Gelman and colleagues), which is now used in several fields (even for studying wine!).

Indeed, running a Bayesian regression model in R is now as simple as

# I use the sample PlantGrowth dataset in R
library("rstanarm")
library("bayesplot")
pg <- PlantGrowth
model1 <- stan_glm(weight ~ group, 
                   data = pg, refresh = 0)
summary(model1)

Model Info:
 function:     stan_glm
 family:       gaussian [identity]
 formula:      weight ~ group
 algorithm:    sampling
 sample:       4000 (posterior sample size)
 priors:       see help('prior_summary')
 observations: 30
 predictors:   3

Estimates:
              mean   sd   10%   50%   90%
(Intercept)  5.0    0.2  4.8   5.0   5.3 
grouptrt1   -0.4    0.3 -0.7  -0.4   0.0 
grouptrt2    0.5    0.3  0.1   0.5   0.9 
sigma        0.6    0.1  0.5   0.6   0.8 

Fit Diagnostics:
           mean   sd   10%   50%   90%
mean_PPD 5.1    0.2  4.9   5.1   5.3  

The mean_ppd is the sample average posterior predictive distribution of the outcome variable (for details see help('summary.stanreg')).

MCMC diagnostics
              mcse Rhat n_eff
(Intercept)   0.0  1.0  2844 
grouptrt1     0.0  1.0  2882 
grouptrt2     0.0  1.0  2642 
sigma         0.0  1.0  3279 
mean_PPD      0.0  1.0  3354 
log-posterior 0.0  1.0  1645 

For each parameter, mcse is Monte Carlo standard error, n_eff is a crude measure of effective sample size, and Rhat is the potential scale reduction factor on split chains (at convergence Rhat=1).
plot(model1)

Another key difference between the first edition and the new edition is that the 2006 book attempted to cover several topics at once. It contained instructions on how to fit simple models in a classical framework all the way up to multilevel models in a Bayesian framework. The new edition attempts to reduce this information overload by splitting itself into two volumes.

The first volume (Regression and Other Stories) covers fitting simple and complex models using R and Stan, and is oriented towards the applied researcher or statistician, who wants a smooth introduction to fitting Bayesian models using Stan without diving into much theory or math.

A draft copy of the table of contents in the new edition can be found here, though it’s very likely that the published edition will have some changes.

The book does not cover much of multilevel modeling, which is reserved for the second volume, Advanced Regression and Multilevel Models (planned to be released in the next year or two).

Make no mistake, although both of these books are unlikely to touch on a serious amount of theory or math, they are not books that can be read without serious engagement and practice. Every chapter contains enough math for the reader to understand the concepts being discussed with exercises at the end to solidify these concepts.

The chapter exercises are incredibly similar to these exam questions that Gelman created for his Applied Regression class.

Question: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, solution to 15.

I suspect that many of the commenters of the blog who had some difficulty with these questions would have had an easier time had they had the opportunity to read the book.

The new edition also covers several news stories from the past few years (some of which long-time blog readers will be familiar with) and gives readers a set of tools to think critically about these stories and how proper statistical thinking could’ve prevented mishaps. In addition, it incorporates concepts that Gelman and colleagues have developed and solidified over the years, since the first edition was published, such as the concept of Type-M and Type-S errors.

Overall, the book is quite comprehensive and will leave the reader with a rich set of tools to think critically about statistics and to fit models in the real world. I look forward to grabbing a hard copy once the book is out, which seems to be in the summer to fall of 2020.

Update: Looks like the book is out!

Back to top ]]> https://lesslikely.com/statistics/regression-stories.html Tue, 11 Jun 2019 04:00:00 GMT Book Review: Fisher, Neyman, and the Creation of Classical Statistics Zad Rafi https://lesslikely.com/statistics/classical-lehmann.html

Erich Lehmann’s last book,1 which was published after his death, is on the history of classical statistics and its creators. Specifically, how his mentor Jerzy Neyman and his adversary Ronald Fisher helped lay the foundations for the methods that are used today in several fields.

Picture of the giants who founded frequentist statistics such as Egon Pearson, Ronald Fisher, and Jerzy Neyman

A Very Brief History of Classical Statistics

This post is intended to be a general review/summary of the book, which I recommend to everyone and anyone who is interested in statistics and science. The book clears up several misconceptions people have about how frequentist statistics came to be the dominant school of statistics. Thus, I want to go over four topics from Lehmann’s book that I believe people should know more about:

  • How the founders of classical statistics viewed Bayesian inference

  • What they each developed

  • How they came to become so conflicted

  • And how their views changed over time

Where Are The Bayesians?

As Stephen Senn points out in his Fisher Memorial Lecture at the Royal Statistical Society, there is a common myth that everyone who practiced applied statistics before the early 20th century was using Bayesian inference and doing everything correctly, but then Fisher came in and created significance testing, thus giving researchers a powerful tool to easily hack their data and produce publishable results, and now we have several replication crises because of this.

Of course, this is far from the truth and any thorough investigation into the history of statistics will clear up this up amongst many other misconceptions.

As several individuals may know, it was Thomas Bayes who came up with Bayes theorem and it was Richard Price who disseminated most of his writings after Bayes’s death. However, as many self-identified Bayesians will attest, using Bayes’ theorem does not make one a Bayesian. It is actually quite hard to know how Bayes would react to modern Bayesian inference. The Bayesian inference that we are familiar with today can be attributed to Pierre-Simon Laplace, who popularized what is now known as “objective Bayes.”

Portrait of Pierre Simon Laplace

Back then, it was not called “Bayesian inference” but was referred to as “inverse probability” and it was a method used by many before the dominance of classical statistics. So this is one part that common myths get right. Inverse probability did indeed have a moment in history before the dominance of frequentist statistics.

Laplace, and several others popularized such methods, but around the end of the 19th century, the tides began to shift. Several mathematicians and statisticians began to discourage the use of inverse probability because they saw it as a nonrigorous method of data analysis.

This can be seen in the following passages about Fisher.

“His first publication on this new approach to inference was a 1930 paper “Inverse probability.” The paper begins with a critique of the inverse (Bayesian) method. This section ends with Fisher’s asking:

If, then, we follow writers like Boole, Venn and Chrystal in rejecting the inverse argument as devoid of foundation and incapable even of consistent application, how are we to avoid the staggering falsity of saying that however extensive our knowledge of the values of x may be, yet we know nothing and can know nothing about the values of ?” (78)

Thus, Fisher was not the first to reject inverse probability, he was building on arguments from proto frequentists who already began to condemn inverse probability. Neyman was also a serious critic of inverse probability. In fact, he was probably more of a critic of it at a later point in time then Fisher (much on that later)!

“On one subject, Fisher and Neyman agreed. Fisher, after 1922, and Neyman, after 1937, were united in their strong opposition to the use of prior distributions (unless they were based on substantial empirical evidence).” (90)

Although the two giants of classical statistics both condemned inverse probability, it withstood their influential criticisms.

“It seems ironic that one of the most significant developments after Fisher and Neyman had established their foundations was to rejuvenate an approach they both had strongly opposed and thought to have vanquished: inverse probability…

The nineteenth century approach to inverse probability, championed particularly by Laplace, considered the prior distribution to represent complete ignorance. This concept, now called objective Bayes, was taken up and improved by the Cambridge geophysicist Harold Jeffreys, culminating in his 1939 book, “Theory of Probability.”

A different Bayesian approach, called subjective, was proposed by Ramsey (1926) and Bruno de Finetti in the 1930s. It considered probability as a measure of a person’s subjective degree of uncertainty about a situation. This view came into its own with the publication in 1954 of L. J. Savage’s book, “Foundations of Statistics,” in which he derives the existence of such subjective probabilities from a few, quite plausible, axioms.” (91)

Now that we have looked at how the founders of classical statistics viewed and attempted to discourage the use of inverse probability, we can move onto a brief summary of each of their individual contributions.

Fisher’s Contributions

Picture of Ronald Fisher sitting and smoking

Much of Fisher’s early work was a result of two individuals, Karl Pearson and William Gosset. Pearson’s work on the method of moments to estimate parameters led to Fisher developing his superior estimation method, maximum likelihood, which he presented in his 1922 foundations paper, “On the mathematical foundations of theoretical statistics.”

“Having defined the problem of statistics to be the estimation of parameters, Fisher states the properties that he desires for his estimators. They are consistency, efficiency, and sufficiency…

He then proposes what he had already suggested earlier in the section on the solution of the estimation problem, the method of maximum likelihood, which “consists, then, simply of choosing such values of these parameters as have the maximum likelihood.” Fisher believes that this method satisfies his three criteria, in particular that it satisfied the criterion of sufficiency, although he states that he “is not satisfied as to the mathematical rigor of any proof which I can put forward to that effect.” He also claims that sufficiency implies efficiency…

Thus, in this paper Fisher has not only formulated the general problem of optimal estimation, but he has also provided a solution. It is a stunning achievement.” (10)

Gosset’s initial work on test statistics, his inability derive proofs for small sample methods, and constant prodding led to Fisher developing several statistical tests which ended up being published in his highly influential book, Statistical Methods For Research Workers,

“For testing the value of a population mean, it had been customary to use a statistic equivalent to what today is called Student’s t, and to refer to the normal distribution. For large samples, this provided a good approximation.

However, Gosset soon realized that for the small samples with which he had to work, the approximation was inadequate. He then had the crucial insight that exact results could be obtained by making an additional assumption, namely that the form of the distribution of the observations is known. Gosset undertook to determine it for the case that the underlying distribution is normal, and he obtained the correct result, although he was not able to give a rigorous proof.

The first proof was obtained (although not published) by Fisher in 1912. His proof was finally published in 1915 [4], together with the corresponding proof for the correlation coefficient that Student had conjectured in a second paper of 1908(b). Fisher followed this in 1921 [14] with a derivation of the distribution of the intraclass correlation coefficient. And then, as a result of constant prodding and urging by Gosset, he found a number of additional small-sample distributions, and in 1925 presented the totality of these results in his book, “Statistical Methods for Research Workers.” (6)

In the book, Fisher’s main focus was on statistical testing and not estimation, and he made this clear,

“…the prime object of this book is to put into the hands of research workers… the means of applying statistical tests accurately to numerical data accumulated in their own laboratories … and later refers to the exact distributions with the use of which this book is chiefly concerned… Thus, the book does not primarily deal with estimation but with significance testing. In fact, estimation is never again mentioned.” (16)

His section about chi-squared tests and significance testing became highly influential,

“In preparing this table we have borne in mind that in practice we do not want to know the exact value of P for any observed , but, in the first place, whether or not the observed value is open to suspicion. If P is between .1 and .9 there is certainly no reason to suspect the hypothesis tested. If it is below .02 it is strongly indicated that the hypothesis fails to account for the whole of the facts. We shall not often be astray if we draw a conventional line at .05 and consider that higher values of indicate a real discrepancy.” (17)

He also presents examples,

“In the first of these, in particular, he finds a p-value between 0.01 and 0.02 and concludes: “If we take P = 0.05 as the limit of significant deviation, we shall say that in this case the deviations from expectation are significant.” (17)

And he expanded on significance testing with analysis of variance, which he had derived while working at Rothamsted analyzing crop data. The book was a large success,

“The first edition of 1,050 copies was sold out after three years, and the second edition of 1,250 copies in another two. Every two to three years necessitated a new edition, which usually contained some improvements and often additions. The size of the editions steadily increased and the eleventh edition of 1950 ran to 7,500 copies. The last edition, the fourteenth, was published posthumously in 1970 from notes Fisher had prepared before his death in 1962.” (25)

And set the groundwork for his next task, discussing experimental methods, which would be published in his second book, The Design of Experiments. In it, he discusses how techniques like randomization were necessary for the validity of statistical tests and how they perform amongst a wide variety of distributions,

“Randomisation properly carried out … ensures that the estimates of error will take proper care of all such causes of different growth rates, and relieves the experimenter from the anxiety of considering and estimating the magnitude of the innumerable causes by which his data may be disturbed. The one flaw in Darwin’s procedure was the absence of randomisation…

It seems to have escaped recognition that the physical act of randomisation which, as has been shown, is necessary for the validity of any test of significance, affords the means, in respect of any particular body of data, of examining the wider hypothesis in which no normality of distribution is implied.” (66)

Although he had proposed randomization tests as a way of dealing with nonnormal distributions, due to their tedious calculations, they never became popular at the time. The book also touched on several other concepts such as randomized blocks, Latin squares, and factorial designs. He also made his position very clear on significance tests and the null hypothesis,

“By increasing the size of the experiment, we can render it more sensitive, meaning by this that it will allow of the detection of a lower degree of sensory discrimination… . Since in every case the experiment is capable of disproving, but never of proving this hypothesis, we may say that the value of the experiment is increased whenever it permits the null hypothesis to be more readily disproved.” (64)

Here we can see Fisher’s concept of statistical power, though “sensitivity” was never a quantified concept. He also clearly states his position on the null hypothesis, that we can never accept it, a mistake that many researchers continue to make today.

Now that we have discussed some of Fisher’s contributions to classical statistics, we can discuss the contributions of Jerzy Neyman.

Neyman’s Contributions

Photo of Jerzy Neyman and his colleagues

Just like Fisher, Neyman was also impacted by Gosset. However, the influence was indirect. In the 1920s, Egon Pearson, had come across the small-sample tests that both Fisher and Gosset had popularized and had the realization that he must make a name for himself if he ever wished to be free of his father’s influence.

“In 1925-6, I was in a state of puzzlement, and realized that, if I was to continue an academic career as a mathematical statistician, I must construct for myself what might be termed a statistical philosophy, which would have to combine what I accepted from K. P.’s large- sample tradition with the newer ideas of Fisher.” (7)

Thus, he contacted Gosset about practical usage of the t-test, to which Gosset replied,

“Even if the chance is very small, say .00001, that doesn’t in itself necessarily prove that the sample is not drawn randomly from the population [specified by the hypothesis]; what it does is to show that if there is any alternative hypothesis which will explain the occurrence of the sample with a more reasonable probability, say .05 (such as that it belongs to a different population or that the sample wasn’t random or whatever will do the trick), you will be very much more inclined to consider that the original hypothesis is not true.” (E. S. Pearson, 1939.)

“In his obituary of Gosset, Pearson continues, Gosset’s reply had a tremendous influence on the direction of my subsequent work, for the first paragraph contains the germ of that idea which has formed the basis of all the later joint researches of Neyman and myself. It is the simple suggestion that the only valid reason for rejecting a statistical hypothesis is that some alternative explains the observed events with a greater degree of probability.” (7)

As a result, Pearson decided to collaborate with someone who was not taught by his father, but who also had the mathematical abilities to create a generalizable theorem that he had in mind. Thus, began the collaboration between Neyman and Pearson.

In 1928, they published a paper in Biometrika titled, “On the use and interpretation of certain test criteria,” where they introduced two kinds of errors,

  1. Sometimes,when hypothesis A is rejected, will in fact have been drawn from .

  2. More often, in accepting hypothesis A, will really have been drawn from [some alternative population] . (31)

As Lehmann notes, the paper was a great achievement,

“It introduces the consideration of alternatives, the two kinds of error, and the distinction between simple and composite hypotheses. In addition, of course, it proposes the likelihood ratio test. This test is intuitively appealing, and Neyman and Pearson show that in a number of important cases it leads to very satisfactory solutions. It has become the standard approach to new testing problems.” (34)

But the Neyman-Pearson lemma was still incomplete. It was between the years of 1930 and 1933 that Neyman had several insights into how to improve the theory, which Pearson already had felt satisfied with.

“In the next letter, dated March 8, Neyman suggests that he and Egon must “fix a certain plan, as we have lot of problems already started and then left in the wood.” He lists several such problems, among them: to finish what I have started to do with the variation calculus. You will understand it in a moment.

To reduce for a given level the errors of rejecting a true hypothesis, we may use any test. Now we want to find a test which would 1) reduce the probability of rejecting a true hypothesis to the level and 2) such that the probability of accepting a false hypothesis should be minimum. – We find that if such a test exists, then it is the -test. I am now shure [sic] that in a few days I shall be ready. This will show that the “ principle” is not only a principle but that there are arguments to prove that it is really “the best test.”” (35)

These correspondences led to their 1933 paper,

“On the Problem of the Most Efficient Tests of Statistical Hypotheses.” in which both authors introduce the novel idea of behavioral guidance,

“Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behavior with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong.” (36)

As Lehmann notes,

“After outlining the general theory, the paper in the next section deals with the case of simple hypotheses and brings the statement and proof of the basic result, now known as the Neyman-Pearson Fundamental Lemma. It states that for testing a simple hypothesis against a simple alternative, the test that at a given level maximizes the probability of rejection is the likelihood ratio test at that level.” (36)

Lehmann summarizes much of the collaboration with the following,

“The collaboration falls into two quite distinct parts. In the early stages, the important ideas, including in particular that of the likelihood ratio principle, all come from Pearson. In fact, Neyman frequency misunderstands them, and continually tries to interpret them in terms of inverse probability.

On the other hand, Pearson is sold on the likelihood ratio principle, which is intuitively appealing and which seems to give reasonable solutions in the cases on which they try it out. But for Neyman, as he is gradually catching on, intuitive appeal is not enough. If the principle is really as good as it appears to be, there ought to be logical justification.

And then one day in early 1930, he sees the light. Since there are two sources of error, one of which is being controlled, the best test is the one minimizing the other one. And from then on, it is Neyman who has the new ideas and Pearson is the reluctant follower. Neyman formulates, and shortly thereafter proves, the Fundamental Lemma and realizes that in some special cases there exist what they later call uniformly most powerful tests. These turn out to coincide with the likelihood ratio tests.” (39)

The Fallout Between The Creators Of Classical Statistics

The conflict between Neyman and Fisher is well known, however, very few are able to accurately point out what lead to each individual strongly detesting the other.

In fact, early correspondences between Neyman and Fisher showed that they were incredibly friendly towards one another. In 1932, Neyman asked Fisher to review their 1933 paper before they submitted it, to which Fisher replied,

“I should be very much interested to see your paper on “the best tests,” as the whole question of tests of significance seems to me to be of immense philosophical importance, and the work you showed me was surely of great promise. It is quite probable that if the work is submitted to the Royal Society, I might be asked to act as referee, and in that case I shall certainly not refuse.” (45)

Fisher not only read the paper, but read it so carefully that he was able to catch a mathematical error and point it out to Neyman and Pearson before it was published,

“When the paper appeared in 1933, the omission was corrected, and a footnote acknowledged that, “We are indebted to Dr. R. A. Fisher – for kindly calling our attention to the fact that we had originally omitted to refer to this restriction.” (46)

Neyman thanked Fisher for his help,

Neyman: “Pearson writes that you have recommended our paper for publication. Although it maybe considered ridiculous to thank a judge, I have intense feeling of gratefulness, which I hope you will kindly accept…” (57)

Fisher replies, “It was a great pleasure to hear from you again.” (57)

Neyman: “I am often thinking that it would be very useful for me to work with you. Unfortunately, this requires considerable amount of money – without speaking of your consent – of course…” (57)

Fisher answers, “You may be sure of my consent,” and in the next letter, “I like hearing from Poland. Best wishes for a Merry Christmas.” (58)

Unfortunately, their relationship began to degrade after the retirement of Karl Pearson. The department of applied statistics that he was the head of was split into the department of statistics, which would be led by his son Egon, and the department of genetics, where Fisher was appointed as Galton professor. Thus, Fisher, one of the creators of classical statistics, was not allowed to teach statistics, while in the floor downstairs, Egon Pearson (a man that he was surely not fond of) was leading the new statistics department.

This change in tone could be seen by the correspondence between Neyman and Fisher following Fisher’s appointment as Galton professor,

“Dr. Pearson writes me that soon you will be Galton Professor at the University College, London. Very probably this means a general reorganization of the Department of Applied Statistics and possibly new people will be needed. I know that there are many statisticians in England and that many of them would be willing to work under you. But improbable things do happen sometimes and you may have a vacant position in your laboratory. In that case please consider whether I can be of any use.” (58)

Fisher replies,

“Many thanks for your letter of congratulation. You will be interested to hear that the Dept. of Statistics has now been separated officially from the Galton Laboratory. I think Egon Pearson is designated as Reader in Statistics. This arrangement will be much laughed at, but it will be rather a poor joke, I fancy, for both Pearson and myself. I think, however, we will make the best of it.

I shall not lecture on statistics, but probably on “the logic of experimentation,” so that my lectures will not be troubled by students who cannot see through a wire fence. I wish I had a fine place for you, but it will be long before my new department can be given any sort of unity and coherence, and you will be head of a faculty before I shall be able to get much done. If in England, do not fail to see me at University College.” (58)

Of course, there is little doubt that both Karl and Egon Pearson contributed to the fallout between Neyman and Fisher. In 1929, Egon Pearson had submitted a critical review of the second edition of Statistical Methods For Research Workers to Nature,

“There is one criticism, however, which must be made from the statistical point of view. A large number of the tests developed are based…on the assumption that the population sampled is of the “normal” form. That this is the case may be gathered from a careful reading of the text, but the point is not sufficiently emphasized.

It does not appear reasonable to lay stress on the “exactness” of the tests when no means whatever are given of appreciating how rapidly they become inexact as the population sampled diverges from normality. That the tests, for example, connected with the analysis of variance are far more dependent on normality than those involving Student’s z (or t) distribution is almost certain, but no clear indication of the need for caution in their application is given.” (22)

As Lehmann points out, Fisher was deeply offended by this review. Nearly six years later (1935), Neyman encountered a similar reaction when he submitted a paper titled, “Statistical problems in agricultural experimentation” pointing out problems with some of the concepts that Fisher had introduced in his book, The Design of Experiments. Fisher was furious,

“I had hoped that Dr. Neyman’s paper would be on a subject with which the author was fully acquainted, and on which he could speak with authority, as in the case of his address to the Society delivered last summer. Since seeing the paper, I have come to the conclusion that Dr. Neyman had been somewhat unwise in his choice of topics… (59)

Were it not for the persistent effort which Dr. Neyman and Dr. Pearson had made to treat what they speak of as problems of estimation, by means merely of tests of significance, I have no doubt that Dr. Neyman would not have been in any danger of falling into the series of misunderstandings which his paper revealed.” (59)

Correspondences from there on out had become hostile,

“Neyman later (Reid 1982, p. 126) recalls that a week after this meeting, Fisher stopped by his room at University College:

And he said to me that he and I are in the same building… . That, as I know, he had published a book – and that’s Statistical Methods for Research Workers – and he is upstairs from me so he knows something about my lectures – that from time to time I mention his ideas, this and that – and that this would be quite appropriate if I were not here in the College but, say, in California – but if I am going to be at University College, then this is not acceptable to him.

And then I said, “Do you mean that if I am here, I should just lecture using your book?” And then he gave an affirmative answer. And I said, “Sorry, no. I cannot promise that.” And then he said, “Well, if so, then from now on I shall oppose you in all my capacities.”

Reid also reports (p. 124) that, After the Royal Statistical Society meeting of March 28, relations between workers on the two floors of K. P.’s old preserve became openly hostile. One evening, late that spring, Neyman and Pearson returned to their department after dinner to do some work.

Entering, they were startled to find strewn on the floor the wooden models which Neyman had used to illustrate his talk on the relative advantages of randomized blocks and Latin squares. They were regularly kept in a cupboard in the laboratory. Both Neyman and Pearson always believed that the models were removed by Fisher in a fit of anger.” (59)

There Is No One Neyman Nor One Fisher

When Fisher released his first edition of Statistical Methods For Research Workers (SMRW), he recommended 5% or 1% as good choices for significance levels, with the latter being used when a “more stringent requirement was necessary.” Fisher was also not interested in exact P-values as pointed out in the section discussing his contributions.

Many of these views changed as he released later editions of his SMRW and his new book, Statistical Methods for Scientific Inference (SMSI). For example, he no longer recommended a particular level of significance,

“In his late, 1956, book SMSI, Fisher protested that “no scientific worker has a fixed level of significance at which from year to year, and in all circumstances, he rejects hypotheses; he rather gives his mind to each particular case, and his ideas” (52)

In the 13th edition of SMRW he stated,

“The actual value of P obtainable from the table by interpolation indicates the strength of the evidence against the hypothesis. A value of exceeding the 5 per cent. point is seldom to be disregarded.” (52)

Thus, Fisher had changed his mind on the topic.

Neyman too had a significant change of mind during the course of his collaboration with Pearson. At first, he constantly defaulted to inverse probability methods, as noted by Pearson’s letter to him in 1978,

“I have eight letters which you wrote to me during February and March 1929, trying to persuade me to put my name as a joint author. But you had introduced an a priori law of probability…, and I was not willing to start from this basis. True we had given the inverse probability as an alternative approach in our 1928 Part I paper, but I must in 1927-28 still have been ready to concede to your line of thought.

However, by 1929 I had come down firmly to agree with Fisher that prior distributions should not be used, except in cases where they were based on real knowledge, e.g., in some Mendelian problems. You were disappointed, but accepted my decision; after all, the whole mathematical development in the paper was yours.” (42)

Though eventually Neyman abandoned his interest in inverse probability and became a serious critic,

“His conviction of the inapplicability of the inverse method had by then become a fundamental part of his statistical philosophy, from which he never wavered.” (42)

Although this post is mainly fixated on the book by Lehmann, I would like to at least paste this one relevant passage from Hulbert & Lombardi, 2009,2

“In a later philosophical essay, Neyman (1977: 112) recounted their cloud-seeding studies, and labeled P values of 0.09, 0.03, and < 0.01 reported in their earlier paper (Lovasich et al. 1971), as “approximately significant,” “significant,” and “highly significant,” respectively. The dichotomies of the paleoFisherian and Neyman-Pearsonian frameworks were quietly admitted to be less appropriate than more nebulous interpretations — at least in cloud work!

Indeed, Cox (2006a: 43, 195) has noted that “the differences between Fisher and Neyman … were not nearly as great as the asperity of the arguments between them might suggest … [and in] actual practice … Neyman … often reported p-values whereas some of Fisher’s use of tests … was much more dichotomous”!

As we can see from a summary of Lehmann’s book, the individuals who founded classical statistics were skilled and talented individuals who were also complex and had various reasons for doing what they did. I hope this blog post encourages readers to fully dive into Lehmann’s book where he gives a far more detailed account of Fisher and Neyman’s contributions to classical statistics.

References

Back to top

References

1. Lehmann EL. (2011). “Fisher, Neyman, and the Creation of Classical Statistics.” Springer New York. doi: 10.1007/978-1-4419-9500-1.
2. Hurlbert SH, Lombardi CM. (2009). “Final collapse of the Neyman-Pearson decision theoretic framework and rise of the neoFisherian.” Ann Zool Fennici. 46:311–349. doi: 10.5735/086.046.0501.
]]> https://lesslikely.com/statistics/classical-lehmann.html Sun, 30 Dec 2018 05:00:00 GMT GAMS Zad Rafi Sir Panda https://lesslikely.com/statistics/gams.html
Code 
library(mgcv) # Fit and interrogate GAMs
library(tidyverse) # Tidy and flexible data manipulation
library(marginaleffects) # Compute conditional and marginal effects
library(ggplot2) # Flexible plotting
library(patchwork) # Combining ggplot objects


plant <- CO2 |>
  as_tibble() |>
  rename(plant = Plant, type = Type, treatment = Treatment) |>
  mutate(plant = factor(plant, ordered = FALSE))
Code 
model_1 <- gam(
  uptake ~ treatment * type +
    s(plant, bs = "re") +
    s(conc, by = treatment, k = 7),
  data = plant,
  method = "REML",
  family = Gamma(link = "log")
)
summary(model_1)
#> 
#> Family: Gamma 
#> Link function: log 
#> 
#> Formula:
#> uptake ~ treatment * type + s(plant, bs = "re") + s(conc, by = treatment, 
#>     k = 7)
#> 
#> Parametric coefficients:
#>                                  Estimate Std. Error t value Pr(>|t|)    
#> (Intercept)                        3.5161     0.0638   55.14  < 2e-16 ***
#> treatmentchilled                  -0.1132     0.0902   -1.26  0.21399    
#> typeMississippi                   -0.3120     0.0902   -3.46  0.00098 ***
#> treatmentchilled:typeMississippi  -0.3604     0.1275   -2.83  0.00631 ** 
#> ---
#> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#> 
#> Approximate significance of smooth terms:
#>                              edf Ref.df     F p-value    
#> s(plant)                    7.03   8.00  8.02  <2e-16 ***
#> s(conc):treatmentnonchilled 5.19   5.68 83.89  <2e-16 ***
#> s(conc):treatmentchilled    5.01   5.55 58.97  <2e-16 ***
#> ---
#> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#> 
#> R-sq.(adj) =  0.957   Deviance explained = 95.5%
#> -REML = 239.08  Scale est. = 0.010327  n = 84
coef(model_1)
#>                      (Intercept)                 treatmentchilled                  typeMississippi 
#>                          3.51609                         -0.11321                         -0.31196 
#> treatmentchilled:typeMississippi                       s(plant).1                       s(plant).2 
#>                         -0.36041                         -0.04123                         -0.01529 
#>                       s(plant).3                       s(plant).4                       s(plant).5 
#>                          0.05652                         -0.04010                          0.02693 
#>                       s(plant).6                       s(plant).7                       s(plant).8 
#>                          0.01317                         -0.05593                          0.04989 
#>                       s(plant).9                      s(plant).10                      s(plant).11 
#>                          0.00604                         -0.21558                          0.09712 
#>                      s(plant).12    s(conc):treatmentnonchilled.1    s(conc):treatmentnonchilled.2 
#>                          0.11846                          5.23232                          1.13324 
#>    s(conc):treatmentnonchilled.3    s(conc):treatmentnonchilled.4    s(conc):treatmentnonchilled.5 
#>                         -1.90483                          0.12919                          0.23606 
#>    s(conc):treatmentnonchilled.6       s(conc):treatmentchilled.1       s(conc):treatmentchilled.2 
#>                          1.20260                          3.69059                          2.11750 
#>       s(conc):treatmentchilled.3       s(conc):treatmentchilled.4       s(conc):treatmentchilled.5 
#>                         -2.36770                          0.06347                          0.23654 
#>       s(conc):treatmentchilled.6 
#>                          0.96775
Code 
plot(model_1, select = 2, shade = TRUE)
abline(h = 0, lty = "dashed")

Code 
plot_predictions(model_1,
  condition = "conc",
  type = "link"
) +
  labs(
    y = "Linear predictor (link scale)",
    title = "Average smooth effect of concentration",
    subtitle = "Aggregated across treatments and types"
  )

Code 
plot_predictions(model_1,
  condition = c("conc", "treatment", "type"),
  type = "link"
) +
  labs(
    y = "Linear predictor (link scale)",
    title = "Average smooth effect of concentration",
    subtitle = "Per treatment, per type"
  )

Code 
plot_slopes(model_1,
  variables = "conc",
  condition = c("conc", "treatment"),
  type = "link"
) +
  geom_hline(yintercept = 0, linetype = "dashed") +
  labs(
    y = "1st derivative of the linear predictor",
    title = "Conditional slopes of the concentration effect",
    subtitle = "Per treatment, per type"
  )

Code 
plot_predictions(model_1,
  condition = "conc",
  type = "response", points = 0.5,
  rug = TRUE
) +
  labs(
    y = "Expected response",
    title = "Average smooth effect of concentration",
    subtitle = "Aggregated across treatments and types"
  )

Code 
plot_predictions(model_1,
  condition = c("conc", "treatment", "type"),
  type = "response", points = 0.5,
  rug = TRUE
) +
  labs(
    y = "Expected response",
    title = "Average smooth effect of concentration",
    subtitle = "Per treatment, per type"
  )

Code 
plot_slopes(model_1,
  variables = "conc",
  condition = c("conc", "treatment", "type"),
  type = "response"
) +
  geom_hline(yintercept = 0, linetype = "dashed") +
  labs(
    y = "1st derivative of the expected response",
    title = "Conditional slopes of the concentration effect",
    subtitle = "Per treatment, per type"
  )

Code 
Xp <- predict(model_1, type = "lpmatrix")
dim(Xp)
#> [1] 84 28
colnames(Xp)
#>  [1] "(Intercept)"                      "treatmentchilled"                 "typeMississippi"                 
#>  [4] "treatmentchilled:typeMississippi" "s(plant).1"                       "s(plant).2"                      
#>  [7] "s(plant).3"                       "s(plant).4"                       "s(plant).5"                      
#> [10] "s(plant).6"                       "s(plant).7"                       "s(plant).8"                      
#> [13] "s(plant).9"                       "s(plant).10"                      "s(plant).11"                     
#> [16] "s(plant).12"                      "s(conc):treatmentnonchilled.1"    "s(conc):treatmentnonchilled.2"   
#> [19] "s(conc):treatmentnonchilled.3"    "s(conc):treatmentnonchilled.4"    "s(conc):treatmentnonchilled.5"   
#> [22] "s(conc):treatmentnonchilled.6"    "s(conc):treatmentchilled.1"       "s(conc):treatmentchilled.2"      
#> [25] "s(conc):treatmentchilled.3"       "s(conc):treatmentchilled.4"       "s(conc):treatmentchilled.5"      
#> [28] "s(conc):treatmentchilled.6"
beta <- coef(model_1)
all.equal(names(beta), colnames(Xp))
#> [1] TRUE
Code 
preds <- as.vector(Xp %*% beta)
all.equal(fitted(model_1), exp(preds))
#> [1] TRUE
newXp <- predict(model_1,
  type = "lpmatrix",
  newdata = data.frame(
    plant = "Qn1",
    treatment = "nonchilled",
    type = "Mississippi",
    conc = 278
  )
)
dim(newXp)
#> [1]  1 28
exp(newXp %*% beta)
#>   [,1]
#> 1 27.6
Code 
conc_seq <- seq(from = min(plant$conc), max(plant$conc), length.out = 500)
newdat <- data.frame(
  conc = conc_seq,
  plant = "Qn1",
  treatment = "nonchilled",
  type = "Mississippi"
)
newXp <- predict(model_1, type = "lpmatrix", newdata = newdat)
conc_coefs <- model_1$smooth[[2]]$first.para:model_1$smooth[[2]]$last.para
conc_coefs
#> [1] 17 18 19 20 21 22

Trace plot of imputed datasets.

Python

Code 
model_2 <- glm(
  uptake ~ treatment * type +
    poly(conc, 4) * treatment + plant,
  data = plant,
  family = Gamma(link = "log")
)
plot_predictions(model_1,
  condition = c("conc", "treatment", "type"),
  type = "response", points = 0.5,
  rug = TRUE
) +
  labs(
    y = "Expected response",
    title = "Average smooth effect of concentration",
    subtitle = "Per treatment, per type"
  )

Code 
plot_predictions(model_2,
  condition = c("conc", "treatment", "type"),
  type = "response", points = 0.5,
  rug = TRUE
) +
  labs(
    y = "Expected response",
    title = "Average smooth effect of concentration",
    subtitle = "Per treatment, per type"
  )

R

Code 
avg_comparisons(model_1,
  newdata = datagrid(
    conc = conc_seq,
    treatment = unique,
    type = unique
  ),
  variables = "treatment",
  by = "type"
)
#> 
#>         type Estimate Std. Error     z Pr(>|z|)    S 2.5 % 97.5 %
#>  Quebec         -4.76       3.04 -1.57    0.117  3.1 -10.7   1.19
#>  Mississippi   -10.71       2.20 -4.87   <0.001 19.8 -15.0  -6.40
#> 
#> Term: treatment
#> Type: response
#> Comparison: chilled - nonchilled
avg_comparisons(model_1,
  newdata = datagrid(
    conc = conc_seq,
    treatment = unique,
    type = unique
  ),
  variables = "treatment",
  by = c("conc", "type")
)
#> 
#>    conc        type Estimate Std. Error      z Pr(>|z|)    S  2.5 % 97.5 %
#>    95.0 Quebec         0.413       1.60  0.258  0.79644  0.3  -2.73   3.55
#>    95.0 Mississippi   -3.114       1.02 -3.057  0.00224  8.8  -5.11  -1.12
#>    96.8 Quebec         0.373       1.61  0.232  0.81662  0.3  -2.78   3.53
#>    96.8 Mississippi   -3.183       1.03 -3.102  0.00192  9.0  -5.19  -1.17
#>    98.6 Quebec         0.332       1.62  0.205  0.83731  0.3  -2.84   3.51
#> --- 990 rows omitted. See ?print.marginaleffects ---
#>   996.4 Mississippi  -11.426       2.75 -4.156  < 0.001 14.9 -16.81  -6.04
#>   998.2 Quebec        -4.436       3.97 -1.119  0.26334  1.9 -12.21   3.34
#>   998.2 Mississippi  -11.427       2.76 -4.147  < 0.001 14.9 -16.83  -6.03
#>  1000.0 Quebec        -4.435       3.98 -1.115  0.26467  1.9 -12.23   3.36
#>  1000.0 Mississippi  -11.428       2.76 -4.138  < 0.001 14.8 -16.84  -6.02
#> Term: treatment
#> Type: response
#> Comparison: chilled - nonchilled
plot_comparisons(model_1,
  newdata = datagrid(
    conc = conc_seq,
    treatment = unique,
    type = unique
  ),
  variables = "treatment",
  by = c("conc", "type"),
  type = "link"
) +
  geom_hline(yintercept = 0, linetype = "dashed") +
  labs(
    y = "Estimated difference",
    title = "Difference between treatment levels",
    subtitle = "Chilled - nonchilled, per type"
  )

Stata

Code 
max_growth <- function(hi, lo, x) {
  dydx <- (hi - lo) / 1e-6
  dydx_max <- max(dydx)
  x[dydx == dydx_max][1]
}

comparisons(model_1,
  newdata = datagrid(
    conc = conc_seq,
    treatment = unique,
    type = unique
  ),
  variables = list("conc" = 1e-6),
  vcov = FALSE,
  by = "treatment",
  comparison = max_growth
)
#> 
#>   treatment Estimate
#>  nonchilled      157
#>  chilled         151
#> 
#> Term: conc
#> Type: response
#> Comparison: +1e-06
hypotheses(slopes(model_1,
  newdata = datagrid(
    conc = conc_seq,
    treatment = unique,
    type = unique
  ),
  variables = "conc",
  by = c("conc", "treatment"),
  type = "link"
)) %>%
  dplyr::filter(p.value <= 0.05)
#> 
#>  Estimate Std. Error    z Pr(>|z|)    S    2.5 %  97.5 %
#>   0.00809   0.000876 9.23   <0.001 65.0 6.37e-03 0.00980
#>   0.00647   0.000830 7.79   <0.001 47.1 4.84e-03 0.00810
#>   0.00809   0.000877 9.22   <0.001 64.9 6.37e-03 0.00980
#>   0.00647   0.000829 7.81   <0.001 47.3 4.84e-03 0.00809
#>   0.00808   0.000876 9.22   <0.001 64.9 6.37e-03 0.00980
#> --- 196 rows omitted. See ?print.marginaleffects ---
#>   0.00127   0.000503 2.53   0.0115  6.4 2.85e-04 0.00226
#>   0.00122   0.000518 2.36   0.0184  5.8 2.06e-04 0.00224
#>   0.00117   0.000523 2.25   0.0247  5.3 1.49e-04 0.00220
#>   0.00113   0.000530 2.12   0.0337  4.9 8.69e-05 0.00217
#>   0.00108   0.000542 1.99   0.0465  4.4 1.69e-05 0.00214
#> Term: conc

Computational Environments

Overall Platforms

Code 
sessioninfo::session_info(info = c("platform", "external"))
#> ─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.6.1 (2026-06-24)
#>  os       macOS Golden Gate 27.0
#>  system   aarch64, darwin25.4.0
#>  ui       unknown
#>  language (EN)
#>  collate  en_US
#>  ctype    C.UTF-8
#>  tz       America/New_York
#>  date     2026-09-13
#>  pandoc   3.11 @ /opt/homebrew/bin/ (via rmarkdown)
#>  quarto   1.10.18 @ /usr/local/bin/quarto
#> 
#> ─ External software ────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting        value
#>  cairo          1.18.4
#>  cairoFT        2.14.3/2.18.1
#>  pango
#>  png
#>  jpeg
#>  tiff
#>  tcl            9.0
#>  curl           8.7.1
#>  zlib           1.2.12
#>  bzlib          1.0.8, 13-Jul-2019
#>  xz             5.8.3
#>  deflate
#>  zstd           1.5.7
#>  PCRE           10.48 2026-08-31
#>  ICU            78.1
#>  TRE            TRE 0.8.0 R_fixes (BSD)
#>  iconv          Apple or GNU libiconv 1.11 /usr/lib/libiconv.2.dylib
#>  readline       8.3
#>  BLAS           /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib
#>  lapack         /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib
#>  lapack_version 3.12.1
#> 
#> ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

R Packages

Code 
sessioninfo::session_info(info = "packages")
#> ═ Session info ═════════════════════════════════════════════════════════════════════════════════════════════════════════════════
#> ─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  package           * version  date (UTC) lib source
#>  abind               1.4-8    2024-09-12 [1] CRAN (R 4.6.1)
#>  Amelia            * 1.8.3    2024-11-08 [1] CRAN (R 4.6.1)
#>  arm                 1.15-3   2026-04-15 [1] CRAN (R 4.6.1)
#>  arrayhelpers        1.1-2    2026-07-24 [1] CRAN (R 4.6.1)
#>  backports           1.5.1    2026-04-03 [1] CRAN (R 4.6.1)
#>  base64enc           0.1-6    2026-02-02 [1] CRAN (R 4.6.1)
#>  bayesplot         * 1.16.0   2026-08-25 [1] CRAN (R 4.6.1)
#>  bitops              1.1-0    2026-07-30 [1] CRAN (R 4.6.1)
#>  blogdown          * 1.24     2026-06-19 [1] CRAN (R 4.6.1)
#>  boot              * 1.3-32   2025-08-29 [1] CRAN (R 4.6.1)
#>  bootImpute        * 1.3.0    2025-12-15 [1] CRAN (R 4.6.1)
#>  bridgesampling      1.2-1    2025-11-19 [1] CRAN (R 4.6.1)
#>  brms              * 2.23.0   2025-09-09 [1] CRAN (R 4.6.1)
#>  Brobdingnag         1.2-9    2022-10-19 [1] CRAN (R 4.6.1)
#>  broom             * 1.0.13   2026-05-14 [1] CRAN (R 4.6.1)
#>  broom.mixed       * 0.2.9.7  2026-02-17 [1] CRAN (R 4.6.1)
#>  Cairo             * 1.7-0    2025-10-29 [1] CRAN (R 4.6.1)
#>  callr               3.8.0    2026-06-05 [1] CRAN (R 4.6.1)
#>  car               * 3.1-5    2026-02-03 [1] CRAN (R 4.6.1)
#>  carData           * 3.0-6    2026-01-30 [1] CRAN (R 4.6.1)
#>  caTools             1.18.4   2026-07-20 [1] CRAN (R 4.6.1)
#>  checkmate         * 2.3.4    2026-02-03 [1] CRAN (R 4.6.1)
#>  class               7.3-24   2026-08-03 [1] CRAN (R 4.6.1)
#>  cli                 3.6.6    2026-04-09 [1] CRAN (R 4.6.1)
#>  clipr               0.8.1    2026-05-25 [1] CRAN (R 4.6.1)
#>  cluster             2.1.8.3  2026-07-30 [1] CRAN (R 4.6.1)
#>  coda              * 0.19-4.1 2024-01-31 [1] CRAN (R 4.6.1)
#>  codetools           0.2-20   2024-03-31 [1] CRAN (R 4.6.0)
#>  colorspace        * 2.1-3    2026-07-12 [1] CRAN (R 4.6.1)
#>  concurve          * 3.0.5    2026-09-13 [1] local
#>  cowplot           * 1.2.0    2025-07-07 [1] CRAN (R 4.6.1)
#>  crayon              1.5.3    2024-06-20 [1] CRAN (R 4.6.1)
#>  curl                8.0.0    2026-08-25 [1] CRAN (R 4.6.1)
#>  data.table          1.18.6.1 2026-08-24 [1] CRAN (R 4.6.1)
#>  DBI                 1.3.0    2026-02-25 [1] CRAN (R 4.6.1)
#>  DEoptimR            1.2-1    2026-08-20 [1] CRAN (R 4.6.1)
#>  desc                1.4.3    2023-12-10 [1] CRAN (R 4.6.1)
#>  details             0.4.0    2025-02-09 [1] CRAN (R 4.6.0)
#>  dichromat           2.0-1    2026-07-22 [1] CRAN (R 4.6.1)
#>  digest              0.6.39   2025-11-19 [1] CRAN (R 4.6.1)
#>  distributional      0.9.0    2026-09-08 [1] CRAN (R 4.6.1)
#>  doParallel        * 1.0.17   2022-02-07 [1] CRAN (R 4.6.1)
#>  doRNG               1.8.6.3  2026-02-05 [1] CRAN (R 4.6.1)
#>  dplyr             * 1.2.1    2026-04-03 [1] CRAN (R 4.6.1)
#>  e1071               1.7-17   2025-12-18 [1] CRAN (R 4.6.1)
#>  emmeans             2.0.4    2026-07-15 [1] CRAN (R 4.6.1)
#>  estimability        2.0.0    2026-06-26 [1] CRAN (R 4.6.1)
#>  evaluate            1.0.5    2025-08-27 [1] CRAN (R 4.6.1)
#>  extremevalues       2.4.1    2024-12-17 [1] CRAN (R 4.6.1)
#>  farver              2.1.2    2024-05-13 [1] CRAN (R 4.6.1)
#>  fastmap             1.2.0    2024-05-15 [1] CRAN (R 4.6.1)
#>  forcats           * 1.0.1    2025-09-25 [1] CRAN (R 4.6.1)
#>  foreach           * 1.5.2    2022-02-02 [1] CRAN (R 4.6.1)
#>  foreign             0.8-91   2026-01-29 [1] CRAN (R 4.6.1)
#>  Formula             1.2-6    2026-08-03 [1] CRAN (R 4.6.1)
#>  fs                  2.1.0    2026-04-18 [1] CRAN (R 4.6.1)
#>  furrr               0.4.0    2026-03-31 [1] CRAN (R 4.6.1)
#>  future            * 1.75.0   2026-07-20 [1] CRAN (R 4.6.1)
#>  future.apply      * 1.20.2   2026-02-20 [1] CRAN (R 4.6.1)
#>  gamlss            * 5.5-0    2025-08-19 [1] CRAN (R 4.6.1)
#>  gamlss.data       * 6.0-7    2025-09-04 [1] CRAN (R 4.6.1)
#>  gamlss.dist       * 6.1-11   2026-09-10 [1] CRAN (R 4.6.1)
#>  generics            0.1.4    2025-05-09 [1] CRAN (R 4.6.1)
#>  ggcorrplot        * 0.3.0    2026-07-24 [1] CRAN (R 4.6.1)
#>  ggdist              3.3.3    2025-04-23 [1] CRAN (R 4.6.1)
#>  ggplot2           * 4.0.3    2026-04-22 [1] CRAN (R 4.6.1)
#>  ggtext            * 0.2.0    2026-08-28 [1] CRAN (R 4.6.1)
#>  glmnet              5.0      2026-05-04 [1] CRAN (R 4.6.1)
#>  globals             0.19.1   2026-03-13 [1] CRAN (R 4.6.1)
#>  glue                1.8.1    2026-04-17 [1] CRAN (R 4.6.1)
#>  gridExtra           2.3.1    2026-06-25 [1] CRAN (R 4.6.1)
#>  gridtext            0.1.6    2026-02-19 [1] CRAN (R 4.6.0)
#>  gtable              0.3.6    2024-10-25 [1] CRAN (R 4.6.1)
#>  here              * 1.0.2    2025-09-15 [1] CRAN (R 4.6.1)
#>  Hmisc             * 5.3-0    2026-09-06 [1] CRAN (R 4.6.1)
#>  hms                 1.1.4    2025-10-17 [1] CRAN (R 4.6.1)
#>  htmlTable           2.5.0    2026-04-22 [1] CRAN (R 4.6.1)
#>  htmltools         * 0.5.9    2025-12-04 [1] CRAN (R 4.6.1)
#>  htmlwidgets         1.6.4    2023-12-06 [1] CRAN (R 4.6.1)
#>  httr                1.4.9    2026-09-01 [1] CRAN (R 4.6.1)
#>  ImputeRobust      * 1.3-1    2018-11-30 [1] CRAN (R 4.6.1)
#>  inline              0.3.21   2025-01-09 [1] CRAN (R 4.6.1)
#>  insight             1.5.4    2026-09-05 [1] CRAN (R 4.6.1)
#>  iterators         * 1.0.14   2022-02-05 [1] CRAN (R 4.6.1)
#>  itertools           0.1-3    2014-03-12 [1] CRAN (R 4.6.1)
#>  jomo                2.7-6    2023-04-15 [1] CRAN (R 4.6.1)
#>  jsonlite            2.0.0    2025-03-27 [1] CRAN (R 4.6.1)
#>  JuliaCall         * 0.17.6   2024-12-07 [1] CRAN (R 4.6.1)
#>  kableExtra        * 1.4.1    2026-07-08 [1] CRAN (R 4.6.1)
#>  knitr             * 1.52     2026-09-06 [1] CRAN (R 4.6.1)
#>  labeling            0.4.3    2023-08-29 [1] CRAN (R 4.6.1)
#>  laeken              0.5.3    2024-01-25 [1] CRAN (R 4.6.1)
#>  latex2exp         * 0.9.8    2026-01-09 [1] CRAN (R 4.6.1)
#>  lattice           * 0.23-1   2026-08-12 [1] CRAN (R 4.6.1)
#>  lifecycle           1.0.5    2026-01-08 [1] CRAN (R 4.6.1)
#>  listenv             1.0.0    2026-06-22 [1] CRAN (R 4.6.1)
#>  lme4                2.0-6    2026-07-16 [1] CRAN (R 4.6.1)
#>  lmtest              0.9-40   2022-03-21 [1] CRAN (R 4.6.1)
#>  loo               * 2.10.1   2026-07-24 [1] CRAN (R 4.6.1)
#>  lubridate         * 1.9.5    2026-02-04 [1] CRAN (R 4.6.1)
#>  magick              2.9.1    2026-02-28 [1] CRAN (R 4.6.1)
#>  magrittr          * 2.0.5    2026-04-04 [1] CRAN (R 4.6.1)
#>  marginaleffects   * 1.0.0    2026-09-03 [1] CRAN (R 4.6.1)
#>  MASS              * 7.3-66   2026-07-15 [1] CRAN (R 4.6.1)
#>  Matrix            * 1.7-6    2026-07-25 [1] CRAN (R 4.6.1)
#>  MatrixModels        0.5-4    2025-03-26 [1] CRAN (R 4.6.1)
#>  matrixStats         1.5.0    2025-01-07 [1] CRAN (R 4.6.1)
#>  mcmc                0.9-8    2023-11-16 [1] CRAN (R 4.6.1)
#>  MCMCpack          * 1.7-1    2024-08-27 [1] CRAN (R 4.6.1)
#>  mgcv              * 1.9-4    2025-11-07 [1] CRAN (R 4.6.0)
#>  mi                * 1.3.1    2026-07-28 [1] CRAN (R 4.6.1)
#>  mice              * 3.19.0   2025-12-10 [1] CRAN (R 4.6.1)
#>  miceadds          * 3.20-10  2026-05-28 [1] CRAN (R 4.6.1)
#>  miceFast          * 0.9.1    2026-02-26 [1] CRAN (R 4.6.1)
#>  minqa               1.2.8    2024-08-17 [1] CRAN (R 4.6.1)
#>  missForest        * 1.6.1    2025-10-26 [1] CRAN (R 4.6.1)
#>  mitml             * 0.4-5    2023-03-08 [1] CRAN (R 4.6.1)
#>  mitools             2.7      2026-08-31 [1] CRAN (R 4.6.1)
#>  multcomp            1.4-32   2026-08-21 [1] CRAN (R 4.6.1)
#>  mvtnorm           * 1.4-2    2026-07-12 [1] CRAN (R 4.6.1)
#>  nlme              * 3.1-171  2026-09-01 [1] CRAN (R 4.6.1)
#>  nloptr              2.2.1    2025-03-17 [1] CRAN (R 4.6.1)
#>  nnet                7.3-21   2026-08-03 [1] CRAN (R 4.6.1)
#>  opdisDownsampling   1.7      2026-09-04 [1] CRAN (R 4.6.1)
#>  otel                0.2.0    2025-08-29 [1] CRAN (R 4.6.1)
#>  pan                 2.0      2026-06-30 [1] CRAN (R 4.6.1)
#>  pander              0.6.6    2025-03-01 [1] CRAN (R 4.6.1)
#>  parallelly        * 1.48.0   2026-06-29 [1] CRAN (R 4.6.1)
#>  patchwork         * 1.3.2    2025-08-25 [1] CRAN (R 4.6.1)
#>  pbmcapply         * 1.5.1    2022-04-28 [1] CRAN (R 4.6.1)
#>  performance       * 0.18.2   2026-09-10 [1] CRAN (R 4.6.1)
#>  pillar              1.11.1   2025-09-17 [1] CRAN (R 4.6.1)
#>  pkgbuild            1.4.8    2025-05-26 [1] CRAN (R 4.6.1)
#>  pkgconfig           2.0.3    2019-09-22 [1] CRAN (R 4.6.1)
#>  plyr                1.8.9    2023-10-02 [1] CRAN (R 4.6.1)
#>  png                 0.1-9    2026-03-15 [1] CRAN (R 4.6.0)
#>  polspline           1.1.25   2024-05-10 [1] CRAN (R 4.6.1)
#>  posterior         * 1.7.0    2026-04-01 [1] CRAN (R 4.6.1)
#>  pracma              2.4.6    2025-10-22 [1] CRAN (R 4.6.1)
#>  prettyunits         1.2.0    2023-09-24 [1] CRAN (R 4.6.1)
#>  processx            3.9.0    2026-04-22 [1] CRAN (R 4.6.1)
#>  ProfileLikelihood * 1.3      2023-08-25 [1] CRAN (R 4.6.1)
#>  progress          * 1.2.3    2023-12-06 [1] CRAN (R 4.6.1)
#>  proxy               0.4-29   2025-12-29 [1] CRAN (R 4.6.1)
#>  purrr             * 1.2.2    2026-04-10 [1] CRAN (R 4.6.1)
#>  qqconf              1.3.2    2023-04-14 [1] CRAN (R 4.6.1)
#>  qqplotr           * 0.0.7    2025-09-05 [1] CRAN (R 4.6.1)
#>  quantreg          * 6.1      2025-03-10 [1] CRAN (R 4.6.1)
#>  QuickJSR            1.11.0   2026-08-21 [1] CRAN (R 4.6.1)
#>  R6                  2.6.1    2025-02-15 [1] CRAN (R 4.6.1)
#>  randomForest      * 4.7-1.2  2024-09-22 [1] CRAN (R 4.6.1)
#>  ranger              0.18.0   2026-01-16 [1] CRAN (R 4.6.1)
#>  rapportools         1.2      2025-02-28 [1] CRAN (R 4.6.1)
#>  rbibutils           2.4.1    2026-01-21 [1] CRAN (R 4.6.1)
#>  rcartocolor         2.1.2    2025-07-23 [1] CRAN (R 4.6.1)
#>  RColorBrewer        1.1-3    2022-04-03 [1] CRAN (R 4.6.1)
#>  Rcpp              * 1.1.2    2026-07-05 [1] CRAN (R 4.6.1)
#>  RcppParallel        6.2.1    2026-08-27 [1] CRAN (R 4.6.1)
#>  Rdpack              2.6.6    2026-02-08 [1] CRAN (R 4.6.1)
#>  readr             * 2.2.0    2026-02-19 [1] CRAN (R 4.6.1)
#>  reformulas          0.4.4    2026-02-02 [1] CRAN (R 4.6.1)
#>  rematch2            2.1.2    2020-05-01 [1] CRAN (R 4.6.1)
#>  reshape2          * 1.4.5    2025-11-12 [1] CRAN (R 4.6.1)
#>  reticulate        * 1.47.0   2026-09-03 [1] CRAN (R 4.6.1)
#>  rlang               1.3.0    2026-07-05 [1] CRAN (R 4.6.1)
#>  rmarkdown         * 2.32     2026-09-01 [1] CRAN (R 4.6.1)
#>  rms               * 8.1-1    2026-02-18 [1] CRAN (R 4.6.1)
#>  rngtools            1.5.2    2021-09-20 [1] CRAN (R 4.6.1)
#>  robustbase          0.99-7   2026-02-05 [1] CRAN (R 4.6.1)
#>  rpart               4.1.27   2026-03-27 [1] CRAN (R 4.6.0)
#>  rprojroot           2.1.1    2025-08-26 [1] CRAN (R 4.6.1)
#>  rstan             * 2.32.7   2025-03-10 [1] CRAN (R 4.6.1)
#>  rstantools          2.7.1    2026-09-04 [1] Github (stan-dev/rstantools@4d84659)
#>  rstudioapi          0.19.0   2026-06-11 [1] CRAN (R 4.6.1)
#>  S7                  0.2.2    2026-04-22 [1] CRAN (R 4.6.1)
#>  sandwich            3.1-3    2026-08-03 [1] CRAN (R 4.6.1)
#>  scales              1.4.0    2025-04-24 [1] CRAN (R 4.6.1)
#>  sessioninfo         1.2.4    2026-06-04 [1] CRAN (R 4.6.1)
#>  shape               1.4.6.1  2024-02-23 [1] CRAN (R 4.6.1)
#>  showtext          * 0.9-8    2026-03-21 [1] CRAN (R 4.6.1)
#>  showtextdb        * 3.0      2020-06-04 [1] CRAN (R 4.6.1)
#>  sp                  2.2-3    2026-07-19 [1] CRAN (R 4.6.1)
#>  SparseM           * 1.84-2   2024-07-17 [1] CRAN (R 4.6.1)
#>  StanHeaders       * 2.39.1   2026-09-02 [1] CRAN (R 4.6.1)
#>  Statamarkdown     * 1.0.0    2026-08-21 [1] CRAN (R 4.6.1)
#>  stringi             1.8.9    2026-08-04 [1] CRAN (R 4.6.1)
#>  stringr           * 1.6.0    2025-11-04 [1] CRAN (R 4.6.1)
#>  summarytools      * 1.1.5    2026-06-25 [1] Github (dcomtois/summarytools@2b613cf)
#>  survival            3.8-12   2026-09-09 [1] CRAN (R 4.6.1)
#>  svglite           * 2.2.2    2025-10-21 [1] CRAN (R 4.6.1)
#>  svgPanZoom          0.3.4    2020-02-15 [1] CRAN (R 4.6.1)
#>  svUnit              1.0.8    2025-08-26 [1] CRAN (R 4.6.1)
#>  sysfonts          * 0.8.9    2024-03-02 [1] CRAN (R 4.6.1)
#>  systemfonts         1.3.2    2026-03-05 [1] CRAN (R 4.6.1)
#>  tensorA             0.36.2.1 2023-12-13 [1] CRAN (R 4.6.0)
#>  texPreview        * 2.1.0    2024-01-24 [1] CRAN (R 4.6.0)
#>  textshaping         1.0.5    2026-03-06 [1] CRAN (R 4.6.0)
#>  TH.data             1.1-5    2025-11-17 [1] CRAN (R 4.6.0)
#>  tibble            * 3.3.1    2026-01-11 [1] CRAN (R 4.6.1)
#>  tidybayes         * 3.0.7    2024-09-15 [1] CRAN (R 4.6.1)
#>  tidyr             * 1.3.2    2025-12-19 [1] CRAN (R 4.6.1)
#>  tidyselect          1.2.1    2024-03-11 [1] CRAN (R 4.6.1)
#>  tidyverse         * 2.0.0    2023-02-22 [1] CRAN (R 4.6.1)
#>  timechange          0.4.0    2026-01-29 [1] CRAN (R 4.6.1)
#>  tinytex           * 0.60     2026-06-16 [1] CRAN (R 4.6.1)
#>  twosamples          2.0.1    2023-06-23 [1] CRAN (R 4.6.1)
#>  tzdb                0.5.0    2025-03-15 [1] CRAN (R 4.6.1)
#>  V8                  8.2.0    2026-04-21 [1] CRAN (R 4.6.0)
#>  vcd                 1.4-14   2026-07-29 [1] CRAN (R 4.6.1)
#>  vctrs               0.7.3    2026-04-11 [1] CRAN (R 4.6.1)
#>  VIM               * 7.3.1    2026-09-11 [1] CRAN (R 4.6.1)
#>  viridisLite         0.4.3    2026-02-04 [1] CRAN (R 4.6.1)
#>  wesanderson       * 0.3.7    2023-10-31 [1] CRAN (R 4.6.1)
#>  whisker             0.4.1    2022-12-05 [1] CRAN (R 4.6.1)
#>  withr               3.0.3    2026-06-19 [1] CRAN (R 4.6.1)
#>  xfun              * 0.60     2026-07-09 [1] CRAN (R 4.6.1)
#>  xgboost             3.2.1.1  2026-03-18 [1] CRAN (R 4.6.1)
#>  xml2                1.6.0    2026-06-22 [1] CRAN (R 4.6.1)
#>  xtable              1.8-8    2026-02-22 [1] CRAN (R 4.6.1)
#>  yaml                2.3.12   2025-12-10 [1] CRAN (R 4.6.1)
#>  yardstick         * 1.4.0    2026-04-07 [1] CRAN (R 4.6.1)
#>  zoo                 1.9-0    2026-07-31 [1] CRAN (R 4.6.1)
#> 
#>  [1] /opt/homebrew/lib/R/4.6/site-library
#>  [2] /opt/homebrew/Cellar/r/4.6.1/lib/R/library
#>  * ── Packages attached to the search path.
#> 
#> ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

References

Back to top ]]> statistics https://lesslikely.com/statistics/gams.html Sun, 11 Nov 2018 05:00:00 GMT P-values Are Tough And S-values Can Help Zad Rafi https://lesslikely.com/statistics/s-values.html

The -value doesn’t have many fans. There are those who don’t understand it, often treating it as a measure it’s not, whether that’s a posterior probability, the probability of getting results due to chance alone, or some other bizarre/incorrect interpretation.1–3 Then there are those who dislike it because they think the concept is too difficult to understand or because they see it as a noisy statistic we’re not interested in.

However, the groups of people mentioned above aren’t mutually exclusive. Many who dislike and criticize the -value also do not understand its properties and behavior. This is unfortunate, given how important and widely used they are. In this article, which could also have been titled, -values: More Than You Ever Wanted to Know, I take on the task of explaining:

  • what -values are
  • the assumptions behind them
  • their properties and behavior
  • different schools of interpretation
  • misleading criticisms of -values
  • some valid issues in interpretation
  • how these issues can be resolved

What is a P-value Anyway?

Some Definitions & Descriptions

The -value is the probability of getting a result (specifically, a test statistic) at least as extreme as what was observed if every model assumption, in addition to the targeted test hypothesis (usually a null hypothesis), used to compute it were correct.3–5

A simple, mathematically rigorous definition of a -value (for those interested) is given by Stark (2015).

Let be the probability distribution of the data , which takes values in the measurable space . Let be a collection of -measurable subsets of such that (1) and (2) If then . Then the -value of for data is inf .

A descriptive but technical definition is given by Sander Greenland below. The description can seem dense, so feel free to skip over it for now and revisit it after reading the rest of the post.

A single -value is the quantile location of a directional measure of divergence = of the data point (usually, the vector in -space formed by individual observations) from a test model manifold in the -dimensional expectation space defined the logical structure of the data generator (“experiment” or causal structure) that produced the data . is the subset of the -space into which the conjunction of the model constraints (assumptions) force the data expectation or predict where y would be were there no ‘random’ variability. I also use to denote the set of all the model constraints, as well as their conjunction.

With this logical set-up, the observed -value is the quantile for the observed value of = . This is read off a reference distribution for derived from . This formulation is essentially that of the “value of P” appearing in Pearson’s seminal 1900 paper on goodness-of-fit tests. Notably, his famed chi-squared statistic is the squared Euclidean distance from to , with coordinates expressed in standard-deviation units derived from .

More broadly, the statistic can be taken as a measure of divergence of a more general embedding or background model manifold (which includes all ‘auxiliary’ assumptions) from a more restrictive model , with the goodness-of-fit case taking as a saturated model covering the entire observation space, and the more common “hypothesis testing” case taking M as the conjunction of an unsaturated with a targeted ‘test’ constraint (or set of constraints) . This is logically independent of and consistent with , with = & in logical terms, or = + in set-theoretic terms with + being union (in particular, we assume no element in is entailed or contradicted by and no element in is entailed or contradicted by ).

Misleading Definitions

It is very common to see the -value defined as

The probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct.

Indeed, this is the actual definition currently given on the Wikipedia page for the topic, however, it is inadequate and misleading because it hides and reifies the other assumptions used to compute the -value and exclusively focuses on the null hypothesis.

The test hypothesis (often the null hypothesis) is only one component of the entire model that is being tested. This is reflected in the first definition I gave above, which explicitly emphasizes that every model assumption must be true. Thus, the -value is sensitive to all these assumptions and their violation(s).

Auxilliary Assumptions

Some of these key assumptions behind the computation of a -value are that some sort of random process was employed (random sampling, random assignment, etc.), that there are no uncontrolled sources of bias (confounding, programming errors, equipment defects, sparse-data bias)6 in the results, and that the test hypothesis (often the null hypothesis) is correct. Some of these assumptions can be seen in the figure below from7, which will be discussed later on. This entire set of assumptions is generally referred to as the test model, and that is because the entire assumed model is being tested.

P-value assumptions

Conditional versus unconditional interpretations of P-values, S-values, and compatibility intervals (CIs). (A) Conditional interpretation, in which background model assumptions, such as no systematic error, are assumed to be correct; thus,the information provided by the P-value and S-value is targeted towards the test hypothesis. (B)Unconditional interpretation, in which no aspect of the statistical model is assumed to be correct; thus,the information provided by the P-value and S-value is targeted toward the entire test model.

We often start from the position that all those assumptions are correct (hence, we “condition” on them, even though they are often not correct)7 when calculating the -value, so that any deviation of the data from what was expected under those assumptions would be purely random error. But in reality such deviations could also be the result of any assumptions being false, including but not limited to the test hypothesis.

Note: “Conditioning” here refers to taking the assumptions in the model as given, and should not be confused with conditional probability.

For example, in high-energy physics, neutrinos were found in one study to be faster than light due to the resulting large test statistic and corresponding small -value, but this result was later found to be a result of a defect in the fiber-optic timing system for that experiment.8 Thus, the low -value was not because the assumed null hypothesis was false, but instead due to a bias in the procedure.

So the -value cannot be the probability of one of these assumptions, such as “the probability of getting results due to chance alone.” A statement like this is backwards because it’s quantifying one of the assumptions behind the computation of a -value.

This assumption of chance causing the results is assumed to be true (aka 100%) along with several other things, when calculating the -value, but this does not mean it is actually correct and the calculation of the -value cannot be the probability of one of those assumptions.

Probability of What?

It is also important to clarify that -values are not probabilities of data or parameter values, which many like to say to differentiate from probabilities of hypotheses. Rather, -values are probabilities of “data features”, such as test statistics (i.e. a z-score or statistic) or can be interpreted as the percentile at which the observed test statistic falls within the expected distribution for the test statistic, assuming all the model assumptions are true.9, 10

Properties (Uniformity)

A -value is considered to be valid if over repeated trials it would be uniform when the tested hypothesis and all other assumptions used to compute the -value are correct (see the histogram below to see what this looks like). Typically, this test hypothesis is a null hypothesis where the tested parameter value is usually 0 or 1, but this property applies to any test hypothesis for any parameter value. Thus, there is the random variable , which (when valid) follows this uniform distribution, and the realization of this random variable, , which is the observed -value. The latter is what most researchers are interpreting from studies.

Thus, if we were to simulate two variables that are practically the same (meaning there’s no difference between them) and then compare them, say, using a t-test, and we were to iterate this process 10000 times and plot the distribution of the observed P-values, it would be uniform, indicating that any P-value within the interval from 0-1 is just as likely as any other to be observed.

#' @title Simulation of valid P-values where test hypothesis is true
#' @param X The first variable we are simulating
#' @param Y The second variable we are simulating
#' @param n.sim # The number of simulations
#' @param t The object storing the t-test results
#' @param t.sim # Empty numeric vector to contain values
#' @param n.samp # Sample size in each group
#' @NOTE The null hypothesis does not have to be 0, it can be any value.

n.sim <- 10000
t.sim <- numeric(n.sim)
n.samp <- 1000

for (i in 1:n.sim) {
  X <- rnorm(n.samp, mean = 0, sd = 1)
  Y <- rnorm(n.samp, mean = 0, sd = 1)
  df <- data.frame(X, Y)
  t <- t.test(X, Y, mu = 0, paired = FALSE,
              var.equal = TRUE, data = df)
  t.sim[i] <- t[[3]]
}

Many frequentist statisticians do not consider -values to be valid/useful if they fail to meet this validity criterion of being uniform, hence they do not recognize variants such as the posterior predictive -value (which concentrates around values such as 0.5, rather than being uniform) to be valid.

Indeed, there have been great efforts to calibrate the -value which ranges from mathematical solutions such as taking the which gives the lower bound on the conditional type I error,11, 12 to taking the of the -value (the square-root calibrator), yielding a test martingale,13 or even empirically attempting to recalibrate the -value by collecting observed -values from observational studies with negative controls (“test-hypotheses where the exposure is not believed to cause the outcome”) and using them to calculate the empirical null distribution.14

The latter is done since observational studies are prone to several more biases than controlled, randomized experiments, thus the observed -values and estimated effect sizes are used to calculate the systematic errors within the sampling distribution and are used for recalibration of the -value. Whether or not this approach is effective, however, is a different matter.15 In short, calibration is an often sought-out property of -values.

Many frequentist statisticians do not consider -values to be valid/useful if they fail to meet this validity criterion of being uniform, hence they do not recognize variants such as the posterior predictive -value (which concentrates around values such as 0.5, rather than being uniform) to be valid.

Indeed, there have been great efforts to calibrate the -value which ranges from mathematical solutions such as taking the which gives the lower bound on the conditional type I error,11, 12 to taking the of the -value (the square-root calibrator), yielding a test martingale,13 or even empirically attempting to recalibrate the -value by collecting observed -values from observational studies with negative controls (“test-hypotheses where the exposure is not believed to cause the outcome”) and using them to calculate the empirical null distribution.14

The latter is done since observational studies are prone to several more biases than controlled, randomized experiments, thus the observed -values and estimated effect sizes are used to calculate the systematic errors within the sampling distribution and are used for recalibration of the -value. Whether or not this approach is effective, however, is a different matter.15 In short, calibration is an often sought-out property of -values.

The Different Interpretations

The Decision-Theoretic Approach

Many researchers interpret the -value in a behavioral, decision-guiding way such as being statistically significant or not (defined below) depending on whether observed p from a study (the realization of the random variable ) falls below a fixed cutoff level (, which is the maximum tolerable type I error rate).16

Statistical Significance

Thus, in this approach, users do not care how small or large the observed -value is, but simply, whether or not it fell beneath the pre-specified level (often 0.05). If it falls below they behave inline with the rejection of this test hypothesis, and if it fails to fall below , then they must behave in a manner where they accept this test hypothesis. The phrase statistical significance, simply indicates that the observed -value fell below this pre-specified level, and nothing else. It does not indicate any meaningful significance on its own.

The pioneers of this approach, Jerzy Neyman and Egon Pearson, define this behavioral guidance in their 1933 paper, “On the Problem of the Most Efficient Tests of Statistical Hypotheses”16

Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behavior with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong.

This decision-making framework may be useful in certain scenarios,17 where some sort of randomization is possible, where experiments can be repeated, and where there is large control over the experimental conditions, with one of the most notable historical examples being Egon Pearson (son of Karl Pearson and coauthor of Jerzy Neyman) using it to improve quality control in industrial settings.

Contrary to some claims,18 this approach does NOT require exact replications of the experiments, instead, it requires that a valid level is used consistently.16, 19 In this approach, the exact, observed -value from a study is not as relevant and cannot validly be interpreted without an entire set of studies that are compared to the fixed error rate ().

Picture of the giants who founded frequentist statistics such as Egon Pearson, Ronald Fisher, and Jerzy Neyman

From left to right: Ronald A. Fisher, Jerzy Neyman, and Egon Pearson.

The Inductive Approach

Others interpret the -value in an inductive inferential/evidential (Fisherian) way,20, 21 as a continuous measure of evidence against the very test hypothesis and entire model (all assumptions) used to compute it (let’s go with this for now, even though there are some problems with this interpretation, more on that below).

This interpretation as a continuous measure of evidence against the test hypothesis and the entire model used to compute it can be seen in the figure below from7. In one framework (left panel), we may assume certain assumptions to be true (“conditioning” on them, i.e, use of random assignment), and in the other (right panel), we question all assumptions, hence the “unconditional” interpretation. Unlike the Neyman-Pearson approach, this inferential approach allows interpretation of -values from single studies, and indeed, lower values of it are taken as more evidence against the tested hypothesis.

Null-Hypothesis Significance Testing

However, it is also worth pointing out that most individuals do not interpret -values from a Neyman-Pearson or Fisherian standpoint, rather, they fuse both approaches together, which is what we commonly know today as “null-hypothesis significance testing.” This approach is regarded by most as being a incompatible hybrid given that it often confuses error rates (, ), which are fixed before a study, with the -value, which is not a fixed error-rate, and the fusion of these approaches often has been blamed for the replication crisis in science by many statisticians. Though some believe these approaches can be reconciled and are useful.22

P-value assumptions

Conditional versus unconditional interpretations of P-values, S-values, and compatibility intervals (CIs). (A) Conditional interpretation, in which background model assumptions, such as no systematic error, are assumed to be correct; thus,the information provided by the P-value and S-value is targeted towards the test hypothesis. (B)Unconditional interpretation, in which no aspect of the statistical model is assumed to be correct; thus,the information provided by the P-value and S-value is targeted toward the entire test model.

Back to the Fisherian approach, the interpretation of the -value as a continuous measure of evidence against the test model that produced it shouldn’t be confused with other statistics that serve as support measures. Likelihood ratios and Bayes factors are absolute measures of evidence for a model compared to another model, whereas the -value is a relative measure of “evidence” (more on that below) that can be tricky to interpret.23–25 Indeed, this is why the -value is converted by some Bayesians to a lower bound of the Bayes factor by taking .11, 12

Measure of Compatibility

The -value is not an absolute measure of evidence for a model (such as the null/alternative model), it is a continuous measure of the compatibility of the observed data with the model used to compute it.3

If it’s high, it means the observed data are very compatible with the model used to compute it. If it’s very low, then it indicates that the data are not as compatible with the model used to calculate it, and this low compatibility may be due to random variation and/or it may be due to a violation of assumptions (such as the null model not being true, not using randomization, a programming error or equipment defect such as that seen with neutrinos, etc.).

Low compatibility of the data with the model can be implied as evidence against the test hypothesis, if we accept the rest of the model used to compute the -value. Thus, lower -values from a Fisherian perspective are seen as stronger evidence against the test hypothesis given the rest of the model.

Common, Misleading Criticisms

Estimation and Intervals

A common criticism put forth by many is that -values are useless, given that they cannot tell you the size of the effect and because they are confounded by sample size and effect size, and that researchers should instead give compatibility (confidence) intervals. However, this criticism is nonsensical as they can both be given and serve different purposes.

A -value for a particular parameter value gives the compatibility between the test model in question, which will vary from one parameter value to the next, and the data. An interval estimate such as a 95% frequentist interval simply gives the region of parameter values with -values above the corresponding level, and which are more consistent with the data than the parameter values outside the interval limits. An interval estimate by itself does not explicitly tell one how consistent a parameter value is with the data, which the -value does.

Overstating the Evidence

-values are routinely criticized for overstating the amount of evidence from a study. Such statements are also often given using Bayesian arguments, of which many are skeptical. However, the -value cannot overstate evidence as it is simply providing the location at which the test statistic fell in the expected distribution, given that every model assumption were true. It is simply indicative of how surprising/extreme the observed result was, given certain assumptions.

Any overstating of evidence, is not an issue of the statistic itself, but rather users. If we treat the value as nothing more or less than a continuous measure of compatibility of the observed data with the model used to compute it (observed ) given certain model assumptions, we won’t run into some of the common misinterpretations such as “the -value is the probability of a hypothesis”, or the “probability of chance alone”, or “the probability of being incorrect”.3

Indeed, many of the “problems” commonly associated with the -value are not due to the actual statistic itself, but rather researchers’ misinterpretations of what it is and what it means for a study.

The answer to these misconceptions may be compatibilism, with less compatibility (smaller -values) indicating a poor fit between the data and the test model and hence more evidence against the test hypothesis.

A -value of 0.04 means that assuming that all the assumptions of the model used to compute the -value are correct, we won’t get data (a test statistic) at least as extreme as what was observed by random variation more than 4% of the time.

To many, such low compatibility between the data and the model may lead them to reject the test hypothesis (the null hypothesis).

Some Valid Issues

Mismatch With Direction

If you recall from above, I wrote that the -value is seen by many as being a continuous measure of evidence against the test hypothesis and model. Technically speaking, it would be incorrect to define it this way because as the -value goes up (with the highest value being 1 or 100%), there is less evidence against the test hypothesis since the data are more compatible with the test model. 1 = perfect compatibility of the data with the test model.

As the -value gets lower (with the lowest value being 0), there is less compatibility between the data and the model, hence more evidence against the test hypothesis used to compute .

Thus, saying that -values are measures of evidence against the hypothesis used to compute them is a backward definition. This definition would be correct if higher -values inferred more evidence against the test hypothesis and vice versa.

Difficulties Due to Scale

Another problem with -values and their interpretation is scaling. Since the statistic is meant to be a continuous measure of compatibility (and relative evidence against the test model + hypothesis), we would hope that differences between -values would be equal (on an additive scale), as this makes it easier to interpret.

For example, the difference between 0 and 10 dollars is the same as the difference between 90 and 100 dollars, in that both are a difference of 10 dollars. And this property remains consistent across various intervals, 120 and 130, 1,000,000 and 1,000,010.

Unfortunately, this doesn’t apply to the -value because it is on the inverse-exponential scale. The difference between a -value of 0.01 and 0.10 is not the same as the difference between 0.90 and 0.99.

Gaussian distribution

A gaussian probability densitiy with the standard deviations annotated. Data points further away from the mean, are more extreme and unlikely events. I also must admit that this is one of my favorite figures of a gaussian distribution.

For example, with a normal distribution (above), a z-score of 0 results in a -value of 1 (perfect compatibility). If we now move to a z-score of 1, the -value is 0.31. Thus, we saw a dramatic decrease from a -value of 1 to 0.31 with one z-score. A 0.69 decrease in the -value.

Now let’s move from a z-score of 1 to a z-score of 2. We saw a decrease of 0.69 with the change in one z-score before, so the new -value must be 0.31 - 0.69 = -0.38 right? No. The -value for a z-score of 2 is 0.045. The -value for a z-score of 3 is 0.003. Even though we’ve only been moving by one z-score at a time, the changes in -values don’t remain constant; the decreases become larger and larger.

Thus, the difference between the -values of 0.01 and 0.10, in terms of z-score, is substantially larger than the difference between 0.90 and 0.99. Again, this makes it difficult to interpret as a statistic across the board, especially as a continuous measure. This can further be seen in the figure from Rafi & Greenland (2020).

Resolution with Surprisals

The issues described above such as the backward definition and the problem of scaling can make it difficult to conceptualize the -value as being an evidence measure against the test hypothesis and test model. However, these issues can be addressed by taking the negative log of the -value , which yields something known as the Shannon information value or surprisal () value,4, 5, 26 named after Claude Shannon, the father of information theory.27

Unlike the -value, this value is not a probability but a continuous measure of information in bits of information against the test hypothesis and is taken from the observed test statistic computed by the test model.

It also provides a more intuitive way to think about -values. Imagine that the variable is always the nearest integer to the calculated value of . Now, take for example a -value of 0.05, the -value for this would be = which equals 4.3 bits of information embedded in the test statistic, which can be implied as evidence against the test hypothesis.

How much evidence is this? can help us think about this. The nearest integer to 4.3 is 4. Thus, the data which yield a -value of 0.05 which results in an value of 4.3 bits of information is no more surprising than getting all heads on 4 fair coin tosses.

Another example. Let’s say our study gives us a -value of 0.005, which would indicate to many very low compatibility between the test model and the observed data; this would yield an value of bits of information. which is the closest integer to would be 8. Thus, these data which yield a -value of 0.005 are no more surprising than getting all heads on 8 fair coin tosses.

A table of various -values and their corresponding -values, maximum-likelihood ratios, and likelihood-ratio statistics can be found below from Rafi & Greenland (2020), which includes the general cutoffs used in different scientific fields such as high-energy physics and genome-wide association studies. It also shows how the traditional cutoffs used in these fields can be problematic.

For example, an of 0.05, which only corresponds to seeing all heads on 4 fair coin tosses, is practically nothing when compared to the cutoffs used in particle physics and GWAS, which correspond to seeing all heads on 22 and 30 fair coin tosses, respectively.

P-value (compatibility) S-value (bits) Maximum Likelihood Ratio Deviance Statistic 2ln(MLR)
0.99 0.01 1.00e+00 0.00
0.9 0.15 1.01e+00 0.02
0.5 1.00 1.26e+00 0.45
0.25 2.00 1.94e+00 1.32
0.1 3.32 3.87e+00 2.71
0.05 4.32 6.83e+00 3.84
0.025 5.32 1.23e+01 5.02
0.01 6.64 2.76e+01 6.63
0.005 7.64 5.14e+01 7.88
1e-04 13.29 1.94e+03 15.10
5 sigma (~ 2.9 in 10 million) 21.70 5.20e+05 26.30
1 in 100 million (GWAS) 26.60 1.40e+07 32.80
6 sigma (~ 1 in a billion) 29.90 1.30e+08 37.40
Abbreviations:
Table 1: $P$-values and binary $S$-values, with corresponding maximum-likelihood ratios (MLR) and deviance (likelihood-ratio) statistics for a simple test hypothesis H under background assumptions A

Unlike the -value, the -value is more intuitive as a measure of refutational evidence against the test hypothesis since its value (bits of information against the test hypothesis) increases with less compatibility, whereas the opposite is true for the -value.

Some Examples

Let’s try using some data to see this in action. I’ll take a sample experimental dataset from R on the effects of different conditions on dried plant weight. We can plot the data and run a one-way ANOVA.

pg <- force(PlantGrowth)
(Hmisc::describe(pg))
#> pg 
#> 
#>  2  Variables      30  Observations
#> ----------------------------------------------------------------------------------------------------------------------------------
#> weight 
#>        n  missing distinct     Info     Mean  pMedian      Gmd      .05      .10      .25      .50      .75      .90      .95 
#>       30        0       29        1    5.073     5.09   0.8131    3.983    4.170    4.550    5.155    5.530    6.038    6.132 
#> 
#> lowest : 3.59 3.83 4.17 4.32 4.41, highest: 5.87 6.03 6.11 6.15 6.31
#> ----------------------------------------------------------------------------------------------------------------------------------
#> group 
#>        n  missing distinct 
#>       30        0        3 
#>                             
#> Value       ctrl  trt1  trt2
#> Frequency     10    10    10
#> Proportion 0.333 0.333 0.333
#> ----------------------------------------------------------------------------------------------------------------------------------

Looks interesting. We can see some differences from the graph. Here’s what our test output gives us,

res <- anova(lm(weight ~ group, data = pg))
ztable(res)
Df Sum Sq Mean Sq F value Pr(>F)
group 2 3.77 1.88 4.85 0.02
Residuals 27 10.49 0.39 NA NA 
(obs_p <- res[1, 5])
#> [1] 0.0159

If we had set our to the traditional 0.05 level before the experiment, we can reject the test hypothesis (the null hypothesis), but that is not as interesting from a continuous evidential perspective. How can I interpret this -value of 0.0159 more intuitively?

Let’s convert it into an -value.

-log2(obs_p)
#> [1] 5.97

That is 5.97 bits of information against the null hypothesis.

Remember, is the nearest integer to the calculated value of and in this case, would be 6. So these results (the test statistic, (4.85)) are as surprising as getting all heads on 6 fair coin tosses. Somewhat surprising, depending on the individual interpreting the results.

How would we interpret it within the context of a given confidence interval? The -value tells us that values within the computed 95% CI: have at most 4.3 bits of information against them. That is because all parameter values within a 95% CI have -values greater than 0.05.

So those parameter values that are inside the 95% interval estimate have less bits of information against them, than the parameter values that go further and further away from the center of the 95% interval estimate. The point estimate is the most compatible with the data (meaning it has the least refutational information against it), while those values near the limits have more information against them.

In other words, as values head in the directions outside the interval, there is more refutational information against them, as depicted by the following function from Rafi & Greenland, 2020, which is known as the surprisal function.

The -value is not meant to replace the -value, and it isn’t superior to the -value. It is merely a logarithmic transformation of it that rescales it on an additive scale and tells us how much information is embedded within the test statistic and can be used as evidence against the test hypothesis. It is meant to be a device to help interpret the information one obtains from a calculated -value.

I’ve constructed a calculator that converts observed -values into -values and provides an intuitive way to think about them. For a more detailed discussion of -values, see this article, in addition to the references below them.

S-value Calculator

Acknowledgments: I’m very grateful to Sander Greenland for his extensive commentary and corrections on several versions of this article. My acknowledgment does not imply endorsement of my views by these colleagues, and I remain solely responsible for the views expressed herein.

The analyses were run on:

#> ─ Session info ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.6.1 (2026-06-24)
#>  os       macOS Golden Gate 27.0
#>  system   aarch64, darwin25.4.0
#>  ui       unknown
#>  language (EN)
#>  collate  C.UTF-8
#>  ctype    C.UTF-8
#>  tz       America/New_York
#>  date     2026-09-11
#>  pandoc   3.11 @ /opt/homebrew/bin/ (via rmarkdown)
#>  quarto   1.10.18 @ /usr/local/bin/quarto
#> 
#> ─ Packages ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  package           * version  date (UTC) lib source
#>  abind               1.4-8    2024-09-12 [1] CRAN (R 4.6.1)
#>  Amelia            * 1.8.3    2024-11-08 [1] CRAN (R 4.6.1)
#>  arm                 1.15-3   2026-04-15 [1] CRAN (R 4.6.1)
#>  arrayhelpers        1.1-2    2026-07-24 [1] CRAN (R 4.6.1)
#>  backports           1.5.1    2026-04-03 [1] CRAN (R 4.6.1)
#>  base              * 4.6.1    2026-06-24 [2] local
#>  base64enc           0.1-6    2026-02-02 [1] CRAN (R 4.6.1)
#>  bayesplot         * 1.16.0   2026-08-25 [1] CRAN (R 4.6.1)
#>  bitops              1.1-0    2026-07-30 [1] CRAN (R 4.6.1)
#>  blogdown          * 1.24     2026-06-19 [1] CRAN (R 4.6.1)
#>  boot              * 1.3-32   2025-08-29 [1] CRAN (R 4.6.1)
#>  bootImpute        * 1.3.0    2025-12-15 [1] CRAN (R 4.6.1)
#>  bridgesampling      1.2-1    2025-11-19 [1] CRAN (R 4.6.1)
#>  brms              * 2.23.0   2025-09-09 [1] CRAN (R 4.6.1)
#>  Brobdingnag         1.2-9    2022-10-19 [1] CRAN (R 4.6.1)
#>  broom             * 1.0.13   2026-05-14 [1] CRAN (R 4.6.1)
#>  broom.mixed       * 0.2.9.7  2026-02-17 [1] CRAN (R 4.6.1)
#>  Cairo             * 1.7-0    2025-10-29 [1] CRAN (R 4.6.1)
#>  callr               3.8.0    2026-06-05 [1] CRAN (R 4.6.1)
#>  car               * 3.1-5    2026-02-03 [1] CRAN (R 4.6.1)
#>  carData           * 3.0-6    2026-01-30 [1] CRAN (R 4.6.1)
#>  caTools             1.18.4   2026-07-20 [1] CRAN (R 4.6.1)
#>  checkmate         * 2.3.4    2026-02-03 [1] CRAN (R 4.6.1)
#>  class               7.3-24   2026-08-03 [1] CRAN (R 4.6.1)
#>  cli                 3.6.6    2026-04-09 [1] CRAN (R 4.6.1)
#>  clipr               0.8.1    2026-05-25 [1] CRAN (R 4.6.1)
#>  cluster             2.1.8.3  2026-07-30 [1] CRAN (R 4.6.1)
#>  coda              * 0.19-4.1 2024-01-31 [1] CRAN (R 4.6.1)
#>  codetools           0.2-20   2024-03-31 [1] CRAN (R 4.6.0)
#>  colorspace        * 2.1-3    2026-07-12 [1] CRAN (R 4.6.1)
#>  compiler            4.6.1    2026-06-24 [2] local
#>  concurve          * 3.0.5    2026-09-10 [1] local
#>  cowplot           * 1.2.0    2025-07-07 [1] CRAN (R 4.6.1)
#>  crayon              1.5.3    2024-06-20 [1] CRAN (R 4.6.1)
#>  curl                8.0.0    2026-08-25 [1] CRAN (R 4.6.1)
#>  data.table          1.18.6.1 2026-08-24 [1] CRAN (R 4.6.1)
#>  datasets          * 4.6.1    2026-06-24 [2] local
#>  DBI                 1.3.0    2026-02-25 [1] CRAN (R 4.6.1)
#>  DEoptimR            1.2-1    2026-08-20 [1] CRAN (R 4.6.1)
#>  desc                1.4.3    2023-12-10 [1] CRAN (R 4.6.1)
#>  details             0.4.0    2025-02-09 [1] CRAN (R 4.6.0)
#>  dichromat           2.0-1    2026-07-22 [1] CRAN (R 4.6.1)
#>  digest              0.6.39   2025-11-19 [1] CRAN (R 4.6.1)
#>  distributional      0.9.0    2026-09-08 [1] CRAN (R 4.6.1)
#>  doParallel        * 1.0.17   2022-02-07 [1] CRAN (R 4.6.1)
#>  doRNG               1.8.6.3  2026-02-05 [1] CRAN (R 4.6.1)
#>  dplyr             * 1.2.1    2026-04-03 [1] CRAN (R 4.6.1)
#>  e1071               1.7-17   2025-12-18 [1] CRAN (R 4.6.1)
#>  emmeans             2.0.4    2026-07-15 [1] CRAN (R 4.6.1)
#>  estimability        2.0.0    2026-06-26 [1] CRAN (R 4.6.1)
#>  evaluate            1.0.5    2025-08-27 [1] CRAN (R 4.6.1)
#>  extremevalues       2.4.1    2024-12-17 [1] CRAN (R 4.6.1)
#>  farver              2.1.2    2024-05-13 [1] CRAN (R 4.6.1)
#>  fastmap             1.2.0    2024-05-15 [1] CRAN (R 4.6.1)
#>  forcats           * 1.0.1    2025-09-25 [1] CRAN (R 4.6.1)
#>  foreach           * 1.5.2    2022-02-02 [1] CRAN (R 4.6.1)
#>  foreign             0.8-91   2026-01-29 [1] CRAN (R 4.6.1)
#>  Formula             1.2-6    2026-08-03 [1] CRAN (R 4.6.1)
#>  fs                  2.1.0    2026-04-18 [1] CRAN (R 4.6.1)
#>  furrr               0.4.0    2026-03-31 [1] CRAN (R 4.6.1)
#>  future            * 1.75.0   2026-07-20 [1] CRAN (R 4.6.1)
#>  future.apply      * 1.20.2   2026-02-20 [1] CRAN (R 4.6.1)
#>  gamlss            * 5.5-0    2025-08-19 [1] CRAN (R 4.6.1)
#>  gamlss.data       * 6.0-7    2025-09-04 [1] CRAN (R 4.6.1)
#>  gamlss.dist       * 6.1-11   2026-09-10 [1] CRAN (R 4.6.1)
#>  generics            0.1.4    2025-05-09 [1] CRAN (R 4.6.1)
#>  ggcorrplot        * 0.3.0    2026-07-24 [1] CRAN (R 4.6.1)
#>  ggdist              3.3.3    2025-04-23 [1] CRAN (R 4.6.1)
#>  ggplot2           * 4.0.3    2026-04-22 [1] CRAN (R 4.6.1)
#>  ggtext            * 0.2.0    2026-08-28 [1] CRAN (R 4.6.1)
#>  glmnet              5.0      2026-05-04 [1] CRAN (R 4.6.1)
#>  globals             0.19.1   2026-03-13 [1] CRAN (R 4.6.1)
#>  glue                1.8.1    2026-04-17 [1] CRAN (R 4.6.1)
#>  graphics          * 4.6.1    2026-06-24 [2] local
#>  grDevices         * 4.6.1    2026-06-24 [2] local
#>  grid              * 4.6.1    2026-06-24 [2] local
#>  gridExtra           2.3.1    2026-06-25 [1] CRAN (R 4.6.1)
#>  gridtext            0.1.6    2026-02-19 [1] CRAN (R 4.6.0)
#>  gtable              0.3.6    2024-10-25 [1] CRAN (R 4.6.1)
#>  here              * 1.0.2    2025-09-15 [1] CRAN (R 4.6.1)
#>  Hmisc             * 5.3-0    2026-09-06 [1] CRAN (R 4.6.1)
#>  hms                 1.1.4    2025-10-17 [1] CRAN (R 4.6.1)
#>  htmlTable           2.5.0    2026-04-22 [1] CRAN (R 4.6.1)
#>  htmltools         * 0.5.9    2025-12-04 [1] CRAN (R 4.6.1)
#>  htmlwidgets         1.6.4    2023-12-06 [1] CRAN (R 4.6.1)
#>  httr                1.4.9    2026-09-01 [1] CRAN (R 4.6.1)
#>  ImputeRobust      * 1.3-1    2018-11-30 [1] CRAN (R 4.6.1)
#>  inline              0.3.21   2025-01-09 [1] CRAN (R 4.6.1)
#>  insight             1.5.4    2026-09-05 [1] CRAN (R 4.6.1)
#>  iterators         * 1.0.14   2022-02-05 [1] CRAN (R 4.6.1)
#>  itertools           0.1-3    2014-03-12 [1] CRAN (R 4.6.1)
#>  jomo                2.7-6    2023-04-15 [1] CRAN (R 4.6.1)
#>  jsonlite            2.0.0    2025-03-27 [1] CRAN (R 4.6.1)
#>  kableExtra        * 1.4.1    2026-07-08 [1] CRAN (R 4.6.1)
#>  knitr             * 1.52     2026-09-06 [1] CRAN (R 4.6.1)
#>  laeken              0.5.3    2024-01-25 [1] CRAN (R 4.6.1)
#>  latex2exp         * 0.9.8    2026-01-09 [1] CRAN (R 4.6.1)
#>  lattice           * 0.23-1   2026-08-12 [1] CRAN (R 4.6.1)
#>  lifecycle           1.0.5    2026-01-08 [1] CRAN (R 4.6.1)
#>  listenv             1.0.0    2026-06-22 [1] CRAN (R 4.6.1)
#>  lme4                2.0-6    2026-07-16 [1] CRAN (R 4.6.1)
#>  lmtest              0.9-40   2022-03-21 [1] CRAN (R 4.6.1)
#>  loo               * 2.10.1   2026-07-24 [1] CRAN (R 4.6.1)
#>  lubridate         * 1.9.5    2026-02-04 [1] CRAN (R 4.6.1)
#>  magick              2.9.1    2026-02-28 [1] CRAN (R 4.6.1)
#>  magrittr          * 2.0.5    2026-04-04 [1] CRAN (R 4.6.1)
#>  MASS              * 7.3-66   2026-07-15 [1] CRAN (R 4.6.1)
#>  Matrix            * 1.7-6    2026-07-25 [1] CRAN (R 4.6.1)
#>  MatrixModels        0.5-4    2025-03-26 [1] CRAN (R 4.6.1)
#>  matrixStats         1.5.0    2025-01-07 [1] CRAN (R 4.6.1)
#>  mcmc                0.9-8    2023-11-16 [1] CRAN (R 4.6.1)
#>  MCMCpack          * 1.7-1    2024-08-27 [1] CRAN (R 4.6.1)
#>  methods           * 4.6.1    2026-06-24 [2] local
#>  mgcv              * 1.9-4    2025-11-07 [1] CRAN (R 4.6.0)
#>  mi                * 1.3.1    2026-07-28 [1] CRAN (R 4.6.1)
#>  mice              * 3.19.0   2025-12-10 [1] CRAN (R 4.6.1)
#>  miceadds          * 3.20-10  2026-05-28 [1] CRAN (R 4.6.1)
#>  miceFast          * 0.9.1    2026-02-26 [1] CRAN (R 4.6.1)
#>  minqa               1.2.8    2024-08-17 [1] CRAN (R 4.6.1)
#>  missForest        * 1.6.1    2025-10-26 [1] CRAN (R 4.6.1)
#>  mitml               0.4-5    2023-03-08 [1] CRAN (R 4.6.1)
#>  mitools             2.7      2026-08-31 [1] CRAN (R 4.6.1)
#>  multcomp            1.4-32   2026-08-21 [1] CRAN (R 4.6.1)
#>  mvtnorm           * 1.4-2    2026-07-12 [1] CRAN (R 4.6.1)
#>  nlme              * 3.1-171  2026-09-01 [1] CRAN (R 4.6.1)
#>  nloptr              2.2.1    2025-03-17 [1] CRAN (R 4.6.1)
#>  nnet                7.3-21   2026-08-03 [1] CRAN (R 4.6.1)
#>  opdisDownsampling   1.7      2026-09-04 [1] CRAN (R 4.6.1)
#>  otel                0.2.0    2025-08-29 [1] CRAN (R 4.6.1)
#>  pan                 2.0      2026-06-30 [1] CRAN (R 4.6.1)
#>  pander              0.6.6    2025-03-01 [1] CRAN (R 4.6.1)
#>  parallel          * 4.6.1    2026-06-24 [2] local
#>  parallelly        * 1.48.0   2026-06-29 [1] CRAN (R 4.6.1)
#>  pbmcapply         * 1.5.1    2022-04-28 [1] CRAN (R 4.6.1)
#>  performance       * 0.18.2   2026-09-10 [1] CRAN (R 4.6.1)
#>  pillar              1.11.1   2025-09-17 [1] CRAN (R 4.6.1)
#>  pkgbuild            1.4.8    2025-05-26 [1] CRAN (R 4.6.1)
#>  pkgconfig           2.0.3    2019-09-22 [1] CRAN (R 4.6.1)
#>  plyr                1.8.9    2023-10-02 [1] CRAN (R 4.6.1)
#>  png                 0.1-9    2026-03-15 [1] CRAN (R 4.6.0)
#>  polspline           1.1.25   2024-05-10 [1] CRAN (R 4.6.1)
#>  posterior         * 1.7.0    2026-04-01 [1] CRAN (R 4.6.1)
#>  pracma              2.4.6    2025-10-22 [1] CRAN (R 4.6.1)
#>  prettyunits         1.2.0    2023-09-24 [1] CRAN (R 4.6.1)
#>  processx            3.9.0    2026-04-22 [1] CRAN (R 4.6.1)
#>  ProfileLikelihood * 1.3      2023-08-25 [1] CRAN (R 4.6.1)
#>  progress          * 1.2.3    2023-12-06 [1] CRAN (R 4.6.1)
#>  proxy               0.4-29   2025-12-29 [1] CRAN (R 4.6.1)
#>  purrr             * 1.2.2    2026-04-10 [1] CRAN (R 4.6.1)
#>  qqconf              1.3.2    2023-04-14 [1] CRAN (R 4.6.1)
#>  qqplotr           * 0.0.7    2025-09-05 [1] CRAN (R 4.6.1)
#>  quantreg          * 6.1      2025-03-10 [1] CRAN (R 4.6.1)
#>  QuickJSR            1.11.0   2026-08-21 [1] CRAN (R 4.6.1)
#>  R6                  2.6.1    2025-02-15 [1] CRAN (R 4.6.1)
#>  randomForest      * 4.7-1.2  2024-09-22 [1] CRAN (R 4.6.1)
#>  ranger              0.18.0   2026-01-16 [1] CRAN (R 4.6.1)
#>  rapportools         1.2      2025-02-28 [1] CRAN (R 4.6.1)
#>  rbibutils           2.4.1    2026-01-21 [1] CRAN (R 4.6.1)
#>  rcartocolor         2.1.2    2025-07-23 [1] CRAN (R 4.6.1)
#>  RColorBrewer        1.1-3    2022-04-03 [1] CRAN (R 4.6.1)
#>  Rcpp              * 1.1.2    2026-07-05 [1] CRAN (R 4.6.1)
#>  RcppParallel        6.2.1    2026-08-27 [1] CRAN (R 4.6.1)
#>  Rdpack              2.6.6    2026-02-08 [1] CRAN (R 4.6.1)
#>  readr             * 2.2.0    2026-02-19 [1] CRAN (R 4.6.1)
#>  reformulas          0.4.4    2026-02-02 [1] CRAN (R 4.6.1)
#>  rematch2            2.1.2    2020-05-01 [1] CRAN (R 4.6.1)
#>  reshape2          * 1.4.5    2025-11-12 [1] CRAN (R 4.6.1)
#>  reticulate        * 1.47.0   2026-09-03 [1] CRAN (R 4.6.1)
#>  rlang               1.3.0    2026-07-05 [1] CRAN (R 4.6.1)
#>  rmarkdown         * 2.32     2026-09-01 [1] CRAN (R 4.6.1)
#>  rms               * 8.1-1    2026-02-18 [1] CRAN (R 4.6.1)
#>  rngtools            1.5.2    2021-09-20 [1] CRAN (R 4.6.1)
#>  robustbase          0.99-7   2026-02-05 [1] CRAN (R 4.6.1)
#>  rpart               4.1.27   2026-03-27 [1] CRAN (R 4.6.0)
#>  rprojroot           2.1.1    2025-08-26 [1] CRAN (R 4.6.1)
#>  rstan             * 2.32.7   2025-03-10 [1] CRAN (R 4.6.1)
#>  rstantools          2.7.1    2026-09-04 [1] Github (stan-dev/rstantools@4d84659)
#>  rstudioapi          0.19.0   2026-06-11 [1] CRAN (R 4.6.1)
#>  S7                  0.2.2    2026-04-22 [1] CRAN (R 4.6.1)
#>  sandwich            3.1-3    2026-08-03 [1] CRAN (R 4.6.1)
#>  scales              1.4.0    2025-04-24 [1] CRAN (R 4.6.1)
#>  sessioninfo         1.2.4    2026-06-04 [1] CRAN (R 4.6.1)
#>  shape               1.4.6.1  2024-02-23 [1] CRAN (R 4.6.1)
#>  showtext          * 0.9-8    2026-03-21 [1] CRAN (R 4.6.1)
#>  showtextdb        * 3.0      2020-06-04 [1] CRAN (R 4.6.1)
#>  sp                  2.2-3    2026-07-19 [1] CRAN (R 4.6.1)
#>  SparseM           * 1.84-2   2024-07-17 [1] CRAN (R 4.6.1)
#>  splines           * 4.6.1    2026-06-24 [2] local
#>  StanHeaders       * 2.39.1   2026-09-02 [1] CRAN (R 4.6.1)
#>  Statamarkdown     * 1.0.0    2026-08-21 [1] CRAN (R 4.6.1)
#>  stats             * 4.6.1    2026-06-24 [2] local
#>  stats4            * 4.6.1    2026-06-24 [2] local
#>  stringi             1.8.9    2026-08-04 [1] CRAN (R 4.6.1)
#>  stringr           * 1.6.0    2025-11-04 [1] CRAN (R 4.6.1)
#>  summarytools      * 1.1.5    2026-06-25 [1] Github (dcomtois/summarytools@2b613cf)
#>  survival            3.8-12   2026-09-09 [1] CRAN (R 4.6.1)
#>  svglite           * 2.2.2    2025-10-21 [1] CRAN (R 4.6.1)
#>  svgPanZoom          0.3.4    2020-02-15 [1] CRAN (R 4.6.1)
#>  svUnit              1.0.8    2025-08-26 [1] CRAN (R 4.6.1)
#>  sysfonts          * 0.8.9    2024-03-02 [1] CRAN (R 4.6.1)
#>  systemfonts         1.3.2    2026-03-05 [1] CRAN (R 4.6.1)
#>  tcltk               4.6.1    2026-06-24 [2] local
#>  tensorA             0.36.2.1 2023-12-13 [1] CRAN (R 4.6.0)
#>  texPreview        * 2.1.0    2024-01-24 [1] CRAN (R 4.6.0)
#>  textshaping         1.0.5    2026-03-06 [1] CRAN (R 4.6.0)
#>  TH.data             1.1-5    2025-11-17 [1] CRAN (R 4.6.0)
#>  tibble            * 3.3.1    2026-01-11 [1] CRAN (R 4.6.1)
#>  tidybayes         * 3.0.7    2024-09-15 [1] CRAN (R 4.6.1)
#>  tidyr             * 1.3.2    2025-12-19 [1] CRAN (R 4.6.1)
#>  tidyselect          1.2.1    2024-03-11 [1] CRAN (R 4.6.1)
#>  tidyverse         * 2.0.0    2023-02-22 [1] CRAN (R 4.6.1)
#>  timechange          0.4.0    2026-01-29 [1] CRAN (R 4.6.1)
#>  tinytex           * 0.60     2026-06-16 [1] CRAN (R 4.6.1)
#>  tools               4.6.1    2026-06-24 [2] local
#>  twosamples          2.0.1    2023-06-23 [1] CRAN (R 4.6.1)
#>  tzdb                0.5.0    2025-03-15 [1] CRAN (R 4.6.1)
#>  utils             * 4.6.1    2026-06-24 [2] local
#>  V8                  8.2.0    2026-04-21 [1] CRAN (R 4.6.0)
#>  vcd                 1.4-14   2026-07-29 [1] CRAN (R 4.6.1)
#>  vctrs               0.7.3    2026-04-11 [1] CRAN (R 4.6.1)
#>  VIM               * 7.3.1    2026-09-11 [1] CRAN (R 4.6.1)
#>  viridisLite         0.4.3    2026-02-04 [1] CRAN (R 4.6.1)
#>  wesanderson       * 0.3.7    2023-10-31 [1] CRAN (R 4.6.1)
#>  whisker             0.4.1    2022-12-05 [1] CRAN (R 4.6.1)
#>  withr               3.0.3    2026-06-19 [1] CRAN (R 4.6.1)
#>  xfun              * 0.60     2026-07-09 [1] CRAN (R 4.6.1)
#>  xgboost             3.2.1.1  2026-03-18 [1] CRAN (R 4.6.1)
#>  xml2                1.6.0    2026-06-22 [1] CRAN (R 4.6.1)
#>  xtable              1.8-8    2026-02-22 [1] CRAN (R 4.6.1)
#>  yaml                2.3.12   2025-12-10 [1] CRAN (R 4.6.1)
#>  yardstick         * 1.4.0    2026-04-07 [1] CRAN (R 4.6.1)
#>  zoo                 1.9-0    2026-07-31 [1] CRAN (R 4.6.1)
#> 
#>  [1] /opt/homebrew/lib/R/4.6/site-library
#>  [2] /opt/homebrew/Cellar/r/4.6.1/lib/R/library
#>  * ── Packages attached to the search path.
#> 
#> ─ External software ────────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  setting        value
#>  cairo          1.18.4
#>  cairoFT        2.14.3/2.18.1
#>  pango
#>  png
#>  jpeg
#>  tiff
#>  tcl            9.0
#>  curl           8.7.1
#>  zlib           1.2.12
#>  bzlib          1.0.8, 13-Jul-2019
#>  xz             5.8.3
#>  deflate
#>  zstd           1.5.7
#>  PCRE           10.48 2026-08-31
#>  ICU            78.1
#>  TRE            TRE 0.8.0 R_fixes (BSD)
#>  iconv          Apple or GNU libiconv 1.11 /usr/lib/libiconv.2.dylib
#>  readline       8.3
#>  BLAS           /opt/homebrew/Cellar/openblas/0.3.34/lib/libopenblasp-r0.3.34.dylib
#>  lapack         /opt/homebrew/Cellar/r/4.6.1/lib/R/lib/libRlapack.dylib
#>  lapack_version 3.12.1
#> 
#> ─ Python configuration ─────────────────────────────────────────────────────────────────────────────────────────────────────────
#>  Python is not available
#> 
#> ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#> $RNGkind
#> [1] "Mersenne-Twister" "Inversion"        "Rejection"

References

Back to top

References

1. Gigerenzer G. (2018). “Statistical Rituals: The Replication Delusion and How We Got There.” Advances in Methods and Practices in Psychological Science. 1:198–218. doi: 10.1177/2515245918771329.
2. Goodman S. (2008). “A Dirty Dozen: Twelve P-Value Misconceptions.” Seminars in Hematology. 45:135–140. doi: 10.1053/j.seminhematol.2008.04.003.
3. Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. (2016). “Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations.” European Journal of Epidemiology. 31:337–350. doi: 10.1007/s10654-016-0149-3.
4. Rafi Z, Greenland S. (2020). “Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise.” BMC Medical Research Methodology. 20:244. doi: 10.1186/s12874-020-01105-9.
5. Greenland S. (2019). “Valid P-values behave exactly as they should: Some misleading criticisms of P-values and their resolution with S-values.” The American Statistician. 73:106–114. doi: 10.1080/00031305.2018.1529625.
6. Greenland S, Mansournia MA, Altman DG. (2016). “Sparse data bias: A problem hiding in plain sight.” BMJ. 352:i1981. doi: 10.1136/bmj.i1981.
7. Greenland S, Rafi Z. (2020). “To Aid Scientific Inference, Emphasize Unconditional Descriptions of Statistics.” arXiv:190908583 [statME]. https://arxiv.org/abs/1909.08583.
8. Moskowitz C. (2012). “Faster-than-light neutrinos aren’t.” Scientific American.
9. Perezgonzalez JD. (2015). “P-values as percentiles. Commentary on: ‘Null hypothesis significance tests. A mixup of two different theories: The basis for widespread confusion and numerous misinterpretations’.” Frontiers in Psychology. 6. doi: 10.3389/fpsyg.2015.00341.
10. Fraser DAS. (2019). “The P-value function and statistical inference.” The American Statistician. 73:135–147. doi: 10.1080/00031305.2018.1556735.
11. Sellke T, Bayarri MJ, Berger JO. (2001). “Calibration of values for testing precise null hypotheses.” The American Statistician. 55:62–71. doi: 10.1198/000313001300339950.
12. Greenland S, Rafi Z. (2020). “Technical Issues in the Interpretation of S-values and Their Relation to Other Information Measures.” arXiv:200812991 [statME]. https://arxiv.org/abs/2008.12991.
13. Shafer G, Shen A, Vereshchagin N, Vovk V. (2011). “Test Martingales, Bayes Factors and p-Values.” Statistical Science. 26:84–101. doi: 10/fkcvt5.
14. Schuemie MJ, Hripcsak G, Ryan PB, Madigan D, Suchard MA. (2016). “Robust empirical calibration of p-values using observational data.” Statistics in Medicine. 35:3883–3888. doi: 10/ghqmsb.
15. Gruber S, Tchetgen ET. (2016). “Limitations of empirical calibration of p-values using observational data.” Statistics in Medicine. 35:3869–3882. doi: 10/ghqmtn.
16. Neyman J, Pearson ES. (1933). “On the Problem of the Most Efficient Tests of Statistical Hypotheses.” Philosophical Transactions of the Royal Society of London Series A, Containing Papers of a Mathematical or Physical Character. 231:289–337. doi: 10.1098/rsta.1933.0009.
17. Whitehead J. (1993). “The case for frequentism in clinical trials.” Statistics in Medicine. 12:1405–1413. doi: 10.1002/sim.4780121506.
18. Rubin M. (2019). “What type of Type I error? Contrasting the Neyman and Fisherian approaches in the context of exact and direct replications.” Synthese. doi: 10.1007/s11229-019-02433-0.
19. Lehmann EL. (2011). “Fisher, Neyman, and the Creation of Classical Statistics.” Springer New York. doi: 10.1007/978-1-4419-9500-1.
20. Fisher RA. (1935). “The Design of Experiments.” Oxford, England: Oliver & Boyd.
21. Fisher R. (1955). “Statistical Methods and Scientific Induction.” Journal of the Royal Statistical Society Series B (Methodological). 17:69–78. doi: 10.1111/j.2517-6161.1955.tb00180.x.
22. Bickel DR. (2019). “Null Hypothesis Significance Testing Defended and Calibrated by Bayesian Model Checking.” The American Statistician. 0:1–16. doi: 10.1080/00031305.2019.1699443.
23. Jeffreys H. (1935). “Some Tests of Significance, Treated by the Theory of Probability.” Mathematical Proceedings of the Cambridge Philosophical Society. 31:203–222. doi: 10.1017/S030500410001330X.
24. Jeffreys H. (1998). “The Theory of Probability.” OUP Oxford.
25. Royall R. (1997). “Statistical Evidence: A Likelihood Paradigm.” CRC Press.
26. Cole SR, Edwards JK, Greenland S. (2020). “Surprise!” American Journal of Epidemiology. doi: 10/gg63md.
27. Shannon CE. (1948). “A mathematical theory of communication.” The Bell System Technical Journal. 27:379–423. doi: 10.1002/j.1538-7305.1948.tb01338.x.

Citation

BibTeX citation:
@online{rafi2018,
  author = {Rafi, Zad and Rafi, Zad},
  title = {P-Values {Are} {Tough} {And} {S-values} {Can} {Help}},
  date = {2018-11-11},
  url = {https://lesslikely.com/statistics/s-values.html},
  langid = {en-US}
}
]]> statistics https://lesslikely.com/statistics/s-values.html Sun, 11 Nov 2018 05:00:00 GMT We May Not Understand Control Groups Zad Rafi https://lesslikely.com/statistics/control-group-effects.html

It’s well known that randomized trials are some of the most efficient ways to make causal inferences and to determine how much something (an intervention) differs from the comparator (some sort of control). Random assignment helps make these goals easier by minimizing selection bias and making the distribution of prognostic factors between groups random (not balanced).1

Discussions (similar to the one above) praising the efficiency of randomized trials are widespread, however, few of these discussions take a close look at some of the common assumptions that individuals hold regarding randomized trials. And unfortunately, these common assumptions may be based on outdated evidence and simplistic ideas.

The Placebo Effect Isn’t What We Often Think

For example, in placebo-controlled trials, many individuals make the assumption that much of the improvement seen in the control group over time is due to the placebo effect, as modeled by the graph below.

Bar graph depicting the placebo-dominant model

However, inquiries into this topic have yielded contradictory results, in that the placebo effect may not be as powerful as we assume.

One systematic review2 that looked at 130 clinical trials concluded the following,

“We found little evidence in general that placebos had powerful clinical effects. Although placebos had no significant effects on objective or binary outcomes, they had possible small benefits in studies with continuous subjective outcomes and for the treatment of pain. Outside the setting of clinical trials, there is no justification for the use of placebos.”

Thus, the placebo effect had some small effects in areas where it is difficult to objectively measure a phenomenon and where there is a higher likelihood for encountering measurement error. However, measurement error is not the only thing that could explain some of the improvements seen in control groups, which we often attribute to the placebo effect.

We simply need to reread two sections of a paper by the very anesthesiologist who popularized the placebo effect3 and who claimed that it had the ability to put patients’ conditions into remission.

Here, Henry Beecher (the anesthesiologist) claims that placebo effects are powerful:

“It is evident that placebos have a high degree of therapeutic effectiveness in treating subjective responses, decided improvement, interpreted under the unknowns technique as a real therapeutic effect, being produced in 35.2 ± 2.2% of cases….

Placebos have not only remarkable therapeutic power but also toxic effects. These are both subjective and objective. The reaction (psychological) component of suffering has power to produce gross physical change.”

Then he reports another observation from his patients.

“The evidence is that placebos are most effective when the stress is greatest.”

If you measured someone’s stress (let’s say objectively, via cortisol measurements) and they were extremely stressed and if you followed up after a certain period of time, it would be very likely that the next measurement of their stress (cortisol levels) would be less than the first measurement.

A slightly reverse situation would likely apply too: if you measured someone’s happiness (in some theoretical objective way) and they were extremely happy, it’s very likely that their next measurement of happiness would not be so high. This phenomenon is known as regression to the mean,4 and it can explain why many individuals who are in an abnormal state, feel slightly more normal over time.

We can see this displayed below in the graph of simulated blood pressure data5 where the change in blood pressure is predicted by baseline blood pressure. The individuals with higher blood pressure at baseline experienced a larger decrease in blood pressure over time.

Graph depicting regression to the mean with simulated blood pressure data

This could possibly explain why Beecher observed that placebo was most effective for situations in which stress was the greatest.

It may not have been just the patients’ minds that led to remission of symptoms, rather the combination of regressing towards an average from a more extreme value, the placebo effect, along with the subjectivity and measurement error of the instruments may have led to this phenomenon of experiencing relief as a function of time.6

This isn’t to say that placebo effects are nonexistent or that they’re not relevant, they still are because they serve as controls for various other phenomena that come along with them. For example, it seems very unlikely that in a clinical trial that recruits healthy participants and aims to improve their intelligence with an experimental drug, that regression to the mean could explain the improvements seen in the control group.

In a trial aiming to improve biomarkers in healthy participants, could regression towards the mean or noise explain all the variance we see in the control group? Probably not. It seems very feasible that the placebo effect can result in changes to both subjective and objective outcomes, irrespective of the effects of regression to the mean and measurement error, however, I’m inclined to believe that many of the claims regarding the placebo effect may be exaggerated.

Thus, a more accurate way to think of control group effects would be to incorporate phenomena such as regression to the mean, the placebo effect, and other factors, as shown below.

Bar graph depicting the unspecific model

Of course, the proportion of these individual effects would likely vary depending on the scenario(s). Here, the proportions of effects in the control group reflect my beliefs about what contributes to improvements seen in sick individuals who are followed over time.

While the model above is certainly an improvement over the first bar graph in the article all the way above, it too may also be simplistic… and may even be wrong.7 I’ll first discuss some theoretical examples and then present some evidence.

The Additive Model Is Too Simple

If we add up all the possible effects that lead to improvements in the control group, such as regression to the mean, the placebo effect, measurement error, unspecific effects etc, we can call this giant block (the blue block) the “total control group effect”, at least for the sake of this discussion. This is shown in the bar graph below. Notice that it has a total effect of 6.

Bar graph depicting the additive model, where treatment effects add on to control group effects

Again, here we are assuming that all the possible things that lead to improvements in the control group are represented by the blue bar. Thus, we design a clinical trial with the assumption that the total control group effect, AKA that represented by the blue bar, will also be the same in the treatment group. This is the “Additive Model” because it simply adds the treatment effect (red bar) on top of the control effect (blue bar), giving a total effect of 10. We subtract the total effect of the treatment group (10) from the total control group effect (6) to get:

Treatment-specific effect (4) = Total treatment group effect (10) - Total control group effect (6)

Therefore, we estimate that the drug has an effect of 4 whereas all the effects that contribute to the control result in an effect of 6. In the form of a linear model, we would model this as y = constant + b(group), where b is the difference between the treatment group and the control group, and group would be coded as 0 or 1, etc.

This idea may be too simplistic as it assumes that the control effect (the blue bar) will be constant between groups.

However, several studies have shown disparities in placebo/control effects between groups, with placebo/control effects sometimes being larger in control groups than in treatment groups.8–10

Experimental studies have also found that the sum of the drug effect and the placebo effect is often larger than the total treatment effect.11, 12 This doesn’t seem to support the additive model, where we assume the total control effect is constant in both groups.

It doesn’t take into account that there may be an interaction13, 14 that occurs between the control group effects and the treatment effects, especially if there are possible factors that could lead the participants in the clinical trial to discover that they’ve been allocated to the treatment group.

Thus, a more inclusive model of these possible factors, in the form of a linear model, would be:

y = *constant + βgroup + βinteractio**n(s) + ϵ*

Bar graph depicting the interactive model

These interactions could result in many scenarios, but let’s look at two simple ones.

If a participant in a drug trial is aware of the side effects of the drug beforehand, and receives the drug and experiences side effects, he/she may now have some confirmation that they’ve received the actual intervention, and may overestimate their improvement in a subjective outcome and/or may experience placebo/physiological mechanisms that overlap and interact, thus amplifying the control effects. The overall treatment group effect would increase. This would be a synergistic interaction14, as shown below. Notice the control group has a total effect of 6, while the drug group has a total effect of 13.

Bar graph depicting the synergistic interactive model

Now let’s say in another scenario, a patient in a drug trial was unaware of the possible side effects of the drug and experienced some side effects, which has somehow convinced the participant that they may not improve, or the physiological interaction in the mechanisms between the drug and placebo result in negative outcomes. This may lead to a reduction in the placebo effect and a reduction in other unspecific control effects. This would be an antagonistic interaction14, as shown below. It’s reduced the overall treatment group effect from 13 to 8.

Bar graph depicting the antagonistic interactive model

Thus, when we design clinical trials with comparators, we’re often assuming that the treatment-specific effect can be estimated by taking the difference between the total treatment group effect and the total control group effect, because we assume the control effect is constant between groups. But again, as shown above, this may be far too simplistic.

Possible Directions

So what’s the solution? How can we design clinical trials to better account for these possible interactions between the intervention and the control group effects? A potential solution that’s been put forth by several researchers over the years is the balanced-placebo design.7, 11, 12, 15, 16 The balanced-placebo design is essentially a 2 x 2 factorial design where two groups will receive the drug and two groups will receive placebo.

Depiction of a balanced placebo design

Out of the two groups receiving the drug, one will be told that they’re receiving placebo, while the other is told they’re receiving the drug. The same would apply to the two groups receiving the placebo. One will be told that they’re receiving the placebo, while the other is told they’re receiving the drug. While this design may account for interactions between control group effects and the intervention, I can’t say I’m much of a fan of it as it requires more groups (therefore more participants) and doesn’t use blinding. It may also be unethical.

A very simple solution to me (out of several proposed designs) is the use of an “active placebo.”7 Active placebos are substances that don’t yield any beneficial effects, but are designed to mimic some of the side effects of the intervention (pretty sure a more appropriate term for them would be nocebo). Thus, controlling for the possible interaction(s) that could occur if the participant is affected by the side effects of the intervention. This will make it harder for the participant to know whether they got the intervention or the placebo.

Depiction of using a design that includes an active placebo and an inert placebo

Psychiatrist Scott Alexander of Slate Star Codex and LessWrong is skeptical of active placebos and believes that the research doesn’t support their usage.17 In his article about selective serotonin reuptake inhibitors he writes,

“In other words, active placebo research has fallen out of favor in the modern world. Most studies that used active placebo are very old studies that were not very well conducted. Those studies failed to find an active-placebo-vs.-drug difference because they weren’t good enough to do this. But they also failed to find an active-placebo-vs.-inactive-placebo difference. So they provide no support for the idea that active placebos are stronger than inactive placebos in depression and in fact somewhat weigh against it.”

Scott uses a narrative review from 200017 that looked at previous active-placebo trials to support this perspective, but several other studies that have used active placebos have been carried out since

  1. Several of these are of much higher quality than the studies discussed in the review he cites. A more recent methodological review,18 published by one of the authors who systematically searched the literature to find that placebo effects were underwhelming in trials with objective outcomes,2 concludes the following,

Pharmacological active placebo control interventions are rarely used in randomized clinical trials, but they constitute a methodological tool which merits serious consideration. We suggest that active placebos are used more often in trials of drugs with noticeable side effects, especially in situations where the expected therapeutic effects are modest and the risk of bias due to unblinding is high.

Luckily, active placebo research doesn’t seem to have fallen out of favor. Is active placebo research the solution to the problems mentioned above? I’m not sure. There are several other proposed designs that attempt to account for possible interactions that can occur and they may be more practical or less biased. It seems like recognizing that such interactions could occur in the first place would be the first step towards finding a solution.

Acknowledgements: I’d like to thank Andrew Vigotsky for bringing my attention to several of the pieces cited in this article and for offering comments on an early version of this piece.

References

Back to top

References

1. Zhao W, Berger V. (2018). “Imbalance control in clinical trial subject randomization-from philosophy to strategy.” J Clin Epidemiol. 101:116–118. doi: 10.1016/j.jclinepi.2018.02.022.
2. Hróbjartsson A, Gøtzsche PC. (2001). “Is the placebo powerless? An analysis of clinical trials comparing placebo with no treatment.” N Engl J Med. 344:1594–1602. doi: 10.1056/NEJM200105243442106.
3. Beecher HK. (1955). “The powerful placebo.” J Am Med Assoc. 159:1602–1606. doi: 10.1001/jama.1955.02960340022006.
4. Senn S. (2011). “Francis Galton and regression to the mean.” Significance. 8:124–126. doi: 10.1111/j.1740-9713.2011.00509.x.
5. Atkinson G, Batterham AM. (2015). “True and false interindividual differences in the physiological response to an intervention.” Exp Physiol. 100:577–588. doi: 10.1113/EP085070.
6. McDonald CJ, Mazzuca SA, McCabe GP. (1983). “How much of the placebo ‘effect’ is really statistical regression?” Statistics in Medicine. 2:417–427. doi: 10.1002/sim.4780020401.
7. Kube T, Rief W. (2017). “Are placebo and drug-specific effects additive? Questioning basic assumptions of double-blinded randomized clinical trials and presenting novel study designs.” Drug Discov Today. 22:729–735. doi: 10.1016/j.drudis.2016.11.022.
8. Finnerup NB, Sindrup SH, Jensen TS. (2010). “The evidence for pharmacological treatment of neuropathic pain.” Pain. 150:573–581. doi: 10.1016/j.pain.2010.06.019.
9. Finnerup NB, Attal N, Haroutounian S, McNicol E, Baron R, Dworkin RH, et al. (2015). “Pharmacotherapy for neuropathic pain in adults: A systematic review and meta-analysis.” Lancet Neurol. 14:162–173. doi: 10.1016/S1474-4422(14)70251-0.
10. Rutherford BR, Tandler J, Brown PJ, Sneed JR, Roose SP. (2014). “Clinic visits in late-life depression trials: Effects on signal detection and therapeutic outcome.” Am J Geriatr Psychiatry. 22:1452–1461. doi: 10.1016/j.jagp.2013.09.003.
11. Hammami MM, Al-Gaai EA, Alvi S, Hammami MB. (2010). “Interaction between drug and placebo effects: A cross-over balanced placebo design trial.” Trials. 11:110. doi: 10.1186/1745-6215-11-110.
12. Lund K, Vase L, Petersen GL, Jensen TS, Finnerup NB. (2014). “Randomised controlled trials may underestimate drug effects: Balanced placebo trial design.” PLoS One. 9:e84104. doi: 10.1371/journal.pone.0084104.
13. Corraini P, Olsen M, Pedersen L, Dekkers OM, Vandenbroucke JP. (2017). “Effect modification, interaction and mediation: An overview of theoretical insights for clinical investigators.” Clin Epidemiol. 9:331–338. doi: 10.2147/CLEP.S129728.
14. Rothman KJ. (1974). “Synergy and antagonism in cause-effect relationships.” Am J Epidemiol. 99:385–388. doi: 10.1093/oxfordjournals.aje.a121626.
15. Atlas LY, Whittington RA, Lindquist MA, Wielgosz J, Sonty N, Wager TD. (2012). “Dissociable influences of opiates and expectations on pain.” J Neurosci. 32:8053–8064. doi: 10.1523/JNEUROSCI.0383-12.2012.
16. Schenk LA, Sprenger C, Geuter S, Büchel C. (2014). “Expectation requires treatment to boost pain relief: An fMRI study.” Pain. 155:150–157. doi: 10.1016/j.pain.2013.09.024.
17. Quitkin FM, Rabkin JG, Gerald J, Davis JM, Klein DF. (2000). “Validity of clinical trials of antidepressants.” Am J Psychiatry. 157:327–337. doi: 10.1176/appi.ajp.157.3.327.
18. Jensen JS, Bielefeldt AØ, Hróbjartsson A. (2017). “Active placebo control groups of pharmacological interventions were rarely used but merited serious consideration: A methodological overview.” J Clin Epidemiol. 87:35–46. doi: 10.1016/j.jclinepi.2017.03.001.
]]> https://lesslikely.com/statistics/control-group-effects.html Sun, 28 Oct 2018 04:00:00 GMT Misplaced Confidence in Observed Power Zad Rafi https://lesslikely.com/statistics/misplaced-power.html

Two months ago, a study came out in JAMA which compared the effectiveness of the antidepressant escitalopram to placebo for long-term major adverse cardiac events (MACE).

The authors explained in the methods section of their paper how they calculated their sample size and what differences they were looking for between groups.

First, they used some previously published data to get an idea for incidence rates,

“Because previous studies in this field have shown conflicting results, there was no appropriate reference for power calculation within the designated sample size. The KAMIR study reported a 10.9% incidence of major adverse cardiac events (MACE) over 1 year… Therefore, approximately 50% MACE incidence was expected during a 5-year follow-up.”

Then, they calculated their sample size based on some differences they were interested in finding,

“Assuming 2-sided tests, α = .05, and a follow-up sample size of 300, the expected power was 70% and 96% for detecting 10% and 15% group differences, respectively.”

So far so good.

Then, we get to the results,

Survival curve showing data from associations between antidepressants and major adverse cardiac events

“A significant difference was found: composite MACE incidence was 40.9% (61/149) in the escitalopram group and 53.6% (81/151) in the placebo group (hazard ratio [HR], 0.69; 95% CI, 0.49-0.96; P = .03). The model assumption was met (Schoenfeld P = .48). The estimated statistical power to detect the observed difference in MACE incidence rates between the 2 groups was 89.7%.”

Ouch. This issue ended up bothering me so much that I wrote a letter to the editor (LTE) to point out the issue. Unfortunately, the LTE got rejected, but Andrew Althouse suggested that I discuss this over at DataMethods, so I did, and I also discussed it on Twitter but also wanted to publish the LTE on my blog. Here it is.

This letter has now been preprinted on arXiv.

In a similar tale, a group of surgeons published a methodological article advocating this practice of calculating observed power, which I further discuss here.

Back to top

Citation

BibTeX citation:
@misc{rafi2018,
  author = {Rafi, Zad and Rafi, Zad},
  title = {Misplaced {Confidence} in {Observed} {Power}},
  date = {2018-09-30},
  url = {https://lesslikely.com/statistics/misplaced-power},
  langid = {en-US},
  abstract = {Another misinterpretation of what statistical power is and
    how trial results should be interpreted.}
}
]]> statistics https://lesslikely.com/statistics/misplaced-power.html Sun, 30 Sep 2018 04:00:00 GMT