The P-value Is Not a Verdict: An Interactive Consonance Curve

Drag an estimate and its interval and watch the entire P-value function, every compatibility interval, and the S-value move together.

A single P-value gives us one number about one hypothesis. The P-value function takes the same test statistic and applies it to every candidate parameter value at once, and (as a result) it contains every compatibility interval at every level as a horizontal slice.¹ None of the mathematics below is new. It is the usual normal approximation that most of us already rely on, except that here we draw it in full instead of evaluating it at one point.

¹ If we slice the curve at height $p = 0.05$ and read off the x-coordinates, we get the 95% limits. If we slice at $p = 0.01$, we get the 99% limits. Thus, the interval was never a separate calculation.

The display itself is quite old. Poole called for it in 1987¹, Sullivan and Foster catalogued its uses in 1990², and it has sat in Modern Epidemiology ever since. So what has changed? For one, it now costs nothing to render. We also have what I think is a better vocabulary for the vertical axis, where we speak of compatibility in place of confidence and surprisal in place of significance³.

TipDefinitions, once

A P-value function (equivalently, a consonance or compatibility curve) plots, for every candidate parameter value $\mu$, the P-value from testing $\mu$ against the data under the full set of analysis assumptions.

An S-value is the Shannon transform $s = -\log_2 p$: the number of consecutive heads from a fair coin that would be exactly as surprising as the observed test statistic, if $\mu$ and every other assumption were correct.

A 95% compatibility interval is the set of $\mu$ whose P-value exceeds 0.05. It is a horizontal slice through the curve, so we don’t need to treat it as a separate object.

The explorer

Let’s start with the explorer itself. When we move the sliders, everything recomputes in the browser (there is no server involved).

// Observable Plot's tip/pointer marks need >= 0.6.11; Quarto embeds an older
// build, so pin an explicit version. See README for how to vendor this locally
// if you'd rather not depend on a CDN at page load.
Plot = import("https://cdn.jsdelivr.net/npm/@observablehq/plot@0.6/+esm")

Plot = Module {Area: class, Arrow: class, BarX: class, BarY: class, Cell: class, Contour: class, Density: class, Dot: class, Frame: class, Geo: class, Hexgrid: class, Image: class, Line: class, Link: class, Mark: class, Raster: class, Rect: class, RuleX: class, RuleY: class, Text: class, …}

import { consonanceCurve, testValue, seFromRatioCI } from "./ojs/stats.js"

  import {consonanceCurve as consonanceCurve, testValue as testValue, seFromRatioCI as seFromRatioCI} from "./ojs/stats.js"

// Inverse standard normal CDF (Acklam's rational approximation; |error| < 4e-9
// over the levels used here, verified against scipy).
//
// TODO(stats.js): seFromRatioCI already needs this internally. Export it from
// ./ojs/stats.js and delete this cell rather than maintaining two copies.
zq = function (L) {
  const p = (1 + L) / 2;
  const a = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02,
             1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00];
  const b = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02,
             6.680131188771972e+01, -1.328068155288572e+01];
  const c = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00,
             -2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00];
  const d = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00,
             3.754408661907416e+00];
  const lo = 0.02425, hi = 1 - lo;
  let q, r;
  if (p < lo) {
    q = Math.sqrt(-2 * Math.log(p));
    return (((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) /
           ((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1);
  }
  if (p <= hi) {
    q = p - 0.5; r = q * q;
    return (((((a[0]*r+a[1])*r+a[2])*r+a[3])*r+a[4])*r+a[5]) * q /
           (((((b[0]*r+b[1])*r+b[2])*r+b[3])*r+b[4])*r+1);
  }
  q = Math.sqrt(-2 * Math.log(1 - p));
  return -(((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) /
          ((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1);
}

zq = ƒ(L)

viewof est = Inputs.range([0.5, 4], {
  value: 1.61, step: 0.01, label: "Point estimate (rate ratio)"
})

Point estimate (rate ratio)

est = 1.61

viewof cir = Inputs.range([1.05, 8], {
  value: 2.60, step: 0.01, label: "Interval ratio (upper ÷ lower)"
})

Interval ratio (upper ÷ lower)

cir = 2.6

viewof mu = Inputs.range([0.5, 4], {
  value: 1, step: 0.01, label: "Value being tested (μ)"
})

Value being tested (μ)

mu = 1

viewof level = Inputs.range([0.50, 0.995], {
  value: 0.95, step: 0.005, label: "Compatibility level to slice at"
})

Compatibility level to slice at

level = 0.95

// Parameterising by the interval *ratio* rather than by two independent limits
// guarantees the interval always brackets the estimate — one less way for a
// reader to produce a nonsensical figure.
limits = ({
  lower: est / Math.sqrt(cir),
  upper: est * Math.sqrt(cir)
})

limits = Object {lower: 0.9984796134431281, upper: 2.5960469949521334}

se = seFromRatioCI(limits.lower, limits.upper, 0.95)

se = 0.24375739874925995

// The slice at an arbitrary level, derived from the same SE. This is the whole
// claim of footnote 1 made draggable: the interval is a horizontal cut, not a
// second computation.
slice = ({
  p: 1 - level,
  lower: est * Math.exp(-zq(level) * se),
  upper: est * Math.exp(zq(level) * se)
})

slice = Object {p: 0.050000000000000044, lower: 0.9984796130585426, upper: 2.5960469959520553}

curve = consonanceCurve(est, se, {from: 0.4, to: 5, n: 600})

curve = Array(601) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

atMu = testValue(est, se, mu)

atMu = Object {mu: 1, p: 0.050734109671123656, s: 4.300900160352262}

html`<div style="
  border-left: 3px solid currentColor;
  padding: 0.6em 1em;
  margin: 1.2em 0;
  font-size: 0.95em;
  line-height: 1.7;
">
  <strong>${est.toFixed(2)}</strong>
  (95% compatibility interval
  <strong>${limits.lower.toFixed(2)}</strong> to
  <strong>${limits.upper.toFixed(2)}</strong>)
  <br>
  Testing μ = <strong>${mu.toFixed(2)}</strong>:
  <em>P</em> = <strong>${atMu.p < 0.001 ? atMu.p.toExponential(2) : atMu.p.toFixed(3)}</strong>,
  <em>S</em> = <strong>${atMu.s.toFixed(1)} bits</strong>
  <br>
  <span style="opacity:0.75">
    The data are no more surprising against μ = ${mu.toFixed(2)} than
    ${atMu.s.toFixed(1)} consecutive heads from a fair coin.
  </span>
  <br>
  <span style="opacity:0.75">
    ${(100 * level).toFixed(1)}% slice (at <em>P</em> = ${slice.p.toFixed(3)}):
    <strong>${slice.lower.toFixed(2)}</strong> to
    <strong>${slice.upper.toFixed(2)}</strong>
  </span>
</div>`

1.61 (95% compatibility interval 1.00 to 2.60)
Testing μ = 1.00: P = 0.051, S = 4.3 bits
The data are no more surprising against μ = 1.00 than 4.3 consecutive heads from a fair coin.
95.0% slice (at P = 0.050): 1.00 to 2.60

Plot.plot({
  height: 280,
  marginLeft: 55,
  marginBottom: 40,
  style: {fontSize: "13px"},
  x: {type: "log", label: "Rate ratio →", domain: [0.4, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
  y: {label: "↑ P-value", domain: [0, 1], grid: true, percent: false},
  marks: [
    Plot.areaY(curve, {x: "mu", y: "p", fillOpacity: 0.10}),
    Plot.lineY(curve, {x: "mu", y: "p", strokeWidth: 2}),
    // The draggable slice, drawn as a segment between the two limits rather
    // than a full-width rule: the horizontal extent *is* the interval.
    Plot.ruleY([slice.p], {strokeDasharray: "3 3", strokeOpacity: 0.35}),
    Plot.link([slice], {x1: "lower", x2: "upper", y1: "p", y2: "p", strokeWidth: 3, strokeOpacity: 0.85}),
    Plot.dot([{x: slice.lower, y: slice.p}, {x: slice.upper, y: slice.p}], {x: "x", y: "y", r: 3}),
    Plot.text([{x: 0.42, y: slice.p}], {x: "x", y: "y", text: [`${(100 * level).toFixed(0)}% slice`], dy: -8, textAnchor: "start", fontSize: 11, fillOpacity: 0.6}),
    Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
    Plot.ruleX([mu], {strokeWidth: 1.5, strokeOpacity: 0.9}),
    Plot.dot([atMu], {x: "mu", y: "p", r: 4.5}),
    Plot.tip(curve, Plot.pointerX({
      x: "mu", y: "p",
      title: d => [
        `Rate ratio  ${d.mu.toFixed(2)}`,
        `P-value     ${d.p < 0.001 ? d.p.toExponential(2) : d.p.toFixed(3)}`,
        `S-value     ${d.s.toFixed(1)} bits`,
        `Sits on the ${(100 * d.level).toFixed(0)}% interval limit`
      ].join("\n")
    }))
  ]
})

Plot.plot({
  height: 220,
  marginLeft: 55,
  marginBottom: 40,
  style: {fontSize: "13px"},
  x: {type: "log", label: "Rate ratio →", domain: [0.4, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
  y: {label: "↑ S-value (bits of information against)", domain: [0, 20], grid: true, clamp: true},
  marks: [
    Plot.lineY(curve, {x: "mu", y: "s", strokeWidth: 2}),
    Plot.ruleY([4.32], {strokeDasharray: "3 3", strokeOpacity: 0.5}),
    Plot.text([{x: 0.42, y: 4.32}], {x: "x", y: "y", text: ["S = 4.3 bits (P = 0.05)"], dy: -8, textAnchor: "start", fontSize: 11, fillOpacity: 0.6}),
    Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
    Plot.ruleX([mu], {strokeWidth: 1.5, strokeOpacity: 0.9}),
    Plot.dot([atMu], {x: "mu", y: "s", r: 4.5}),
    Plot.tip(curve, Plot.pointerX({
      x: "mu", y: "s",
      title: d => `Rate ratio  ${d.mu.toFixed(2)}\nS-value     ${d.s.toFixed(1)} bits`
    }))
  ]
})

Calibrating bits

Why bother with the S-value at all? Mostly because “P = 0.05” doesn’t carry much intuition, and “significant” tends to carry the wrong one. Bits seem to be easier to think about, since most of us have a physical sense of what flipping a coin is like. If we drag μ above, we can watch the coins change.

// A physical calibration strip. Whole bits are drawn as coins; the fractional
// remainder is drawn as a partial coin, because rounding 4.31 up to 5 would
// quietly overstate the evidence by a factor of ~1.6.
coins = {
  const whole = Math.floor(Math.min(atMu.s, 24));
  const frac = Math.min(atMu.s, 24) - whole;
  const cells = [];
  for (let i = 0; i < whole; i++) cells.push(1);
  if (frac > 0.02) cells.push(frac);
  return html`<div style="margin: 0.8em 0 1.4em;">
    <div style="display:flex; flex-wrap:wrap; gap:5px; align-items:center; min-height:22px;">
      ${cells.map(f => html`<span style="
        display:inline-block; width:18px; height:18px; border-radius:50%;
        border:1.5px solid currentColor;
        background: linear-gradient(to right, currentColor ${(100*f).toFixed(0)}%, transparent ${(100*f).toFixed(0)}%);
        opacity:0.8;"></span>`)}
      ${atMu.s > 24 ? html`<span style="opacity:0.7; font-size:0.9em;">… and ${(atMu.s - 24).toFixed(0)} more</span>` : ``}
    </div>
    <div style="opacity:0.7; font-size:0.9em; margin-top:0.5em;">
      ${atMu.s.toFixed(2)} bits against μ = ${mu.toFixed(2)}.
      ${atMu.s < 1 ? "Less surprising than a single coin flip."
        : atMu.s < 4.32 ? "Less surprising than the conventional 0.05 threshold."
        : "More surprising than the conventional 0.05 threshold."}
    </div>
  </div>`;
}

4.30 bits against μ = 1.00. Less surprising than the conventional 0.05 threshold.

WarningWhat the coins do not license

The S-value measures how surprising the data are given the whole model, and $\mu$ is only one part of that model. Every bit is also a bit against the assumed absence of uncontrolled confounding, selection effects, measurement error, and misspecified model form. Thus, a large S-value tells us that the data are hard to reconcile with $\mu$ together with everything else that we assumed. However, it cannot tell us which of those assumptions to give up. In observational work, the “everything else” is usually the weaker link.

A worked example: one data set, four analyses

Brown and colleagues studied serotonergic antidepressant use in pregnancy and autism spectrum disorder in 35,906 Ontario births, of which 2,837 pregnancies (7.9%) were exposed⁴. They reported four hazard ratios from the same cohort under progressively stronger confounding control. This is the example reanalysed at length in³, and the working code for that paper lives here.

Before anything else, let’s look at the point estimates.

brown = [
  {key: "crude",  label: "Crude",                 hr: 2.16, lower: 1.64,  upper: 2.86, ciLabel: "1.64 – 2.86",  control: "None"},
  {key: "adj",    label: "Covariate-adjusted",    hr: 1.59, lower: 1.17,  upper: 2.17, ciLabel: "1.17 – 2.17",  control: "Regression adjustment"},
  {key: "iptw",   label: "HdPS IPT-weighted",     hr: 1.61, lower: 0.997, upper: 2.59, ciLabel: "0.997 – 2.59", control: "High-dimensional propensity score"},
  {key: "sib",    label: "Sibling-matched",       hr: 1.60, lower: 0.69,  upper: 3.74, ciLabel: "0.69 – 3.74",  control: "Within-family (shared confounders)"}
]

brown = Array(4) [Object, Object, Object, Object]

brownStats = brown.map(d => {
  const s = seFromRatioCI(d.lower, d.upper);
  const t = testValue(d.hr, s, 1);
  return {...d, se: s, pNull: t.p, sNull: t.s, ratio: d.upper / d.lower};
})

brownStats = Array(4) [Object, Object, Object, Object]

Inputs.table(
  brownStats.map(d => ({
    Analysis: d.label,
    "Confounding control": d.control,
    HR: d.hr.toFixed(2),
    "95% CI": d.ciLabel,
    "Interval ratio": d.ratio.toFixed(2),
    "P (vs HR = 1)": d.pNull < 0.001 ? d.pNull.toExponential(1) : d.pNull.toFixed(4),
    "S (bits)": d.sNull.toFixed(2)
  })),
  {rows: 6, layout: "auto"}
)

	Analysis	Confounding control	HR	95% CI	Interval ratio	P (vs HR = 1)	S (bits)
	Crude	None	2.16	1.64 – 2.86	1.74	5.7e-8	24.07
	Covariate-adjusted	Regression adjustment	1.59	1.17 – 2.17	1.85	0.0033	8.26
	HdPS IPT-weighted	High-dimensional propensity score	1.61	0.997 – 2.59	2.60	0.0505	4.31
	Sibling-matched	Within-family (shared confounders)	1.60	0.69 – 3.74	5.42	0.2757	1.86

Three of the four point estimates are 1.59, 1.60 and 1.61. By the conventional reading, one of them is a finding and two of them are nothing. However, the estimate barely changes between them. What changes is the width of the interval, which grows from a ratio of 1.9 to 5.4 across the three adjusted analyses, as each one trades precision for stronger confounding control. So the verdicts flip while the estimate stays where it was.

The published conclusion was that exposure “was not associated with” autism. I don’t think the arithmetic supports that sentence, since the IPT-weighted analysis puts $P = 0.0505$ against the null (4.31 bits of surprise), and its interval runs to 2.59.

viewof shown = Inputs.checkbox(brown, {
  value: brown,
  format: d => d.label,
  label: "Analyses to overlay"
})

Analyses to overlay

CrudeCovariate-adjustedHdPS IPT-weightedSibling-matched

shown = Array(4) [Object, Object, Object, Object]

brownCurves = shown.flatMap(d => {
  const s = seFromRatioCI(d.lower, d.upper);
  return consonanceCurve(d.hr, s, {from: 0.5, to: 5, n: 500})
    .map(pt => ({...pt, analysis: d.label}));
})

brownCurves = Array(2004) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

Plot.plot({
  height: 320,
  marginLeft: 55,
  marginBottom: 40,
  style: {fontSize: "13px"},
  color: {legend: true},
  x: {type: "log", label: "Hazard ratio →", domain: [0.5, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
  y: {label: "↑ P-value", domain: [0, 1], grid: true},
  marks: [
    Plot.ruleY([0.05], {strokeDasharray: "3 3", strokeOpacity: 0.4}),
    Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
    Plot.lineY(brownCurves, {x: "mu", y: "p", z: "analysis", stroke: "analysis", strokeWidth: 2}),
    // P = 1 exactly at the point estimate, so each dot sits at the apex of its curve.
    Plot.dot(shown, {x: "hr", y: () => 1, fill: "label", r: 4}),
    Plot.tip(brownCurves, Plot.pointer({
      x: "mu", y: "p", z: "analysis",
      title: d => [
        d.analysis,
        `Hazard ratio  ${d.mu.toFixed(2)}`,
        `P-value       ${d.p < 0.001 ? d.p.toExponential(2) : d.p.toFixed(3)}`,
        `S-value       ${d.s.toFixed(1)} bits`
      ].join("\n")
    }))
  ]
})

Covariate-adjustedCrudeHdPS IPT-weightedSibling-matched

If we uncheck the crude analysis, the remaining three curves are nearly concentric, with the same peak and different widths. That is what a precision problem looks like, and it gives us little reason to think the analyses disagree about the size of the effect. Now, let’s test a value other than the null:

viewof muBrown = Inputs.range([0.5, 4], {
  value: 1.5, step: 0.01, label: "Test every analysis against this hazard ratio"
})

Test every analysis against this hazard ratio

muBrown = 1.5

Inputs.table(
  brownStats.map(d => {
    const t = testValue(d.hr, d.se, muBrown);
    return {
      Analysis: d.label,
      HR: d.hr.toFixed(2),
      [`P (vs ${muBrown.toFixed(2)})`]: t.p < 0.001 ? t.p.toExponential(1) : t.p.toFixed(3),
      "S (bits)": t.s.toFixed(2),
      Reading: t.s < 1 ? "Highly compatible" : t.s < 4.32 ? "Compatible" : "Poorly compatible"
    };
  }),
  {rows: 6, layout: "auto"}
)

	Analysis	HR	P (vs 1.50)	S (bits)	Reading
	Crude	2.16	0.010	6.62	Poorly compatible
	Covariate-adjusted	1.59	0.712	0.49	Highly compatible
	HdPS IPT-weighted	1.61	0.771	0.37	Highly compatible
	Sibling-matched	1.60	0.881	0.18	Highly compatible

If we set the slider to 1.5, the four analyses give $S$ = 6.6, 0.5, 0.4 and 0.2 bits. Three of the four find a 50% increase in hazard less surprising than a single coin flip. Thus, against this hypothesis, the analysis that was “not significant” and the analysis that was “significant” are saying nearly the same thing.

NoteWhy the null gets a curve and not a verdict

Testing HR = 1 is a legitimate question. However, if we report only that test, and only its dichotomised outcome, we discard the answer to every other question that the same data address. Some of those are questions the reader may care about more, such as whether an effect large enough to change prescribing can be ruled out. On these data, it cannot be ruled out, since the IPT-weighted 95% interval reaches 2.59, and even the 50% interval runs from 1.37 to 1.90.

What more data would do

Precision and the parameter value are two separate things, but a single P-value fuses them into one number. To pull them apart, we can hold the IPT-weighted estimate fixed and vary only the amount of information (the standard error scales as $1/\sqrt{k}$ when the sample size is multiplied by $k$).

viewof kInfo = Inputs.range([0.25, 16], {
  value: 1, step: 0.05, label: "Sample size multiplier (k)"
})

Sample size multiplier (k)

kInfo = 1

base = ({hr: 1.61, se: seFromRatioCI(0.997, 2.59)})

base = Object {hr: 1.61, se: 0.24354079775495885}

scaled = ({se: base.se / Math.sqrt(kInfo)})

scaled = Object {se: 0.24354079775495885}

scaledCurve = consonanceCurve(base.hr, scaled.se, {from: 0.5, to: 5, n: 500})

scaledCurve = Array(501) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

baseCurve = consonanceCurve(base.hr, base.se, {from: 0.5, to: 5, n: 500})

baseCurve = Array(501) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

scaledAtNull = testValue(base.hr, scaled.se, 1)

scaledAtNull = Object {mu: 1, p: 0.050528853158511446, s: 4.306748753957083}

html`<div style="border-left:3px solid currentColor; padding:0.6em 1em; margin:1.2em 0; font-size:0.95em; line-height:1.7;">
  At <strong>${kInfo.toFixed(2)}×</strong> the original sample size, with the estimate held at
  <strong>1.61</strong>:
  95% interval
  <strong>${(base.hr * Math.exp(-zq(0.95) * scaled.se)).toFixed(2)}</strong> to
  <strong>${(base.hr * Math.exp(zq(0.95) * scaled.se)).toFixed(2)}</strong>,
  <em>P</em> = <strong>${scaledAtNull.p < 0.001 ? scaledAtNull.p.toExponential(2) : scaledAtNull.p.toFixed(4)}</strong>
  against the null,
  <em>S</em> = <strong>${scaledAtNull.s.toFixed(1)} bits</strong>.
</div>`

At 1.00× the original sample size, with the estimate held at 1.61: 95% interval 1.00 to 2.59, P = 0.0505 against the null, S = 4.3 bits.

Plot.plot({
  height: 280,
  marginLeft: 55,
  marginBottom: 40,
  style: {fontSize: "13px"},
  x: {type: "log", label: "Hazard ratio →", domain: [0.5, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
  y: {label: "↑ P-value", domain: [0, 1], grid: true},
  marks: [
    Plot.ruleY([0.05], {strokeDasharray: "3 3", strokeOpacity: 0.4}),
    Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
    Plot.lineY(baseCurve, {x: "mu", y: "p", strokeWidth: 1.5, strokeOpacity: 0.30, strokeDasharray: "2 3"}),
    Plot.areaY(scaledCurve, {x: "mu", y: "p", fillOpacity: 0.10}),
    Plot.lineY(scaledCurve, {x: "mu", y: "p", strokeWidth: 2}),
    Plot.dot([{x: base.hr, y: 1}], {x: "x", y: "y", r: 4.5})
  ]
})

The dashed ghost curve is the study as it was published. If we push $k$ to 4, the null falls well outside the 95% interval, even though the estimate hasn’t moved at all. If we pull $k$ down to 0.3, the same estimate becomes compatible with almost everything between 0.7 and 3.9.

I think this deserves some attention, because it seems to be the mechanism behind a lot of “failure to replicate” reporting. A study that reports $P = 0.04$ and a study that reports $P = 0.12$ can have identical point estimates and differ only in how many people they enrolled. Neither of those numbers is a property of the effect.

ImportantThe one thing this widget cannot show

Scaling the standard error only models random error. Multiplying the sample size by 16 does nothing to confounding, selection, or measurement error, since those biases do not shrink with $\sqrt{n}$, and past some sample size they dominate entirely. Thus, a very narrow curve can still be centred in the wrong place, and it will look far more convincing than it should. If we want those biases quantified (instead of assumed away), we need a bias analysis, and adding more subjects won’t help.

Do two estimates conflict?

We often hear something like “Study A was significant, study B was not, so they disagree.” However, that compares two verdicts and says nothing direct about the two estimates. The comparison that actually answers the question has a curve of its own. Under independence, the log ratio-of-ratios has standard error $\sqrt{se_A^2 + se_B^2}$, and the null of no difference sits at a ratio of 1.

Study A is the explorer at the top of this page, and we can set study B here.

viewof estB = Inputs.range([0.5, 4], {value: 1.05, step: 0.01, label: "Study B: point estimate"})

Study B: point estimate

estB = 1.05

viewof cirB = Inputs.range([1.05, 8], {value: 1.60, step: 0.01, label: "Study B: interval ratio"})

Study B: interval ratio

cirB = 1.6

limitsB = ({lower: estB / Math.sqrt(cirB), upper: estB * Math.sqrt(cirB)})

limitsB = Object {lower: 0.8300978857941996, upper: 1.3281566172707193}

seB = seFromRatioCI(limitsB.lower, limitsB.upper, 0.95)

seB = 0.11990108822230006

diff = ({ratio: est / estB, se: Math.sqrt(se * se + seB * seB)})

diff = Object {ratio: 1.5333333333333334, se: 0.2716504010707466}

// The ratio of ratios can leave a fixed window entirely (A/B ranges from 0.125
// to 8 across the slider limits), so the domain follows the estimate instead of
// silently cropping the peak off the plot.
diffDomain = ({
  from: Math.min(0.25, diff.ratio / 3),
  to: Math.max(4, diff.ratio * 3)
})

diffDomain = Object {from: 0.25, to: 4.6000000000000005}

diffCurve = consonanceCurve(diff.ratio, diff.se, {from: diffDomain.from, to: diffDomain.to, n: 500})

diffCurve = Array(501) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

diffAtNull = testValue(diff.ratio, diff.se, 1)

diffAtNull = Object {mu: 1, p: 0.11560129228171975, s: 3.1127705693843435}

html`<div style="border-left:3px solid currentColor; padding:0.6em 1em; margin:1.2em 0; font-size:0.95em; line-height:1.7;">
  Study A: <strong>${est.toFixed(2)}</strong> (${limits.lower.toFixed(2)}–${limits.upper.toFixed(2)}),
  <em>P</em> vs 1 = <strong>${testValue(est, se, 1).p.toFixed(3)}</strong><br>
  Study B: <strong>${estB.toFixed(2)}</strong> (${limitsB.lower.toFixed(2)}–${limitsB.upper.toFixed(2)}),
  <em>P</em> vs 1 = <strong>${testValue(estB, seB, 1).p.toFixed(3)}</strong><br>
  <span style="opacity:0.9">Ratio of ratios A ÷ B = <strong>${diff.ratio.toFixed(2)}</strong>
  (95% CI ${(diff.ratio * Math.exp(-zq(0.95) * diff.se)).toFixed(2)}–${(diff.ratio * Math.exp(zq(0.95) * diff.se)).toFixed(2)}),
  <em>P</em> for no difference = <strong>${diffAtNull.p.toFixed(3)}</strong>,
  <em>S</em> = <strong>${diffAtNull.s.toFixed(1)} bits</strong></span>
</div>`

Study A: 1.61 (1.00–2.60), P vs 1 = 0.051
Study B: 1.05 (0.83–1.33), P vs 1 = 0.684
Ratio of ratios A ÷ B = 1.53 (95% CI 0.90–2.61), P for no difference = 0.116, S = 3.1 bits

Plot.plot({
  height: 250,
  marginLeft: 55,
  marginBottom: 40,
  style: {fontSize: "13px"},
  x: {type: "log", label: "Ratio of ratios (A ÷ B) →", domain: [diffDomain.from, diffDomain.to], tickFormat: d => d},
  y: {label: "↑ P-value for the difference", domain: [0, 1], grid: true},
  marks: [
    Plot.areaY(diffCurve, {x: "mu", y: "p", fillOpacity: 0.10}),
    Plot.lineY(diffCurve, {x: "mu", y: "p", strokeWidth: 2}),
    Plot.ruleY([0.05], {strokeDasharray: "3 3", strokeOpacity: 0.4}),
    Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
    Plot.dot([diffAtNull], {x: "mu", y: "p", r: 4.5}),
    Plot.tip(diffCurve, Plot.pointerX({
      x: "mu", y: "p",
      title: d => `Ratio of ratios  ${d.mu.toFixed(2)}\nP-value          ${d.p.toFixed(3)}\nS-value          ${d.s.toFixed(1)} bits`
    }))
  ]
})

With the defaults (A at 1.61 with $P = 0.051$, and B at 1.05 with $P = 0.684$), the two studies land on opposite sides of the conventional threshold. However, their ratio of ratios is 1.53 (95% interval 0.90 to 2.61), and the difference carries 3.1 bits of information. That is fewer than the 4.3 bits that the 0.05 convention itself demands, and fewer than four coin flips. So the two verdicts disagree, while the direct comparison gives us little evidence of a difference.

WarningThis does not apply to the four Brown analyses

The overlay in the previous section shows four analyses of the same subjects. Their estimates are strongly correlated, so $\sqrt{se_A^2 + se_B^2}$ badly overstates the standard error of their difference, and the resulting P-value is not interpretable. Thus, this widget should only be used for studies that are actually independent. For nested analyses of one data set, I’d stick with the overlay itself as the summary (same peak, different width).

NoteWhy two panels instead of one with two axes

A P-value and an S-value are the same quantity on different scales, so a dual axis is technically defensible. However, I still think it is a bad idea. Readers reliably misread which curve belongs to which axis, and the visual slope of the P-value curve near the tails is uninformative in the same region where the S-value is most informative. Two stacked panels that share an x-axis cost one extra inch of page, and they remove the ambiguity.

NoteThings this figure deliberately does not do

• No shading of “significant” regions. The whole point of the curve is that there is no cliff at 0.05.
• No default null at 1. The μ slider starts at 1 only because that is the conventional test value, and I wouldn’t treat it as privileged.
• No P-value below about 1e-300. Double precision runs out. The implementation stays accurate to roughly z = 37, which is far past the point where the normal approximation itself becomes the binding constraint.
• No normality diagnostics. Every curve here assumes that the log estimate is approximately normal with the reported standard error. The published interval already made that same assumption, so the curve adds nothing new here and only makes the assumption visible across the whole range.

Interactive table

Here’s the same idea applied to tabular results. The table is searchable, and it can be sorted by clicking the column headers, which is most of what makes a long results table usable. These are the Brown analyses again, with each row carrying its own P-value against whichever hypothesis we choose.

viewof muTable = Inputs.range([0.5, 4], {value: 1, step: 0.01, label: "Hypothesis tested in the table (μ)"})

Hypothesis tested in the table (μ)

muTable = 1

tableRows = brownStats.map(d => {
  const t = testValue(d.hr, d.se, muTable);
  return {
    Analysis: d.label,
    "Hazard ratio": d.hr,
    "95% CI": d.ciLabel,
    "SE (log scale)": +d.se.toFixed(3),
    "P": t.p < 0.001 ? t.p.toExponential(1) : +t.p.toFixed(4),
    "S (bits)": +t.s.toFixed(2)
  };
})

tableRows = Array(4) [Object, Object, Object, Object]

viewof tableSearch = Inputs.search(tableRows, {placeholder: "Filter analyses…"})

4 results

tableSearch = Array(4) [Object, Object, Object, Object]

Inputs.table(tableSearch, {
  sort: "S (bits)",
  reverse: true,
  rows: 12
})

	Analysis	Hazard ratio	95% CI	SE (log scale)	P	▾S (bits)
	Crude	2.16	1.64 – 2.86	0.142	5.7e-8	24.07
	Covariate-adjusted	1.59	1.17 – 2.17	0.158	0.003	8.26
	HdPS IPT-weighted	1.61	0.997 – 2.59	0.244	0.051	4.31
	Sibling-matched	1.6	0.69 – 3.74	0.431	0.276	1.86

How to report this

None of this requires a new statistic, a new threshold, or new software. I’d suggest four sentences that can go straight into a results section:

1. Report the estimate and interval without the word “significant” (for example, “HR 1.61, 95% compatibility interval 0.997 to 2.59”).
2. Report the P-value as a number instead of as an inequality against 0.05, and say which hypothesis it tests.
3. Report at least one P-value for a non-null hypothesis that matters clinically or practically. If the data cannot distinguish the null from a 60% increase, then that is the finding.
4. Say that the interval and the P-value are conditional on the entire model, and that the parameter is only one part of it.

What if a reviewer asks for the significance verdict back? I’d point them to the curve, since it contains their verdict as one horizontal slice, along with every slice they did not ask for.

Reader infrastructure that is already on

Lastly, three of the things that people associate with Gwern’s pages happen to be Quarto defaults, and they need no code at all:

• Citation popups. Hover any @citekey reference.
• Footnote popups. Hover the marker on ² above.
• Cross-reference popups. Hover any @fig- or @tbl- reference.

² It helps to put the 0.05 convention in bits. It amounts to 4.32 bits of information against the tested hypothesis, or roughly the surprise of four consecutive heads. I doubt that many of us would abandon a belief on four coin flips.

All three are controlled by citations-hover, footnotes-hover and crossrefs-hover, which default to true. The margin placement of footnotes here comes from reference-location: margin in this document’s front matter.

References

1. Poole C. (1987). “Beyond the confidence interval.” American Journal of Public Health. 77:195–199. doi: 10.2105/ajph.77.2.195.

Abstract

Beyond the confidence interval. C PooleCopyRight https://doi.org/10.2105/AJPH.77.2.195 Published Online: October 07, 2011

2. Sullivan KM, Foster DA. (1990). “Use of the confidence interval function.” Epidemiology. 1:39–42. doi: 10.1097/00001648-199001000-00009.

Abstract

Graphics displaying all confidence intervals around a point estimate have been referred to as P-value functions and consonance intervals. We recommend use of the term confidence interval function (CI function) rather than P-value function. The CI function is useful because it simultaneously depicts point estimation, variability, and the relation of these two factors to the null value. The usefulness of the CI function in demonstrating the concepts of effect modification and confounding, in meta-analysis, and in the comparison of various confidence interval procedures is evaluated. Software packages that produce CI functions are described.

3. Rafi Z, Greenland S. (2020). “Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise.” BMC Medical Research Methodology. 20:244. doi: 10.1186/s12874-020-01105-9.

Abstract

BACKGROUND: Researchers often misinterpret and misrepresent statistical outputs. This abuse has led to a large literature on modification or replacement of testing thresholds and P-values with confidence intervals, Bayes factors, and other devices. Because the core problems appear cognitive rather than statistical, we review some simple methods to aid researchers in interpreting statistical outputs. These methods emphasize logical and information concepts over probability, and thus may be more robust to common misinterpretations than are traditional descriptions. METHODS: (p), to provide a measure of the information supplied by the testing procedure, and to help calibrate intuitions against simple physical experiments like coin tossing. We also use tables or graphs of test statistics for alternative hypotheses, and interval estimates for different percentile levels, to thwart fallacies arising from arbitrary dichotomies. Finally, we reinterpret P-values and interval estimates in unconditional terms, which describe compatibility of data with the entire set of analysis assumptions. We illustrate these methods with a reanalysis of data from an existing record-based cohort study …

4. Brown HK, Ray JG, Wilton AS, Lunsky Y, Gomes T, Vigod SN. (2017). “Association between serotonergic antidepressant use during pregnancy and autism spectrum disorder in children.” JAMA. 317:1544–1552. doi: 10.1001/jama.2017.3415.

Abstract

IMPORTANCE: Previous observations of a higher risk of child autism spectrum disorder with serotonergic antidepressant exposure during pregnancy may have been confounded. OBJECTIVE: To evaluate the association between serotonergic antidepressant exposure during pregnancy and child autism spectrum disorder. DESIGN, SETTING, AND PARTICIPANTS: Retrospective cohort study. Health administrative data sets were used to study children born to mothers who were receiving public prescription drug coverage during pregnancy in Ontario, Canada, from 2002-2010, reflecting 4.2% of births. Children were followed up until March 31, 2014. EXPOSURES: Serotonergic antidepressant exposure was defined as 2 or more consecutive maternal prescriptions for a selective serotonin or serotonin-norepinephrine reuptake inhibitor between conception and delivery. MAIN OUTCOMES AND MEASURES: Child autism spectrum disorder identified after the age of 2 years …

Linked from (5)

• All Posts
• Blog § All Posts
• Less Likely
• Site Map § Search & Navigation
• Statistics § Latest Statistics Articles