The P-value Is Not a Verdict: An Interactive Consonance Curve

statistics

interactive

Drag an estimate and its interval and watch the entire P-value function, every compatibility interval, and the S-value move together.

Reading time

9 min readLeave a comment

Author

Zad Rafi

Published

September 16, 2026

A single P-value reports one number about one hypothesis. The P-value function reports the same test statistic applied to every candidate parameter value at once — and it contains every compatibility interval at every level as horizontal slices.¹ Nothing below is new mathematics. It is the same normal approximation you already use, drawn in full instead of evaluated at one point.

¹ Slice the curve at height $p = 0.05$ and read off the x-coordinates: those are the 95% limits. Slice at $p = 0.01$ and you have the 99% limits. The interval was never a separate calculation.

The display is old. Poole called for it in 1987¹, Sullivan and Foster catalogued its uses in 1990², and it has sat in Modern Epidemiology ever since. What has changed is that it now costs nothing to render, and that we have a better vocabulary for the vertical axis: compatibility rather than confidence, and surprisal rather than significance³.

Definitions, once

A P-value function (equivalently, a consonance or compatibility curve) plots, for every candidate parameter value $\mu$ , the P-value from testing $\mu$ against the data under the full set of analysis assumptions.

An S-value is the Shannon transform $s = -\log_2 p$ : the number of consecutive heads from a fair coin that would be exactly as surprising as the observed test statistic, if $\mu$ and every other assumption were correct.

A 95% compatibility interval is the set of $\mu$ whose P-value exceeds 0.05 — a horizontal slice through the curve, not a separate object.

The explorer

Move the sliders. Everything recomputes in the browser — no server, no round trip.

// Observable Plot's tip/pointer marks need >= 0.6.11; Quarto embeds an older
// build, so pin an explicit version. See README for how to vendor this locally
// if you'd rather not depend on a CDN at page load.
Plot = import("https://cdn.jsdelivr.net/npm/@observablehq/plot@0.6/+esm")

Plot = Module {Area: class, Arrow: class, BarX: class, BarY: class, Cell: class, Contour: class, Density: class, Dot: class, Frame: class, Geo: class, Hexgrid: class, Image: class, Line: class, Link: class, Mark: class, Raster: class, Rect: class, RuleX: class, RuleY: class, Text: class, …}

import { consonanceCurve, testValue, seFromRatioCI } from "./ojs/stats.js"

  import {consonanceCurve as consonanceCurve, testValue as testValue, seFromRatioCI as seFromRatioCI} from "./ojs/stats.js"

// Inverse standard normal CDF (Acklam's rational approximation; |error| < 4e-9
// over the levels used here, verified against scipy).
//
// TODO(stats.js): seFromRatioCI already needs this internally. Export it from
// ./ojs/stats.js and delete this cell rather than maintaining two copies.
zq = function (L) {
  const p = (1 + L) / 2;
  const a = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02,
             1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00];
  const b = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02,
             6.680131188771972e+01, -1.328068155288572e+01];
  const c = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00,
             -2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00];
  const d = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00,
             3.754408661907416e+00];
  const lo = 0.02425, hi = 1 - lo;
  let q, r;
  if (p < lo) {
    q = Math.sqrt(-2 * Math.log(p));
    return (((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) /
           ((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1);
  }
  if (p <= hi) {
    q = p - 0.5; r = q * q;
    return (((((a[0]*r+a[1])*r+a[2])*r+a[3])*r+a[4])*r+a[5]) * q /
           (((((b[0]*r+b[1])*r+b[2])*r+b[3])*r+b[4])*r+1);
  }
  q = Math.sqrt(-2 * Math.log(1 - p));
  return -(((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) /
          ((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1);
}

zq = ƒ(L)

viewof est = Inputs.range([0.5, 4], {
  value: 1.61, step: 0.01, label: "Point estimate (rate ratio)"
})

est = 1.61

viewof cir = Inputs.range([1.05, 8], {
  value: 2.60, step: 0.01, label: "Interval ratio (upper ÷ lower)"
})

cir = 2.6

viewof mu = Inputs.range([0.5, 4], {
  value: 1, step: 0.01, label: "Value being tested (μ)"
})

mu = 1

viewof level = Inputs.range([0.50, 0.995], {
  value: 0.95, step: 0.005, label: "Compatibility level to slice at"
})

level = 0.95

// Parameterising by the interval *ratio* rather than by two independent limits
// guarantees the interval always brackets the estimate — one less way for a
// reader to produce a nonsensical figure.
limits = ({
  lower: est / Math.sqrt(cir),
  upper: est * Math.sqrt(cir)
})

limits = Object {lower: 0.9984796134431281, upper: 2.5960469949521334}

se = seFromRatioCI(limits.lower, limits.upper, 0.95)

se = 0.24375739874925995

// The slice at an arbitrary level, derived from the same SE. This is the whole
// claim of footnote 1 made draggable: the interval is a horizontal cut, not a
// second computation.
slice = ({
  p: 1 - level,
  lower: est * Math.exp(-zq(level) * se),
  upper: est * Math.exp(zq(level) * se)
})

slice = Object {p: 0.050000000000000044, lower: 0.9984796130585426, upper: 2.5960469959520553}

curve = consonanceCurve(est, se, {from: 0.4, to: 5, n: 600})

curve = Array(601) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

atMu = testValue(est, se, mu)

atMu = Object {mu: 1, p: 0.050734109671123656, s: 4.300900160352262}

html`<div style="
  border-left: 3px solid currentColor;
  padding: 0.6em 1em;
  margin: 1.2em 0;
  font-size: 0.95em;
  line-height: 1.7;
">
  <strong>${est.toFixed(2)}</strong>
  (95% compatibility interval
  <strong>${limits.lower.toFixed(2)}</strong> to
  <strong>${limits.upper.toFixed(2)}</strong>)
  <br>
  Testing μ = <strong>${mu.toFixed(2)}</strong>:
  <em>P</em> = <strong>${atMu.p < 0.001 ? atMu.p.toExponential(2) : atMu.p.toFixed(3)}</strong>,
  <em>S</em> = <strong>${atMu.s.toFixed(1)} bits</strong>
  <br>
  <span style="opacity:0.75">
    The data are no more surprising against μ = ${mu.toFixed(2)} than
    ${atMu.s.toFixed(1)} consecutive heads from a fair coin.
  </span>
  <br>
  <span style="opacity:0.75">
    ${(100 * level).toFixed(1)}% slice (at <em>P</em> = ${slice.p.toFixed(3)}):
    <strong>${slice.lower.toFixed(2)}</strong> to
    <strong>${slice.upper.toFixed(2)}</strong>
  </span>
</div>`

1.61 (95% compatibility interval 1.00 to 2.60)
Testing μ = 1.00: P = 0.051, S = 4.3 bits
The data are no more surprising against μ = 1.00 than 4.3 consecutive heads from a fair coin.
95.0% slice (at P = 0.050): 1.00 to 2.60

Plot.plot({
  height: 280,
  marginLeft: 55,
  marginBottom: 40,
  style: {fontSize: "13px"},
  x: {type: "log", label: "Rate ratio →", domain: [0.4, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
  y: {label: "↑ P-value", domain: [0, 1], grid: true, percent: false},
  marks: [
    Plot.areaY(curve, {x: "mu", y: "p", fillOpacity: 0.10}),
    Plot.lineY(curve, {x: "mu", y: "p", strokeWidth: 2}),
    // The draggable slice, drawn as a segment between the two limits rather
    // than a full-width rule: the horizontal extent *is* the interval.
    Plot.ruleY([slice.p], {strokeDasharray: "3 3", strokeOpacity: 0.35}),
    Plot.link([slice], {x1: "lower", x2: "upper", y1: "p", y2: "p", strokeWidth: 3, strokeOpacity: 0.85}),
    Plot.dot([{x: slice.lower, y: slice.p}, {x: slice.upper, y: slice.p}], {x: "x", y: "y", r: 3}),
    Plot.text([{x: 0.42, y: slice.p}], {x: "x", y: "y", text: [`${(100 * level).toFixed(0)}% slice`], dy: -8, textAnchor: "start", fontSize: 11, fillOpacity: 0.6}),
    Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
    Plot.ruleX([mu], {strokeWidth: 1.5, strokeOpacity: 0.9}),
    Plot.dot([atMu], {x: "mu", y: "p", r: 4.5}),
    Plot.tip(curve, Plot.pointerX({
      x: "mu", y: "p",
      title: d => [
        `Rate ratio  ${d.mu.toFixed(2)}`,
        `P-value     ${d.p < 0.001 ? d.p.toExponential(2) : d.p.toFixed(3)}`,
        `S-value     ${d.s.toFixed(1)} bits`,
        `Sits on the ${(100 * d.level).toFixed(0)}% interval limit`
      ].join("\n")
    }))
  ]
})

Plot.plot({
  height: 220,
  marginLeft: 55,
  marginBottom: 40,
  style: {fontSize: "13px"},
  x: {type: "log", label: "Rate ratio →", domain: [0.4, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
  y: {label: "↑ S-value (bits of information against)", domain: [0, 20], grid: true, clamp: true},
  marks: [
    Plot.lineY(curve, {x: "mu", y: "s", strokeWidth: 2}),
    Plot.ruleY([4.32], {strokeDasharray: "3 3", strokeOpacity: 0.5}),
    Plot.text([{x: 0.42, y: 4.32}], {x: "x", y: "y", text: ["S = 4.3 bits (P = 0.05)"], dy: -8, textAnchor: "start", fontSize: 11, fillOpacity: 0.6}),
    Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
    Plot.ruleX([mu], {strokeWidth: 1.5, strokeOpacity: 0.9}),
    Plot.dot([atMu], {x: "mu", y: "s", r: 4.5}),
    Plot.tip(curve, Plot.pointerX({
      x: "mu", y: "s",
      title: d => `Rate ratio  ${d.mu.toFixed(2)}\nS-value     ${d.s.toFixed(1)} bits`
    }))
  ]
})

Calibrating bits

The S-value exists because “P = 0.05” carries no intuition and “significant” carries the wrong one. Bits do carry intuition, because everyone has a physical model of coin flipping. Drag μ above and watch the coins.

// A physical calibration strip. Whole bits are drawn as coins; the fractional
// remainder is drawn as a partial coin, because rounding 4.31 up to 5 would
// quietly overstate the evidence by a factor of ~1.6.
coins = {
  const whole = Math.floor(Math.min(atMu.s, 24));
  const frac = Math.min(atMu.s, 24) - whole;
  const cells = [];
  for (let i = 0; i < whole; i++) cells.push(1);
  if (frac > 0.02) cells.push(frac);
  return html`<div style="margin: 0.8em 0 1.4em;">
    <div style="display:flex; flex-wrap:wrap; gap:5px; align-items:center; min-height:22px;">
      ${cells.map(f => html`<span style="
        display:inline-block; width:18px; height:18px; border-radius:50%;
        border:1.5px solid currentColor;
        background: linear-gradient(to right, currentColor ${(100*f).toFixed(0)}%, transparent ${(100*f).toFixed(0)}%);
        opacity:0.8;"></span>`)}
      ${atMu.s > 24 ? html`<span style="opacity:0.7; font-size:0.9em;">… and ${(atMu.s - 24).toFixed(0)} more</span>` : ``}
    </div>
    <div style="opacity:0.7; font-size:0.9em; margin-top:0.5em;">
      ${atMu.s.toFixed(2)} bits against μ = ${mu.toFixed(2)}.
      ${atMu.s < 1 ? "Less surprising than a single coin flip."
        : atMu.s < 4.32 ? "Less surprising than the conventional 0.05 threshold."
        : "More surprising than the conventional 0.05 threshold."}
    </div>
  </div>`;
}

4.30 bits against μ = 1.00. Less surprising than the conventional 0.05 threshold.

What the coins do not license

The S-value measures surprise at the whole model, not at $\mu$ alone. Every bit is also a bit against the assumed absence of uncontrolled confounding, selection effects, measurement error, and model form. A large S-value says the data are hard to reconcile with the conjunction of $\mu$ and everything else you assumed; it does not tell you which conjunct to abandon. In observational work, the “everything else” is usually the weaker link.

A worked example: one data set, four analyses

Brown and colleagues studied serotonergic antidepressant use in pregnancy and autism spectrum disorder in 35,906 Ontario births, of which 2,837 pregnancies (7.9%) were exposed⁴. They reported four hazard ratios from the same cohort under progressively stronger confounding control. This is the example reanalysed at length in³, and the working code for that paper lives here.

Look at the point estimates before you look at anything else.

brown = [
  {key: "crude",  label: "Crude",                 hr: 2.16, lower: 1.64,  upper: 2.86, ciLabel: "1.64 – 2.86",  control: "None"},
  {key: "adj",    label: "Covariate-adjusted",    hr: 1.59, lower: 1.17,  upper: 2.17, ciLabel: "1.17 – 2.17",  control: "Regression adjustment"},
  {key: "iptw",   label: "HdPS IPT-weighted",     hr: 1.61, lower: 0.997, upper: 2.59, ciLabel: "0.997 – 2.59", control: "High-dimensional propensity score"},
  {key: "sib",    label: "Sibling-matched",       hr: 1.60, lower: 0.69,  upper: 3.74, ciLabel: "0.69 – 3.74",  control: "Within-family (shared confounders)"}
]

brown = Array(4) [Object, Object, Object, Object]

brownStats = brown.map(d => {
  const s = seFromRatioCI(d.lower, d.upper);
  const t = testValue(d.hr, s, 1);
  return {...d, se: s, pNull: t.p, sNull: t.s, ratio: d.upper / d.lower};
})

brownStats = Array(4) [Object, Object, Object, Object]

Inputs.table(
  brownStats.map(d => ({
    Analysis: d.label,
    "Confounding control": d.control,
    HR: d.hr.toFixed(2),
    "95% CI": d.ciLabel,
    "Interval ratio": d.ratio.toFixed(2),
    "P (vs HR = 1)": d.pNull < 0.001 ? d.pNull.toExponential(1) : d.pNull.toFixed(4),
    "S (bits)": d.sNull.toFixed(2)
  })),
  {rows: 6, layout: "auto"}
)

Analysis	Confounding control	HR	95% CI	Interval ratio	P (vs HR = 1)	S (bits)
Crude	None	2.16	1.64 – 2.86	1.74	5.7e-8	24.07
Covariate-adjusted	Regression adjustment	1.59	1.17 – 2.17	1.85	0.0033	8.26
HdPS IPT-weighted	High-dimensional propensity score	1.61	0.997 – 2.59	2.60	0.0505	4.31
Sibling-matched	Within-family (shared confounders)	1.60	0.69 – 3.74	5.42	0.2757	1.86

Three of the four point estimates are 1.59, 1.60 and 1.61. By the conventional reading, one of them is a finding and two of them are nothing. What actually changed between them is not the estimate — it is the width of the interval, which grows from a ratio of 1.9 to 5.4 across the three adjusted analyses as each trades precision for stronger confounding control. The verdicts flip; the estimate does not move.

The published conclusion was that exposure “was not associated with” autism. That sentence is doing work the arithmetic will not support: the IPT-weighted analysis puts $P = 0.0505$ against the null, 4.31 bits of surprise, and its interval runs to 2.59.

viewof shown = Inputs.checkbox(brown, {
  value: brown,
  format: d => d.label,
  label: "Analyses to overlay"
})

shown = Array(4) [Object, Object, Object, Object]

brownCurves = shown.flatMap(d => {
  const s = seFromRatioCI(d.lower, d.upper);
  return consonanceCurve(d.hr, s, {from: 0.5, to: 5, n: 500})
    .map(pt => ({...pt, analysis: d.label}));
})

brownCurves = Array(2004) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

Plot.plot({
  height: 320,
  marginLeft: 55,
  marginBottom: 40,
  style: {fontSize: "13px"},
  color: {legend: true},
  x: {type: "log", label: "Hazard ratio →", domain: [0.5, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
  y: {label: "↑ P-value", domain: [0, 1], grid: true},
  marks: [
    Plot.ruleY([0.05], {strokeDasharray: "3 3", strokeOpacity: 0.4}),
    Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
    Plot.lineY(brownCurves, {x: "mu", y: "p", z: "analysis", stroke: "analysis", strokeWidth: 2}),
    // P = 1 exactly at the point estimate, so each dot sits at the apex of its curve.
    Plot.dot(shown, {x: "hr", y: () => 1, fill: "label", r: 4}),
    Plot.tip(brownCurves, Plot.pointer({
      x: "mu", y: "p", z: "analysis",
      title: d => [
        d.analysis,
        `Hazard ratio  ${d.mu.toFixed(2)}`,
        `P-value       ${d.p < 0.001 ? d.p.toExponential(2) : d.p.toFixed(3)}`,
        `S-value       ${d.s.toFixed(1)} bits`
      ].join("\n")
    }))
  ]
})

Covariate-adjustedCrudeHdPS IPT-weightedSibling-matched

Uncheck the crude analysis and the remaining three curves are near-concentric: same peak, different width. That is the shape of a precision problem, not a disagreement about effect size. Now test a value other than the null:

viewof muBrown = Inputs.range([0.5, 4], {
  value: 1.5, step: 0.01, label: "Test every analysis against this hazard ratio"
})

muBrown = 1.5

Inputs.table(
  brownStats.map(d => {
    const t = testValue(d.hr, d.se, muBrown);
    return {
      Analysis: d.label,
      HR: d.hr.toFixed(2),
      [`P (vs ${muBrown.toFixed(2)})`]: t.p < 0.001 ? t.p.toExponential(1) : t.p.toFixed(3),
      "S (bits)": t.s.toFixed(2),
      Reading: t.s < 1 ? "Highly compatible" : t.s < 4.32 ? "Compatible" : "Poorly compatible"
    };
  }),
  {rows: 6, layout: "auto"}
)

Analysis	HR	P (vs 1.50)	S (bits)	Reading
Crude	2.16	0.010	6.62	Poorly compatible
Covariate-adjusted	1.59	0.712	0.49	Highly compatible
HdPS IPT-weighted	1.61	0.771	0.37	Highly compatible
Sibling-matched	1.60	0.881	0.18	Highly compatible

Set the slider to 1.5 and all four analyses agree: $S$ = 6.6, 0.5, 0.4 and 0.2 bits. Three of the four find a 50% increase in hazard less surprising than a single coin flip. The analysis that was “not significant” and the analysis that was “significant” are, against this hypothesis, saying nearly the same thing.

Why the null gets a curve and not a verdict

Testing HR = 1 is a legitimate question. Reporting only that test, and only its dichotomised outcome, discards the answer to every other question the same data address — including questions the reader may care about more, such as whether an effect large enough to change prescribing can be ruled out. On these data it cannot: the IPT-weighted 95% interval reaches 2.59, and even the 50% interval runs from 1.37 to 1.90.

What more data would do

Precision and parameter value are separate things that a single P-value fuses into one number. Hold the IPT-weighted estimate fixed and vary only the information: the standard error scales as $1/\sqrt{k}$ when the sample size is multiplied by $k$ .

viewof kInfo = Inputs.range([0.25, 16], {
  value: 1, step: 0.05, label: "Sample size multiplier (k)"
})

kInfo = 1

base = ({hr: 1.61, se: seFromRatioCI(0.997, 2.59)})

base = Object {hr: 1.61, se: 0.24354079775495885}

scaled = ({se: base.se / Math.sqrt(kInfo)})

scaled = Object {se: 0.24354079775495885}

scaledCurve = consonanceCurve(base.hr, scaled.se, {from: 0.5, to: 5, n: 500})

scaledCurve = Array(501) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

baseCurve = consonanceCurve(base.hr, base.se, {from: 0.5, to: 5, n: 500})

baseCurve = Array(501) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

scaledAtNull = testValue(base.hr, scaled.se, 1)

scaledAtNull = Object {mu: 1, p: 0.050528853158511446, s: 4.306748753957083}

html`<div style="border-left:3px solid currentColor; padding:0.6em 1em; margin:1.2em 0; font-size:0.95em; line-height:1.7;">
  At <strong>${kInfo.toFixed(2)}×</strong> the original sample size, with the estimate held at
  <strong>1.61</strong>:
  95% interval
  <strong>${(base.hr * Math.exp(-zq(0.95) * scaled.se)).toFixed(2)}</strong> to
  <strong>${(base.hr * Math.exp(zq(0.95) * scaled.se)).toFixed(2)}</strong>,
  <em>P</em> = <strong>${scaledAtNull.p < 0.001 ? scaledAtNull.p.toExponential(2) : scaledAtNull.p.toFixed(4)}</strong>
  against the null,
  <em>S</em> = <strong>${scaledAtNull.s.toFixed(1)} bits</strong>.
</div>`

At 1.00× the original sample size, with the estimate held at 1.61: 95% interval 1.00 to 2.59, P = 0.0505 against the null, S = 4.3 bits.

Plot.plot({
  height: 280,
  marginLeft: 55,
  marginBottom: 40,
  style: {fontSize: "13px"},
  x: {type: "log", label: "Hazard ratio →", domain: [0.5, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
  y: {label: "↑ P-value", domain: [0, 1], grid: true},
  marks: [
    Plot.ruleY([0.05], {strokeDasharray: "3 3", strokeOpacity: 0.4}),
    Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
    Plot.lineY(baseCurve, {x: "mu", y: "p", strokeWidth: 1.5, strokeOpacity: 0.30, strokeDasharray: "2 3"}),
    Plot.areaY(scaledCurve, {x: "mu", y: "p", fillOpacity: 0.10}),
    Plot.lineY(scaledCurve, {x: "mu", y: "p", strokeWidth: 2}),
    Plot.dot([{x: base.hr, y: 1}], {x: "x", y: "y", r: 4.5})
  ]
})

The dashed ghost is the study as published. Push $k$ to 4 and the null falls well outside the 95% interval without the estimate having moved a hair. Pull $k$ down to 0.3 and the same estimate becomes compatible with almost everything between 0.7 and 3.9.

This is worth sitting with, because it is the mechanism behind most “failure to replicate” reporting. A study that reports $P = 0.04$ and a study that reports $P = 0.12$ can have identical point estimates and differ only in how many people they enrolled. Neither number is a property of the effect.

The one thing this widget cannot show

Scaling the standard error models random error only. Multiplying the sample size by 16 does nothing to confounding, selection, or measurement error — those biases do not shrink with $\sqrt{n}$ , and past some sample size they dominate entirely. A very narrow curve centred in the wrong place is the most confident way to be wrong. If you want that quantified rather than assumed away, the tool is bias analysis, not more subjects.

Do two estimates conflict?

“Study A was significant, study B was not, so they disagree” is a comparison of two verdicts, not of two estimates. The comparison that answers the question is a curve in its own right: under independence, the log ratio-of-ratios has standard error $\sqrt{se_A^2 + se_B^2}$ , and the null of no difference sits at a ratio of 1.

Study A is the explorer at the top of this page. Set study B here.

viewof estB = Inputs.range([0.5, 4], {value: 1.05, step: 0.01, label: "Study B: point estimate"})

estB = 1.05

viewof cirB = Inputs.range([1.05, 8], {value: 1.60, step: 0.01, label: "Study B: interval ratio"})

cirB = 1.6

limitsB = ({lower: estB / Math.sqrt(cirB), upper: estB * Math.sqrt(cirB)})

limitsB = Object {lower: 0.8300978857941996, upper: 1.3281566172707193}

seB = seFromRatioCI(limitsB.lower, limitsB.upper, 0.95)

seB = 0.11990108822230006

diff = ({ratio: est / estB, se: Math.sqrt(se * se + seB * seB)})

diff = Object {ratio: 1.5333333333333334, se: 0.2716504010707466}

// The ratio of ratios can leave a fixed window entirely (A/B ranges from 0.125
// to 8 across the slider limits), so the domain follows the estimate instead of
// silently cropping the peak off the plot.
diffDomain = ({
  from: Math.min(0.25, diff.ratio / 3),
  to: Math.max(4, diff.ratio * 3)
})

diffDomain = Object {from: 0.25, to: 4.6000000000000005}

diffCurve = consonanceCurve(diff.ratio, diff.se, {from: diffDomain.from, to: diffDomain.to, n: 500})

diffCurve = Array(501) [Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, Object, …]

diffAtNull = testValue(diff.ratio, diff.se, 1)

diffAtNull = Object {mu: 1, p: 0.11560129228171975, s: 3.1127705693843435}

html`<div style="border-left:3px solid currentColor; padding:0.6em 1em; margin:1.2em 0; font-size:0.95em; line-height:1.7;">
  Study A: <strong>${est.toFixed(2)}</strong> (${limits.lower.toFixed(2)}–${limits.upper.toFixed(2)}),
  <em>P</em> vs 1 = <strong>${testValue(est, se, 1).p.toFixed(3)}</strong><br>
  Study B: <strong>${estB.toFixed(2)}</strong> (${limitsB.lower.toFixed(2)}–${limitsB.upper.toFixed(2)}),
  <em>P</em> vs 1 = <strong>${testValue(estB, seB, 1).p.toFixed(3)}</strong><br>
  <span style="opacity:0.9">Ratio of ratios A ÷ B = <strong>${diff.ratio.toFixed(2)}</strong>
  (95% CI ${(diff.ratio * Math.exp(-zq(0.95) * diff.se)).toFixed(2)}–${(diff.ratio * Math.exp(zq(0.95) * diff.se)).toFixed(2)}),
  <em>P</em> for no difference = <strong>${diffAtNull.p.toFixed(3)}</strong>,
  <em>S</em> = <strong>${diffAtNull.s.toFixed(1)} bits</strong></span>
</div>`

Study A: 1.61 (1.00–2.60), P vs 1 = 0.051
Study B: 1.05 (0.83–1.33), P vs 1 = 0.684
Ratio of ratios A ÷ B = 1.53 (95% CI 0.90–2.61), P for no difference = 0.116, S = 3.1 bits

Plot.plot({
  height: 250,
  marginLeft: 55,
  marginBottom: 40,
  style: {fontSize: "13px"},
  x: {type: "log", label: "Ratio of ratios (A ÷ B) →", domain: [diffDomain.from, diffDomain.to], tickFormat: d => d},
  y: {label: "↑ P-value for the difference", domain: [0, 1], grid: true},
  marks: [
    Plot.areaY(diffCurve, {x: "mu", y: "p", fillOpacity: 0.10}),
    Plot.lineY(diffCurve, {x: "mu", y: "p", strokeWidth: 2}),
    Plot.ruleY([0.05], {strokeDasharray: "3 3", strokeOpacity: 0.4}),
    Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
    Plot.dot([diffAtNull], {x: "mu", y: "p", r: 4.5}),
    Plot.tip(diffCurve, Plot.pointerX({
      x: "mu", y: "p",
      title: d => `Ratio of ratios  ${d.mu.toFixed(2)}\nP-value          ${d.p.toFixed(3)}\nS-value          ${d.s.toFixed(1)} bits`
    }))
  ]
})

With the defaults — A at 1.61 ( $P = 0.051$ ) and B at 1.05 ( $P = 0.684$ ) — the two studies land on opposite sides of the conventional threshold. Their ratio of ratios is 1.53, 95% interval 0.90 to 2.61, and the difference carries 3.1 bits of information: fewer than the 4.3 bits the 0.05 convention itself demands, and fewer than four coin flips. Two verdicts that disagree, one comparison that does not.

This does not apply to the four Brown analyses

The overlay in the previous section shows four analyses of the same subjects. Their estimates are strongly correlated, so $\sqrt{se_A^2 + se_B^2}$ badly overstates the standard error of their difference and the resulting P-value is not interpretable. Use this widget for genuinely independent studies. For nested analyses of one data set, the honest summary is the overlay itself: same peak, different width.

Why two panels instead of one with two axes

A P-value and an S-value are the same quantity on different scales, so a dual axis is technically defensible. It is still a bad idea: readers reliably misread which curve belongs to which axis, and the visual slope of the P-value curve near the tails is uninformative precisely where the S-value is most informative. Two stacked panels sharing an x-axis cost one extra inch of page and remove the ambiguity.

Things this figure deliberately does not do

No shading of “significant” regions. The point of the curve is that there is no cliff at 0.05.
No default null at 1. The μ slider starts at 1 because that is the conventional test value, not because it is privileged.
No P-value below about 1e-300. Double precision runs out. The implementation stays accurate to roughly z = 37, which is far past the point where the normal approximation itself is the binding constraint.
No normality diagnostics. Every curve here assumes the log estimate is approximately normal with the reported standard error. That is the same assumption the published interval already made; the curve does not add it, it only makes it visible across the whole range.

Interactive table

The same idea applied to tabular results — searchable and sortable by clicking column headers, which is most of what makes a long results table usable. These are the Brown analyses again, with each row carrying its own P-value against a hypothesis you choose.

viewof muTable = Inputs.range([0.5, 4], {value: 1, step: 0.01, label: "Hypothesis tested in the table (μ)"})

muTable = 1

tableRows = brownStats.map(d => {
  const t = testValue(d.hr, d.se, muTable);
  return {
    Analysis: d.label,
    "Hazard ratio": d.hr,
    "95% CI": d.ciLabel,
    "SE (log scale)": +d.se.toFixed(3),
    "P": t.p < 0.001 ? t.p.toExponential(1) : +t.p.toFixed(4),
    "S (bits)": +t.s.toFixed(2)
  };
})

tableRows = Array(4) [Object, Object, Object, Object]

viewof tableSearch = Inputs.search(tableRows, {placeholder: "Filter analyses…"})

tableSearch = Array(4) [Object, Object, Object, Object]

Inputs.table(tableSearch, {
  sort: "S (bits)",
  reverse: true,
  rows: 12
})

Analysis	Hazard ratio	95% CI	SE (log scale)	P	▾S (bits)
Crude	2.16	1.64 – 2.86	0.142	5.7e-8	24.07
Covariate-adjusted	1.59	1.17 – 2.17	0.158	0.003	8.26
HdPS IPT-weighted	1.61	0.997 – 2.59	0.244	0.051	4.31
Sibling-matched	1.6	0.69 – 3.74	0.431	0.276	1.86

How to report this

Nothing above requires a new statistic, a new threshold, or a new software stack. It requires four sentences you can paste into a results section:

Report the estimate and interval without the word “significant” — “HR 1.61, 95% compatibility interval 0.997 to 2.59”.
Report the P-value as a number, not as an inequality against 0.05, and say what hypothesis it tests.
Report at least one P-value for a non-null hypothesis that matters clinically or practically. If the data cannot distinguish the null from a 60% increase, that is the finding.
Say that the interval and P-value are conditional on the entire model, not just on the parameter.

If a reviewer asks for the significance verdict back, the curve is the reply: it contains their verdict as one horizontal slice, plus every slice they did not ask for.

Reader infrastructure that is already on

Three things people associate with Gwern’s pages are Quarto defaults and need no code at all:

Citation popups. Hover any @citekey reference.
Footnote popups. Hover the marker on ² above.
Cross-reference popups. Hover any @fig- or @tbl- reference.

² The convention that 0.05 marks a boundary is worth pricing in bits: it is 4.32 bits of information against the tested hypothesis, or roughly the surprise of four consecutive heads. Few people would abandon a belief on four coin flips.

All three are controlled by citations-hover, footnotes-hover and crossrefs-hover, which default to true. The margin placement of footnotes here comes from reference-location: margin in this document’s front matter.

References

1. Poole C. (1987). “Beyond the confidence interval.” American Journal of Public Health. 77:195–199. doi: 10.2105/ajph.77.2.195.

Abstract

Beyond the confidence interval. C PooleCopyRight https://doi.org/10.2105/AJPH.77.2.195 Published Online: October 07, 2011

2. Sullivan KM, Foster DA. (1990). “Use of the confidence interval function.” Epidemiology. 1:39–42. doi: 10.1097/00001648-199001000-00009.

Abstract

Graphics displaying all confidence intervals around a point estimate have been referred to as P-value functions and consonance intervals. We recommend use of the term confidence interval function (CI function) rather than P-value function. The CI function is useful because it simultaneously depicts point estimation, variability, and the relation of these two factors to the null value. The usefulness of the CI function in demonstrating the concepts of effect modification and confounding, in meta-analysis, and in the comparison of various confidence interval procedures is evaluated. Software packages that produce CI functions are described.

3. Rafi Z, Greenland S. (2020). “Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise.” BMC Medical Research Methodology. 20:244. doi: 10.1186/s12874-020-01105-9.

Abstract

BACKGROUND: Researchers often misinterpret and misrepresent statistical outputs. This abuse has led to a large literature on modification or replacement of testing thresholds and P-values with confidence intervals, Bayes factors, and other devices. Because the core problems appear cognitive rather than statistical, we review some simple methods to aid researchers in interpreting statistical outputs. These methods emphasize logical and information concepts over probability, and thus may be more robust to common misinterpretations than are traditional descriptions. METHODS: (p), to provide a measure of the information supplied by the testing procedure, and to help calibrate intuitions against simple physical experiments like coin tossing. We also use tables or graphs of test statistics for alternative hypotheses, and interval estimates for different percentile levels, to thwart fallacies arising from arbitrary dichotomies. Finally, we reinterpret P-values and interval estimates in unconditional terms, which describe compatibility of data with the entire set of analysis assumptions. We illustrate these methods with a reanalysis of data from an existing record-based cohort study …

4. Brown HK, Ray JG, Wilton AS, Lunsky Y, Gomes T, Vigod SN. (2017). “Association between serotonergic antidepressant use during pregnancy and autism spectrum disorder in children.” JAMA. 317:1544–1552. doi: 10.1001/jama.2017.3415.

Abstract

IMPORTANCE: Previous observations of a higher risk of child autism spectrum disorder with serotonergic antidepressant exposure during pregnancy may have been confounded. OBJECTIVE: To evaluate the association between serotonergic antidepressant exposure during pregnancy and child autism spectrum disorder. DESIGN, SETTING, AND PARTICIPANTS: Retrospective cohort study. Health administrative data sets were used to study children born to mothers who were receiving public prescription drug coverage during pregnancy in Ontario, Canada, from 2002-2010, reflecting 4.2% of births. Children were followed up until March 31, 2014. EXPOSURES: Serotonergic antidepressant exposure was defined as 2 or more consecutive maternal prescriptions for a selective serotonin or serotonin-norepinephrine reuptake inhibitor between conception and delivery. MAIN OUTCOMES AND MEASURES: Child autism spectrum disorder identified after the age of 2 years …

The P-value Is Not a Verdict: An Interactive Consonance Curve

The explorer

Calibrating bits

A worked example: one data set, four analyses

What more data would do

Do two estimates conflict?

Interactive table

How to report this

Reader infrastructure that is already on

References

Linked from (5)

Comments

Webmentions