---
title: "The P-value Is Not a Verdict: An Interactive Consonance Curve"
description: "Drag an estimate and its interval and watch the entire P-value function, every compatibility interval, and the S-value move together."
author: "Zad Rafi"
date: today
categories: [statistics, interactive]
zotero: TRUE
bibliography: references.bib
csl: custom.csl
format:
html:
toc: true
code-tools: true
---
A single P-value reports one number about one hypothesis. The *P-value function* reports the same test statistic applied to **every** candidate parameter value at once — and it contains every compatibility interval at every level as horizontal slices.[^1] Nothing below is new mathematics. It is the same normal approximation you already use, drawn in full instead of evaluated at one point.
[^1]: Slice the curve at height $p = 0.05$ and read off the x-coordinates: those are the 95% limits. Slice at $p = 0.01$ and you have the 99% limits. The interval was never a separate calculation.
The display is old. Poole called for it in 1987 [@poole1987], Sullivan and Foster catalogued its uses in 1990 [@sullivan1990], and it has sat in *Modern Epidemiology* ever since. What has changed is that it now costs nothing to render, and that we have a better vocabulary for the vertical axis: **compatibility** rather than confidence, and **surprisal** rather than significance [@rafi2020].
::: callout-tip
## Definitions, once
A **P-value function** (equivalently, a consonance or compatibility curve) plots, for every candidate parameter value $\mu$, the P-value from testing $\mu$ against the data under the full set of analysis assumptions.
An **S-value** is the Shannon transform $s = -\log_2 p$: the number of consecutive heads from a fair coin that would be exactly as surprising as the observed test statistic, if $\mu$ and every other assumption were correct.
A **95% compatibility interval** is the set of $\mu$ whose P-value exceeds 0.05 — a horizontal slice through the curve, not a separate object.
:::
## The explorer
Move the sliders. Everything recomputes in the browser — no server, no round trip.
```{ojs}
//| echo: false
// Observable Plot's tip/pointer marks need >= 0.6.11; Quarto embeds an older
// build, so pin an explicit version. See README for how to vendor this locally
// if you'd rather not depend on a CDN at page load.
Plot = import("https://cdn.jsdelivr.net/npm/@observablehq/plot@0.6/+esm")
```
```{ojs}
//| echo: false
import { consonanceCurve, testValue, seFromRatioCI } from "./ojs/stats.js"
```
```{ojs}
//| echo: false
// Inverse standard normal CDF (Acklam's rational approximation; |error| < 4e-9
// over the levels used here, verified against scipy).
//
// TODO(stats.js): seFromRatioCI already needs this internally. Export it from
// ./ojs/stats.js and delete this cell rather than maintaining two copies.
zq = function (L) {
const p = (1 + L) / 2;
const a = [-3.969683028665376e+01, 2.209460984245205e+02, -2.759285104469687e+02,
1.383577518672690e+02, -3.066479806614716e+01, 2.506628277459239e+00];
const b = [-5.447609879822406e+01, 1.615858368580409e+02, -1.556989798598866e+02,
6.680131188771972e+01, -1.328068155288572e+01];
const c = [-7.784894002430293e-03, -3.223964580411365e-01, -2.400758277161838e+00,
-2.549732539343734e+00, 4.374664141464968e+00, 2.938163982698783e+00];
const d = [7.784695709041462e-03, 3.224671290700398e-01, 2.445134137142996e+00,
3.754408661907416e+00];
const lo = 0.02425, hi = 1 - lo;
let q, r;
if (p < lo) {
q = Math.sqrt(-2 * Math.log(p));
return (((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) /
((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1);
}
if (p <= hi) {
q = p - 0.5; r = q * q;
return (((((a[0]*r+a[1])*r+a[2])*r+a[3])*r+a[4])*r+a[5]) * q /
(((((b[0]*r+b[1])*r+b[2])*r+b[3])*r+b[4])*r+1);
}
q = Math.sqrt(-2 * Math.log(1 - p));
return -(((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) /
((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1);
}
```
```{ojs}
//| echo: false
viewof est = Inputs.range([0.5, 4], {
value: 1.61, step: 0.01, label: "Point estimate (rate ratio)"
})
```
```{ojs}
//| echo: false
viewof cir = Inputs.range([1.05, 8], {
value: 2.60, step: 0.01, label: "Interval ratio (upper ÷ lower)"
})
```
```{ojs}
//| echo: false
viewof mu = Inputs.range([0.5, 4], {
value: 1, step: 0.01, label: "Value being tested (μ)"
})
```
```{ojs}
//| echo: false
viewof level = Inputs.range([0.50, 0.995], {
value: 0.95, step: 0.005, label: "Compatibility level to slice at"
})
```
```{ojs}
//| echo: false
// Parameterising by the interval *ratio* rather than by two independent limits
// guarantees the interval always brackets the estimate — one less way for a
// reader to produce a nonsensical figure.
limits = ({
lower: est / Math.sqrt(cir),
upper: est * Math.sqrt(cir)
})
```
```{ojs}
//| echo: false
se = seFromRatioCI(limits.lower, limits.upper, 0.95)
```
```{ojs}
//| echo: false
// The slice at an arbitrary level, derived from the same SE. This is the whole
// claim of footnote 1 made draggable: the interval is a horizontal cut, not a
// second computation.
slice = ({
p: 1 - level,
lower: est * Math.exp(-zq(level) * se),
upper: est * Math.exp(zq(level) * se)
})
```
```{ojs}
//| echo: false
curve = consonanceCurve(est, se, {from: 0.4, to: 5, n: 600})
```
```{ojs}
//| echo: false
atMu = testValue(est, se, mu)
```
```{ojs}
//| echo: false
html`<div style="
border-left: 3px solid currentColor;
padding: 0.6em 1em;
margin: 1.2em 0;
font-size: 0.95em;
line-height: 1.7;
">
<strong>${est.toFixed(2)}</strong>
(95% compatibility interval
<strong>${limits.lower.toFixed(2)}</strong> to
<strong>${limits.upper.toFixed(2)}</strong>)
<br>
Testing μ = <strong>${mu.toFixed(2)}</strong>:
<em>P</em> = <strong>${atMu.p < 0.001 ? atMu.p.toExponential(2) : atMu.p.toFixed(3)}</strong>,
<em>S</em> = <strong>${atMu.s.toFixed(1)} bits</strong>
<br>
<span style="opacity:0.75">
The data are no more surprising against μ = ${mu.toFixed(2)} than
${atMu.s.toFixed(1)} consecutive heads from a fair coin.
</span>
<br>
<span style="opacity:0.75">
${(100 * level).toFixed(1)}% slice (at <em>P</em> = ${slice.p.toFixed(3)}):
<strong>${slice.lower.toFixed(2)}</strong> to
<strong>${slice.upper.toFixed(2)}</strong>
</span>
</div>`
```
```{ojs}
//| echo: false
Plot.plot({
height: 280,
marginLeft: 55,
marginBottom: 40,
style: {fontSize: "13px"},
x: {type: "log", label: "Rate ratio →", domain: [0.4, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
y: {label: "↑ P-value", domain: [0, 1], grid: true, percent: false},
marks: [
Plot.areaY(curve, {x: "mu", y: "p", fillOpacity: 0.10}),
Plot.lineY(curve, {x: "mu", y: "p", strokeWidth: 2}),
// The draggable slice, drawn as a segment between the two limits rather
// than a full-width rule: the horizontal extent *is* the interval.
Plot.ruleY([slice.p], {strokeDasharray: "3 3", strokeOpacity: 0.35}),
Plot.link([slice], {x1: "lower", x2: "upper", y1: "p", y2: "p", strokeWidth: 3, strokeOpacity: 0.85}),
Plot.dot([{x: slice.lower, y: slice.p}, {x: slice.upper, y: slice.p}], {x: "x", y: "y", r: 3}),
Plot.text([{x: 0.42, y: slice.p}], {x: "x", y: "y", text: [`${(100 * level).toFixed(0)}% slice`], dy: -8, textAnchor: "start", fontSize: 11, fillOpacity: 0.6}),
Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
Plot.ruleX([mu], {strokeWidth: 1.5, strokeOpacity: 0.9}),
Plot.dot([atMu], {x: "mu", y: "p", r: 4.5}),
Plot.tip(curve, Plot.pointerX({
x: "mu", y: "p",
title: d => [
`Rate ratio ${d.mu.toFixed(2)}`,
`P-value ${d.p < 0.001 ? d.p.toExponential(2) : d.p.toFixed(3)}`,
`S-value ${d.s.toFixed(1)} bits`,
`Sits on the ${(100 * d.level).toFixed(0)}% interval limit`
].join("\n")
}))
]
})
```
```{ojs}
//| echo: false
Plot.plot({
height: 220,
marginLeft: 55,
marginBottom: 40,
style: {fontSize: "13px"},
x: {type: "log", label: "Rate ratio →", domain: [0.4, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
y: {label: "↑ S-value (bits of information against)", domain: [0, 20], grid: true, clamp: true},
marks: [
Plot.lineY(curve, {x: "mu", y: "s", strokeWidth: 2}),
Plot.ruleY([4.32], {strokeDasharray: "3 3", strokeOpacity: 0.5}),
Plot.text([{x: 0.42, y: 4.32}], {x: "x", y: "y", text: ["S = 4.3 bits (P = 0.05)"], dy: -8, textAnchor: "start", fontSize: 11, fillOpacity: 0.6}),
Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
Plot.ruleX([mu], {strokeWidth: 1.5, strokeOpacity: 0.9}),
Plot.dot([atMu], {x: "mu", y: "s", r: 4.5}),
Plot.tip(curve, Plot.pointerX({
x: "mu", y: "s",
title: d => `Rate ratio ${d.mu.toFixed(2)}\nS-value ${d.s.toFixed(1)} bits`
}))
]
})
```
### Calibrating bits
The S-value exists because "P = 0.05" carries no intuition and "significant" carries the wrong one. Bits do carry intuition, because everyone has a physical model of coin flipping. Drag μ above and watch the coins.
```{ojs}
//| echo: false
// A physical calibration strip. Whole bits are drawn as coins; the fractional
// remainder is drawn as a partial coin, because rounding 4.31 up to 5 would
// quietly overstate the evidence by a factor of ~1.6.
coins = {
const whole = Math.floor(Math.min(atMu.s, 24));
const frac = Math.min(atMu.s, 24) - whole;
const cells = [];
for (let i = 0; i < whole; i++) cells.push(1);
if (frac > 0.02) cells.push(frac);
return html`<div style="margin: 0.8em 0 1.4em;">
<div style="display:flex; flex-wrap:wrap; gap:5px; align-items:center; min-height:22px;">
${cells.map(f => html`<span style="
display:inline-block; width:18px; height:18px; border-radius:50%;
border:1.5px solid currentColor;
background: linear-gradient(to right, currentColor ${(100*f).toFixed(0)}%, transparent ${(100*f).toFixed(0)}%);
opacity:0.8;"></span>`)}
${atMu.s > 24 ? html`<span style="opacity:0.7; font-size:0.9em;">… and ${(atMu.s - 24).toFixed(0)} more</span>` : ``}
</div>
<div style="opacity:0.7; font-size:0.9em; margin-top:0.5em;">
${atMu.s.toFixed(2)} bits against μ = ${mu.toFixed(2)}.
${atMu.s < 1 ? "Less surprising than a single coin flip."
: atMu.s < 4.32 ? "Less surprising than the conventional 0.05 threshold."
: "More surprising than the conventional 0.05 threshold."}
</div>
</div>`;
}
```
::: {.callout-warning collapse="true"}
## What the coins do not license
The S-value measures surprise *at the whole model*, not at $\mu$ alone. Every bit is also a bit against the assumed absence of uncontrolled confounding, selection effects, measurement error, and model form. A large S-value says the data are hard to reconcile with the conjunction of $\mu$ and everything else you assumed; it does not tell you which conjunct to abandon. In observational work, the "everything else" is usually the weaker link.
:::
## A worked example: one data set, four analyses
Brown and colleagues studied serotonergic antidepressant use in pregnancy and autism spectrum disorder in 35,906 Ontario births, of which 2,837 pregnancies (7.9%) were exposed [@brown2017]. They reported four hazard ratios from the same cohort under progressively stronger confounding control. This is the example reanalysed at length in @rafi2020, and the working code for that paper lives [here].
[here]: /statistics/rg2020bmc
Look at the point estimates before you look at anything else.
```{ojs}
//| echo: false
brown = [
{key: "crude", label: "Crude", hr: 2.16, lower: 1.64, upper: 2.86, ciLabel: "1.64 – 2.86", control: "None"},
{key: "adj", label: "Covariate-adjusted", hr: 1.59, lower: 1.17, upper: 2.17, ciLabel: "1.17 – 2.17", control: "Regression adjustment"},
{key: "iptw", label: "HdPS IPT-weighted", hr: 1.61, lower: 0.997, upper: 2.59, ciLabel: "0.997 – 2.59", control: "High-dimensional propensity score"},
{key: "sib", label: "Sibling-matched", hr: 1.60, lower: 0.69, upper: 3.74, ciLabel: "0.69 – 3.74", control: "Within-family (shared confounders)"}
]
```
```{ojs}
//| echo: false
brownStats = brown.map(d => {
const s = seFromRatioCI(d.lower, d.upper);
const t = testValue(d.hr, s, 1);
return {...d, se: s, pNull: t.p, sNull: t.s, ratio: d.upper / d.lower};
})
```
```{ojs}
//| echo: false
Inputs.table(
brownStats.map(d => ({
Analysis: d.label,
"Confounding control": d.control,
HR: d.hr.toFixed(2),
"95% CI": d.ciLabel,
"Interval ratio": d.ratio.toFixed(2),
"P (vs HR = 1)": d.pNull < 0.001 ? d.pNull.toExponential(1) : d.pNull.toFixed(4),
"S (bits)": d.sNull.toFixed(2)
})),
{rows: 6, layout: "auto"}
)
```
Three of the four point estimates are 1.59, 1.60 and 1.61. By the conventional reading, one of them is a finding and two of them are nothing. What actually changed between them is not the estimate — it is the width of the interval, which grows from a ratio of 1.9 to 5.4 across the three adjusted analyses as each trades precision for stronger confounding control. The verdicts flip; the estimate does not move.
The published conclusion was that exposure "was not associated with" autism. That sentence is doing work the arithmetic will not support: the IPT-weighted analysis puts $P = 0.0505$ against the null, 4.31 bits of surprise, and its interval runs to 2.59.
```{ojs}
//| echo: false
viewof shown = Inputs.checkbox(brown, {
value: brown,
format: d => d.label,
label: "Analyses to overlay"
})
```
```{ojs}
//| echo: false
brownCurves = shown.flatMap(d => {
const s = seFromRatioCI(d.lower, d.upper);
return consonanceCurve(d.hr, s, {from: 0.5, to: 5, n: 500})
.map(pt => ({...pt, analysis: d.label}));
})
```
```{ojs}
//| echo: false
Plot.plot({
height: 320,
marginLeft: 55,
marginBottom: 40,
style: {fontSize: "13px"},
color: {legend: true},
x: {type: "log", label: "Hazard ratio →", domain: [0.5, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
y: {label: "↑ P-value", domain: [0, 1], grid: true},
marks: [
Plot.ruleY([0.05], {strokeDasharray: "3 3", strokeOpacity: 0.4}),
Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
Plot.lineY(brownCurves, {x: "mu", y: "p", z: "analysis", stroke: "analysis", strokeWidth: 2}),
// P = 1 exactly at the point estimate, so each dot sits at the apex of its curve.
Plot.dot(shown, {x: "hr", y: () => 1, fill: "label", r: 4}),
Plot.tip(brownCurves, Plot.pointer({
x: "mu", y: "p", z: "analysis",
title: d => [
d.analysis,
`Hazard ratio ${d.mu.toFixed(2)}`,
`P-value ${d.p < 0.001 ? d.p.toExponential(2) : d.p.toFixed(3)}`,
`S-value ${d.s.toFixed(1)} bits`
].join("\n")
}))
]
})
```
Uncheck the crude analysis and the remaining three curves are near-concentric: same peak, different width. That is the shape of a precision problem, not a disagreement about effect size. Now test a value other than the null:
```{ojs}
//| echo: false
viewof muBrown = Inputs.range([0.5, 4], {
value: 1.5, step: 0.01, label: "Test every analysis against this hazard ratio"
})
```
```{ojs}
//| echo: false
Inputs.table(
brownStats.map(d => {
const t = testValue(d.hr, d.se, muBrown);
return {
Analysis: d.label,
HR: d.hr.toFixed(2),
[`P (vs ${muBrown.toFixed(2)})`]: t.p < 0.001 ? t.p.toExponential(1) : t.p.toFixed(3),
"S (bits)": t.s.toFixed(2),
Reading: t.s < 1 ? "Highly compatible" : t.s < 4.32 ? "Compatible" : "Poorly compatible"
};
}),
{rows: 6, layout: "auto"}
)
```
Set the slider to 1.5 and all four analyses agree: $S$ = 6.6, 0.5, 0.4 and 0.2 bits. Three of the four find a 50% increase in hazard *less* surprising than a single coin flip. The analysis that was "not significant" and the analysis that was "significant" are, against this hypothesis, saying nearly the same thing.
::: {.callout-note collapse="true"}
## Why the null gets a curve and not a verdict
Testing HR = 1 is a legitimate question. Reporting only that test, and only its dichotomised outcome, discards the answer to every other question the same data address — including questions the reader may care about more, such as whether an effect large enough to change prescribing can be ruled out. On these data it cannot: the IPT-weighted 95% interval reaches 2.59, and even the 50% interval runs from 1.37 to 1.90.
:::
## What more data would do
Precision and parameter value are separate things that a single P-value fuses into one number. Hold the IPT-weighted estimate fixed and vary only the information: the standard error scales as $1/\sqrt{k}$ when the sample size is multiplied by $k$.
```{ojs}
//| echo: false
viewof kInfo = Inputs.range([0.25, 16], {
value: 1, step: 0.05, label: "Sample size multiplier (k)"
})
```
```{ojs}
//| echo: false
base = ({hr: 1.61, se: seFromRatioCI(0.997, 2.59)})
```
```{ojs}
//| echo: false
scaled = ({se: base.se / Math.sqrt(kInfo)})
```
```{ojs}
//| echo: false
scaledCurve = consonanceCurve(base.hr, scaled.se, {from: 0.5, to: 5, n: 500})
```
```{ojs}
//| echo: false
baseCurve = consonanceCurve(base.hr, base.se, {from: 0.5, to: 5, n: 500})
```
```{ojs}
//| echo: false
scaledAtNull = testValue(base.hr, scaled.se, 1)
```
```{ojs}
//| echo: false
html`<div style="border-left:3px solid currentColor; padding:0.6em 1em; margin:1.2em 0; font-size:0.95em; line-height:1.7;">
At <strong>${kInfo.toFixed(2)}×</strong> the original sample size, with the estimate held at
<strong>1.61</strong>:
95% interval
<strong>${(base.hr * Math.exp(-zq(0.95) * scaled.se)).toFixed(2)}</strong> to
<strong>${(base.hr * Math.exp(zq(0.95) * scaled.se)).toFixed(2)}</strong>,
<em>P</em> = <strong>${scaledAtNull.p < 0.001 ? scaledAtNull.p.toExponential(2) : scaledAtNull.p.toFixed(4)}</strong>
against the null,
<em>S</em> = <strong>${scaledAtNull.s.toFixed(1)} bits</strong>.
</div>`
```
```{ojs}
//| echo: false
Plot.plot({
height: 280,
marginLeft: 55,
marginBottom: 40,
style: {fontSize: "13px"},
x: {type: "log", label: "Hazard ratio →", domain: [0.5, 5], ticks: [0.5, 0.75, 1, 1.5, 2, 3, 4, 5], tickFormat: d => d},
y: {label: "↑ P-value", domain: [0, 1], grid: true},
marks: [
Plot.ruleY([0.05], {strokeDasharray: "3 3", strokeOpacity: 0.4}),
Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
Plot.lineY(baseCurve, {x: "mu", y: "p", strokeWidth: 1.5, strokeOpacity: 0.30, strokeDasharray: "2 3"}),
Plot.areaY(scaledCurve, {x: "mu", y: "p", fillOpacity: 0.10}),
Plot.lineY(scaledCurve, {x: "mu", y: "p", strokeWidth: 2}),
Plot.dot([{x: base.hr, y: 1}], {x: "x", y: "y", r: 4.5})
]
})
```
The dashed ghost is the study as published. Push $k$ to 4 and the null falls well outside the 95% interval without the estimate having moved a hair. Pull $k$ down to 0.3 and the same estimate becomes compatible with almost everything between 0.7 and 3.9.
This is worth sitting with, because it is the mechanism behind most "failure to replicate" reporting. A study that reports $P = 0.04$ and a study that reports $P = 0.12$ can have identical point estimates and differ only in how many people they enrolled. Neither number is a property of the effect.
::: callout-important
## The one thing this widget cannot show
Scaling the standard error models *random* error only. Multiplying the sample size by 16 does nothing to confounding, selection, or measurement error — those biases do not shrink with $\sqrt{n}$, and past some sample size they dominate entirely. A very narrow curve centred in the wrong place is the most confident way to be wrong. If you want that quantified rather than assumed away, the tool is bias analysis, not more subjects.
:::
## Do two estimates conflict?
"Study A was significant, study B was not, so they disagree" is a comparison of two verdicts, not of two estimates. The comparison that answers the question is a curve in its own right: under independence, the log ratio-of-ratios has standard error $\sqrt{se_A^2 + se_B^2}$, and the null of *no difference* sits at a ratio of 1.
Study A is the explorer at the top of this page. Set study B here.
```{ojs}
//| echo: false
viewof estB = Inputs.range([0.5, 4], {value: 1.05, step: 0.01, label: "Study B: point estimate"})
```
```{ojs}
//| echo: false
viewof cirB = Inputs.range([1.05, 8], {value: 1.60, step: 0.01, label: "Study B: interval ratio"})
```
```{ojs}
//| echo: false
limitsB = ({lower: estB / Math.sqrt(cirB), upper: estB * Math.sqrt(cirB)})
```
```{ojs}
//| echo: false
seB = seFromRatioCI(limitsB.lower, limitsB.upper, 0.95)
```
```{ojs}
//| echo: false
diff = ({ratio: est / estB, se: Math.sqrt(se * se + seB * seB)})
```
```{ojs}
//| echo: false
// The ratio of ratios can leave a fixed window entirely (A/B ranges from 0.125
// to 8 across the slider limits), so the domain follows the estimate instead of
// silently cropping the peak off the plot.
diffDomain = ({
from: Math.min(0.25, diff.ratio / 3),
to: Math.max(4, diff.ratio * 3)
})
```
```{ojs}
//| echo: false
diffCurve = consonanceCurve(diff.ratio, diff.se, {from: diffDomain.from, to: diffDomain.to, n: 500})
```
```{ojs}
//| echo: false
diffAtNull = testValue(diff.ratio, diff.se, 1)
```
```{ojs}
//| echo: false
html`<div style="border-left:3px solid currentColor; padding:0.6em 1em; margin:1.2em 0; font-size:0.95em; line-height:1.7;">
Study A: <strong>${est.toFixed(2)}</strong> (${limits.lower.toFixed(2)}–${limits.upper.toFixed(2)}),
<em>P</em> vs 1 = <strong>${testValue(est, se, 1).p.toFixed(3)}</strong><br>
Study B: <strong>${estB.toFixed(2)}</strong> (${limitsB.lower.toFixed(2)}–${limitsB.upper.toFixed(2)}),
<em>P</em> vs 1 = <strong>${testValue(estB, seB, 1).p.toFixed(3)}</strong><br>
<span style="opacity:0.9">Ratio of ratios A ÷ B = <strong>${diff.ratio.toFixed(2)}</strong>
(95% CI ${(diff.ratio * Math.exp(-zq(0.95) * diff.se)).toFixed(2)}–${(diff.ratio * Math.exp(zq(0.95) * diff.se)).toFixed(2)}),
<em>P</em> for no difference = <strong>${diffAtNull.p.toFixed(3)}</strong>,
<em>S</em> = <strong>${diffAtNull.s.toFixed(1)} bits</strong></span>
</div>`
```
```{ojs}
//| echo: false
Plot.plot({
height: 250,
marginLeft: 55,
marginBottom: 40,
style: {fontSize: "13px"},
x: {type: "log", label: "Ratio of ratios (A ÷ B) →", domain: [diffDomain.from, diffDomain.to], tickFormat: d => d},
y: {label: "↑ P-value for the difference", domain: [0, 1], grid: true},
marks: [
Plot.areaY(diffCurve, {x: "mu", y: "p", fillOpacity: 0.10}),
Plot.lineY(diffCurve, {x: "mu", y: "p", strokeWidth: 2}),
Plot.ruleY([0.05], {strokeDasharray: "3 3", strokeOpacity: 0.4}),
Plot.ruleX([1], {strokeDasharray: "4 3", strokeOpacity: 0.55}),
Plot.dot([diffAtNull], {x: "mu", y: "p", r: 4.5}),
Plot.tip(diffCurve, Plot.pointerX({
x: "mu", y: "p",
title: d => `Ratio of ratios ${d.mu.toFixed(2)}\nP-value ${d.p.toFixed(3)}\nS-value ${d.s.toFixed(1)} bits`
}))
]
})
```
With the defaults — A at 1.61 ($P = 0.051$) and B at 1.05 ($P = 0.684$) — the two studies land on opposite sides of the conventional threshold. Their ratio of ratios is 1.53, 95% interval 0.90 to 2.61, and the difference carries 3.1 bits of information: fewer than the 4.3 bits the 0.05 convention itself demands, and fewer than four coin flips. Two verdicts that disagree, one comparison that does not.
::: callout-warning
## This does not apply to the four Brown analyses
The overlay in the previous section shows four analyses of the *same* subjects. Their estimates are strongly correlated, so $\sqrt{se_A^2 + se_B^2}$ badly overstates the standard error of their difference and the resulting P-value is not interpretable. Use this widget for genuinely independent studies. For nested analyses of one data set, the honest summary is the overlay itself: same peak, different width.
:::
::: {.callout-note collapse="true"}
## Why two panels instead of one with two axes
A P-value and an S-value are the same quantity on different scales, so a dual axis is technically defensible. It is still a bad idea: readers reliably misread which curve belongs to which axis, and the visual slope of the P-value curve near the tails is uninformative precisely where the S-value is most informative. Two stacked panels sharing an x-axis cost one extra inch of page and remove the ambiguity.
:::
::: {.callout-note collapse="true"}
## Things this figure deliberately does not do
- **No shading of "significant" regions.** The point of the curve is that there is no cliff at 0.05.
- **No default null at 1.** The μ slider starts at 1 because that is the conventional test value, not because it is privileged.
- **No P-value below about 1e-300.** Double precision runs out. The implementation stays accurate to roughly z = 37, which is far past the point where the normal approximation itself is the binding constraint.
- **No normality diagnostics.** Every curve here assumes the log estimate is approximately normal with the reported standard error. That is the same assumption the published interval already made; the curve does not add it, it only makes it visible across the whole range.
:::
## Interactive table
The same idea applied to tabular results — searchable and sortable by clicking column headers, which is most of what makes a long results table usable. These are the Brown analyses again, with each row carrying its own P-value against a hypothesis you choose.
```{ojs}
//| echo: false
viewof muTable = Inputs.range([0.5, 4], {value: 1, step: 0.01, label: "Hypothesis tested in the table (μ)"})
```
```{ojs}
//| echo: false
tableRows = brownStats.map(d => {
const t = testValue(d.hr, d.se, muTable);
return {
Analysis: d.label,
"Hazard ratio": d.hr,
"95% CI": d.ciLabel,
"SE (log scale)": +d.se.toFixed(3),
"P": t.p < 0.001 ? t.p.toExponential(1) : +t.p.toFixed(4),
"S (bits)": +t.s.toFixed(2)
};
})
```
```{ojs}
//| echo: false
viewof tableSearch = Inputs.search(tableRows, {placeholder: "Filter analyses…"})
```
```{ojs}
//| echo: false
Inputs.table(tableSearch, {
sort: "S (bits)",
reverse: true,
rows: 12
})
```
## How to report this
Nothing above requires a new statistic, a new threshold, or a new software stack. It requires four sentences you can paste into a results section:
1. Report the estimate and interval **without** the word "significant" — "HR 1.61, 95% compatibility interval 0.997 to 2.59".
2. Report the P-value **as a number**, not as an inequality against 0.05, and say what hypothesis it tests.
3. Report at least one P-value for a **non-null** hypothesis that matters clinically or practically. If the data cannot distinguish the null from a 60% increase, that is the finding.
4. Say that the interval and P-value are conditional on the **entire** model, not just on the parameter.
If a reviewer asks for the significance verdict back, the curve is the reply: it contains their verdict as one horizontal slice, plus every slice they did not ask for.
## Reader infrastructure that is already on
Three things people associate with Gwern's pages are Quarto defaults and need no code at all:
- **Citation popups.** Hover any `@citekey` reference.
- **Footnote popups.** Hover the marker on [^2] above.
- **Cross-reference popups.** Hover any `@fig-` or `@tbl-` reference.
[^2]: The convention that 0.05 marks a boundary is worth pricing in bits: it is 4.32 bits of information against the tested hypothesis, or roughly the surprise of four consecutive heads. Few people would abandon a belief on four coin flips.
All three are controlled by `citations-hover`, `footnotes-hover` and `crossrefs-hover`, which default to `true`. The margin placement of footnotes here comes from `reference-location: margin` in this document's front matter.
## References
::: {#refs}
:::
Comments