How to Be a Good Collaborator to a Statistician and a Data Engineer
statistical collaboration, statistical consulting, data engineering, reproducibility, analytic variability, estimand, data quality
Most projects that fail seem to fail because of the collaboration, and much less often because of anything technical. They usually fail early, and for reasons that rarely get written up (a question that was never pinned down, a column whose meaning changed in 2023, a “quick pull” that later ended up as the number in the board deck). The statistician gets blamed for the estimate and the data engineer gets blamed for the pipeline. However, the decisive errors were usually committed by well-meaning people who thought they were being helpful by not bothering anyone.
The influential statistician and geneticist Ronald A. Fisher put it elegantly in his 1938 presidential address to the Indian Statistical Congress:
To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.1.
In this post, I’d like to go through the habits that help us avoid the post mortem. I’ve deliberately made it two-sided, since collaborators owe analysts some specific things and analysts owe collaborators some specific things back (and I don’t think “the data wasn’t clean” counts as a professional deliverable).
Why This Is Harder Than It Looks
Many of us have the intuition that a well-specified question has one obvious analysis. That intuition is wrong, and we can measure how wrong it is.
In the best-known demonstration, 29 teams comprising 61 analysts were given the same dataset and the same question, which was whether soccer referees give more red cards to dark-skin-toned players. The 29 analyses used 21 unique combinations of covariates. The estimated effects ranged from an odds ratio of 0.89 to 2.93 (median 1.31), and 20 teams (69%) reported a statistically significant positive effect while 9 (31%) did not. Neither the analysts’ prior beliefs, nor their expertise, nor peer ratings of analysis quality explained the spread2.
The same pattern shows up in neuroimaging. Seventy teams analysed one fMRI dataset against nine pre-specified hypotheses, and no two teams chose identical workflows. The resulting variation in hypothesis-test outcomes was substantial, even between teams whose intermediate statistical maps were highly correlated3.
Does this mean that analysis is arbitrary? I don’t think so. What it suggests is that the analysis is underdetermined by the question, and that the gap gets filled by defensible-looking judgment calls (the same garden of forking paths that makes untested model assumptions so consequential). Every one of those calls is a place where the collaborator’s context (what the variable actually measures, which exclusions are clinically or commercially sensible, what decision the number feeds) either arrives in time or doesn’t.
The variation in results across competent analysts is large enough that a collaborator’s contribution (supplying the context that narrows the set of defensible choices) should be seen as part of the estimate, and I wouldn’t call it a courtesy.
What You Owe the Statistician
In Research Settings
Bring them in at the design stage. The change that probably matters most is about timing. Kass and colleagues put “statistical analysis is more than a set of computations” and “plan ahead, really ahead” near the top of their ten rules for effective statistical practice, because analysis choices are constrained by design choices that have already been locked in by the time most people call4. A statistician who is consulted at design can still change the sample size. However, one who is consulted at submission can do little more than change how the results are worded.
State the estimand before the method. Asking “can you run a regression” skips the question that matters, which is what quantity we want, in what population, under what contrast, and with what handling of intercurrent events. If we cannot say it in a sentence, then that is a finding in itself, and it is worth knowing before any data are collected.
Disclose everything that has already been looked at. I don’t think of selective reporting as a moral failing that has to be confessed. It is a fact about the data-generating process, and it changes what the numbers mean. An analyst who learns on submission day that eleven subgroup comparisons were already run has been handed a different problem from the one they thought they were solving.
Settle authorship and credit at the start. The ASA’s Ethical Guidelines for Statistical Practice (eight principles across 72 elements, approved in 2022) put responsibilities to stakeholders and the integrity of data and methods on the same footing5. Statisticians are expected to refuse to produce misleading analyses. That obligation is much easier to honour when they are a named collaborator than when they are an unacknowledged service provider who is paid by the hour.
Accept that the honest answer may be “this design can’t answer that.” Sometimes the most valuable thing a statistician does is to say that the study is not powered to detect the thing we care about. If we treat that as obstruction when it is really information, we are likely to end up with an underpowered study and a strained relationship.
In Industry and Analytics Settings
Bring the decision along with the query. “Pull last quarter’s same-store sales by region” is a specification. “We’re deciding whether to close three underperforming stores and I need to know whether the recent dip is real” is a problem. The second one lets the analyst point out that the first request would have been misleading (because, say, the comp-store definition excludes the exact stores being evaluated).
Define the metric once, in writing, and stop redefining it in meetings. Most “the numbers don’t match” escalations seem to be two correct calculations of two different things. Thus, a metric definition should name its grain, its filters, its date logic, its denominator, and its owner.
Say what happens if the number is wrong. An estimate for a monthly newsletter and an estimate that feeds a pricing change deserve different levels of scrutiny, and analysts cannot calibrate effort that they cannot see. So, we should tell them how far the consequences would reach (the blast radius).
Give real deadlines and real slack. I wouldn’t call “EOD” a deadline when the request requires joining three source systems. In practice, it amounts to asking for an unvalidated guess that will carry the requester’s name.
What You Owe the Data Engineer
Data engineers are the ones who find out that the business logic is inconsistent, usually at 2 a.m. Their constraints are rarely matters of preference, so it helps to treat them as fixed.
Send the reproduction. “The dashboard looks off” costs an engineer hours. Compare that with “Row for store 412, 2026-07-14, shows fuel gallons of 0 but the POS extract shows 8,431 (here’s the query)”, which costs minutes.
A defect report has four parts. Telling an engineer or statistician that a number is wrong (and nothing more) hands them a search problem with no search criteria. A usable report specifies:
- What is wrong: the exact rows, cells, or figures, with the query or filter that reproduces them.
- Why it is wrong: the business or domain fact that contradicts it (“these stores were acquired in March, so they can’t have January comp sales”).
- What correct looks like: the expected value or behaviour, with the independent source it comes from (the POS extract, the vendor invoice, the trial protocol).
- How to build the correct logic: the rule in implementable terms (“exclude stores from the comp base until they have 13 full months of history, keyed on
acquisition_date, notopen_date”).
The fourth part is the one that collaborators most often skip, probably because it feels like doing the engineer’s job. However, the collaborator is the only person who knows the rule. The engineer can translate a rule into SQL in an afternoon, but they cannot come up with the rule itself. If they are forced to guess, the business logic has been delegated to the person with the least context for it, which is how the definition-drift problem below gets started in the first place.
Announce upstream changes before they ship. A renamed field, a new store format, a changed promotion code, a vendor migration, and so on. Every one of these breaks something downstream. A two-line heads-up costs roughly nothing, whereas finding out through a silently wrong report can cost quite a lot.
Understand that “just add a column” is rarely just adding a column. It may mean a backfill, a grain change, a contract change with every downstream consumer, and a reconciliation against numbers that people have already quoted externally.
Stop building shadow pipelines. The spreadsheet that we maintain because the official table was inconvenient will eventually be cited in a meeting as a source of truth, and then someone will have to reconcile it. If the official table is wrong, it is better to file that as a defect than to route around it.
Treat data quality as a shared problem. In dbt Labs’ 2025 practitioner survey (459 respondents, collected October–December 2024, and a self-selected industry sample as opposed to a probability sample), poor data quality was the most commonly cited challenge, named by more than 56% of respondents6. Data quality originates in operational systems and business processes. Thus, most of it sits outside the data team’s control, and escalating it to the data team (as if it were a service-desk ticket) is unlikely to fix it.
What Statisticians and Data Engineers Owe Back
The obligations run in both directions, and this seems to be the part that pieces like this one tend to skip over.
Learn the domain well enough to be wrong intelligently. Vance and Smith’s ASCCR framework (Attitude, Structure, Content, Communication, Relationship) is drawn from directing statistical collaboration labs across more than 900 direct projects and the supervision of 4,000 more, and it puts Attitude and Relationship on equal footing with Content7. The framework exists because technically correct work that nobody can understand has an effect size of zero.
Report uncertainty as uncertainty. We should give the interval and the assumptions that it rests on. Treating a P-value as a binary verdict throws away most of what the analysis produced, since it is continuous evidence. This is the case for S-values and for the full compatibility/consonance function over a single dichotomized interval.
Say what would change your mind. Before running the analysis, we can state which results would be surprising and what we would do about them. This is the analyst-side equivalent of pre-registration, and it costs nothing.
Make it reproducible or don’t ship it. Given the demonstrated spread across analysts2, 3, I’d hesitate to call an analysis a result if its choices aren’t recorded. That means code in version control, the query saved next to the number, and the exclusions written down.
Push back early and in writing. An analyst who accepts an impossible framing in the meeting, and then privately resents it for six weeks, has probably chosen the worst option available.
Explain the constraint along with the refusal. “That’s a three-week change because the fact table’s grain is transaction-line and you’re asking for basket-level” is a sentence that a business partner can act on. “That’s not possible” gives them nothing to act on.
Everything Runs on Assumptions, So Write Them Down
Every model is a bundle of assumptions: linearity, exchangeability, missingness mechanisms, and the proposition that the measurement instrument measures the thing. However, so is everything else in the collaboration. We assume that the chair holds, the laptop boots, the warehouse job ran last night, the engineer understood the ticket, and the stakeholder meant what the ticket says. I don’t believe there is any assumption-free work. There is work whose assumptions are written down, and there is work whose assumptions are discovered later, at a worse time, by someone angrier.
The problem doesn’t come from making assumptions. It comes from what I’d call laundering them, which is letting them pass silently into the result so that the final number looks cleaner than the process that produced it. The analyst-variability results above2, 3 are an example of this, with dozens of defensible assumption sets, each of them invisible in its own write-up and only visible when they are laid side by side.
So the practice that I’d suggest, for both sides of the table, is a two-column ledger. For each assumption, we state:
- The assumption, in one falsifiable sentence.
- What is different if it’s wrong, meaning which conclusions survive, which die, and what we’d check first.
Here’s a short worked example:
| Assumption | If it’s wrong |
|---|---|
| The nightly load completed before the snapshot | Yesterday’s figures are partial; totals are biased downward; check the job log before anything else |
acquisition_date marks when a store enters the comp base |
Comp growth mixes organic and acquired stores; the trend is uninterpretable; re-key the base |
| Missingness in loyalty ID is unrelated to basket size | Basket-level estimates are biased toward small transactions; bound the effect or model the missingness |
| The stakeholder’s “sales” means net of returns | Every reconciliation against finance fails by the return rate; re-confirm the definition |
Two things tend to happen when we do this. First, most of the second column turns out to be checkable in minutes, so a general worry about the number becomes a short list of things to verify. Second, the disagreements move to where they belong. “Your number is wrong” becomes “I don’t accept row three,” which is a conversation that two professionals can actually finish. An assumption that is stated along with the consequence of its failure can be tested like a hypothesis, whereas one that is left implicit tends to surface later, when it is much more expensive.
An Intake Template
I’d put this at the top of every request. If it cannot be filled in, then the work probably isn’t ready to start.
| Field | Why it matters |
|---|---|
| Decision | What action changes based on the answer |
| Question | One sentence, in domain language |
| Estimand / metric | Quantity, population, contrast, grain, denominator |
| Population & exclusions | Who’s in, who’s out, and why |
| Time window | Including how “current” is defined |
| Known data issues | Migrations, backfills, definition changes |
| Prior looks | What has already been run or explored |
| Blast radius | External? Regulatory? Internal only? |
| Deadline & slack | Real date, and what happens if it slips |
| Owner | Who signs off on the definition |
| Assumption ledger | Each assumption, and what changes if it’s wrong |
Two Checklists
Before You Ask
Before You Deliver
The Failure Modes Worth Naming
The post-mortem request. The design is locked, the data are collected, and the ask is to find something (see Fisher, above).
The definition drift. There are two teams, two correct numbers, and one metric name. I suspect this costs more meeting-hours annually than any modelling error.
The heroic pull. An engineer bypasses the pipeline under deadline pressure, the one-off ends up being relied upon, and six months later nobody remembers the filters.
The silent constraint. The analyst knows that the design cannot support the claim, says so once, softly, in a meeting, and is not heard. The remedy here is simply to write it down.
None of this is sophisticated, and I think that’s rather the point. The returns to being a good collaborator seem to come almost entirely from unglamorous, cheap, and early habits, and hardly at all from anyone being cleverer.
Bring the decision, define the metric, disclose the prior looks, and warn about upstream changes. On the other side, explain the constraint, report the uncertainty, and make it reproducible.
References
Abstract
Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players. Analytic approaches varied widely across the teams, and the estimated effect sizes ranged from 0.89 to 2.93 ( Mdn = 1.31) in odds-ratio units. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship. Overall, the 29 different analyses used 21 unique combinations of covariates. Neither analysts’ prior beliefs about the effect of interest nor their level of expertise readily explained the variation in the outcomes of the analyses. Peer ratings of the quality of the analyses also did not account for the variability. These findings suggest that significant variation in the results of analyses of complex data may be difficult to avoid, even by experts with honest intentions …
Abstract
Several months ago, Phil Bourne, the initiator and frequent author of the wildly successful and incredibly useful “Ten Simple Rules” series, suggested that some statisticians put together a Ten Simple Rules article related to statistics. (One of the rules for writing a PLOS Ten Simple Rules article is to be Phil Bourne [1]. In lieu of that, we hope effusive praise for Phil will suffice.) Implicit in the guidelines for writing Ten Simple Rules [1] is “know your audience.” We developed our list of rules with researchers in mind: researchers having some knowledge of statistics, possibly with one or more statisticians available in their building, or possibly with a healthy do-it-yourself attitude and a handful of statistical packages on their laptops …
Abstract
Statistics and data science are especially collaborative disciplines that typically require practitioners to interact with many different people or groups. Consequently, interdisciplinary collaboration skills are part of the personal and professional skills essential for success as an applied statistician or data scientist. These skills are learnable and teachable, and learning and improving collaboration skills provides a way to enhance one’s practice of statistics and data science. To help individuals learn these skills and organizations to teach them, we have developed a framework covering five essential components of statistical collaboration: Attitude, Structure, Content, Communication, and Relationship. We call this the ASCCR Frame. This framework can be incorporated into formal training programs in the classroom or on the job and can also be used by individuals through self-study. We show how this framework can be applied specifically to statisticians and data scientists to improve their collaboration skills and their interdisciplinary impact …
Citation
@online{rafi2026,
author = {Rafi, Zad and Rafi, Zad},
title = {How to {Be} a {Good} {Collaborator} to a {Statistician} and a
{Data} {Engineer}},
date = {2026-08-11},
url = {https://lesslikely.com/statistics/good-collaborator.html},
langid = {en-US}
}
Comments