Research & Development: The Repro Loop

Reproducible ≠ understood. A field guide for R&D and diagnostic teams.

1. The problem in one sentence

A reproducible bug did not protect us from wasting the investigation — it pulled us into it.

This is not about the replication crisis. That literature is about results that fail to reproduce. This guide is about what happens when they succeed: once a failure reproduces reliably, a second dynamic starts that nobody warns you about. The reproduction itself becomes the activity. Runs get cheap, each one feels like progress, and the team ends the day with a mountain of detail about the symptom and no eliminated hypotheses.

Reproducibility is still the first thing every engineer should want. It answers "is this real?" — and then, quietly, it removes the reason to stop and think.

This guide names the loop, explains why it catches competent people, and gives a small set of rules that make it structurally hard to fall in.

2. Why it happens

A successful reproduction answers exactly two questions:

Is the symptom real?
Can we trigger it on demand?

It says nothing about whether the current hypothesis is right or whether we are looking in the right place. Yet every successful run feels like confirmation, because it confirms the frame we are working in ("yes, it still crashes here"). Four forces then compound:

Marginal cost collapses. Once the repro takes 40 seconds instead of a day, there is no natural reason to stop before the next run. The budget that used to force reflection disappears. This is the part that is specific to reproducibility: the other three forces exist in every investigation, but a cheap, reliable repro is what feeds them.
Detail accumulates at the symptom site. Each run adds logs, traces, register dumps — all from the place where the failure shows, which is rarely the place where it originates. More detail feels like more understanding. The repro is a streetlight; it lights the place where the failure shows, and that is where everyone searches.
Sunk cost and momentum. "We've reproduced it 30 times, we must be close." The number of runs becomes evidence in its own right.
Social signalling. Reporting "reproducible, investigating" reads as progress to everyone outside the room. Reporting "we stopped to question our assumptions" does not.

The result is a loop: tweak a parameter → run → observe → tweak → run. It is indistinguishable from work from the outside and often from the inside.

3. How to recognize you are in it

Any two of these on the same day is a warning; three is a stop signal.

You have run the reproduction more than three times since a hypothesis was last ruled out.
Nobody wrote down, before the last run, what result would have surprised them.
You are varying inputs to "see what happens" rather than to distinguish between two named hypotheses.
The evidence discussed in the last hour is all symptom evidence (what it looks like) and none of it is cause evidence (which code, which state, which instruction).
A fix moved the failure to a different place and this was treated as "getting closer" rather than as proof the cause is upstream.
Someone says "let's just run it once more with X" and nobody can state what X is supposed to decide.
The investigation has no exclusion list — you cannot say, in one minute, which explanations are dead and why.
You cannot say, for run #12, exactly which version of the code and the test rig produced it.

4. The rules

Rule 1 — No run without a prediction

Before every reproduction, write one sentence: "If hypothesis H is true, this run will show ___. If it shows ___ instead, H is dead."

A run without a falsifiable prediction produces no information, however interesting the output. If you cannot write the sentence, you are not ready to run — you are ready to think.

Beware the pro-forma prediction: a sentence that has the right shape but would come true whether or not H holds. "The program will crash" is not a prediction about H; it is a description of the symptom.

Rule 2 — Falsification is the unit of progress

Count runs since the last hypothesis was eliminated, not runs in total. A dead hypothesis is worth more than ten successful reproductions. Say it out loud in the team: "We killed H3 today" is a result. "We reproduced it again" is not.

Rule 3 — The three-run stop

Three runs without a falsification: stop. Not "wrap up this one", stop. Then, in this order:

Write the exclusion list (what is ruled out, by which evidence).
Question the search location, not the parameters. "Are we looking where it shows or where it starts?"
Bring it to someone who was not in the loop (Rule 6).

Three is arbitrary. Two is too tight for messy systems, five is too loose to interrupt momentum. Pick a number and make it a team rule, not a personal judgement, so that invoking it is not a confession.

Rule 4 — Ask the counterfactual

"What would we do if this bug were not reproducible?"

The answer is always a list: read the code path, bisect versions, check invariants, disassemble at the fault address, diff configurations, read the spec. Do at least one of those before the next run. Reproducibility should widen the toolset, not narrow it to a single button.

Rule 5 — Vary to map, don't repeat to confirm

Repeating the same reproduction proves existence. Varying it — different inputs, a different code path, a different platform, a different build — maps the boundary of the failure, and boundaries point at causes. If the last three runs used the same scenario with cosmetic changes, they were repetitions.

Rule 6 — Separate the investigator from the reviewer

The person running the loop cannot see the loop. Assign a reviewer who is explicitly not doing the runs and whose job is to ask, at fixed points:

What did the last three runs rule out?
What is the evidence for the current search location, other than "that's where it crashes"?
What is the one experiment that would make us abandon this direction entirely?

The reviewer does not need to know the system better than the investigator. Distance is the qualification. The reviewer's most useful contribution is often not a verdict but an alternative hypothesis — the one the investigator stopped being able to see.

Rule 7 — Keep two lightweight artifacts

Hypothesis register — numbered, with status: open / testing / eliminated (by run #). New hypotheses get an entry before they get a run.

Run log — one line per run: timestamp, hypothesis tested, prediction, outcome (confirmed / falsified), state of the system under test (commit hash, build id, or a note of what was changed since the last run), what varied. Tab-separated or a table — anything that a grep or a filter can count.

The state column is not bureaucracy. A falsification is a claim about a specific version of the code and the rig. Without it, "H4 is dead" has no referent, and a patch made between run #7 and run #8 can silently turn into evidence against a hypothesis it never tested.

Together these make the loop visible. At the end of the day the log answers, without anyone's opinion: how many runs, how many falsifications, how many hypotheses touched. Many runs and zero falsifications is a number, not an accusation.

Rule 8 — Check the rig before you trust the count

Three green runs on a test that cannot fail are three units of false confidence. Before treating a repeated result as evidence, verify once that the measurement can detect what you are measuring: make the failure happen on purpose, or remove the component under test and confirm the test goes red. If it stays green, you have been measuring the rig, not the system.

This rule exists because the loop does not need a bug to run. It needs a symptom and cheap runs — and a test rig that has drifted from what you think it measures supplies both.

5. Anti-patterns worth naming

These deserve their own vocabulary so the team can call them out in one word.

Name	What it looks like	Why it misleads
Symptom similarity	"This looks like the X bug we had last year"	Similar symptoms, different causes. Similarity is a hypothesis, not evidence.
Bug displacement	A patch makes the failure appear somewhere else	Treated as progress; actually proof the cause is upstream of both locations.
Local → global	Drawing an architectural conclusion from one failing call site	The evidence supports a local claim only.
Magnitude as proof	"CPU drops 80% right there, that must be it"	Size of a symptom says nothing about its position in the causal chain.
Run count as evidence	"We've seen it 50 times, it's definitely in module M"	Fifty observations of the same symptom are one data point about the cause.
Drift to closure	Reframing an intermediate state as "good enough for now"	Comfortable, and usually the moment the real cause was one step away.
Drift to depth	Ever finer detail at the same location	The mirror image of closure drift: activity without direction.
Unverified rig	Repeated green results, nobody has seen the test go red	The test may be measuring the setup, not the system.
Pro-forma prediction	A prediction that would hold whether or not the hypothesis does	Satisfies the ritual, produces no information.

6. Team rituals

Keep them small enough to survive a busy week.

Stand-up question: "What did we rule out yesterday?" — replaces "what did we find".
Run-log review, five minutes, daily: the reviewer reads the log, not the traces. Counts runs vs. falsifications. Flags any hypothesis with more than three runs and no elimination, and any falsification where the code changed between runs.
Stop is a team rule, not a personal call. Anyone can invoke the three-run stop on anyone's investigation without it being a criticism. Make this explicit; otherwise nobody invokes it on a senior engineer.
Exclusion list before handover. No investigation moves between people or across a weekend without a current exclusion list. It is the only artifact that transfers.

7. Quick reference

BEFORE A RUN
  [ ] Which hypothesis? (number from the register)
  [ ] What result would kill it — and would that result still
      be possible if the hypothesis were false?
  [ ] Is this a variation or a repetition?
  [ ] Has the code or the rig changed since the last run? Note it.

AFTER A RUN
  [ ] Confirmed or falsified? Log it, with the state under test.
  [ ] Falsified → hypothesis eliminated, counter to zero.
  [ ] Confirmed → counter + 1.

COUNTER AT 3
  [ ] Stop.
  [ ] Write the exclusion list.
  [ ] Question the search location.
  [ ] Reviewer.

ANY TIME
  "What would we do if this were not reproducible?"
  Do one of those first.
  "Has this test ever gone red?" If not, make it.

8. A note on tooling

Everything above works with a text file and discipline. If a team wants it enforced rather than remembered, the same rules translate directly into automation: a wrapper around the reproduction command that refuses to run without a prediction on file, a counter that blocks after three unfalsified runs, a log in a format that can be counted, a recorded state of the tree under test. The mechanism does not change; only who has to remember it.

One observation from building such a tool: the person building it fell into the loop while building it — three green runs on a test that could not fail, taken as three facts about the code. The mechanism caught it, one level up. That is the strongest argument for building the mechanism rather than trusting the rule.

Reproducibility is a status, not a proof. A reproduced bug is a bug you can study. Whether you are studying its cause or its costume is a separate question — and the only one that ends the investigation.

Appendix — What this article is based on

The term "repro loop" is ours. The phenomenon is assembled from four bodies of work that each cover part of the mechanism, plus practitioner observation. An earlier draft was titled "The Reproducibility Trap"; the title was dropped because "reproducibility" in the literature belongs to the replication crisis — the failure of results to reproduce — which is the opposite problem.

A. Cognitive biases in software engineering

Mohanani, Salman, Turhan, Rodríguez, Ralph — "Cognitive Biases in Software Engineering: A Systematic Mapping Study." IEEE Transactions on Software Engineering (online 2018; vol. 46, 2020). The reference survey. Identifies confirmation bias, anchoring and optimism bias as the most frequently studied; notes that research on debiasing interventions is scarce and rarely validated empirically. Source for §2 and for the caution that awareness alone does not remove a bias.
Çalıklı & Bener — "Empirical analyses of the factors affecting confirmation bias and the effects of confirmation bias on software developer/tester performance." PROMISE 2010. Shows that developers and testers prefer positive tests (those expected to pass) over negative ones, and identifies time pressure as a major antecedent. The same tendency, one stage earlier in the process than the repro loop. Source for Rules 1, 2 and 8.
Chattopadhyay et al. — "A Tale from the Trenches: Cognitive Biases and Software Development." ICSE 2020. Field observation of anchoring and related biases in working developers. Source for the anti-pattern table (§5).

B. Debugging as hypothesis testing

Alaboudi & LaToza — "Using Hypotheses as a Debugging Aid." IEEE VL/HCC 2020. The closest empirical work to this guide. Findings: incorrect hypotheses lead developers to investigate irrelevant information and block progress; developers hold few hypotheses (about two per defect); an early correct hypothesis strongly predicts success. Crucially, offering fault locations did not improve outcomes, while offering candidate hypotheses made success about six times more likely. This is the evidence behind the hypothesis register (Rule 7), the reviewer's job of supplying alternatives (Rule 6), and the warning against accumulating detail at the symptom site.
Alaboudi & LaToza — "Edit-Run Behavior in Programming and Debugging." IEEE VL/HCC 2021. Characterises the edit-run cycle itself — the loop this guide is about — in terms of frequency and duration.
Alaboudi & LaToza — "What Constitutes Debugging? An Exploratory Study of Debugging Episodes." Empirical Software Engineering 28(5), 2023. Observational data on how professional developers actually structure debugging, including forward and backward reasoning strategies.
"A Grounded Theory of Debugging in Professional Software Engineering Practice." arXiv:2602.11435, 2026. Observes that developers often voice uncertainty and speculative statements that resemble hypotheses without being testable ones — the "pro-forma prediction" in Rule 1 and §5.
Zeller — Why Programs Fail: A Guide to Systematic Debugging. 2nd ed., Morgan Kaufmann 2009. Applies the scientific method (hypothesis → prediction → experiment → observation) explicitly to debugging. The methodological backbone of Rules 1–5.

C. Older roots in philosophy and methodology of science

Platt — "Strong Inference." Science 146:347–353, 1964. Formulate competing hypotheses, design the experiment that kills one of them, repeat. Rule 2 and the counterfactual in Rule 4 are Platt applied to a repro loop.
Chamberlin — "The Method of Multiple Working Hypotheses." Science, 1890 (reprinted 1965). The predecessor: keeping several hypotheses alive at once as a defence against attachment to the first one.
Wason — "On the failure to eliminate hypotheses in a conceptual task." Quarterly Journal of Experimental Psychology 12:129–140, 1960. The 2-4-6 task; the first experimental demonstration that people test hypotheses by seeking confirmation rather than refutation.
Popper — The Logic of Scientific Discovery. 1959 (German original 1934). Falsifiability as the criterion that separates a testable claim from an unfalsifiable one. Rule 1 in its original form.
The streetlight effect. Folklore older than any citation (the drunk searching for his keys under the lamp); popularised for research practice by Freedman, "Why Scientific Studies Are So Often Wrong: The Streetlight Effect," Discover, 2010. The repro loop is a special case: the repro is the streetlight.

D. Diagnostic error in medicine

Croskerry — "The importance of cognitive errors in diagnosis and strategies to minimize them." Academic Medicine 78(8), 2003. Names "premature closure", "anchoring" and "search satisficing" — the medical vocabulary for the closure-drift and depth-drift anti-patterns in §5 — and proposes "cognitive forcing strategies": deliberate, rule-based interruptions of the diagnostic process. The tooling note in §8 is a cognitive forcing strategy for engineers.

E. Practitioner origin

The guide grew out of a diagnostic session (September 2026) in which an investigation with a fast, reliable reproduction ran for hours without eliminating a hypothesis. The one-line lesson — reproducibility didn't protect the investigation, it pulled it in — is the first sentence of §1. The three-run stop, the prediction-before-run rule and the run log were first designed as enforced tooling for an AI coding agent and then translated into the team practice described here, since the mechanism turned out not to be specific to the agent. Rule 8 and the state column in Rule 7 were added after the tool's own construction and review produced two further instances of the loop — one caught by the builder, one by an external reviewer.

What the literature does not (yet) cover

None of the above isolates the specific effect that a cheap, reliable reproduction removes the natural stopping point of an investigation. The edit-run studies measure how often and how long developers cycle, not the information yield per cycle. Whether run count without falsification is a usable leading indicator of a stalled investigation is, to the authors' knowledge, an open empirical question. A run log in the format described in Rule 7 would produce exactly the data needed to test it.