Parachute Problem

As a general rule, we try to not believe things without sufficient evidence. What defines sufficient evidence is complicated. 1

However, not everything can be discovered via direct comparison, randomised trials, or similar efforts.

Software Engineering is its own discipline and trying to directly transplant, e.g. medical research techniques aren’t conducive to generating understanding. "It needs to be studied with tools that borrow as much from the social and cognitive sciences as they do from the mathematical theories of computation."

Sometimes things are so useful, or so obvious, that you won’t find research supporting the claim. For example, no evidence that parachutes actually prevent death from falling exists. 2 In fact, in one study, it was shown that parachutes do not reduce injury above a normal backpack. 3

Similarly, “obviously good” practices, such as using tests, exist in software engineering.

  1. Tim Menzies & Forrest Shull. (2011). Chapter 1: The Quest for Convincing Evidence. In Andy Oram & Greg Wilson (Eds.), Making software: what really works, and why we believe it: Theory in practice (1st ed). O’Reilly. ↩︎

  2. Gordon C S Smith & Jill P Pell. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ : British Medical Journal, 327(7429), 1459–1461. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/ ↩︎

  3. Robert W. Yeh, Linda R. Valsdottir, Michael W. Yeh, Changyu Shen, Daniel B. Kramer, Jordan B. Strom, Secemsky, E. A., Joanne L. Healy, Robert M. Domeier, Dhruv S. Kazi, & Brahmajee K. Nallamothu. (2018). Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial. BMJ, 363, k5094. https://doi.org/10.1136/bmj.k5094 ↩︎