Introduction

Reproducible Software Pipelines

We consider a software pipeline that processes data to produce results (e.g. reports, plots, presentations)

Scenario 1

  • You install some libraries
  • You develop your analysis those libraries
  • You share the analysis with someone else
  • The analysis breaks in mysterious and subtle ways

Scenario 2

  • You install some libraries
  • You develop your analysis those libraries
  • You start a new project, for which you need an updated version of the libraries
  • After a while, you go back to your first project, and it’s broken in mysterious and sublte ways!

Scenario 3

  • You download some raw data
  • You preprocess it manually
  • You carry out some analysis of the preprocessed data
  • After one year you have to repeat the analysis, but you lost access to the preprocessed data

reproducibility

  1. The quality of being reproducible.
  2. The closeness of agreement among repeated measurements of a variable made under the same operating conditions over a period of time, or by different people.

— wikitionary

The ACM definitions

The Association of Computing Machinery (ACM) defines three different flavors of reproducibility:

  • Repeatability
  • Reproducibility
  • Replicability

Repeatability (Same team, same experimental setup)

The measurement can be obtained with stated precision by the same team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same location on multiple trials. For computational experiments, this means that a researcher can reliably repeat her own computation.

— ACM

Reproducibility (Different team, same experimental setup)

The measurement can be obtained with stated precision by a different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using the author’s own artifacts.

— ACM

Replicability (Different team, different experimental setup)

The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently.

— ACM

We want our software pipelines to be at least repeatable.

  • manage the data
  • manage the code
  • manage the environment
  • manage the process