ReproHacks

A sandbox environment

for practicing reproducible research

---

Dr Anna Krystalli (r-rse) @annakrystalli

TIER2 @ FENS Forum 2024 ReproHack

👋 Hello

me: Dr Anna Krystalli

• Research Software Engineering Consultant, r-rse

  • twitter @annakrystalli
  • github @annakrystalli
  • email r.rse.eu[at]gmail.com

• 2018 Software Sustainability Fellow

• Founder & Core Team member ReproHack

slides: bit.ly/tier2-reprohack-slides

Motivation

I’m going to start with some motivating background and in particular focus on two key benefits of open science

Why Open Science

---

Benefit #1

Transparency as a means of verification

Benefit #2

Transparency as a means of supercharging research cycle

I’m likely preaching to the coverted here and there has been much discussion over the passed decade + on the topic but let’s just recap some the benefits of whats really driving this activity today: Open Science

• In the shadow of the reproducibility crisis, the first benefit of Open Science is transparency, which allows us to impose a minimum standard of verification of research results in the form of reproducibility. Being able to verify by recreating a result from the original code and data.

• In addition, Making underlying materials openly available means it’s much easier to for anyone to build on previous research, allowing us to supercharging the research cycle.

So how are we doing?

So how are we doing in terms of opening up science?

The points have actually been widely recognised and generally accepted by now.

There’s certainly much more open research code and data out there! It’s a very crude metric but just a quick search of GitHub for research-paper returns over 11K repositories!

• How far have we actually come in terms of more open, reusable and robust science?

• Well that’s actually a really hard question to answer because,

• despite an increase in the publication of more research code and data (by no means sufficient yet), there is still no systematic review of such materials to confirm that research is indeed reproducible

Is code and data enough?

• Indeed trying to work with many materials can feel a lot like this where you download some materials, start playing around, can’t get them to work and soon give up.

• It also begs the question:

slide: Karthik Ram: rstudio::conf 2019 talk

If a paper claims to be reproducible but nobody checks it, is it really reproducible?

Open materials != functional materials.

In truth, this situation is not really surprising because formal training in practices and standards for researchers is still lagging despite demands for reproducibility.

The lack of systematic review leads to an additional problem, that there is no opportunity to practice and get feedback on the reproducibility of our research.

All skills require practice

Photo by Alexander Grey on Unsplash

Reprohacks

One day reproducibility hackathons

---

• How reproducible are papers?

• How can we provide a sandbox environment to practice reproducibility?

So these were really the key issues that the ReproHack project sought to address:

• How reproducible are papers with associated published code and data?

• How can we provide a sandbox environment where both authors and reviewers can practice reproducibility?

ReproHack History

• OpenCon Satellite: Berlin, 2016

• OpenCon Satellite: London, 2017

Inspired by Reproducible Research in Ecology, Evolution, Behaviour, and Environmental Studies course:

• Reproduce published results from raw data
• Over a few months and a number of sessions

ReproHack mission: Reproduce paper in a day from code and data

2018 SSI Fellowship

In 2018 I also became a Software Sustainability Institute Fellow and the project I proposed was to run a bunch more ReproHacks. This was a great opportunity to develop the project further.

At the time, these fellowships were only for UK based researchers but the project has since opened up to international applicants so I string recommend applying for one if you have an idea for a project to promote open reproducible science.

The support also lead to the development of the ReproHack Hub site which we built to make it easier to engage and facilitate all reprohack activities.

How does it work?

Call for papers

As organisers, In the current format, leading up to the event,

we have a call for papers

where we encourage authors to submit their papers for reproduction.

We like this because it means authors get engaged from the start

And this hopefully results in a nice paper list for participants to work with on the day but either way, there is always the central paper list to fall back on.

On the day

---

• Intro & Set the Tone

• Select paper and form groups

• Work with materials and reproduce

• Regroup & Discuss

• Feed back to authors

On the day, we start with a bit of an introduction in which we stress that:

• Reproducibility is hard!
• Authors are incredibly brave to invite us into their work
• So we set the tone to invite only constructive feedback.

Participants are then free to review and select papers they wish to work on and attempt to reproduce them

At some point we regroup to share progress and close with a final share out.

Most importantly, we ask participants to feed their experiences back to the authors through a structured feedback form.

Benefits

ReproHacks are fun

Crucially though the events are fun!

Participants really enjoyed the low pressure friendly and collaborative environments, felt they learnt a lot and found being able to reproduce others work extremely satisfying!

Tips for reviewing

Code of Conduct

Event governed by ReproHack Code of Conduct

Additional Considerations

• Reproducibility is hard!

• Submitting authors are incredibly brave!

Thank you Authors! 🙌

• Without them there would be no ReproHack.

• Show gratitude and appreciation for their effort and bravery. 🙏

• Constructive criticism only please!

Selecting Papers

• Author comments: paper description and why you should choose to reproduce.
• Tags: Tools, languages & domains
• No. attempts: No. times reproduction has been attempted
• Mean Repro Score: Mean reproducibility score (out of 10)
  • lower == harder!

Review as an auditor 📑

Looking for FAIR principles

• Findable
• Accessible
• Interoperable
• Reusable

Access

• How easy was it to gain access to the materials?

Installation

• How easy / automated was installation?
• Did you have any problems?

Data

• Were data clearly separated from code and other items?
• Were large data files deposited in a trustworthy data repository and referred to using a persistent identifier?
• Were data documented …somehow…

Documentation

Was there adequate documentation describing:

• how to install necessary software including non-standard dependencies?
• how to use materials to reproduce the paper?
• how to cite the materials, ideally in a form that can be copy and pasted?

Analysis

• Were you able to fully reproduce the paper? ✅
• How automated was the process of reproducing the paper?
• How easy was it to link analysis code to:
  • the plots it generates
  • sections in the manuscript in which it is described

Analysis

If the analysis was not fully reproducible 🚫

• Did results (e.g. model outputs, tables, figures) differ to those published? By how much?
• Were there missing dependencies?
• Was the computational environment not adequately described / captured?

Review as a user 🎮

What did you find easy / intuitive?

What did you find confusing / difficult

What did you enjoy?

Feedback as a community member

Acknowledge author effort

Give feedback in good faith

Focus on community benefits and system level solutions

There are many ways to ReproHack and I’ve mentioned two of these alreay, but what I’d really like is for research groups to start reproducing and reviewing their work before publication to start normalising the process and developing the necessary skills

Ideally we’d love to see the project become obsolete because reproducibility reviewing to become the norm in research practice.

slide: Karthik Ram: rstudio::conf 2019 talk

Next Steps!

Submit your own papers!

Host your own event!

reprohack.org

---

Chat to us:

👋 Thanks for 👀 | ❓