n8cir-reprohacks
Welcome to the N8 CIR ReproHack Series!
Event Repository: http://bit.ly/n8-reprohacks
Contains all event information and links to materials
Introductions
Who am I?
Dr Anna Krystalli (@annakrystalli)
- Research Software Engineer University of Sheffield
- 2019 Fellow Software Sustainability Institute
- Software Peer Review Editor rOpenSci
- Co-organiser Sheffield R Users Group
Why am I here?
I believe there’s lots to learn about Reproducibility from working with real published projects.
Who is my favorite animated character?
Stitch!
Who are you?
Why are you here?
Who is your favorite animated character?
Plan of Action
Reproduce paper
-
Project review and team formation
-
Select and register your project
-
Work on your project!
-
Re-group part-way through.
-
Feedback at the end (group & authors)
Code of Conduct
Event governed by ReproHack Code of Conduct
Additional Considerations
-
Reproducibility is hard!
-
Submitting authors are incredibly brave!
Thank you Authors! :raised_hands:
-
Without them there would be no ReproHack.
-
Show gratitude and appreciation for their effort and bravery. :pray:
-
Constructive criticism only please!
Reproduce and Review
:mag:
Review as an auditor :bookmark_tabs:
Access
- How easy was it to gain access to the materials?
Installation
- How easy / automated was installation?
- Did you have any problems?
Data
- Were data clearly separated from code and other items?
- Were large data files deposited in a trustworthy data repository and referred to using a persistent identifier?
- Were data documented …somehow…
Documentation
Was there adequate documentation describing:
- how to install necessary software including non-standard dependencies?
- how to use materials to reproduce the paper?
- how to cite the materials, ideally in a form that can be copy and pasted?
Analysis
- Were you able to fully reproduce the paper? :white_check_mark:
- How automated was the process of reproducing the paper?
- How easy was it to link analysis code to:
- the plots it generates
- sections in the manuscript in which it is described
Analysis
If the analysis was not fully reproducible :no_entry_sign:
- Did results (e.g. model outputs, tables, figures) differ to those published? By how much?
- Were missing dependencies?
- Was the computational environment not adequately described / captured?
Review as a user :video_game:
What did you find easy / intuitive?
What did you find confusing / difficult
What did you enjoy?
Feed back
:speech_balloon:
Feedback as a community member
Acknowledge author effort
Give feedback in good faith
Focus on community benefits and system level solutions
Feed back opportunities
- Regroup part way through to discuss progress and troubleshoot any sticking points
- Feedback to authors using form by end of session
- Feedback to group at the end, contribute to discussions
Additional activities
Finished early?
Explore the work more deeply:
- Try and run additional analyses.
- Create new plots.
- Combine materials with your own or other open materials available on the web!
Replicate your paper
- Consider attempting replication!
- Replications could be considered for publication in ReScience C Journal
ReScience C
ReScience C is an open-access peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research is reproducible.
What is a Replication:
- Repeating a published protocol
- Respecting its spirit and intentions
- Varying the technical details, e.g. using different software, initial conditions, etc.
Change something that everyone believes shouldn’t matter, and see if the scientific conclusions are affected
Let’s go! :checkered_flag:
1. Paper review
- Have a look at the papers available for reproduction
2. Team formation / project registration
- Fine to work individually
- Add your details to the hackpad.
- Register your team and paper on the hackpad
3. Mid-point regroup
- Which paper have you selected? Briefly describe what it’s about.
- Briefly describe the approach to reproducibility the paper has taken.
- Anything in particular you like about the paper’s approach so far?
- Anything you’re having difficulty with?
4. Feedback to authors
- Please complete the feedback form for authors
- Feel free to record general findings the hackpad
5. Final regroup
- So, how did you get on?
- Final comments.
- If there’s time, tackle some discussion topics (see hackpad).
- On post-its: One thing you liked, one thing that can be improved.
Further Resources
- The Turing Way: a lightly opinionated guide to reproducible data science.
- Packaging data analytical work reproducibly using R (and friends): how researchers can improve the reproducibility of their work using research compendia based on R packages and related tools
THANK YOU ALL! :pray:
-
Thank you PARTICIPANTS for coming!
-
Thank you AUTHORS for submitting!