3. Assessment

You will evaluate the available materials for reproducing the study and assign a reproducibility score to the display items related to the chosen claims. You will also review the overall reproducibility practices of the paper.

The focus is on analysing the current reproducibility of the reproduction package before suggesting any improvements. By the end of this stage, a comprehensive description of the package's reproducibility will be created, which can be used to make improvements.

You will be asked to record the following input materials.

Raw data: Unmodified data files obtained by the authors from the sources cited in the paper.

Analytic data: Data used as the final input in a workflow to produce a statistic displayed in the paper

Code scripts: A script associated with data cleaning or analysis. Examples include deleting variables or observations, merging data sets, removing outliers, reshaping the data structure, running regressions, running hypothesis tests, computing standard errors, and imputing missing values, generating a series and plotting it.

Once you have identified and recorded the input materials, you can assign reproducibility scores to individual display items.

You can follow the guideline below for assigning reproducibility scores.

• Level 1 (L1): No data or code are available. Possible improvements include adding: raw data, analysis data, cleaning code, and analysis code.

• Level 2 (L2): Code scripts are available (partial or complete), but no data are available. Possible improvements include adding: raw data and analysis data.

• Level 3 (L3): Analytic data and code are partially available, but raw data and cleaning code are missing. Possible improvements include: completing analysis data and/or code, adding raw data, and adding analysis code.

• Level 4 (L4): All analytic data sets and analysis code are available, but the code fails to run or produces results inconsistent with the paper (not CRA). Possible improvements include: debugging the analysis code or obtaining raw data.

• Level 5 (L5): Analytic data sets and analysis code are available and they produce the same results as presented in the paper (CRA). The reproducibility package may be improved by obtaining the original raw data.

• Level 6 (L6): Cleaning code scripts are available (partial or complete), but raw data is missing. Possible improvements include: adding raw data.

• Level 7 (L7): Cleaning code is available and complete, and raw data is partially available. Possible improvements: adding raw data.

• Level 8 (L8): All the materials (raw data, analytic data, cleaning code, and analysis code) are available. However, the cleaning code fails to run or produces different results from those presented in the paper (not CRR) or the analysis code fails to run or produces results inconsistent with the paper (not CRA). Possible improvements: debugging the cleaning or analysis code.

• Level 9 (L9): All the materials (raw data, analytic data, cleaning code, and analysis code) are available. The analysis code produces the same output as presented in the paper (CRA). However, the cleaning code fails to run or produces different results from those presented in the paper (not CRR). Possible improvements: debugging the cleaning code.

• Level 10 (L10): All necessary materials are available and produce consistent results with those presented in the paper. The reproduction involves minimal effort and can be conducted starting from the analytic data ( CRA) and the raw data (CRR). Note that Level 10 is aspirational and may be unattainable for most research published today.