The Limits of Human Predictions of Recidivism

This is the replication code repository for analysis in paper The Limits of Human Predictions of Recidivism.

Run Analysis

To view tables and figures generated by us, results.html contains all figures and tables in the paper.

To rerun all the analysis in the paper, you may execute Rscript run.R in command line.

Runing our analysis scripts requires several R packages. They are being specified at the top of src/pkgs.R file. You can install them via running the following command:

install.packages(
  c("scales", "matrixStats", "boot", "arm", "ROCR", "assertthat", "doMC", 
  "forcats", "furrr", "lubridate",  "kableExtra", "tidyverse")
 )

After running the script, you can view the result by running/knitting results.Rmd.

LSI-R Dataset

The original LSI-R datasets under data/private/individuals/ are only available upon request due to privacy concerns. However, you can still run the analysis using our pre-generated intermediate data.

If the original LSI-R datasets are not available, ID and outcome columns from the LSI-R data (under data/public/individuals/) and pre-calculated summary statistics (under data/public/derived/lsi) are used to reproduce the results. If the original LSI-R datasets do exist under data/private/individuals/, then derived data at data/public/derived/lsi will be recalculated using the original datasets.

Reference

Lin, Zhiyuan (Jerry), Jongbin Jung, Sharad Goel, Jennifer Skeem. "The Limits of Human Predictions of Recidivism." Science Advances (2020).

BibTex

@article {lin2020limits,
	author = {Lin, Zhiyuan (Jerry) and Jung, Jongbin and Goel, Sharad and Skeem, Jennifer},
	title = {The Limits of Human Predictions of Recidivism},
	volume = {6},
	number = {7},
	elocation-id = {eaaz0652},
	year = {2020},
	doi = {10.1126/sciadv.aaz0652},
	publisher = {American Association for the Advancement of Science},
	abstract = {Dressel and Farid recently found that laypeople were as accurate as statistical algorithms in predicting whether a defendant would reoffend, casting doubt on the value of risk assessment tools in the criminal justice system. We report the results of a replication and extension of Dressel and Farid{\textquoteright}s experiment. Under conditions similar to the original study, we found nearly identical results, with humans and algorithms performing comparably. However, algorithms beat humans in the three other datasets we examined. The performance gap between humans and algorithms was particularly pronounced when, in a departure from the original study, participants were not provided with immediate feedback on the accuracy of their responses. Algorithms also outperformed humans when the information provided for predictions included an enriched (versus restricted) set of risk factors. These results suggest that algorithms can outperform human predictions of recidivism in ecologically valid settings.},
	URL = {https://advances.sciencemag.org/content/6/7/eaaz0652},
	eprint = {https://advances.sciencemag.org/content/6/7/eaaz0652.full.pdf},
	journal = {Science Advances}
}