P-hacking Issues

Abstract

Researcher degrees of freedom can lead to a multiple comparisons problem, even in settings
where researchers perform only a single analysis on their data. The problem is there can be a
large number of potential comparisons when the details of data analysis are highly contingent on
data, without the researcher having to perform any conscious procedure of fishing or examining
multiple p-values. We discuss in the context of several examples of published papers where
data-analysis decisions were theoretically-motivated based on previous literature, but where the
details of data selection and analysis were not pre-specified and, as a result, were contingent on
data.

Attachment: 
Resource Type: