Blackwell Reference Online

Extract

Researchers have traditionally but erroneously presumed that statistical significance tests evaluate the replicability of results ( Thompson 1996, 2006 ). But p values evaluate the probability of the sample, assuming the null hypothesis perfectly describes the population, and not the probability of the population. Therefore, p values do not bear upon questions of replicability. As Cohen (1994) noted, the statistical significance test “does not tell us what we want to know, and we so much want to know what we want to know that, out of desperation, we nevertheless believe that it does!” Because isolating relationships that replicate under stated conditions is the ultimate objective of social science research, methods that do evaluate result replicability become fundamentally important. Thompson (1996) suggested that result replicability evaluation methods can be grouped into two classes: “external” and “internal” methods. External replicability analyses involve true replication via data collection with an independent sample. External replication is the ultimate, best method for evaluating result replicability. However, researchers may not have the luxury of external replication of every study they conduct. Internal replicability evaluation methods attempt to approximate real replication studies in various ways. “Internal” evidence for replicability is never as good ... log in or subscribe to read full text