 | This article includes a list ofgeneral references, butit lacks sufficient correspondinginline citations. Please help toimprove this article byintroducing more precise citations.(May 2012) (Learn how and when to remove this message) |
Statistical conclusion validity is the degree to which conclusions about the relationship amongvariables based on the data are correct or "reasonable". This began as being solely about whether the statistical conclusion about the relationship of the variables was correct, but now there is a movement towards moving to "reasonable" conclusions that use: quantitative, statistical, and qualitative data.[1] Fundamentally, two types of errors can occur:type I (finding a difference or correlation when none exists) andtype II (finding no difference or correlation when one exists). Statistical conclusion validity concerns the qualities of the study that make these types of errors more likely. Statistical conclusion validity involves ensuring the use of adequate sampling procedures, appropriate statistical tests, and reliable measurement procedures.[2][3][4]
The most common threats to statistical conclusion validity are:
Power is the probability of correctly rejecting thenull hypothesis when it is false (inverse of the type II error rate). Experiments with low power have a higher probability of incorrectly failing to reject the null hypothesis—that is, committing a type II error and concluding that there is no detectable effect when there is an effect (e.g., there is real covariation between the cause and effect). Low power occurs when the sample size of the study is too small given other factors (smalleffect sizes, large group variability, unreliable measures, etc.).
Most statistical tests (particularlyinferential statistics) involve assumptions about the data that make the analysis suitable fortesting a hypothesis. Violating the assumptions of statistical tests can lead to incorrect inferences about the cause–effect relationship. Therobustness of a test indicates how sensitive it is to violations. Violations of assumptions may make tests more or less likely to maketype I or II errors.
Each hypothesis test involves a set risk of a type I error (the alpha rate). If a researcher searches or "dredges" through their data, testing many different hypotheses to find a significant effect, they are inflating their type I error rate. The more the researcher repeatedly tests the data, the higher the chance of observing a type I error and making an incorrect inference about the existence of a relationship.
If the dependent and/or independent variable(s) are not measuredreliably (i.e. with large amounts ofmeasurement error), incorrect conclusions can be drawn.
Restriction of range, such asfloor and ceiling effects orselection effects, reduce the power of the experiment, and increase the chance of a type II error.[5] This is becausecorrelations are attenuated (weakened) by reduced variability (see, for example, the equation for thePearson product-moment correlation coefficient which uses score variance in its estimation).
Greater heterogeneity of individuals participating in the study can also impact interpretations of results by increasing the variance of results or obscuring true relationships (see alsosampling error). This obscures possible interactions between the characteristics of the units and the cause–effect relationship.
Any effect that can impact theinternal validity of a research study may bias the results and impact the validity of statistical conclusions reached. These threats to internal validity include unreliability of treatment implementation (lack ofstandardization) or failing to control forextraneous variables.