Investigating the Effects of Task Characteristics and Educational Interventions on Intuitive Statistical Biases

Abstract

Past research indicates people have some capacity to intuitively (i.e., informally) detect, estimate, and apply information derived from important data characteristics (e.g., mean, sample size, SD) in various statistical judgment tasks. This research has also found that people are generally capable of making intuitive between-group comparisons, which involve comparing groups of data and making judgments on these data. However, people also tend to exhibit a bias in such comparisons against normatively integrating the withingroup variability (e.g., standard deviation; SD) of the groups with the between-group variability (e.g., mean differences) while making between-group comparisons. This can lead to errors in judgment, because accurate inferences about any differences between the groups requires considering the between-group variability in relation to the within-group variability. Few studies have investigated targeted educational interventions for overcoming this intuitive statistical bias. These studies have tended to be based on incorporating targeted educational material specific to the bias in a semester-long introductory statistics course. A brief laboratory-based educational intervention would allow for a more tightly controlled experiment with a larger sample size. For this purpose, a brief computer-delivered educational intervention was developed to offer subjects practice and feedback opportunities while engaging in between-group comparison tasks. These experimental learning condition tasks, and associated assessment tasks, were presented in the context of making judgments between products based on customers’ product ratings. The ratings for the groups were displayed visually on frequency distribution graphs. Practice opportunities consisted of allowing subjects to manipulate relevant statistical characteristics of the between-group comparisons (i.e., the mean ii difference, SD, sample size), one-at-a-time. Feedback opportunities provided subjects information on how many manipulations of the statistical characteristic being currently focused on would be necessary for there to be a significant difference between the groups. A series of three experiments using a pretest–posttest design were run to test the hypotheses that: (a) subjects will exhibit a pretest bias against normatively integrating within-group variability into their between-group comparisons; (b) that subjects receiving practice, feedback, or both opportunities would show greater improvement on their between-group comparisons at posttest than those in a control condition; and (c) that subjects receiving both practice and feedback will show greater improvement at posttest than those who received either practice or feedback alone. Subjects were randomly assigned to receive practice and/or feedback, or to a control condition with neither practice nor feedback. The dependent measures for the experiments included: (a) novel between-group comparison tasks designed for these experiments (Forced-Choice Task; used in all three experiments; Strength-Of-Evidence Task, used in Experiment 1), which displayed the data for the groups visually on frequency distribution graphs; and (b) a between-group comparison task that has been shown to elicit the bias in previous research (Intuitive ANOVA task, used in Experiments 2 and 3). The findings of these experiments revealed that subjects exhibited the bias strongly when measured with the Intuitive ANOVA task but that the bias was either extremely diminished, or unobservable, when measured with the Forced-Choice Task or the Strength-Of-Evidence Task, consistent with recent findings by other researchers. These experimental learning conditions were also found to have no reliable effect on subjects’ performance on any of the dependent measures. Implications, limitations, and future directions are discussed.