Decades of survey research have indicated that heterogeneous question wording and response options can be problematic when trying to draw comparisons between studies; we cannot assume construct equivalence between studies using different methodologies. Unfortunately, although much research has been dedicated to establishing more preferable question and response option wordings, a clear consensus on these decisions has not emerged. However, on the basis of this literature we can identify a number of relevant dimensions on which self-report questions can vary, and we can reflect on how each dimension might affect participants’ responses. First, however, it will be useful to consider how respondents generally approach answering survey questions.

How people answer survey questions

According to Krosnick and Presser (2010), respondents go through a four-step process to answer a survey question. In particular, respondents attempt to (1) understand the question, (2) find relevant information in memory on which to base a response, (3) extract a single answer based on this information, and (4) map that answer onto one of the provided response options (assuming that it is not an open-ended question). Some of these steps are more relevant to mind-wandering probes than are others. Since the same question is repeated multiple times during a typical mind-wandering study, and since participants usually engage in a practice phase in which they encounter the question and response options, the first step of understanding the question does not need to reoccur at each subsequent probe. Next, since participants are being asked about the moments immediately preceding the probe, they will not usually need to—or ought not to—search long-term memory for a response, but instead should rely on their working memory to indicate where their thoughts were immediately prior to the probe. Steps 3 and 4, however, are quite relevant to answering a probe-caught mind-wandering question: Participants will need to label the thoughts they were having that immediately preceded the probe, and then they will have to map those thoughts onto one of the options provided. This step of identifying the appropriate response option will necessarily differ depending on which response options, and how many, are provided.

Heuristics and biases that affect responses

Optimizing versus satisficing

The effort with which respondents perform the four steps described above can be mapped onto a dimension ranging from optimizing (i.e., choosing the most accurate response option) to arbitrary responding. Along this scale, respondents will show some degree ofsatisficing (Krosnick,1999), which involves choosing the first response option that seems “good enough” and can occur for a variety of reasons—including both factors related to the task itself (specifically, high task difficulty) and factors inherent to the participants (in particular, low motivation or low ability).

Social desirability responding

Participants—particularly those more likely to want to please the experimenter; for example, older adults—may feel that mind-wandering is undesirable. Thus, one form of satisficing may involve the withholding of off-task thought reports. For this reason, some researchers choose to suppress the term “mind-wandering” from the participant instructions (e.g., Jackson & Balota,2012) or take care to emphasize that mind-wandering is natural and to be expected (e.g., Nagamatsu, Kam, Liu-Ambrose, Chan, & Handy,2013). Although this issue is beyond the scope of this review, it would be interesting to examine how different researchers have operationalized mind-wandering in their participant instructions.

Acquiescence bias

One specific form of satisficing involves de facto agreeing with statements; this is known as “acquiescence,” and it is particularly likely when participants are asked yes/no questions such as “Are you mind-wandering?” or “Is your mind on task?” The size of the acquiescence bias observed in any given mind-wandering study is, of course, impossible to determine without a manipulation of the probe itself, but from previous research, we might expect a difference of about 10% (Krosnick & Presser,2010). Thus, asking “Are you mind-wandering?” could bias participants toward higher rates of self-reported mind-wandering, which would presumably not reflect true differences in actual mind-wandering, relative to when the question is formulated as “Is your mind on task?” (see Weinstein, De Lima, & van der Zee,2017).

Scales

One way to avoid the acquiescence bias that could occur with binary statements is to give participants a scale on which to report their answers (in the case of mind-wandering, this could involve a scale from “on-task” to “off-task”, or “vice versa”). However, scales are also potentially problematic in a number of ways.

Number of items on ratings scale

The research on the optimal number of response options on a Likert (or other) rating scale is incredibly rich and nuanced. Some studies (e.g., Lissitz & Green,1975) have shown greater reliability and validity as the number of response options increases up to a certain point, whereas others (e.g., Bendig,1954) have not shown this relationship. Confusingly, the difficulty of a question may either increase or decrease when the number of response options increases (Krosnick & Presser,2010). A greater number of response options could lead to a more difficult choice of which option to pick. On the other hand, a smaller number of response options could lead to confusion if none of the provided response options accurately capture the answer the participant wants to volunteer. In mind-wandering research, confusion could be particularly salient in the case of binary response options if participants do not experience their thought state as a binary phenomenon.

Neutral alternatives

Scales also differ in terms of whether they include a middle (neutral) alternative—a scale with an odd number of response options will typically include a neutral alternative, whereas binary question formats and even-numbered scales do not. Including a neutral response option may increase satisficing; in the case of probe-caught mind-wandering, this would manifest itself as picking the neutral/middle option instead of expending effort to determine one’s though state. However, neutral response options have been shown to increase reliability and validity, presumably because they prevent participants from choosing response options at random when they truly do not have a preference for one option over the other (O’Muircheartaigh, Krosnick, & Helic,2001).

Ordering of scale options

Regardless of the number of response options provided, researchers must pick an order in which the scale options will be displayed (e.g., fromcompletely on task tonot at all on task). In general, research has converged on the notion that participants tend to exhibit a primacy effect, skewing their responses to the left of a response scale (e.g., Payne,1971). Ideally, therefore, counterbalancing would be used to overcome this biasing effect of scale direction (Krosnick & Presser,2010).

Categorical response options

Instead of a scale fromon-task tomind-wandering, researchers may choose to present participants with a set of categories that can be used to describe their thought state. In this situation, a choice has to be made as to the order of the thought categories presented for selection (in theory, this order could also be counterbalanced, but no mind-wandering researchers have chosen to do this). Satisficing will push respondents toward choosing options higher up on the list, because participants may stop evaluating options once they find an option that adequately describes their thought state.

Methodological issues in probe-caught mind-wandering

A handful of studies have begun to address methodological issues in probe-caught mind-wandering research. Evidence is starting to amass regarding the various biases that may apply to this line of inquiry. Vinski and Watter (2012) found that an honesty prime reduced reports of probe-caught mind-wandering, suggesting that demand characteristics may play a role in the high rates of mind-wandering reported in some studies. In addition, priming participants for honesty strengthened the relationship between mind-wandering and performance, suggesting that the effect was not simply one of criterion-setting. In addition, Seli, Jonker, Cheyne, Cortes, and Smilek (2015) found that participants’ confidence in their self-reported attention state (mind-wandering or on-task) was an important moderator of the relationship between mind-wandering and task performance, with low-confidence attentional-state reports correlating less strongly with performance. The finding by Seli, Jonker, et al. that 16% of the responses fell into the low-confidence category suggests that a subset of thought state reports may be made as guesses. These guesses would be most prone to demand characteristics. In addition, Seli, Carriere, Levene, and Smilek (2013) demonstrated that the rate at which probes are presented during a sustained-attention task can drastically affect the rates of reported mind-wandering, with longer interprobe intervals resulting in much greater reported rates of mind-wandering. The authors concluded that this pattern stemmed from response bias and not from a true fluctuation in mind-wandering rates caused by probe frequency, because objective measures they also employed did not fluctuate with probe frequency.

In this review, I have documented the different probe framings and specific wordings of probes and response options used in probe-caught mind-wandering studies published between 2005 and 2015. The goal of this exercise was to provide a comprehensive overview of the different methodological variants that currently exist in the literature for measuring probe-caught mind-wandering.

Criteria

I used three criteria to select studies for inclusion in my review. The first criterion for inclusion was that the researchers must have collected probe-caught mind-wandering data. That is, studies that measured only self-caught mind-wandering, where the participant had to spontaneously indicate a mind-wandering episode (e.g., Grandchamp, Braboszcz, & Delorme,2014), were not included. The second criterion for inclusion was that the probe must have targeted on-line mind-wandering. That is, participants must have been asked to reflect only on the moments just before the probe when reporting whether they were mind-wandering; studies in which participants were asked to reflect back on a longer period, such as a block of trials, were not included (e.g., Farley, Risko, & Kingstone,2013; Stawarczyk, Cassol, & D’Argembeau,2013). Finally, participants must have been forced to classify their mind-wandering (with a binary, category, Likert, or continuous-scale judgment) rather than freely reporting the contents of their thoughts (e.g., Baird et al.,2011). Experience-sampling studies outside the lab (e.g., McVay, Kane, & Kwapil,2009) were included as long as they involved questions about participants’ current attentional state just prior to the occurence of an experience-sampling probe.

Sources

I gathered peer-reviewed journal articles (excluding published conference proceedings) from three sources. First, I systematically examined every article in the most recent review of mind-wandering, by Smallwood and Schooler (2015). From that review I identified 89 studies across 66 publications. Next, to catch other relevant articles that may have not been cited in the review, I searched Google Scholar for articles with the term “mind-wandering” (also “mind wandering” and “mindwandering”) in the title, published during the period from 2005 to August 2015, when I performed the last search. This search yielded an additional 55 studies across 38 publications. Finally, I looked at all of the articles in the Frontiers Research Topic “Towards a Psychological and Neuroscientific Account of the Wandering Mind,” from which I added one final study (the rest had all been picked up using Methods 1 and 2). The review below thus includes 145 studies reported across 105 journal articles. The “Found in” columns of Tables 1a,1b,2a,2b,3a,4a,4b,5a,5b and6 provide information on the source(s) that included each publication.

Information gathering

After identifying the 145 studies of interest, I next examined the methods used in each study. Two of the studies (Franklin, Mrazek, et al.,2014 [experience sampling], and Stawarczyk, Majerus, Van der Linden, & D’Argembeau,2012) were excluded because they did not report any new mind-wandering data. Tables1a,1b,2a,2b,3a,4a,4b,5a,5b, and6 present all of the examined studies, with information regarding the exact wording of the probe and response options. In many cases, details regarding the exacting wording were missing from the publication. To get these details, I contacted the first author of each relevant publication. Information was missing for 52 of the 145 studies; after contacting the authors, I was able to obtain information for all but five of the studies (see Table10).

I discovered five distinct response option methodologies in use across the studies: neutral (or counterbalanced), dichotomous (“yes/no”), dichotomous (“on-task/mind-wandering”), categorical, and scale. With the exception of the neutral methodology (Tables 1a and1b) and the unclassifiable studies (Table6), the remaining four methodologies could be further broken down into those that anchored the participants on-task and those that anchored the participants off-task. Thus, for Tables 2,3,4, and5 versions a and b of each table include the studies that anchored on-task and off-task, respectively.

The breakdown in terms of the number of studies that fell into each type of methodology is presented in Fig. 2. Only one of these five methodologies involved full probe neutrality and/or counterbalancing of response options. A probe was considered neutral if it referenced neither an on-task nor an off-task thought state, such as in the probe “What were you thinking just now?” (Forster & Lavie,2009,2014). Furthermore, for the methodology to be truly neutral, counterbalancing of the response options must also have been employed. That is, the order of the response options or the direction of the scale must have been counterbalanced between participants, as in Christoff, Gordon, Smallwood, Smith, and Schooler (2009), in which “subjects answered using a 7-point Likert scale, ranging from ‘completely on task’ to ‘completely off task’…(with the scale direction counterbalanced across subjects)” (p. 8720). Eight studies across six different articles and four different labs met both of these criteria (Christoff et al.,2009; Thomson, Besner, & Smilek,2013; Thomson, Seli, Besner, & Smilek,2014; Thomson, Smilek, & Besner,2014; Ward & Wegner,2013; Ye, Song, Zhang, & Wang,2014; see Table1a). Five additional studies across four different articles and three different labs (Dixon & Bortolussi,2013, Exp. 1; Dixon & Li,2013; Killingsworth & Gilbert,2010) did not counterbalance the order of the response options or the direction of the response scale, but instead the response options were presented in the opposite order from how they had been presented in the probe, making it unclear which of the two states (on-task or off-task) was emphasized (see Table1b).

Probe-caught mind-wandering studies by category and subtype. This figure summarizes the information presented in more detail in Tables 1a,1b,2a,2b,3a,4a,4b,5a,5b and6

Full size image

The remaining four methods of collecting mind-wandering data were biased to some extent toward one of the two attentional states (on-task or off-task). In some cases, the probe mentioned one of the two attentional states and asked participants to endorse or deny the state by answering either “yes” or “no,” respectively. For example, Deng, Li, and Tang (2014) asked participants, “Did you focus on the task just now?”; analogously, Krawietz, Tamplin, and Radvansky (2012) asked participants, “Are you mind wandering?” However, a comparison of Tables 2a and2b reveals that the latter—asking participants whether they were off-task—was a much more common methodology than asking participants whether they were on-task; in only two studies were participants asked whether they were on-task (Table2a), as compared to 29 studies in which participants were asked whether they were mind-wandering (i.e., off-task; Table2b).

Another very common methodology was to ask participants to choose between two options: on-task or off-task (mind-wandering). Twenty-five studies, summarized in Table3a, posed the question in this way. It is important to note that the order of the response options (on-task vs. mind-wandering) was consistent across these 25 studies; in not a single study were participants asked whether they were “mind-wandering or on-task” (reversing the order; hence there is no Table 3b). This is interesting when considered alongside the yes/no method described in the previous paragraph, which revealed a strong preference for asking participants whether they were mind-wandering (Table2b) rather than whether they were on-task (Table2a).

Another set of studies went beyond the dichotomy of yes/no or on-task/off-task response options, by allowing participants to choose between a set of categorical options to describe their attentional states. There was a great deal of variability in terms of the number and nature of these options. The number of options ranged from three to nine and sometimes referred to the specific nature of the thoughts—for example, temporal context (“past” vs. “future”; Smallwood, Nind, & O’Connor,2009) or intentionality (INTENTIONALLY MIND WANDERING vs. UNINTENTIONALLY MIND WANDERING; Seli, Cheyne, Xu, Purdon, & Smilek,2015). In this set of studies, there was once again a strong preference for listing the on-task option first and off-task last: 42 studies listed the on-task option first (Table4a), whereas only two studies listed mind-wandering first (Table4b).

The fourth general method involved letting participants respond about their attentional state on a scale from on-task to off-task, or vice versa. These scales usually involved a number of response options, ranging from four to nine, although Macdonald, Mathan, and Yeung (2011) used a continuous vertical scale. In this method, as in the dichotomous and multiple-category methods, the on-task option was once again most commonly used as the anchor: In 25 studies the on-task option was presented to the left of the Likert scale (Table5a), whereas in only two studies was the opposite the case (Table5b).

In some cases I was unable to classify the bias of a particular study, because precise information was not available in the article and the authors did not respond to requests for clarification (see Table6). An example of this is Smallwood, McSpadden, and Schooler (2008), in which the probe read “Just prior to being asked, was your attention on- or off-task?” but the response options, as presented in the article, read “tuning out,” “zoning out,” and “on task”—but it was not clear whether this order or the order in the probe was actually presented to participants.

Another way to look at this data set is to identify the number of unique methodological variants employed by the field. A conservative analysis yielded 69 such variants; Fig. 3 summarizes the proliferation of different methodological variants by plotting their cumulative numbers, as well as the numbers of different articles and of studies within those articles, over time.

Cumulative experiments, articles, and methodological variants by publication year. Note that this is a conservative estimate; in cases in which the response options differed subtly but the methodology was functionally similar, the variants were not separated. An example of this is where participants were asked to choose between “on-task” and “off-task” or between “on-task” and “mind-wandering.” This figure includes only 132 of the 145 studies in this review because insufficient information was provided for 13 of the studies to determine whether a new or existing methodological variant was used

Full size image

What methodological observations can be extracted from this compilation of a decade’s worth of work on probe-caught mind-wandering? Perhaps due to the immense interest in the field of mind-wandering and the speed at which has expanded, with many labs jumping on the bandwagon, there has been no standardization of the probe-caught method for collecting mind-wandering. As a result, there have been almost as many different probes and response options as there have been studies. Within each of the distinct categories I described above, variation in the specific wordings of both probes and response options has been abundant; this is particularly striking in the categorical methodology, in which a wide range of semantic categories have been used for thought classification, depending on the authors’ specific research questions.

Although this methodological heterogeneity ought to be addressed so that the field can reach a consensus on how to measure probe-caught mind-wandering most accurately, it could also be seen as a strength of the field. If the same theoretical conclusions can be drawn from studies that measure mind-wandering in a variety of ways, the conclusions are more likely to be accurate. For example, one recurring phenomenon is the negative correlation between probe-caught mind-wandering and task performance on a sustained-attention task. This relationship has been observed with at least three different probe-caught mind-wandering methodologies—binary (e.g., Seli et al.,2014), categorical (e.g., Seli, Cheyne, et al.,2015), and scale (e.g., Seli et al.,2014). Similarly, mind-wandering during study has also been found to correlate negatively with later memory performance, and this relationship has also been observed when mind-wandering was measured with binary response options (e.g., Risko et al.,2012), categorical response options (e.g., Storm & Bui,2015), and scales (e.g., Mrazek, Franklin, Phillips, Baird, & Schooler,2013). The fact that these relationships can be found consistently with a variety of methodologies lends credence to the observed effects.

One striking pattern that can be observed from Fig. 2 is the strong contrast between the binary, yes/no studies (Tables 2a and2b) and all other methodologies. That is, the vast majority (94%) of the yes/no studies asked participants whether they had been mind-wandering (rather than whether they had been on-task). On the other hand, in the remaining methodologies, in the majority of cases (96%) participants were asked whether they had been on-task, versus in another attentional state. Why might this framing matter? Indeed, as of right now no published data directly address how probe framing affects mind-wandering reports (Weinstein et al.,2017). However, we might look to data from other domains in which framing has been manipulated. Seminal work by Tversky and Kahneman (1981) demonstrated that whether decisions are framed positively (in terms of gains) or negatively (in terms of losses) impacts decision-making. More recently, Finn (2008) extended the framing paradigm to metacognitive judgments of memory, showing that people are less likely to be overconfident when they judge their learning in terms of future forgetting rather than future remembering. In addition, the honesty prime study by Vinsky and Watter (2012) suggests that demand characteristics affect not only the levels of reported mind-wandering, but also the relationship between mind-wandering and accuracy measures.

Recommendations

On the basis of this review, I have three broad recommendations: (1) Those intending to conduct probe-caught mind-wandering studies should familiarize themselves with the different methodological variants already employed in the past decade (see Tables 1a,1b,2a,2b,3a,4a,4d,5a,5b and6) and make an informed choice for their own experiments; (2) when making these decisions, mind-wandering researchers should also acquaint themselves with relevant survey research literature (e.g., Krosnick & Presser,2010) and follow the recommendations from that literature, as detailed below; and (3) further research should attempt to determine whether framing effects are a concern in the probe-caught mind-wandering paradigm (Weinstein et al.,2017).

In addition to these broad recommendations, more specific recommendations can be made on the basis of the survey research discussed earlier in this article. Mind-wandering researchers should be aware that the following potential sources of bias could arise from their choice of methodology:

1. 1.

   Binary yes/no probes may lead to acquiescence, such that participants will tend to default to a “yes” response (Krosnick & Presser,2010). In fact, in a forthcoming article (Weinstein et al.,2017) we show that simply changing the mind-wandering probe from “At the time of the ding, my mind was on the text” to “At the time of the ding, my mind was on something other than the text” increased mind-wandering reports by 10%.

2. 2.

   If a scale is used, researchers should think carefully about the number of options provided: Having too few response options can encourage random responding, because participants will not find an option that accurately reflects their desired answer, whereas an overabundance of response options can make the task too difficult (Krosnick & Presser,2010).

3. 3.

   The order of options on a given scale should ideally be counterbalanced between participants (e.g., as in Christoff et al.,2009) to prevent anchoring (Payne,1971) to the leftmost option.

4. 4.

   If categorical response options are used, they should not be too numerous, and they also ought to be counterbalanced to avoid order effects.

5. 5.

   Researchers should be particularly careful when conducting probe-caught mind-wandering research with vulnerable populations, including older adult, children, and participants with lower ability, who may be more prone to satisficing (Krosnick, Narayan, & Smith,1996), especially in combination with fatigue (Clancy & Wachsler,1971).

6. 6.

   Be aware that social desirability may push participants to report on-task thoughts more or less often, depending on whether mind-wandering is presented as being positive or negative (Parry & Crossley,1950).

In the present review I identified, by a conservative estimate (see Fig. 3), 69 different methodological variants of probe-caught mind-wandering among 145 studies. In other words, almost one in every two new studies has used probe or response option wordings that have never been reported in the literature. Moreover, this methodological review has focused only on the probe and response option wordings; further differences would have emerged if I had taken into account the ways that mind-wandering was defined for participants. The lack of methodological consensus in this field may have emerged in part due to a lack of open communication between researchers in terms of their methodological choices. Hopefully the present review will enable mind-wandering researchers to make more informed choices about how they measure probe-caught mind-wandering.