The effects of study conditions on exercise

Our megastudy included a placebo control condition in which participants received 1,500 points when they enrolled in the study (worth US$1.08 when redeemed athttps://www.amazon.com, an amount equal to the expected earnings of participants in a typical experimental condition; see the ‘Descriptions of the 54 conditions in the megastudy’ section of theSupplementary Information). Participants in the placebo control condition received no other intervention content.

We also included a baseline intervention called planning, reminders and microincentives to exercise. This condition combined three low-cost, evidence-based components that are expected to increase exercise. First, as past research has shown that planning prompts facilitate follow-through^21–23, we prompted participants to plan the dates and times when they would exercise each week of the programme. Second, as reminders have been shown to enhance goal achievement²⁴, we texted participants reminders to exercise at these scheduled times. Finally, building on past work showing that cash rewards for exercise that are an order of magnitude larger than this can promote gym attendance^3–6 and that the effects of very small incentives on goal commitment can be surprisingly large²⁵, we offered participants microincentives for each gym visit (300 points per visit, redeemable for approximately US$0.22).

The other 52 experimental conditions in our megastudy augmented this planning, reminders and microincentives to exercise condition by adding new features (Supplementary Table 1).

Compared with the placebo control condition, 45% of the 53 experimental conditions tested in our megastudy produced a statistically significant (two-sidedP < 0.05) increase in an ordinary least squares (OLS) regression model predicting weekly gym visits during our four-week intervention (significantP values range from 2.39 × 10⁷ to 0.045;Fig. 1a andExtended Data Table 6 present these regressions;Table 2 shows the percentage of other treatments each experimental condition outperformed). InExtended Data Table 7, we present parallel analyses of whether study participants attended the gym at least once per week, and we found that, compared with the placebo control condition, approximately 34% of the experimental conditions had significantly more people visiting the gym at least once per week.

Rather than adjusting ourP values for 53 paired comparisons, we report unadjusted standard errors, two-sidedP values and confidence intervals (CI) so readers may choose a preferred correction. Using the Storey–Tibshirani method of computing the false-discovery rate²⁶, we estimate that the results identified as significant at the 5% level have less than a 5.07% chance of being a true null. The 45% of our experimental conditions that increased gym visits produced an estimated 0.14 to 0.40 extra weekly gym visits during the four-week intervention period (the CI lower bounds range from 0.004 to 0.21 and the CI upper bounds range from 0.23 to 0.59), increasing exercise by an estimated 9% to 27% compared with the placebo control condition, in which participants visited the gym a mean of 1.48 times per week during the intervention period. No treatment significantly reduced gym visits. Furthermore, anF-test enables us to reject the null hypothesis that all 53 treatment effects have the same true value (F = 1.392,P = 0.032).

The planning, reminders and microincentives to exercise condition produced an estimated 0.14 more weekly gym visits per participant (a 9% increase in exercise) compared with the placebo control condition (b = 0.14, 95% CI = 0.04–0.23,P = 0.006).

All of the 24 treatments that significantly increased exercise in comparison to the placebo control condition included planning, reminders and incentives to exercise, typically with an additional nudge or reward to visit the gym (Fig. 1). Five of these experimental conditions stood out, significantly outperforming the planning, reminders and microincentives condition according to Wald tests comparing the estimated treatment effects. As some effect-size estimates had wider confidence intervals than others, these five conditions were not exactly the same as the five conditions with the largest estimated effect sizes shown inFig. 1. The conditions in question are presented inTable 1 with their estimated effects on exercise. Note that the criteria used for their selection (that they are the top performers in a distribution) mean that these estimated treatment effects are probably inflated.

As shown inTable 1, we found that rewarding participants with a bonus of 125 points (US$0.09) for returning to the gym after a missed workout produced an estimated 0.40 more weekly gym visits per participant (a 27% increase in exercise) compared with the placebo control (b = 0.40,P < 0.001). This condition produced a 16% increase in exercise relative to planning, reminders and microincentives (b = 0.27,P = 0.010). Second, offering participants larger incentives (that is, 490 points per gym visit, or US$1.75) produced an estimated 0.37 more weekly gym visits per participant (a 25% increase in exercise) compared with the placebo control (b = 0.37,P < 0.001). This condition produced a 14% increase in exercise relative to planning, reminders and microincentives (b = 0.23,P = 0.020). Third, telling participants that the majority of Americans exercise and the fraction is growing produced an estimated 0.35 more weekly gym visits per participant (a 24% increase in exercise) compared with the placebo control (b = 0.35,P < 0.001). This condition produced a 13% increase in exercise relative to planning, reminders and microincentives (b = 0.21,P = 0.020). Fourth, rewarding participants with a bonus of 225 points (US$0.16) for returning to the gym after a missed workout produced an estimated 0.34 more weekly gym visits per participant (a 23% increase in exercise) compared with the placebo control (b = 0.34,P < 0.001). This condition produced a 12% increase in exercise relative to planning, reminders and microincentives (b = 0.20,P = 0.022). Fifth, allowing participants to choose whether their rewards for gym visits would be framed as gains (such that they would earn points each day that they visited the gym) or losses (such that they would lose points each day that they did not visit the gym) produced an estimated 0.28 more weekly gym visits per participant (a 19% increase in exercise) compared with the placebo control (b = 0.28,P < 0.001). This condition produced a 9% increase in exercise relative to planning, reminders and microincentives (b = 0.15,P = 0.021). Note that, in different conditions, points had different cash values (Supplementary Table 1).

Enduring effects of study conditions

Although 45% of the experimental conditions in our megastudy outperformed the placebo control condition during our four-week intervention, only 8% produced significant increases in the frequency of gym visits during the four weeks post-intervention, compared with 2.5% that would be expected to do so by chance (Extended Data Table 9). AnF-test enabled us to reject the null hypothesis that all 53 treatments have null effects beyond the treatment period (F = 1.418,P = 0.024).

Focusing on the 45% of interventions that outperformed the placebo control during the four-week intervention period, each extra gym visit that was generated during the four-week intervention period corresponded to between −0.07 and 0.76 extra gym visits during the ten weeks post-intervention (median = 0.354 extra gym visits post-intervention, 25th percentile = 0.085 extra gym visits post-intervention, 75th percentile = 0.522 extra gym visits post-intervention;Supplementary Table 5). We also pooled data from these interventions into a single category and estimated that they generated a mean of 0.30 extra gym visits during the 10-week post-intervention period for every additional gym visit that they produced during the four-week intervention (skew-corrected 95% CI=0.13–0.54; seeSupplementary Information 3 for details). These post-intervention returns are consistent with those from previous studies of gym attendance and habit formation^3–6, in which analogous returns range from 0.16 to 0.46 extra gym visits post-intervention for every extra gym visit induced during the intervention (Supplementary Table 5).

By selecting on the basis of those interventions that increased exercise significantly during the four-week intervention period, we focused on experimental conditions that will be of the greatest interest to policy makers, but we also probably overstate their post-intervention effects due to the winner’s curse. To address this, we pooled data from all 53 experimental conditions into a single category. We estimate that interventions in our study generated a mean of 0.28 extra gym visits during the 10-week post-intervention period for every additional gym visit that they produced during the four-week intervention (skew-corrected 95% CI = 0.07–0.59).

Prediction accuracy

One could argue that the harder it is to predict the results of experiments, the more valuable the megastudy approach. The more difficult it is to forecast ex ante which interventions will work, the harder it is to decide in advance which interventions to prioritize for testing, and the more useful it is to instead test a large number of treatment approaches.

To assess forecasting accuracy, we conducted a series of separate preregistered studies (see the ‘Data availability’ section) in which third-party observers were asked to predict the impact of three randomly selected interventions from our megastudy. We collected these data 14 months after conducting our megastudy. One study included 301 participants recruited from Prolific (who made a total of 903 predictions, or a mean of 17 predictions per treatment condition); another included 156 professors from the top 50 schools of public health as rated by U.S. News & World Report in 2019 (who made a total of 468 predictions, or a mean of 9 predictions per treatment condition; a list of schools is provided inSupplementary Information 11); and a final study included 90 practitioners recruited from companies that specialize in applied behavioural science (who made a total of 270 predictions, or a mean of 5 predictions per treatment condition). See the ‘Prediction study participants’ section in the Methods for demographic information about the study participants.

We found no robust correlations (weighted pooledr = 0.02,P = 0.89) between these populations’ estimated treatment effects and observed treatment effects (Prolific participantsr = 0.25,P = 0.07; professors’r = −0.07,P = 0.63; practitionersr = −0.18,P = 0.19). Furthermore, predictions about the benefits of our interventions were a mean of 9.1 times too optimistic (Fig. 1b). Predictions of treatment effects for our secondary dependent variable–the likelihood of making a gym visit in a week–were similarly inaccurate and are presented inSupplementary Information 11.

Taken together, these results highlight how difficult it is to predict ex ante the efficacy of interventions and why it is therefore so valuable that megastudies enable the synchronous testing of many different approaches to changing behaviour.

Ethics approval

The Institutional Review Board at the University of Pennsylvania approved our study’s protocols, and this research was deemed to comply with all of the relevant ethical regulations. Informed consent was obtained from all of the study participants as part of the enrolment process. The reference number for the field experiment was 827107 and the reference number for the prediction accuracy studies was 833336.

Megastudy setting

We conducted our megastudy in partnership with 24 Hour Fitness, one of the largest gym chains in the United States. At the time of the study, 24 Hour Fitness had over four million members and 450 gym locations in 14 states (although some members of 24 Hour Fitness reside in states without a 24 Hour Fitness location, so our study participants came from more than 14 US states). The cost of a basic membership at 24 Hour Fitness varies by location, but ranges from approximately US$30 to US$60 per month. Members check in to 24 Hour Fitness gyms by either (1) giving their ID to a staff member at the front desk, (2) swiping or scanning a member card or (3) using a fingerprint reader and unique check-in code. We used 24 Hour Fitness check-in data to track gym attendance.

Participant recruitment and enrolment

All of the approximately 4 million adult members of 24 Hour Fitness gyms whose memberships were active between 21 March 2018 and 31 January 2019 were eligible to participate. Recruitment involved a multichannel marketing campaign advertising “a habit-building, science-based workout program” called StepUp, and 24 Hour Fitness members could sign up online anytime between 21 March 2018 and 31 January 2019. All of the recruitment materials informed members that they could sign up for free for the StepUp Program and earn Amazon cash rewards for exercising. Members were also told that they would earn a chance to receive a US$50 Amazon gift card by simply registering for the programme. Three participants were randomly selected to receive a US$50 gift card.

All of the recruitment materials included a URL that directed gym members to the StepUp Program website, which conveyed that StepUp was a 28-day digital experience being offered exclusively to 24 Hour Fitness members. Participants who visited the StepUp Program website were first prompted to consent to participate in research. Participants then provided their gym check-in code and date of birth to verify their gym membership. Finally, participants were prompted to provide their name, email address and phone number, and they were required to verify that their phone could receive text messages from StepUp (details are provided in the ‘Registration experience’ section of theSupplementary Information). After verifying that they could receive text messages, the participants were randomly assigned to one of twenty different preregistered substudies (all involving different versions of the StepUp Program) aimed at increasing gym visit frequency, and they were then randomly assigned to one of the 54 different experimental conditions within these studies. Participants were blind to study hypotheses.

Our initial, preregistered recruitment goal was to include at least 3,000 participants per experimental condition in our megastudy. However, shortly after launching recruitment, it became apparent that this would take nearly a decade. As a consequence, we updated our preregistrations early on in the 10 month study to reflect a more realistic stopping rule of recruiting at least 400 participants per condition.

In total, 62,746 participants were randomized to one of the 54 study conditions in our megastudy, with at least 455 participants in each condition (Extended Data Table 4). Participants were excluded from analyses if they requested to withdraw (n = 123), signed up more than once for the StepUp Program (n = 355) or experienced severe technology glitches (n = 975). Further details about these exclusions are provided inSupplementary Information 9 and10.

Thus, our final sample includesn = 61,293 study participants. 24 Hour Fitness shared a record of every gym visit made by study participants starting one year before each participant’s enrolment in the programme and continuing until one year after each participant’s programme participation concluded (for a total of 758 d of observations per participant).

As reported inExtended Data Table 6 andSupplementary Information 1 and7, balance checks suggest that randomization was successful. As we obtained informed consent to analyse data on study participants only, we unfortunately cannot determine how representative our final sample is of the 24 Hour Fitness membership.

Megastudy intervention content

After enrolling, participants in all 54 conditions of our megastudy were shown descriptions of the StepUp Program. All of the participants learned that they would receive points during the intervention period that were redeemable for an Amazon gift card after they completed the intervention. Participants in the 53 experimental conditions (that is, every condition except for the placebo control condition) received 100 points for registering and learned how they could earn incentives (through points that were redeemable for an Amazon gift card at the conclusion of the programme; notably, the conversion rate differed by experimental condition). Most conditions awarded points for gym visits. A number of the conditions offered additional bonuses based on the time of a participant’s gym visit or other observable behaviours (such as responding to text messages). Complete information about study stimuli and incentives in each condition is provided in the ‘Descriptions of the 54 conditions in the megastudy’ section of theSupplementary Information.

In 53 experimental conditions (all of the conditions except for the placebo control condition), the participants were prompted to create a weekly schedule of the days and times that they planned to work out during the four-week programme. The registration experience for the experimental conditions also included other content specific to the study condition (such as survey questions, instructions, images and videos). At the conclusion of the registration experience, all of the participants were informed that their four-week programme started the next day.

Participants across all 54 study conditions received a welcome text message shortly after they completed enrolment confirming the points that they received for registering, as well as a final text message on the last (28th) day of the programme confirming the programme’s end.

In all 53 experimental conditions, the participants received workout reminders by text 30 min before each scheduled workout (the language of these texts varied across conditions); most of the experimental conditions included additional text messages reinforcing intervention content. Moreover, the participants in all 53 experimental conditions received an email shortly after registration and once a week thereafter for four weeks. Each email confirmed the workout schedule that they had created and reinforced study-specific content.

The simplest experimental condition was the planning, reminders and microincentives to exercise condition. This condition included components that have previously been shown to increase exercise–prompts to plan workouts, reminders to exercise at planned times and microincentives for gym visits⁶. The study participants in this condition were prompted to create a weekly workout schedule after registering for StepUp. Over the next four weeks, the participants received text message reminders before each scheduled gym visit, weekly emails containing their workout schedules and 300 points (worth a total of US$0.22) each time they visited the gym that were redeemable for an Amazon gift card at the conclusion of the study.

To develop our study’s 52 other experimental conditions, members of an interdisciplinary group of 34 scientists who study behaviour change were invited to independently submit designs (‘tournament’ entries) along with additional collaborators of their choosing, and submissions were then revised in partnership with the project’s principal investigators (a process that required extensive coordination). The first and last author invited all of the scientists affiliated with the University of Pennsylvania’s Behaviour Change for Good Initiative (BCFG) to contribute submissions, and the 23 affiliated scientists who submitted study designs brought 13 of their own collaborators and graduate students to the project.

The participants in the placebo control condition received 1,500 points (US$1.08) when they signed up for our programme. This value was equivalent to the expected earnings of participants in our planning, reminders and microincentives condition, which was determined by calculating the mean historical gym attendance of the 24 Hour Fitness members and the point values that participants would earn in the planning, reminders and microincentives condition if they attended the gym at this frequency (100 points for registering and 300 points per gym visit × 1.17 expected gym visits per week for 4 weeks = 1,500 expected points). The participants in the placebo control condition did not create a workout schedule or receive any additional intervention content.

The other 52 experimental conditions in our megastudy involved augmentations to our planning, reminders and microincentives to exercise condition designed by scientists affiliated with BCFG. Scientists were invited to vary the (1) online registration experience delivered immediately after participants completed study enrolment, (2) text messages and emails sent during the four-week programme and (3) incentives for activities completed during the programme.

Megastudy randomization

The 54 conditions in our megastudy comprised 20 separate preregistered studies (links to all study preregistrations are provided in the ‘Full descriptions of each study condition’ section of theSupplementary Information). To offset the risk of underpowering all studies if we failed to reach our recruitment targets, megastudy participants were randomized using a weighted, time-varying algorithm as follows. At any given time, the plurality of participants (40–60%) was assigned with equal probability to conditions within one of the 20 studies noted above (the target study), 5% of participants were assigned to our placebo control condition and the remaining participants were randomly assigned with equal probability to treatment conditions in the other 19 studies. The randomization algorithm switched to a different target study after a predetermined number of participants enrolled, and this happened 26 times, creating 27 megastudy ‘stratification cohorts’. Our data analyses are weighted to account for these 27 different stratification cohorts, as described below. More details on randomization weighting are included inSupplementary Information 8.

Megastudy statistical analysis

Each of the 20 studies in our megastudy was preregistered on the Open Science Framework (details are provided in the ‘Data availability’ section). For analyses of our megastudy, we scaled up our standard, preregistered regression analysis strategy (including all of the study conditions in one giant regression model) to identify which of the 53 conditions across all 20 preregistered studies increased the frequency of gym visits during our intervention relative to our placebo control condition.

Although all 20 of the substudies in this megastudy were preregistered, the megastudy itself was not. This was an oversight on our part. We had planned to publish analyses on the totality of preregistered substudies within our megastudy, which is why we used a weighted random assignment scheme rather than sequential random assignment. Preregistering the individual substudies obviated concerns about selective inclusion of treatment arms in substudy analyses. We recommend that future megastudies are preregistered themselves.

To identify which experimental conditions were effective at increasing the frequency of gym visits during our megastudy’s four-week intervention period, we evaluated the mean estimated effect of each of the 53 experimental conditions compared with the placebo control condition. We used OLS regressions and weighted observations to account for the different probabilities of assignment across stratification cohorts.

Specifically, we used an OLS regression with participant fixed effects to estimate the following equation:

whereY_ict is the outcome (that is, gym attendance) of participanti from stratification cohortc in weekt,α is a constant,$d_{it}^g$ is an indicator for both whether participanti is in experimental conditiong and whether weekt is during the intervention period,β^g is the effect of experimental conditiong during the intervention period,δ_ct is a cohort-by-week fixed effect,v_i is a participant fixed effect andε_ict is a random error term.G is the number of treatment conditions in the analysis (53 when estimating the treatment effect of experimental conditions relative to the placebo control reference group). We estimate the cohort-by-week fixed effects by including cohort-by-week indicator variables in the regression. To account for clustering, we estimated cluster-robust standard errors that allowed for arbitrary correlations of the error term within individuals over time³¹. This regression estimates the treatment effect of experimental conditiong relative to the reference group (either the placebo control, or the planning, reminders and microincentives treatment) across all of the cohorts. Participant fixed effects are not collinear with the indicators for whether an individual is in an experimental condition during the intervention period ($d_{it}^g$) because even though each individual can be in only one condition (which would normally create collinearity) our model includes data on participants’ preintervention gym visits for up to 52 weeks (fewer weeks are included when fewer are available for new gym members).

To adjust for the compositional differences across cohorts, we weighted each observation such that each condition is equally weighted within a cohort, and each cohort is weighted proportionally to the length of the cohort in days. This weighting, along with the inclusion of individual and cohort-by-week fixed effects described above, accounts for differences in cohort assignment and seasonality and ensures that our regression produces unbiased estimates of treatment effects. By design, the probability of assignment to each study condition differs by cohort, which would produce unbalanced estimates without the use of sample weighting and fixed effects in our regression specification. Thus, we included sample weights that ensure that, for each cohort, each experimental group is equally represented such that the estimates are equivalent to those from an experiment with equal probabilities of assignment and are therefore balanced estimates. Furthermore, to control for chance imbalances and improve statistical precision, our models include individual fixed effects and cohort-by-week fixed effects. As cohorts were determined by when participants signed up for the StepUp Program, these fixed effects should absorb any remaining seasonal variation in gym attendance. Our simulations, which are presented in the ‘Simulation to ensure validity of analyses’ section of theSupplementary Information, show that this approach yields unbiased estimates of the mean treatment effects and our balance tests reveal that experimental groups do not systematically differ in ways that could lead to biases in our estimates (details about our weighting strategy are provided inSupplementary Information 8). We rely on this statistical analysis strategy for additional regression analyses presented inSupplementary Information 5 and6.

Approximately 6.6% of the megastudy participants were not assigned to the experimental condition that they were intended to experience according to a predefined randomization matrix due to a bug that manifested when there was heavy traffic on our website (leading occasional skips or repeats in the conditions to which subsequent participants were assigned). Our weighting accounts for this error because it is based on the number of people who were actually assigned to each condition within a cohort, rather than the number of people to whom we intended to assign each condition within a cohort. Analyses based on the intended condition assignment are provided in theSupplementary Information (seeSupplementary Information 5a–g for robustness checks) and provide very similar results to those presented here.

In addition to estimating treatment effects during the four-week StepUp Program, we also estimated treatment effects during the four-week post-intervention period. To measure the mean estimated effect of experimental conditions on post-intervention gym attendance, we ran a similar regression with an additional indicator term for the post-intervention period:

Here,$p_{it}^g$ is an indicator for whether participanti is both in experimental conditiong and the weekt is during the four-week post-intervention period,${\beta }_1^g$ is the mean effect of experimental conditiong during the intervention period,${\beta }_2^g$ is the mean effect of experimental conditiong during the four-week post-intervention period and all of the other variables are as defined above.

Across all analyses, to identify the most effective interventions, we conducted Wald tests to compare effects across all of the experimental conditions. Specifically, each Wald test assessed the null hypothesis that the estimated treatment effect of experimental conditiong (β^g) minus the estimated treatment effect of experimental conditionk (β^k) equalled 0.

Prediction study participants

Prediction study content

Before beginning the survey (which was the same for all participant populations with the exception of the demographic questions asked at the end), potential participants were screened out if they reported being familiar with any of the results from the megastudy (which were featured on an episode of the Freakonomics Radio podcast³²). The participants were first shown an overall description of the StepUp Program, and they were then asked to compare three of the megastudy’s experimental conditions with the placebo control condition (one at a time). The three conditions that the participants reviewed were randomly selected from the megastudy’s 53 experimental conditions and were presented in a random order.

For each experimental condition that they were prompted to examine, the participants were presented with a summary table comparing the key features of the experimental condition with the placebo control condition. The participants next viewed screenshots of the registration experience and a summary of the text messages and emails sent during the programme in both the experimental condition and the placebo control condition. Sample stimuli comparing the planning, reminders and microincentives to exercise condition with the placebo control condition are available in Prediction Study Stimuli on the Open Science Framework (https://osf.io/kyt7d/?view_only=8bb9282111c24f81a19c2237e7d7eba3). The participants were informed of how many days per week an average participant in the placebo control condition visited the gym during the StepUp Program as well as how likely a participant was to visit the gym in a given week, on average, in the placebo control condition. The participants were then asked to forecast the average number of days per week that gym members would visit the gym and the percentage of the time that members would visit the gym at least once in a given week in the StepUp Program experimental condition that they had just reviewed. Specifically, participants answered these two questions:

1. On average, how many days per week do you think members in the enhanced version of StepUp went to the gym? (For reference, people in the basic version went to the gym 1.5 days per week.)

2. In an average week, what percent of the time do you think members in the enhanced version of StepUp made it to the gym? (For reference, in a given week, members in the basic version of StepUp made it to the gym at least once 57% of the time)

For each study, our key dependent variable was the predicted increase in gym attendance induced by a given experimental condition (compared with the placebo control condition). To determine the extra number of gym visits per week that a participant predicted a condition would induce, we subtracted the placebo control condition’s mean of 1.5 d of gym visits per week from the participants’ estimated total weekly gym visits for a given experimental condition (the possible range of values was −1.5 to 5.5, as weeks include only 7 d). To determine the added likelihood of visiting the gym at least once in a given week that a participant predicted a condition would induce, we subtracted the placebo control condition’s mean visit likelihood of 57% from the participants’ estimated weekly visit likelihood for a given experimental condition (the possible range of values was −57% to 43% as the maximum likelihood was 100%). As any weekly gym attendance is not our primary focus, we present these results inExtended Data Fig. 1,Extended Data Tables 1–3 and7 andSupplementary Information 2. Finally, we computed an unweighted correlation between the actual regression-estimated change in gym attendance induced by a given experimental condition in our megastudy (see estimates inExtended Data Tables 6 and7) and the mean predicted change in gym attendance induced by that same experimental condition.

Extended Data Table 1 |.

Extended Data Table 2 |.

Extended Data Table 3 |.

Extended Data Table 4 |.

Extended Data Table 5 |.

Extended Data Table 6 |.

Extended Data Table 7 |.

Extended Data Table 8 |.

Extended Data Table 9 |.

• 1.Behavioural Insights and Public Policy: Lessons from Around the World (OECD Publishing, 2017).
• 2.Benartzi Set al. Should governments invest more in nudging?Psychol. Sci28, 1041–1055 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
• 3.Charness G & Gneezy UIncentives to EXercise. Econometrica77, 909–931 (2009). [Google Scholar]
• 4.Acland D & Levy MRNaiveté, projection bias, and habit formation in gym attendance. Manage. Sci61, 146–160 (2015). [Google Scholar]
• 5.Royer H, Stehr M & Sydnor JIncentives, commitments, and habit formation in exercise: evidence from a field experiment with workers at a Fortune-500 company. Am. Econ. J. Appl. Econ7, 51–84 (2015). [Google Scholar]
• 6.Beshears J, Lee HN, Milkman KL, Mislavsky R & Wisdom JCreating exercise habits using incentives: the tradeoff between flexibility and routinization. Manage. Sci67, 4139–4171 (2020). [DOI] [PMC free article] [PubMed] [Google Scholar]
• 7.DellaVigna S & Linos ERCTs to Scale: Comprehensive Evidence from Two Nudge Units 65 (National Bureau of Economic Research, 2020). [Google Scholar]
• 8.DellaVigna S & Pope DWhat motivates effort? Evidence and expert forecasts. Rev. Econ. Stud85, 1029–1069 (2018). [Google Scholar]
• 9.DellaVigna S & Pope DPredicting experimental results: who knows what?J. Polit. Econ126, 2410–2456 (2018). [Google Scholar]
• 10.DellaVigna S, Pope D & Vivalt EPredict science to improve science. Science366, 428–429 (2019). [DOI] [PubMed] [Google Scholar]
• 11.Kristal AS & Whillans AVWhat we can learn from five naturalistic field experiments that failed to shift commuter behaviour. Nat. Hum. Behav4, 169–176 (2020). [DOI] [PubMed] [Google Scholar]
• 12.Donoho D50 years of data science. J. Comput. Graph. Stat26, 745–766 (2017). [Google Scholar]
• 13.Liberman MFred Jelinek. Comput. Linguist36, 595–599 (2010). [Google Scholar]
• 14.Lai CKet al. Reducing implicit racial preferences: I. A comparative investigation of 17 interventions. J. Exp. Psychol. Gen143, 1765–1785 (2014). [DOI] [PubMed] [Google Scholar]
• 15.Lai CKet al. Reducing implicit racial preferences: II. Intervention effectiveness across time. J. Exp. Psychol. Gen145, 1001–1016 (2016). [DOI] [PubMed] [Google Scholar]
• 16.Mellers Bet al. Psychological strategies for winning a geopolitical forecasting tournament. Psychol. Sci25, 1106–1115 (2014). [DOI] [PubMed] [Google Scholar]
• 17.Open Science Collaboration Estimating the reproducibility of psychological science. Science349, aac4716 (2015). [DOI] [PubMed] [Google Scholar]
• 18.Milkman KL, Minson JA & Volpp KGMHolding the hunger games hostage at the gym: an evaluation of temptation bundling. Manage. Sci60, 283–299 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
• 19.Ward BW, Clarke TC, Nugent CN & Schiller JSEarly Release of Selected Estimates Based on Data From the 2015 National Health Interview Survey 120 (National Center for Health Statistics, 2015). [Google Scholar]
• 20.Lee I-Met al. Effect of physical inactivity on major non-communicable diseases worldwide: an analysis of burden of disease and life expectancy. Lancet380, 219–229 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
• 21.Gollwitzer PMImplementation intentions: strong effects of simple plans. Am. Psychol54, 493–503 (1999). [Google Scholar]
• 22.Milkman KL, Beshears J, Choi JJ, Laibson D & Madrian BCUsing implementation intentions prompts to enhance influenza vaccination rates. Proc. Natl Acad. Sci. USA108, 10415–10420 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]
• 23.Rogers T, Milkman KL, John LK & Norton MIBeyond good intentions: prompting people to make plans improves follow-through on important tasks. Behav. Sci. Pol1, 33–41 (2015). [Google Scholar]
• 24.Karlan D, McConnell M, Mullainathan S & Zinman JGetting to the top of mind: how reminders increase saving. Manage. Sci62, 3393–3411 (2016). [Google Scholar]
• 25.Homonoff TACan small incentives have large effects? The impact of taxes versus bonuses on disposable bag use. Am. Econ. J. Econ. Pol10, 177–210 (2018). [Google Scholar]
• 26.Storey JD & Tibshirani RStatistical significance for genomewide studies. Proc. Natl Acad. Sci. USA100, 9440–9445 (2003). [DOI] [PMC free article] [PubMed] [Google Scholar]
• 27.Allcott HSocial norms and energy conservation. J. Publ. Econ95, 1082–1095 (2011). [Google Scholar]
• 28.Chapman GB, Li M, Colby H & Yoon HOpting in vs opting out of influenza vaccination. JAMA304, 43–44 (2010). [DOI] [PubMed] [Google Scholar]
• 29.Milkman KL, et al. A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor’s appointment. Proc. Natl Acad. Sci. USA118, e2101165118 (2021). [DOI] [PMC free article] [PubMed] [Google Scholar]
• 30.Lee MR & Shen MWinner’s curse: bias estimation for total effects of features in online controlled experiments In Proc. 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining491–499 (ACM, 2018). [Google Scholar]
• 31.White HAsymptotic Theory for Econometricians (Elsevier, 1984). [Google Scholar]
• 32.Dubner SJHow goes the behavior-change revolution? (Ep. 382). Freakonomicshttps://freakonomics.com/podcast/live-philadelphia/ (2019).

Supplementary Materials

Data Availability Statement

The data analysed in this paper were provided by 24 Hour Fitness and we have their legal permission to share the deidentified data. We have therefore made deidentified data available athttps://osf.io/9av87/?view_only=8bb9282111c24f81a19c2237e7d7eba3. Furthermore, tables of all of the preregistration links for each of the substudies with the interventions and the prediction studies are available inSupplementary Tables 2 and30.