Developing a conceptual framework—Delphi study

Following research team discussion of our previous experience with, and research on, pilot and feasibility studies we initially produced mutually exclusive definitions of pilot and feasibility studies based on, but not identical to, the definitions used by the NIHR. We drew up two draft reporting checklists based on the 2010 CONSORT statement [8], one for what we had defined as feasibility studies and one for what we had defined as pilot studies. We constructed a Delphi survey, administered on-line by Clinvivo [9], to obtain consensus on checklist items for inclusion in a reporting guideline, and views on the definitions. Following user-testing of a draft version of the survey with a purposive sample of researchers active in the field of trials and pilot studies, and a workshop at the 2013 Society for Clinical Trials Conference in Boston, we further refined the definitions, checklists, survey introduction and added additional questions.

The first round of the main Delphi survey included: a description and explanation of our definitions of pilot and feasibility studies including examples (Figs2 and3); questions about participants’ characteristics; 67 proposed items for the two checklists and questions about overall appropriateness of the guidelines for feasibility or pilot studies; and four questions related to the definitions of feasibility and pilot studies:How appropriate do you think our definition for a pilot study conducted in preparation for an RCT is?How appropriate do you think our definition for a feasibility study conducted in preparation for an RCT is?How appropriate is the way we have distinguished between two different types of study conducted in preparation for an RCT?How appropriate are the labels ‘pilot’ and ‘feasibility’ for the two types of study we have distinguished? Participants were asked to rate their answers to the four questions on a nine-point scale from ‘not at all appropriate’ to ‘completely appropriate’. There was also a space for open comments about the definitions. The second round included results from the first round and again asked for further comments about the definitions.

Participants for the main survey were identified as likely users of the checklist including trialists, methodologists, statisticians, funders and journal editors. Three hundred and seventy potential participants were approached by email from the project team or directly from Clinvivo. These were individuals identified based on personal networks, authors of relevant studies in the literature, members of the Canadian Institute of Health Research, Biostatistics section of Statistics Society of Canada, and the American Statistical Society. The International Society for Clinical Biostatistics and the Society for Clinical Trials kindly forwarded our email to their entire membership. There was a link within the email to the on-line questionnaire. Each round lasted three weeks and participants were sent one reminder a week before the closure of each survey. The survey took place between August and October 2013. Ethical approval was granted by the ScHARR research ethics committee at the University of Sheffield.

Developing a conceptual framework—Open meeting and research team meetings

The results of the Delphi survey pertaining to the definitions of feasibility and pilot studies were presented to an open meeting at the 2^nd UK MRC Trials Methodology Conference in Edinburgh in November 2013 [13]. Attendees chose their preferred proposition from four propositions regarding the definitions, based variously on our original definitions, the NIHR and MRC views of pilot and feasibility studies and different views expressed in the Delphi survey. At a subsequent two-day research team meeting we collated the findings from the Delphi survey and the open meeting, and considered definitions of piloting and feasiblity outside the health research context found from on-line searches using the terms ‘pilot definition’, ‘feasiblity definition’, ‘pilot study definition’ and ‘feasibility study definition’ in Google. We expected all searches to give a very large number of hits and examined the first two pages of hits only from each search. From this, we developed a conceptual framework reflecting consensus about the definitions, types and roles of feasibility and pilot studies conducted in preparation for an RCT evaluating the effect of an intervention or therapy. To ensure we incorporated the views of all researchers likely to be conducting pilot/feasiblity studies, two qualitative researchers joined the second day of the meeting which focused on agreeing this framework. Throughout this process we continually referred back to examples that we had identified to check that our emerging definitions were workable.

Validating the conceptual framework—systematic review

To validate the proposed conceptual framework, we identified a selection of recently reported studies that fitted our definition of pilot and feasibility studies, and tested a number of hypotheses in relation to these studies. We expected that approximately 30 reports would be sufficient to test the hypotheses. We conducted a systematic review to identify studies that authors described as pilot or feasibility studies, by searching Medline via PubMed for studies that had the words ‘pilot’ or ‘feasibility’ in the title. To increase the likelihood that the studies would be those conducted in preparation for a randomised controlled trial of the effect of a therapy or intervention we limited our search to those that contained the word ‘trial’ in the title or abstract. For full details of the search strategy seeS1 Fig.

To focus on current practice, we selected the 150 most recent studies from those identified by the electronic search. We did not exclude protocols since we were primarily interested in identifying the way researchers characterised their study and any possible future study and the relationship between them; we expected investigators to describe these aspects of their studies in a similar way in protocols and reports of findings. Two research team members independently reviewed study abstracts to assess whether each study fitted our working definition of a pilot or feasibility study in preparation for an RCT evaluating the effect of an intervention or therapy. Where reviewers disagreed, studies were classed as ‘possible inclusions’ and disagreements resolved by discussion with referral to the full text of the paper as necessary. Given the difficulty of interpreting some reports and to ensure that all research team members agreed on inclusion, the whole team then reviewed relevant extracted sections of the papers provisionally agreed for inclusion. We recognised that abstracts of some studies might not include appropriate information, and therefore that our initial abstract review could have excluded some relevant studies; we explored the extent of this potential omission of studies by reviewing the full texts of a random sample of 30 studies from the original 150. Since our prime goal was to identify a manageable number of relevant studies in order to test our hypotheses rather than identify all possible relevant studies we did not include any additional studies as a result of this exploratory study.

We postulated that the following hypotheses would support our conceptual framework:

1. The words ‘pilot’ and ‘feasibility’ are both used in the literature to describe studies undertaken in preparation for an RCT evaluating the effect of an intervention or therapy

2. It is possible to identify a subset of studies within the literature that are RCTs conducted in preparation for a larger RCT which evaluates the effect of an intervention or therapy. Authors do not use the term ‘pilot trial’ consistently in relation to these studies.

3. Within the literature it is not possible to apply unique mutually exclusive definitions of pilot and feasibility studies in preparation for an RCT evaluating the effect of an intervention or therapy that are consistent with the way authors describe their studies.

4. Amongst feasibility studies in preparation for an RCT which evaluates the effect of an intervention or therapy it is possible to identify some studies that are not pilot studies as defined within our conceptual framework, but are studies that acquire information about the feasibility of applying an intervention in a future study.

In order to explore these hypotheses, we categorised included studies into three groups that tallied with our framework (seeresults for details): randomised pilot studies, non-randomised pilot studies, feasibility studies that are not pilot studies. We also extracted data on objectives, and the phrases that indicated that the studies were conducted in preparation for a subsequent RCT.

Validating the conceptual framework—Consensus meeting

We also took an explanation and visual representation of our framework to an international consensus meeting primarily designed to reach consensus on an extension of the 2010 CONSORT statement to randomised pilot studies. There were 19 invited participants with known expertise, experience, or interest in pilot and feasibility studies, including representatives of CONSORT, funders, journal editors, and those who had been involved in writing the NIHR definitions of pilot and feasibility studies and the MRC guidance on designing and evaluating complex interventions. Thus this was an ideal forum in which to discuss the framework also. This project was not concerned with any specific disease, and was methodological in design; no patients or public were involved.

Developing a conceptual framework—Delphi study

Ninety-three individuals, including chief investigators, statisticians, trial managers, clinicians, research assistants and a funder, participated in the first round of the Delphi survey and 79 in the second round. Over 70% of participants in the first round felt that our definitions, the way we had distinguished between pilot and feasibility studies, and the labels ‘pilot’ and ‘feasibility’ were appropriate. However, these four items had some of the lowest appropriateness ratings in the survey and there were a large number of comments both in direct response to our four survey items related to appropriateness of definitions, and in open comment boxes elsewhere in the survey. Some of these comments are presented inFig 4. Some participants commented favourably on the definitions we had drawn up (quote 1) but others were confused by them (quote 2). Several compared our definitions to the NIHR definitions pointing out the differences (quote 3) and suggesting this might make it particularly difficult for the research community to understand our definitions (quote 4). Some expressed their own views about the definitions (quote 5); largely these tallied with the NIHR definitions. Others noted that both the concept of feasibility and the word itself were often used in relation to studies which investigators referred to as pilot studies (quote 6). Others questioned whether it was practically and/or theoretically possible to make a distinction between pilot and feasibility studies (quote 6, quote 7), suggesting that the two terms are not mutually exclusive and that feasibility was more of an umbrella term for studies conducted prior to the main trial. Some participants felt that, using our definitions, feasibility studies would be less structured and more variable and therefore their quality would be less appropriately assessed via a checklist (quote 8). These responses regarding definitions mirrored what we had found in the user-testing of the Delphi survey, the Society for Clinical Trials workshop, and differences of opinion already apparent in the literature. In the second round of the survey there were few comments about definitions.

Developing a conceptual framework—Open meeting and research team meetings

There was a wide range of participants in the open meeting, including senior quantitative and qualitative methodologists, and a funding body representative. The four propositions we devised to cover different views about definitions of pilot and feasibility studies are shown inFig 5. Fourteen out of the fifteen attendees who voted on these propositions preferred propositions 3 or 4, based on comments from the Delphi survey and the MRC guidance on designing and evaluating complex interventions respectively. Neither of these propositions implied mutually exclusive definitions of pilot and feasibility studies.

Definitions of feasibility outside the health research context focus on the likelihood of being able to do something. For example, the Oxford on-line dictionary defines feasibility as: ‘The state or degree of being easily or conveniently done’ [14] and a feasibility study as: ‘An assessment of the practicality of a proposed plan or method’ [15]. Some definitions also suggest that a feasibility study should help with decision making, for example [16]: ‘The feasibility study is an evaluation and analysis of the potential of a proposed project. It is based on extensive investigation and research to support the process of decision making’. Outside the health research context the word ‘pilot’ has several different meanings but definitions of pilot studies usually focus on an experiment, project or development undertaken in advance of a future wider experiment, project or development. For example the Oxford on-line dictionary describes a pilot study as: ‘Done as an experiment or test before being introduced more widely’ [17]. Several definitions carry with them ideas that the purpose of a pilot study is also to facilitate decision making, for example ‘a small-scale experiment or set of observations undertaken to decide how and whether to launch a full-scale project’ [18] and some definitions specifically mention feasibility, for example: ‘a small scale preliminary study conducted in order to evaluate feasibility’ [19].

In keeping with these definitions not directly related to the health research context, we agreed that feasiblity is a concept encapsulating ideas about whether it is possible to do something and thata feasibility study asks whether something can be done,should we proceed with it,and if so,how. While piloting is also concerned with whether something can be done and whether and how we should proceed with it, it has a further dimension; piloting is implementing something, or part of something, in a way you intend to do it in future to see whether it can be done in practice. We therefore agreed thata pilot study is a study in which a future study or part of a future study,is conducted on a smaller scale to ask the question whether something can be done,should we proceed with it,and if so,how. The corollary of these definitions is that all pilot studies are feasibility studies but not all feasibility studies are pilot studies. Within the context of RCTs, the focus of our research, the ‘something’ in the definitions can be replaced with ‘a future RCT evaluating the effect of an intervention or therapy’. Studies that address the question of whether the RCT can be done, should we proceed with it and if so how, can then be classed as feasibility or pilot studies. Some of these studies may, of course, have other objectives but if they are mainly focusing on feasiblity of the future RCT we would include them as feasiblity studies. All three studies used as examples in our Delphi survey [10–12] satisfy the definition of a feasiblity study. However, a study by Piotet al, that we encountered while developing the Delphi study, does not. This study is described as a pilot trial in the abstract but the authors present only data on effectiveness and although they state that their results require confirmation in a larger study it is not clear that their pilot study was conducted in preparation for such a larger study [20]. On the other hand, Palmeret al ‘performed a feasibility study to determine whether patient and surgeon opinion was permissive for a Randomised Controlled Trial (RCT) comparing operative with non-operative treatment for FAI [femoroacetabular impingement]’ [12]. Heazellet al describe the aim of their randomised study as ‘to address whether a randomised controlled trial (RCT) of the management of RFM [reduced fetal movement] was feasible’ [10]. Their study was piloting many of the aspects they hoped to implement in a larger trial of RFM, thus making this also a pilot study, whereas the study conducted by Palmeret al, which comprised a questionnare to clinicians and seeking patient opinion, is not a pilot study but is a feasibility study.

Within our framework, some important studies conducted in advance of a future RCT to evaluate the effect of a therapy or intervention are not feasibility studies. For example, a systematic review, usually an essential pre-requisite for such an RCT, normally addresses whether the future RCT isnecessary ordesirable, not whether it isfeasible. To reflect this, we developed a comprehensive diagrammatical representation of our framework for studies conducted in preparation for an RCT which, for completeness, includes, on the left hand side, early studies that are not pilot and feasibility studies, such as systematic reviews and, along the bottom, details of existing or planned reporting guidelines for different types of study (S2 Fig).

Validating the conceptual framework—Systematic review

From the 150 most recent studies identified by our electronic search, we identified 27 eligible reports (Fig 6). In keeping with our working definition of a pilot or feasibility study, to be included the reports had to show evidence that investigators were addressing at least some feasibility objectives and that the study was in preparation for a future RCT evaluating the effect of an intervention. Ideally we would have stipulated that the primary objective of the study should be a feasibility objective but, given the nature of the reporting of most of these studies, we felt this would be too restrictive.

The 27 studies are reported inTable 1 and results relating to terminology that authors used summarised inTable 2. Results inTable 2 support our first hypothesis that the words ‘pilot’ and ‘feasibility’ are both used in the literature to describe studies undertaken in preparation for a randomised controlled trial of effectiveness; 63% (17/27) used both terms somewhere in the title or abstract. The table also supports our second hypothesis that amongst the subset of feasibility studies in preparation for an RCT that are themselves RCTs, authors do not use the term ‘pilot trial’ consistently in relation to these studies; of the 18 randomised studies only eight contained the words ‘pilot’ and ‘trial’ in the title. Our third hypothesis, namely that it is not possible to apply unique mutually exclusive definitions of pilot and feasibility studies in preparation for an RCT that are consistent with the way authors describe their studies, is supported by the characteristics of studies presented inTable 1 and summarised inTable 2. We could find no design or other features (such as randomisation or presence of a control group) that distinguished between those that investigators called feasibility studies and those that they called pilot studies. However, the fourth hypothesis, that amongst studies in preparation for an RCT evaluating the effect of an intervention or therapy it is possible to identify some studies that explore the feasibility of a certain intervention or acquire related information about the feasibility of applying an intervention in a future study but are not pilot studies, was not supported; we identified no such studies amongst those reported inTable 1. Nevertheless, we had identified two prior to carrying out the review [10,15].

Out of our exploratory sample of 30 study reports for which we reviewed full texts rather than only titles and abstracts, we identified 10 that could be classed as pilot or feasibility studies using our framework. We had already identified four of these in our sample reported inTable 1, but had failed to identify the other six. As expected, this was because key information to identify them as pilot or feasiblity studies such as the fact that they were in preparation for a larger RCT, or that the main objectives were to do with feasiblity were not included in the abstract. Thus our assumption that an initial screen using only abstracts resulted in the omission of some pilot and feasiblity studies was correct.

Validating the conceptual framework—Consensus meeting

International consensus meeting participants agreed with the general tenets of our conceptual framework including the ideas that all pilot studies are feasibility studies but that some feasibility studies are not pilot studies. They suggested that any definitive diagrammatic representation should more strongly reflect non-linearity in the ordering of feasibility studies. As a result of their input we produced a new, simplified, diagrammatical representation of the framework (Fig 7) which focuses on the key elements represented inside an oval shape on our original diagram, omits the wider context outside this shape, and highlights some features, including the non-linearity, more clearly.

The finalised framework

Fig 7 represents the framework. The figure indicates that where there is uncertainty about future RCT feasibility, a feasibility study is appropriate. Feasibility is thus an overarching concept within which we distinguish between three distinct types of study.Randomised pilot studies are those studies in which the future RCT, or parts of it, including the randomisation of participants, is conducted on a smaller scale (piloted) to see if it can be done. Thus randomised pilot studies can include studies that for the most part reflect the design of a future definitive trial but, if necessary due to remaining uncertainty, may involve trying out alternative strategies, for example, collecting an outcome variable via telephone for some participants and on-line for others. Within the framework randomised pilot studies could also legitimately be called randomised feasibility studies. Two-thirds of the studies presented inTable 1 are of this type.

Non-randomised pilot studies are similar to randomised pilot studies; they are studies in which all or part of the intervention to be evaluated and other processes to be undertaken in a future trial is/are carried out (piloted) but without randomisation of participants. These could also legitimately be called by the umbrella term, feasibility study. These studies cover a wide range from those that are very similar to randomised pilot studies except that the intervention and control groups have not been randomised, to those in which only the intervention, and no other trial processes, are piloted. One-third of studies presented inTable 1 are of this type.

Feasibility studies that are not pilot studies are those in which investigators attempt to answer a question about whether some element of the future trial can be done but do not implement the intervention to be evaluated or other processes to be undertaken in a future trial, though they may be addressing intervention development in some way. Such studies are rarer than the other types of feasibility study and, in fact, none of the studies inTable 1 were of this type. Nevertheless, we include these studies within the framework because they do exist; the Palmer study [15] in which surgeons and patients were asked about the feasibility of randomisation is one such example. Other examples might be interviews to ascertain the acceptability of an intervention, or questionnaires to assess the types of outcomes participants might think important. Within the framework these studies can be called feasibility studies but cannot be called pilot studies since no part of the future randomised controlled trial is being conducted on a smaller scale.

Investigators may conduct a number of studies to assess feasibility of an RCT to test the effect of any intervention or therapy. While it may be most common to carry out what we have referred to asfeasibility studies that are not pilot studies beforenon-randomised pilot studies, andnon-randomised pilot studies prior torandomised pilot studies, the process of feasibility work is not necessarily linear and such studies can in fact be conducted in any order. For completeness the diagram indicates the location of internal pilot studies.

• 1.Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. Journal of evaluation in clinical practice. 2004;10(2):307–12. Epub 2004/06/11. 10.1111/j.2002.384.doc.x . [DOI] [PubMed] [Google Scholar]
• 2.Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. British Medical Journal. 2008;337. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 3.Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC medical research methodology. 2010;10:1 Epub 2010/01/08. 10.1186/1471-2288-10-1 ; PubMed Central PMCID: PMCPmc2824145. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 4.Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC medical research methodology. 2010;10:67 Epub 2010/07/20. 10.1186/1471-2288-10-67 ; PubMed Central PMCID: PMCPmc2912920. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 5.National Institute for Health Research. NIHR Evaluation, Trials and Studies | Glossary 2015. Available:http://www.nets.nihr.ac.uk/glossary/feasibility-studies. Accessed 2015 Mar 17.
• 6.National Institute for Health Research. NIHR Evaluation, Trials and Studies | Pilot studies 2015. Available:http://www.nets.nihr.ac.uk/glossary/pilot-studies. Accessed 2015 Mar 17.
• 7.Shanyinde M, Pickering RM, Weatherall M. Questions asked and answered in pilot and feasibility randomized controlled trials. BMC medical research methodology. 2011;11:117 Epub 2011/08/19. 10.1186/1471-2288-11-117 ; PubMed Central PMCID: PMCPmc3170294. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 8.Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomized trials. Annals of Internal Medicine. 2010;152(11):726–32. Epub 2010/03/26. 10.7326/0003-4819-152-11-201006010-00232 . [DOI] [PubMed] [Google Scholar]
• 9.CLINVIVO. Clinvivo Limited 2015 [cited 2015 9 April]. Available:http://www.clinvivo.com/.
• 10.Heazell AE, Bernatavicius G, Roberts SA, Garrod A, Whitworth MK, Johnstone ED, et al. A randomised controlled trial comparing standard or intensive management of reduced fetal movements after 36 weeks gestation—a feasibility study. BMC pregnancy and childbirth. 2013;13:95 Epub 2013/04/18. 10.1186/1471-2393-13-95 ; PubMed Central PMCID: PMCPmc3640967. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 11.Colon HM, Finlinson HA, Negron J, Sosa I, Rios-Olivares E, Robles RR. Pilot trial of an intervention aimed at modifying drug preparation practices among injection drug users in Puerto Rico. AIDS and behavior. 2009;13(3):523–31. Epub 2009/03/25. 10.1007/s10461-009-9540-3 . [DOI] [PubMed] [Google Scholar]
• 12.Palmer AJ, Thomas GE, Pollard TC, Rombach I, Taylor A, Arden N, et al. The feasibility of performing a randomised controlled trial for femoroacetabular impingement surgery. Bone & joint research. 2013;2(2):33–40. Epub 2013/04/24. 10.1302/2046-3758.22.2000137 ; PubMed Central PMCID: PMCPmc3626218. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 13.In Conference Ltd. 2nd Clinical Trials Methodology Conference | 18–19 November 2013 EICC, Edinburgh, Scotland 2013. Available:http://www.methodologyconference2013.org.uk/. Accessed 2015 Mar 17.
• 14.Oxford Dictionaries. Oxford Dictionaries | feasibility 2015. Available:http://www.oxforddictionaries.com/definition/english/feasibility. Accessed 2015 Mar 17.
• 15.Oxford Dictionaries. Oxford Dictionaries | feasibility study 2015. Available:http://www.oxforddictionaries.com/definition/english/feasibility-study. Accessed 2015 Mar 17.
• 16.Wikipedia. Feasibility study 2015 [cited 2015 17 March]. Available:http://en.wikipedia.org/wiki/Feasibility_study.
• 17.Oxford Dictionaries. Oxford Dictionaries | pilot 2015. Available from:http://www.oxforddictionaries.com/definition/english/pilot. Accessed 2015 Mar 17.
• 18.Collins. Collins English Dictionary | pilot study 2015. Available from:http://www.collinsdictionary.com/dictionary/english/pilot-study. Accessed 2015 Mar 17.
• 19.Wikipedia. Pilot experiment 2015. Available:http://en.wikipedia.org/wiki/Pilot_experiment. Accessed 2015 Mar 17.
• 20.Piot C, Croisille P, Staat P, Thibault H, Rioufol G, Mewton N, et al. Effect of cyclosporine on reperfusion injury in acute myocardial infarction. The New England journal of medicine. 2008;359(5):473–81. Epub 2008/08/02. 10.1056/NEJMoa071142 . [DOI] [PubMed] [Google Scholar]
• 21.Allen J, Stapleton H, Tracy S, Kildea S. Is a randomised controlled trial of a maternity care intervention for pregnant adolescents possible? An Australian feasibility study. BMC medical research methodology. 2013;13:138 Epub 2013/11/15. 10.1186/1471-2288-13-138 ; PubMed Central PMCID: PMCPmc4226005. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 22.Boogerd EA, Noordam C, Kremer JA, Prins JB, Verhaak CM. Teaming up: feasibility of an online treatment environment for adolescents with type 1 diabetes. Pediatric diabetes. 2014;15(5):394–402. Epub 2013/12/20. 10.1111/pedi.12103 . [DOI] [PubMed] [Google Scholar]
• 23.Buse GL, Bhandari M, Sancheti P, Rocha S, Winemaker M, Adili A, et al. Accelerated care versus standard care among patients with hip fracture: the HIP ATTACK pilot trial. Canadian Medical Association Journal. 2013;186(1):E52–E60. 10.1503/cmaj.130901 [DOI] [PMC free article] [PubMed] [Google Scholar]
• 24.Clark WF, Sontrop JM, Huang S-H, Gallo K, Moist L, House AA, et al. The chronic kidney disease Water Intake Trial (WIT): results from the pilot randomised controlled trial. BMJ Open. 2013;3(12):e00366610.1136/bmjopen-2013-003666 [DOI] [PMC free article] [PubMed] [Google Scholar]
• 25.Crawley E, Mills N, Beasant L, Johnson D, Collin SM, Deans Z, et al. The feasibility and acceptability of conducting a trial of specialist medical care and the Lightning Process in children with chronic fatigue syndrome: feasibility randomized controlled trial (SMILE study). Trials. 2013;14:415 Epub 2013/12/07. 10.1186/1745-6215-14-415 ; PubMed Central PMCID: PMCPmc4235039. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 26.Goodall M, Barton GR, Bower P, Byrne P, Cade JE, Capewell S, et al. Food for thought: pilot randomized controlled trial of lay health trainers supporting dietary change to reduce cardiovascular disease in deprived communities. Journal of Public Health (Oxford, England)2014;36(4):635–43. Epub 2013/11/28. 10.1093/pubmed/fdt112 . [DOI] [PubMed] [Google Scholar]
• 27.Higgins J, Koski L, Xie H. Combining rTMS and task-oriented training in the rehabilitation of the arm after stroke: a pilot randomized controlled trial. Stroke Research and Treatment. 2013;201310.1155/2013/539146 [DOI] [PMC free article] [PubMed] [Google Scholar]
• 28.Holt TA, Mant D, Carr A, Gwilym S, Beard D, Toms C, et al. Corticosteroid injection for shoulder pain: single-blind randomized pilot trial in primary care. Trials. 2013;14:425 Epub 2013/12/12. 10.1186/1745-6215-14-425 ; PubMed Central PMCID: PMCPmc3878869. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 29.Hurt CN, Roberts K, Rogers TK, Griffiths GO, Hood K, Prout H, et al. A feasibility study examining the effect on lung cancer diagnosis of offering a chest X-ray to higher-risk patients with chest symptoms: protocol for a randomized controlled trial. Trials. 2013;14:405 Epub 2013/11/28. 10.1186/1745-6215-14-405 ; PubMed Central PMCID: PMCPmc4222751. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 30.Lakes KD, Bryars T, Sirisinahal S, Salim N, Arastoo S, Emmerson N, et al. The Healthy for Life Taekwondo Pilot Study: A Preliminary Evaluation of Effects on Executive Function and BMI, Feasibility, and Acceptability. Mental health and physical activity. 2013;6(3):181–8. Epub 2014/02/25. 10.1016/j.mhpa.2013.07.002 ; PubMed Central PMCID: PMCPmc3927879. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 31.Lee TS, Goh SJ, Quek SY, Phillips R, Guan C, Cheung YB, et al. A brain-computer interface based cognitive training system for healthy elderly: a randomized control pilot study for usability and preliminary efficacy. PloS one. 2013;8(11):e79419 Epub 2013/11/22. 10.1371/journal.pone.0079419 ; PubMed Central PMCID: PMCPmc3832588. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 32.McKenna S, Jones F, Glenfield P, Lennon S. Bridges self-management program for people with stroke in the community: A feasibility randomized controlled trial. International journal of stroke: official journal of the International Stroke Society. 2013. Epub 2013/11/22. 10.1111/ijs.12195 . [DOI] [PubMed] [Google Scholar]
• 33.Powell JE, Carroll FE, Sebire SJ, Haase AM, Jago R. Bristol girls dance project feasibility study: using a pilot economic evaluation to inform design of a full trial. BMJ Open. 2013;3(12):e00372610.1136/bmjopen-2013-003726 [DOI] [PMC free article] [PubMed] [Google Scholar]
• 34.Saez C, Marti-Bonmati L, Alberich-Bayarri A, Robles M, Garcia-Gomez JM. Randomized pilot study and qualitative evaluation of a clinical decision support system for brain tumour diagnosis based on SV (1)H MRS: evaluation as an additional information procedure for novice radiologists. Computers in biology and medicine. 2014;45:26–33. Epub 2014/02/01. 10.1016/j.compbiomed.2013.11.009 . [DOI] [PubMed] [Google Scholar]
• 35.Safdar N, Zahid R, Shah S, Fatima R, Cameron I, Siddiqi K. Tuberculosis patients learning about second-hand smoke (TBLASS): results of a pilot randomised controlled trial. The international journal of tuberculosis and lung disease: the official journal of the International Union against Tuberculosis and Lung Disease. 2015;19(2):237–43. Epub 2015/01/13. 10.5588/ijtld.14.0615 . [DOI] [PubMed] [Google Scholar]
• 36.Schoultz M, Atherton IM, Hubbard G, Watson AJ. The use of mindfulness-based cognitive therapy for improving quality of life for inflammatory bowel disease patients: study protocol for a pilot randomised controlled trial with embedded process evaluation. Trials. 2013;14:431 Epub 2013/12/18. 10.1186/1745-6215-14-431 ; PubMed Central PMCID: PMCPmc3878510. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 37.Siriwardhana C, Adikari A, Van Bortel T, McCrone P, Sumathipala A. An intervention to improve mental health care for conflict-affected forced migrants in low-resource primary care settings: a WHO MhGAP-based pilot study in Sri Lanka (COM-GAP study). Trials. 2013;14:423 Epub 2013/12/11. 10.1186/1745-6215-14-423 ; PubMed Central PMCID: PMCPmc3906999. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 38.Wolf KH, Hetzer K, zu Schwabedissen HM, Wiese B, Marschollek M. Development and pilot study of a bed-exit alarm based on a body-worn accelerometer. Zeitschrift fur Gerontologie und Geriatrie. 2013;46(8):727–33. Epub 2013/11/26. 10.1007/s00391-013-0560-2 . [DOI] [PubMed] [Google Scholar]
• 39.Alers NO, Jenkin G, Miller SL, Wallace EM. Antenatal melatonin as an antioxidant in human pregnancies complicated by fetal growth restriction—a phase I pilot clinical trial: study protocol. BMJ Open. 2013;3(12):e00414110.1136/bmjopen-2013-004141 [DOI] [PMC free article] [PubMed] [Google Scholar]
• 40.Carlesso LC, Macdermid JC, Santaguida PL, Thabane L. Determining adverse events in patients with neck pain receiving orthopaedic manual physiotherapy: a pilot and feasibility study. Physiotherapy Canada Physiotherapie Canada. 2013;65(3):255–65. Epub 2014/01/10. 10.3138/ptc.2012-28 ; PubMed Central PMCID: PMCPmc3740991. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 41.Collado A, Castillo SD, Maero F, Lejuez CW, Macpherson L. Pilot of the brief behavioral activation treatment for depression in latinos with limited english proficiency: preliminary evaluation of efficacy and acceptability. Behavior therapy. 2014;45(1):102–15. Epub 2014/01/15. 10.1016/j.beth.2013.10.001 ; PubMed Central PMCID: PMCPmc4103902. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 42.Galantino ML, Callens ML, Cardena GJ, Piela NL, Mao JJ. Tai chi for well-being of breast cancer survivors with aromatase inhibitor-associated arthralgias: a feasibility study. Alternative therapies in health and medicine. 2013;19(6):38–44. Epub 2013/11/21. . [PubMed] [Google Scholar]
• 43.Garcia MK, Driver L, Haddad R, Lee R, Palmer JL, Wei Q, et al. Acupuncture for treatment of uncontrolled pain in cancer patients: a pragmatic pilot study. Integrative cancer therapies. 2014;13(2):133–40. Epub 2013/11/28. 10.1177/1534735413510558 . [DOI] [PubMed] [Google Scholar]
• 44.Hu X, Hughes J, Fisher P, Lorenc A, Purtell R, Park AL, et al. A pragmatic observational feasibility study on integrated treatment for musculoskeletal disorders: Design and protocol. Chinese journal of integrative medicine. 2013. Epub 2013/12/18. 10.1007/s11655-013-1557-9 . [DOI] [PubMed] [Google Scholar]
• 45.Misumi Y, Nishio M, Takahashi T, Ohyanagi F, Horiike A, Murakami H, et al. A feasibility study of carboplatin plus irinotecan treatment for elderly patients with extensive disease small-cell lung cancer. Japanese journal of clinical oncology. 2014;44(2):116–21. Epub 2013/12/18. 10.1093/jjco/hyt195 . [DOI] [PubMed] [Google Scholar]
• 46.Penn L, Ryan V, White M. Feasibility, acceptability and outcomes at a 12-month follow-up of a novel community-based intervention to prevent type 2 diabetes in adults at high risk: mixed methods pilot study. BMJ Open. 2013;3(11):e00358510.1136/bmjopen-2013-003585 [DOI] [PMC free article] [PubMed] [Google Scholar]
• 47.Pompeu JE, Arduini LA, Botelho AR, Fonseca MB, Pompeu SM, Torriani-Pasin C, et al. Feasibility, safety and outcomes of playing Kinect Adventures! for people with Parkinson's disease: a pilot study. Physiotherapy. 2014;100(2):162–8. Epub 2014/04/08. 10.1016/j.physio.2013.10.003 . [DOI] [PubMed] [Google Scholar]
• 48.Lancaster GA. Pilot and feasibility studies come of age!Pilot and Feasibility Studies. 2015;1(1):1. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 49.Bugge C, Williams B, Hagen S, Logan J, Glazener C, Pringle S, et al. A process for Decision-making after Pilot and feasibility Trials (ADePT): development following a feasibility study of a complex intervention for pelvic organ prolapse. Trials. 2013;14:353 Epub 2013/10/29. 10.1186/1745-6215-14-353 ; PubMed Central PMCID: PMCPmc3819659. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 50.Lee EC, Whitehead AL, Jacques RM, Julious SA. The statistical interpretation of pilot trials: should significance thresholds be reconsidered?BMC medical research methodology. 2014;14:41 Epub 2014/03/22. 10.1186/1471-2288-14-41 ; PubMed Central PMCID: PMCPmc3994566. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 51.Moore CG, Carter RE, Nietert PJ, Stewart PW. Recommendations for planning pilot studies in clinical and translational research. Clinical and translational science. 2011;4(5):332–7. Epub 2011/10/28. 10.1111/j.1752-8062.2011.00347.x ; PubMed Central PMCID: PMCPmc3203750. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 52.Brooks D, Stratford P. Pilot studies and their suitability for publication in physiotherapy Canada. Physiotherapy Canada Physiotherapie Canada. 2009;61(2):66–7. Epub 2010/03/02. 10.3138/physio.61.2.66 ; PubMed Central PMCID: PMCPmc2792235. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 53.Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American journal of public health. 1999;89(9):1322–7. Epub 1999/09/04. ; PubMed Central PMCID: PMCPmc1508772. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 54.Le Reste JY, Nabbe P, Rivet C, Lygidakis C, Doerr C, Czachowski S, et al. The European general practice research network presents the translations of its comprehensive definition of multimorbidity in family medicine in ten European languages. PloS one. 2015;10(1):e0115796 Epub 2015/01/22. 10.1371/journal.pone.0115796 ; PubMed Central PMCID: PMCPmc4301631. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 55.Vanderver A, Prust M, Tonduti D, Mochel F, Hussey HM, Helman G, et al. Case definition and classification of leukodystrophies and leukoencephalopathies. Molecular genetics and metabolism. 2015;114(4):494–500. Epub 2015/02/05. 10.1016/j.ymgme.2015.01.006 . [DOI] [PMC free article] [PubMed] [Google Scholar]
• 56.San L, Serrano M, Canas F, Romero SL, Sanchez-Cabezudo A, Villar M. Towards a pragmatic and operational definition of relapse in schizophrenia: A Delphi consensus approach. International journal of psychiatry in clinical practice. 2015:1–9. Epub 2014/12/31. 10.3109/13651501.2014.1002501 . [DOI] [PubMed] [Google Scholar]
• 57.Post MW. Definitions of quality of life: what has happened and how to move on. Topics in spinal cord injury rehabilitation. 2014;20(3):167–80. Epub 2014/12/09. 10.1310/sci2003-167 ; PubMed Central PMCID: PMCPmc4257148. [DOI] [PMC free article] [PubMed] [Google Scholar]
• 58.Parnell RW. Health examinations of students; pilot survey in Oxford. Lancet. 1947;2(6487):939–41. Epub 1947/12/27. . [PubMed] [Google Scholar]

Supplementary Materials

Data Availability Statement

Due to a requirement by the ethics committee that the authors specified when the data will be destroyed, the authors are not able to give unlimited access to the Delphi study quantitative data. These data are available from Professor Sandra Eldridge. Data will be available upon request to all interested researchers. Qualitative data from the Delphi study are not available because the authors do not have consent from participants for wider distribution of this more sensitive data.