In thedesign of experiments,optimal experimental designs (oroptimum designs^[2]) are a class ofexperimental designs that areoptimal with respect to somestatisticalcriterion. The creation of this field of statistics has been credited to Danish statisticianKirstine Smith.^[3][4]

In thedesign of experiments forestimatingstatistical models,optimal designs allow parameters to beestimated without bias and withminimum variance. A non-optimal design requires a greater number ofexperimental runs toestimate theparameters with the sameprecision as an optimal design. In practical terms, optimal experiments can reduce the costs of experimentation.

The optimality of a design depends on thestatistical model and is assessed with respect to a statistical criterion, which is related to the variance-matrix of the estimator. Specifying an appropriate model and specifying a suitable criterion function both require understanding ofstatistical theory and practical knowledge withdesigning experiments.

Optimal designs offer three advantages over sub-optimalexperimental designs:^[5]

1. Optimal designs reduce the costs of experimentation by allowingstatistical models to be estimated with fewer experimental runs.
2. Optimal designs can accommodate multiple types of factors, such as process, mixture, and discrete factors.
3. Designs can be optimized when the design-space is constrained, for example, when the mathematical process-space contains factor-settings that are practically infeasible (e.g. due to safety concerns).

Experimental designs are evaluated using statistical criteria.^[6]

It is known that theleast squares estimator minimizes thevariance ofmean-unbiasedestimators (under the conditions of theGauss–Markov theorem). In theestimation theory forstatistical models with onerealparameter, thereciprocal of the variance of an ("efficient") estimator is called the "Fisher information" for that estimator.^[7] Because of this reciprocity,minimizing thevariance corresponds tomaximizing theinformation.

When thestatistical model has severalparameters, however, themean of the parameter-estimator is avector and itsvariance is amatrix. Theinverse matrix of the variance-matrix is called the "information matrix". Because the variance of the estimator of a parameter vector is a matrix, the problem of "minimizing the variance" is complicated. Usingstatistical theory, statisticians compress the information-matrix using real-valuedsummary statistics; being real-valued functions, these "information criteria" can be maximized.^[8] The traditional optimality-criteria areinvariants of theinformation matrix; algebraically, the traditional optimality-criteria arefunctionals of theeigenvalues of the information matrix.

• A-optimality ("average" ortrace)
  • One criterion isA-optimality, which seeks to minimize thetrace of theinverse of the information matrix. This criterion results in minimizing the average variance of the estimates of the regression coefficients.
• C-optimality
  • This criterion minimizes the variance of abest linear unbiased estimator of a predetermined linear combination of model parameters.
• D-optimality (determinant)
  • A popular criterion isD-optimality, which seeks to minimize |(X'X)⁻¹|, or equivalently maximize thedeterminant of theinformation matrix X'X of the design. This criterion results in maximizing thedifferential Shannon information content of the parameter estimates.
• E-optimality (eigenvalue)
  • Another design isE-optimality, which maximizes the minimumeigenvalue of the information matrix.
• S-optimality^[9]
  • This criterion maximizes a quantity measuring the mutual column orthogonality of X and thedeterminant of the information matrix.
• T-optimality
  • This criterion maximizes the discrepancy between two proposed models at the design locations.^[10]

Other optimality-criteria are concerned with the variance ofpredictions:

• G-optimality
  • A popular criterion isG-optimality, which seeks to minimize the maximum entry in thediagonal of thehat matrix X(X'X)⁻¹X'. This has the effect of minimizing the maximum variance of the predicted values.
• I-optimality (integrated)
  • A second criterion on prediction variance isI-optimality, which seeks to minimize the average prediction varianceover the design space.
• V-optimality (variance)
  • A third criterion on prediction variance isV-optimality, which seeks to minimize the average prediction variance over a set of m specific points.^[11]

In many applications, the statistician is most concerned with a"parameter of interest" rather than with"nuisance parameters". More generally, statisticians considerlinear combinations of parameters, which are estimated via linear combinations of treatment-means in thedesign of experiments and in theanalysis of variance; such linear combinations are calledcontrasts. Statisticians can use appropriate optimality-criteria for suchparameters of interest and forcontrasts.^[12]

Catalogs of optimal designs occur in books and in software libraries.

In addition, majorstatistical systems likeSAS andR have procedures for optimizing a design according to a user's specification. The experimenter must specify amodel for the design and an optimality-criterion before the method can compute an optimal design.^[13]

Some advanced topics in optimal design require morestatistical theory and practical knowledge in designing experiments.

Since the optimality criterion of most optimal designs is based on some function of the information matrix, the 'optimality' of a given design ismodel dependent: While an optimal design is best for thatmodel, its performance may deteriorate on othermodels. On othermodels, anoptimal design can be either better or worse than a non-optimal design.^[14] Therefore, it is important tobenchmark the performance of designs under alternativemodels.^[15]

The choice of an appropriate optimality criterion requires some thought, and it is useful to benchmark the performance of designs with respect to several optimality criteria. Cornell writes that

since the [traditional optimality] criteria . . . are variance-minimizing criteria, . . . a design that is optimal for a given model using one of the . . . criteria is usually near-optimal for the same model with respect to the other criteria.

Indeed, there are several classes of designs for which all the traditional optimality-criteria agree, according to the theory of "universal optimality" ofKiefer.^[17] The experience of practitioners like Cornell and the "universal optimality" theory of Kiefer suggest that robustness with respect to changes in theoptimality-criterion is much greater than is robustness with respect to changes in themodel.

High-quality statistical software provide a combination of libraries of optimal designs or iterative methods for constructing approximately optimal designs, depending on the model specified and the optimality criterion. Users may use a standard optimality-criterion or may program a custom-made criterion.

All of the traditional optimality-criteria areconvex (or concave) functions, and therefore optimal-designs are amenable to the mathematical theory ofconvex analysis and their computation can use specialized methods ofconvex minimization.^[18] The practitioner need not selectexactly one traditional, optimality-criterion, but can specify a custom criterion. In particular, the practitioner can specify a convex criterion using the maxima of convex optimality-criteria andnonnegative combinations of optimality criteria (since these operations preserveconvex functions). Forconvex optimality criteria, theKiefer-Wolfowitzequivalence theorem allows the practitioner to verify that a given design is globally optimal.^[19] TheKiefer-Wolfowitzequivalence theorem is related with theLegendre-Fenchelconjugacy forconvex functions.^[20]

If an optimality-criterion lacksconvexity, then finding aglobal optimum and verifying its optimality often are difficult.

When scientists wish to test several theories, then a statistician can design an experiment that allows optimal tests between specified models. Such "discrimination experiments" are especially important in thebiostatistics supportingpharmacokinetics andpharmacodynamics, following the work ofCox and Atkinson.^[21]

When practitioners need to consider multiplemodels, they can specify aprobability-measure on the models and then select any design maximizing theexpected value of such an experiment. Such probability-based optimal-designs are called optimalBayesiandesigns. SuchBayesian designs are used especially forgeneralized linear models (where the response follows anexponential-family distribution).^[22]

The use of aBayesian design does not force statisticians to useBayesian methods to analyze the data, however. Indeed, the "Bayesian" label for probability-based experimental-designs is disliked by some researchers.^[23] Alternative terminology for "Bayesian" optimality includes "on-average" optimality or "population" optimality.

Scientific experimentation is an iterative process, and statisticians have developed several approaches to the optimal design of sequential experiments.

Sequential analysis was pioneered byAbraham Wald.^[24] In 1972,Herman Chernoff wrote an overview of optimal sequential designs,^[25] whileadaptive designs were surveyed later by S. Zacks.^[26] Of course, much work on the optimal design of experiments is related to the theory ofoptimal decisions, especially thestatistical decision theory ofAbraham Wald.^[27]

Optimal designs forresponse-surface models are discussed in the textbook by Atkinson, Donev and Tobias, and in the survey of Gaffke and Heiligers and in the mathematical text of Pukelsheim. Theblocking of optimal designs is discussed in the textbook of Atkinson, Donev and Tobias and also in the monograph by Goos.

The earliest optimal designs were developed to estimate the parameters of regression models with continuous variables, for example, byJ. D. Gergonne in 1815 (Stigler). In English, two early contributions were made byCharles S. Peirce andKirstine Smith.

Pioneering designs for multivariateresponse-surfaces were proposed byGeorge E. P. Box. However, Box's designs have few optimality properties. Indeed, theBox–Behnken design requires excessive experimental runs when the number of variables exceeds three.^[28]Box's"central-composite" designs require more experimental runs than do the optimal designs of Kôno.^[29]

The optimization of sequential experimentation is studied also instochastic programming and insystems andcontrol. Popular methods includestochastic approximation and other methods ofstochastic optimization. Much of this research has been associated with the subdiscipline ofsystem identification.^[30]In computationaloptimal control, D. Judin & A. Nemirovskii andBoris Polyak has described methods that are more efficient than the (Armijo-style)step-size rules introduced byG. E. P. Box inresponse-surface methodology.^[31]

Adaptive designs are used inclinical trials, and optimaladaptive designs are surveyed in theHandbook of Experimental Designs chapter by Shelemyahu Zacks.

There are several methods of finding an optimal design, given ana priori restriction on the number of experimental runs or replications. Some of these methods are discussed by Atkinson, Donev and Tobias and in the paper by Hardin andSloane. Of course, fixing the number of experimental runsa priori would be impractical. Prudent statisticians examine the other optimal designs, whose number of experimental runs differ.

In the mathematical theory on optimal experiments, an optimal design can be aprobability measure that issupported on an infinite set of observation-locations. Such optimal probability-measure designs solve a mathematical problem that neglected to specify the cost of observations and experimental runs. Nonetheless, such optimal probability-measure designs can bediscretized to furnishapproximately optimal designs.^[32]

In some cases, a finite set of observation-locations suffices tosupport an optimal design. Such a result was proved by Kôno andKiefer in their works onresponse-surface designs for quadratic models. The Kôno–Kiefer analysis explains why optimal designs for response-surfaces can have discrete supports, which are very similar as do the less efficient designs that have been traditional inresponse surface methodology.^[33]

In 1815, an article on optimal designs forpolynomial regression was published byJoseph Diaz Gergonne, according toStigler.

Charles S. Peirce proposed an economic theory of scientific experimentation in 1876, which sought to maximize the precision of the estimates. Peirce's optimal allocation immediately improved the accuracy of gravitational experiments and was used for decades by Peirce and his colleagues. In his 1882 published lecture atJohns Hopkins University, Peirce introduced experimental design with these words:

Logic will not undertake to inform you what kind of experiments you ought to make in order best to determine the acceleration of gravity, or the value of the Ohm; but it will tell you how to proceed to form a plan of experimentation.

[....] Unfortunately practice generally precedes theory, and it is the usual fate of mankind to get things done in some boggling way first, and find out afterward how they could have been done much more easily and perfectly.^[34]

Kirstine Smith proposed optimal designs for polynomial models in 1918. (Kirstine Smith had been a student of the Danish statisticianThorvald N. Thiele and was working withKarl Pearson in London.)

• Atkinson, A. C.; Donev, A. N.; Tobias, R. D. (2007).Optimum experimental designs, withSAS.Oxford University Press. pp. 511+xvi.ISBN 978-0-19-929660-6.
• Chernoff, Herman (1972).Sequential analysis and optimal design. Society for Industrial and Applied Mathematics.ISBN 978-0-89871-006-9.
• Fedorov, V. V. (1972).Theory of Optimal Experiments. Academic Press.
• Fedorov, Valerii V.; Hackl, Peter (1997).Model-Oriented Design of Experiments. Lecture Notes in Statistics. Vol. 125. Springer-Verlag.
• Goos, Peter (2002).The Optimal Design of Blocked and Split-plot Experiments. Lecture Notes in Statistics. Vol. 164. Springer. Archived fromthe original on 2020-10-31. Retrieved2009-05-28.
• Kiefer, Jack Carl (1985).Brown;Olkin, Ingram;Sacks, Jerome; et al. (eds.).Jack Carl Kiefer: Collected papers III—Design of experiments. Springer-Verlag and the Institute of Mathematical Statistics. pp. 718+xxv.ISBN 978-0-387-96004-3.
• Logothetis, N.;Wynn, H. P. (1989).Quality through design: Experimental design, off-line quality control, and Taguchi's contributions. Oxford U. P. pp. 464+xi.ISBN 978-0-19-851993-5.
• Nordström, Kenneth (May 1999)."The life and work of Gustav Elfving".Statistical Science.14 (2):174–196.doi:10.1214/ss/1009212244.JSTOR 2676737.MR 1722074.
• Pukelsheim, Friedrich (2006).Optimal design of experiments. Classics in Applied Mathematics. Vol. 50 (republication with errata-list and new preface of Wiley (0-471-61971-X) 1993 ed.).Society for Industrial and Applied Mathematics. pp. 454+xxxii.ISBN 978-0-89871-604-7.
• Shah, Kirti R. & Sinha, Bikas K. (1989).Theory of Optimal Designs. Lecture Notes in Statistics. Vol. 54. Springer-Verlag. pp. 171+viii.ISBN 978-0-387-96991-6.

The textbook by Atkinson, Donev and Tobias has been used for short courses for industrial practitioners as well as university courses.

• Atkinson, A. C.; Donev, A. N.; Tobias, R. D. (2007).Optimum experimental designs, withSAS. Oxford University Press. pp. 511+xvi.ISBN 978-0-19-929660-6.
• Logothetis, N.;Wynn, H. P. (1989).Quality through design: Experimental design, off-line quality control, and Taguchi's contributions. Oxford U. P. pp. 464+xi.ISBN 978-0-19-851993-5.

Optimalblock designs are discussed by Bailey and by Bapat. The first chapter of Bapat's book reviews thelinear algebra used by Bailey (or the advanced books below). Bailey's exercises and discussion ofrandomization both emphasize statistical concepts (rather than algebraic computations).

• Bailey, R. A. (2008).Design of Comparative Experiments. Cambridge U. P.ISBN 978-0-521-68357-9. Draft available on-line. (Especially Chapter 11.8 "Optimality")
• Bapat, R. B. (2000).Linear Algebra and Linear Models (Second ed.). Springer.ISBN 978-0-387-98871-9. (Chapter 5 "Block designs and optimality", pages 99–111)

Optimalblock designs are discussed in the advanced monograph by Shah and Sinha and in the survey-articles by Cheng and by Majumdar.

• Chernoff, Herman (1972).Sequential Analysis and Optimal Design.SIAM.ISBN 978-0-89871-006-9.
• Fedorov, V. V. (1972).Theory of Optimal Experiments. Academic Press.
• Fedorov, Valerii V.; Hackl, Peter (1997).Model-Oriented Design of Experiments. Vol. 125. Springer-Verlag.
• Goos, Peter (2002).The Optimal Design of Blocked and Split-plot Experiments. Vol. 164. Springer.
• Goos, Peter & Jones, Bradley (2011).Optimal design of experiments: a case study approach. Chichester Wiley. p. 304.ISBN 978-0-470-74461-1.
• Kiefer, Jack Carl. (1985).Brown, Lawrence D.;Olkin, Ingram; Jerome Sacks; Wynn, Henry P (eds.).Jack Carl Kiefer Collected Papers III Design of Experiments. Springer-Verlag and theInstitute of Mathematical Statistics.ISBN 978-0-387-96004-3.
• Pukelsheim, Friedrich (2006).Optimal Design of Experiments. Vol. 50.Society for Industrial and Applied Mathematics.ISBN 978-0-89871-604-7. Republication with errata-list and new preface of Wiley (0-471-61971-X) 1993
• Shah, Kirti R. & Sinha, Bikas K. (1989).Theory of Optimal Designs. Vol. 54. Springer-Verlag.ISBN 978-0-387-96991-6.

• Chaloner, Kathryn & Verdinelli, Isabella (1995)."Bayesian Experimental Design: A Review".Statistical Science.10 (3):273–304.CiteSeerX 10.1.1.29.5355.doi:10.1214/ss/1177009939.
• Ghosh, S.;Rao, C. R., eds. (1996).Design and Analysis of Experiments. Handbook of Statistics. Vol. 13. North-Holland.ISBN 978-0-444-82061-7.
  • "ModelRobust Designs".Design and Analysis of Experiments. Handbook of Statistics. pp. 1055–1099.
  • Cheng, C.-S. "Optimal Design: Exact Theory".Design and Analysis of Experiments. Handbook of Statistics. pp. 977–1006.
  • DasGupta, A. "Review of OptimalBayesian Designs".Design and Analysis of Experiments. Handbook of Statistics. pp. 1099–1148.
  • Gaffke, N. & Heiligers, B. "Approximate Designs forPolynomial Regression:Invariance,Admissibility, and Optimality".Design and Analysis of Experiments. Handbook of Statistics. pp. 1149–1199.
  • Majumdar, D. "Optimal and Efficient Treatment-Control Designs".Design and Analysis of Experiments. Handbook of Statistics. pp. 1007–1054.
  • Stufken, J. "OptimalCrossover Designs".Design and Analysis of Experiments. Handbook of Statistics. pp. 63–90.
  • Zacks, S. "Adaptive Designs for Parametric Models".Design and Analysis of Experiments. Handbook of Statistics. pp. 151–180.
• Kôno, Kazumasa (1962)."Optimum designs for quadratic regression onk-cube"(PDF).Memoirs of the Faculty of Science. Kyushu University. Series A. Mathematics.16 (2):114–122.doi:10.2206/kyushumfs.16.114.

• Gergonne, J. D. (November 1974) [1815]."The application of the method of least squares to the interpolation of sequences".Historia Mathematica.1 (4) (Translated by Ralph St. John andS. M. Stigler from the 1815 French ed.):439–447.doi:10.1016/0315-0860(74)90034-2.
• Stigler, Stephen M. (November 1974)."Gergonne's 1815 paper on the design and analysis of polynomial regression experiments".Historia Mathematica.1 (4):431–439.doi:10.1016/0315-0860(74)90033-0.
• Peirce, C. S (1876). "Note on the Theory of the Economy of Research".Coast Survey Report:197–201. (Appendix No. 14).NOAA PDF Eprint. Reprinted inCollected Papers of Charles Sanders Peirce. Vol. 7. 1958. paragraphs 139–157, and inPeirce, C. S. (July–August 1967). "Note on the Theory of the Economy of Research".Operations Research.15 (4):643–648.doi:10.1287/opre.15.4.643.JSTOR 168276.
• Smith, Kirstine (1918)."On the Standard Deviations of Adjusted and Interpolated Values of an Observed Polynomial Function and its Constants and the Guidance They Give Towards a Proper Choice of the Distribution of the Observations".Biometrika.12 (1/2):1–85.doi:10.2307/2331929.JSTOR 2331929.

• Design of experiments
• Regression analysis
• Statistical theory
• Optimal decisions
• Mathematical optimization
• Industrial engineering
• Systems engineering
• Statistical process control

• Webarchive template wayback links
• Articles with short description
• Short description matches Wikidata