We learn by solving problems. During evolution humans have developed the ability not only to learn individually, but also to share the acquired knowledge. Indeed, there is little doubt about the role played by the natural selection in the emerging social nature of humans^1,2. Human beings probably have an innate tendency to operate in groups:Pan troglodytes (common chimpanzee), the closest living relative of modern humans, is known to have a noticeable social structure³. The emerging skill of social problem solving has allowed our ancestors to face the challenges of gradually increasing complexity. This factor not only determined our evolutionary success, but it still shapes our cultural and social behavior⁴. In addition, everybody has at least minimal capacity to coordinate the activity of others and direct the process of problem solving^5,6. However, the existence of this ability is by no means obvious, as it requires an intricate synchronization between perception, communication and cognitive representation of the situation-task-others triad. Social Darwinism provides a framework to study the evolution this capacity, not just in terms of neurophysiology, but also in relation to the cognitive activities behind social problem solving.

On the other hand, any activity of intrinsically social nature requires a robust level of collaboration between the interacting subjects. The existence of a large sub-population of cooperators guarantees a stable support for the community, but at the same time creates a socioecologicalniche for the “free-riders” – the subjects benefiting from the available social support while providing little to its development. The free-riders are in fact known to be a well identifiable sub-population, competing to survive within various societies^7,8. Still, their individualism in facing new tasks can eventually create new knowledge that could not have been created by working only as a collective. An individual facing a new challenge in isolation has considerably smaller chances of solving it, despite relying on the available knowledge. Yet, if a solution under these circumstances is ultimately found, this subject learns much more than when the solution is found collectively. Modeling this phenomenon is at the hearth of the paper that follows.

Fast forward to the present day, engaging citizens and stakeholders in gathering constructive ideas and practices is among the most interesting challenges of our time. Referred to ascrowdsourcing^9,10, this process is often defined as a “collective intelligence system”^11,12. Crowdsourcing for ideas or solutions means that many subjects freely contribute independent suggestions. If a functional system of managing these suggestions is established, crowdsourcing can lead to surprising results, in the best sense of collective problem solving. In his 2009 Polymath project, Cambridge mathematician Tim Gowers proposed a very hard problem in his blog, that he estimated would take him months to solve. In only a few weeks, the community of his colleagues solved not only the original problem, but also a far much harder generalized version of it¹³. Needless to say, Wikipedia is probably the most successful example of the applied crowdsourcing^14,15. Indeed, the extraordinary development of information and communication technologies calls for new practices of “virtual participation”, that would enable the simultaneous engagement of a huge number of individuals¹⁶. Diverse studies and applications of crowdsourcing include crowd science^17,18, public participation in planning projects¹⁹, public health²⁰ and crowd funding²¹. Still, the full potentials of crowdsourcing in the digital era remains underused. Social media is probably the most powerful platform for this, which however requires a mechanism for systematic management of users’ engagement. In this context, the precise understanding of the origins and roles of free-riders in crowdsourcing has still to be strengthened, which is the core focus of this paper.

Psychologists have repeatedly shown that certain general intelligence emerges from the correlations among peoples’ performance^22,23. But the true nature of “collective intelligence”, which is expected to determine the capacity of collective problem solving, remains elusive^24,25. Woolleyet al. found a converging evidence of collective intelligence by explaining a group performance on a wide variety of tasks²⁶. Upon establishing the name “c-factor” for the obtained value of collective intelligence, they fund this value not to be strongly correlated with the average or maximum individual intelligence of the group members, but instead with other parameters such as social sensitivity, conversational turn-taking and the proportion of females. On a different front, there are massive efforts to develop computational^27,28,29 and experimental models^30,31,32,33 of collective social behavior, primarily within the framework of game theory and computational social science^34,35,36. These studies systematically show that cooperation is in general always beneficial for the community, which calls for mechanisms to discourage defection and oppose its strong appeal as a temporary benefit for an individual^{37,38,39,40,41,42}.

All these findings reveal an extremely complex scenario, in which too many dimensions must be taken into account to obtain even a nearly realistic model of crowdsourcing. Seeking to implement at least the core assumptions, we propose a stylized model linking the process of crowdsourcing to better studied process of collective problem solving. Our model is grounded in the fact that a (social) player can choose to either share the acquire knowledge with others or not, which in turn determines the amount of help obtained from the community, dictating the improvement of this player’s individual skills. Our model is implemented both computationally as a stochastic evolutionary system and analytically via methods of statistical physics. The model is inspired by the sociopsychological bases of knowledge dynamics that follow from the discussion above. As we show, similarly to defection emergence in game-theoretic models, free-riding here is a direct consequence of the evolutionary nature of our model and the interaction between the involved degrees of freedom. However, despite evolving towards maximizing the knowledge of the individual subjects, our model reveals a critical group size where an interesting balance between collective and individual approaches is maintained. This is, to our best knowledge, an entirely new model of crowdsourcing.

Below we present the computational model, which for better clarity, we do in several steps. First we explain the process that we wish to model. Our interest is to see how dividing a given population with fixed number of subjects (below called players) into several smaller groups, will influence the ability of these groups to cope with the problems (below called tasks) of different difficulty. We look on one end at the situation where everybody is in the same group and on the other at the scenario of each player working alone. Between these two extremes, we look for the optimal group size that would allow for its players to learn the most. We are also interested in how the average intensity of collaboration relates to the group size.

General setting

We consider a population ofN players divided inton groups. Each group has the same size (number of players)S, so that. We takeN = 100 and consider seven specific choices ofS to keepn integer. These sizes areS = 1, 2, 5, 10, 25, 50, 100, respectively corresponding to dividing the population inton = 100, 50, 20, 10, 4, 2, 1 groups. The total number of playersN = 100 always remains fixed.

Collaboration and task assignment

Oncen andS are selected, we proceed by assigning a valuep_i to each playeri in each group. Eachp_i is chosen uniformly at random between 0 and 1.p_i identifies the player’s tendency to collaborate with other group members when solving a certain task (the tendency to work collectively as opposed to individually). Small values ofp_i (close to 0) indicate a preference for individualism, while large values (close to 1) indicate a tendency to collaborate. The valuesp_i do not change and are hence characteristic for each player. Next, a group is assigned a task to be solved, quantified by the valueR, which is randomly uniformly chosen between 0 and 1.R measures the simplicity of the task, in the sense thatR close to 1 indicates an easy task, whereasR close to 0 implies a hard task (in other words, the difficulty of the task is 1 − R). Each ofn groups that work in parallel is assigned the tasks of the same simplicityR. ForR we consider the following thirteen choicesR = 0.0001, 0.001, 0.01, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. Note that our scale for smallerR is exponential, while it is linear for largerR, in order to better examine very hard tasks where collective working is expected to be beneficial.

The definition of a simulation

For each choice of group sizeS and for each choice of tasks simplicityR (both among the pre-specified 7 × 13 values mentioned above), we run a sequence of crowdsourcing games detailed below. Thus, each simulation is carried out for a fixed choice ofS andR, which are the two main parameters in our analysis that follows. Our interest is to examine the task solving behavior of a group ofS players when confronted with a task of simplicityR. The games in each simulation are done separately within each of then groups.

One iteration

One iteration (or a time-step) for a given group consists of the following three steps:

• We establish a subgroup of players which play this iteration together and call them ‘collectivists’. Each playeri joins the collectivists with the probabilityp_i. Each player that did not join the collectivists, plays this iteration alone. We call each such player an ‘individualist’. Thus, a group is in each iteration divided into a subgroup of collectivists, in addition to the remaining number of individualists. Thus, largerp_i means that for most iterations this player will end up as a collectivist.

• For each individualist separately we choose a number for this iterationG_i. ThisG_i is the desired gain for this player, whose meaning is clarified below.G_i is an integer chosen uniformly at random between 1 and 10. LargerG_i means smaller chance to win (solve the task), but with potentially bigger gain if the task is actually solved.

• Now, each collectivist solves the task with the probabilityR, while each individualist solves the task with the probability (sinceR ≤ 1, largerG_i means smaller chance of solving). These two different probabilities model the fact that by approaching the task in isolation, an individualist has in fact to solve aG_i times harder task. In contrast, collectivists face the task together which improves the chance for each of them to solve it. This difference is reflected in their respective scores, as explained below.

Each iteration finishes with some players winning (solving the task) and others losing (not solving the task). Both a collectivist and an individualist can win or lose. Next iteration begins with a new draw of collectivists using the same valuesp_i. These rules apply equally for alln groups that play simultaneously and in all iterations.

Group’s capacity and player’s fitness

Once the winners and losers for a given iteration are known, we proceed with assigning the scores. For a given group we introduce an integer parameter ∑_j called capacity (we usej to index groups, in order to differentiate fromi which indexes players within a group). Capacity is simply the number of iterations in which at least one collectivist has solved the task, regardless ofR. It models the collective knowledge of a group, i.e., the group’s capacity to solve increasingly challenging tasks. It is defined for each group, regardless of its iteration-dependent divisions into collectivists and individualists. Initially we set for all groups ∑_j = 0. We update ∑_j → ∑_j + 1 each time at least one player playing as a collectivist solves the task (and regardless of how many individualists solve it). Next we introduce for each player a parameter called fitness or pay-off π_i. It models player’s own benefit in terms of new knowledge acquired. A collectivist who solves a task contributes to each player in the group (collectivist and individualist) the additional fitness, where ∑_i is the updated capacity. A collectivist who fails to solve the task does not contribute anything. An individualist who solves the task gains additional fitnessG_i without contributing anything to the other players, while an individualist who fails makes no additional gain. Still, each individualist always gains per each collectivist that had solved the task. This organization of pay-offs models the fact that collectivists distribute new knowledge both to themselves and to all others, while individualists keep it for themselves. Collectivists however solve easier tasks since they work together, but with potentially lesser new knowledge (fitness) gained for each of them separately. In contrast, by working alone, individualists solve harder tasks but learn much more when they actually solve them, while avoiding to share this new knowledge with anyone else.

Motivation behind two scoring scales

Our model attempts to portray the incremental nature of human advances, for which there is evidence of superlinear behavior. To illustrate with a toy example, the invention of the microscope (first advancement), linked to the discovery of a new molecule (second advancement), can ultimately lead to the creation of the cure for a dangerous disease (third advancement). This represents a chain of fitness gains, with the total gain much larger than the simple addition of the payoffs due to single advancements in optics and chemistry. Thus, two distinct gaining schemes can be established: the individual scheme related to the skills for solving daily problems in a given time and context and the global scheme which represents the society knowledge accumulated over the history. These two we respectively model as the payoff and the capacity.

Example

To clarify our scoring scheme, say in a given iterations collectivists manage to solve the task in a groupj of sizeS. The group’s capacity increases by one ∑_j → ∑_j + 1. Fitness updates are as follows: each collectivist, regardless of having solved the task or not, gains. Each individualist that solves the task gains. A failing individualist makes the same gain as a collectivist. Thus, an individualist can count one the same gain as a collectivist, but if no collectivist solves that task, even that gain is lost. Both capacity and the fitness increase monotonically over time, depending on the distribution of collaboration tendenciesp_i and the values ofS andR.

Rounds and replacement of players

n groups of sizeS play the game simultaneously for a givenR. After 1000 iterations, which we call one round, the game is interrupted. We compute the mean fitness of all players regardless the group they belong to. We remove at random 20% of players whose fitness is below. They are replaced by new players whosep_i are drawn anew, so that the groups’ sizesS are preserved. All groups’ capacities ∑_j and all players’ fitnesses π_i are reset to 0. The valueR remains the same. Note that the only thing that has changed from one round to another is the structure of groups in terms of players’p_i, i.e., the distribution ofp_i within each group. Better fit players are kept in the game, in addition to 80% of lesser fit players. It is the player’sp_i and its relation with other player’sp_i-s that dictates the player’s overall performance in any game.

Full game and full simulation

A round consists of 1000 iterations and a game consists of 2000 rounds. Thus, the systems evolves over 2000 rounds, with evolutionary selection being applied at the beginning of each round. We checked that the numbers of iterations and rounds are sufficient for reaching a stable configuration. Upon finishing a game, we are left with the distribution ofp_i that optimally fits for solving the task of given simplicityR while operating in a group of sizeS. For better statistics, we conduct 20 games for each choice ofS andR and average the results, so that the final values do not depend on random realizations in our model. As we are interested only in the final distribution ofp_i, the choice of the initial distribution forp_i basically plays no role.

For each simulation done for fixedS andR, we store the following quantities:, the mean of player’s collaboration probabilitiesp_i in one group,, the mean players’ fitness π_i in one group and ∑_max, the capacity ∑ reached by a given group after 2000 rounds (i.e., the final value of ∑). Each quantity is stored after averaging over 20 games. Our first result, as expected by evolutionary nature of our algorithm, is that for any fixedR, the final results for all groups of given sizeS are the same. So for example, alln = 4 groups withS = 25 players ultimately reach the same, and ∑_max for any givenR. This means it suffices to consider only one group of sizeS. Therefore, in what follows we represent our findings via final values of, and ∑_max as function ofR andS.

We begin by examining, the mean collaboration probability in a group of sizeS which is undertaking the task of simplicityR. The results are shown inFig. 1 as a surfaceplot. For very hard problems (smallR), players have no preference on collaborating, so regardless ofS. In contrast, for very easy tasks, players in small groups strongly tend to collaborate, while players in big groups tend to work individually.

The value of as a function of group sizeS and the tasks simplicityR.

Colors denote (red - large, green - medium, blue - small). Note that the scale inR is illustrative.

Full size image

On the other hand, for small groups the tendency to collaborate increases with increasing task simplicity, while for big groups this tendency actually decreases. In between, for groups of size roughlyS = 50, the collaboration probability is basically constant withR and hence independent on the task that such group is facing.

To examine these trends in more detail, we show inFig. 2 the sequence of profiles of for constantR (left) and for constantS (right).

Left: the values of as function of group sizeS for varying values ofR (see legend). Right: the values of as function of task simplicityR for varying values ofS (see legend).

Full size image

Looking at the left plot, for the most difficult tasks the is indeed constant with the group size, while for simpler tasks it exhibits a clear trend: smaller groups tend to collaborate more intensely, while players in the larger groups are predominantly individualists. Groups with approximately 50 players seem always to display regardless ofR. This is also confirmed by the plots on right inFig. 2: all curves with exception ofS = 100 indicate that collaboration propensity increases with task simplicity. This increase becomes more and more moderate with the group size, so that finally, forS = 100 we find a clear negative trend: collaboration propensity decreases with the task simplicity. In between we find an almost constant curve for roughlyS = 50.

We interpret these findings as follows. When the task is extremely difficult, it is hard to solve it collectively as well as individually. Thus, in such cases there is no particular preference for either approach, regardless of the group size. As the task becomes gradually easier, it is less beneficial for players in small groups to work individually, since this quickly reduces the size of the collaborating subgroup, ultimately making it so small that there is little or no fitness gained from it for anybody. This in turn means that by collaborating a player is more safe to gain at least some fitness. In contrast, easier problems encourage the players in large groups to defect collaboration, since the collaborating subgroup is always fairly large and contributes significant fitness. These circumstances allow for more “individually oriented” players to seek their luck by defecting, which they can do safely since at least some fitness is practically guaranteed. This feature captures the concept of free-riders discussed in the Introduction. These players contribute no knowledge of their own to the collective, but at the same time manage to benefit from the novelties reached collectively by the rest of the group. The equilibrium case of groups with approximatelyS = 50 players is theoptimal system where both collaboration and individualism are equally beneficial for task of any difficulty. Such group has a good balance between its collectivist part which slowly but surely generates new knowledge and its individualists who are still on average equally fit as collectivists.

To support these observations with the results for the mean fitness and the maximal capacity ∑_max, we show inFig. 3 the surfaceplots for these two quantities. Both surfaceplots display similar features: as expected, the biggest values are found for the simplest tasks when faced by the largest groups.

Left: the mean fitness reached by a given group of sizeS when solving the task of simplicityR.

Right: the same for maximum capacity ∑_max.

Full size image

Smallest and ∑_max correspond to the most difficult of tasks when approached by the smallest groups. The group withS = 50 players optimizes the gain in both quantities and in fact performs better than the group withS = 100 players. This resonates with the earlier observation that this group has the best balance between collectivism and individualism. Indeed, by working excessively in isolation and looking too eagerly for free rides, the players inS = 100 group at the end achieve lesser total fitness, despite their larger size which could have made them all much better fit. It is easy to conclude that this trend would actually further increase with even larger groups, leavingS = 50 as the only optimal case. This confirms that the knowledge accumulation in too large groups is actually slower than in groups of certain moderate size, when the balanced tendencies for collective and individual approach. On the other hand, one could argue that in a real-life scenario this might in fact be in the long-run beneficial for the society. Namely, the individually gained unique knowledge, that could not have been gained via pure collectivism, might eventually be shared with the rest of the community and hence available to everyone. By running the simulations with a number of players different fromN = 100, we confirmed that the equilibrium group size is indeed robust to this and is always approximatelyN/2.

Analytical approach

We now show that the above numerical findings can be justified via analytical treatment, relying on standard methods from statistical physics. Let us consider the average fitness for a given player, depending on the player’s strategy and the strategy of the opponents. We callj the player’s group and α_j the collectivists density in this group. The average fitness of a collectivist inj at timet is then

where is the capacity reached at timet in groupj and indicate averaging. On its turn, individualist (defector) average fitness is

where, by definition, is the mean value of the integer random number. Of course, within the same groupj it is always, so that in absence of competition between the groups (i.e.,S = N, n = 1) individualism is the best strategy, as confirmed by the simulations. On the other hand, forn ≥ 2, let us think about a groupl made up by only collectivists, compared with a groupj with just one individualist (we consider this scenario as it the best situation for a defector). In this case,l's collectivist have a better fitness thanj's defector if it results

Here we distinguish two cases, the large group size and the opposite regime whereS ~ 1. We treat these two cases separately below.

In the large group size limit, theequation (3) reduces to

Equation (4) implies that in order to have the best fitness, collectivist inl must have reached at timet a capacity such that

This clearly implies that δ increases monotonically withR, going from 0 to 11/5. This means that, with increasingR cooperative groups have to reach a higher capacity to thrive, but this is balanced by the higher ease to solve the tasks: the combined effect is that, given the group size, the final cooperation level does not depend strongly onR, as shown inFig. 1. On the other hand, increasingR enhances the global fitness, because it is easier to solve the tasks and, for a cooperative group, reach a higher capacity with respect to the others, as depicted inequation (5). This behavior is confirmed inFig. 3, where an increase for increasingR is observable in the average agent fitness. The narrow tilt change for can be understood considering that whenR becomes smaller thanR^* = 0.5, also δ gets smaller than, that is, there is practically no need to reach a higher capacity for cooperators to thrive, lowering the global fitness.

In the limit ofS approaching to 1,equation (3) reduces to

relation automatically satisfied for, implying that as a group solves the second task, cooperation becomes the best strategy (in other words, free-riding is no longer convenient). This is also confirmed inFig. 1, where it is clear that for small group size cooperation is in general enhanced and increases more with increasingR. This analytical reasoning hence entirely confirms our numerical findings.

The above numerical and analytical study showed two consistent results. First, when facing the problems whose difficulty is not extreme, small groups tend to collaborate significantly more than large groups. In fact, very large groups end up having lesser average fitness due to excessive individualism being stimulated in these conditions. Small groups are actually known to have weaker knowledge and culture preservation due to not having enough “critical mass” of subjects⁴³. In contrast, large groups have too many free-riders which slows down the knowledge generation process. This agrees with the sociopsychological observations about the sub-population of free-riders, who use the niche provided by the stable knowledge support generated by the collaborating majority. Still, when tasks become more demanding, large groups gradually discourage individualism and cooperation restores to a level equal to individualism. Second, there exist and optimal group size, where the stimulations for collectivism and individualism are in balance so that both are equally encouraged, regardless of the task difficulty. Note that this effect is exclusively the consequence of the critical group size and the evolution which keeps the size fixed. Interestingly, groups of exactly this size ultimately manage to have not only the highest fitness, but also the best capacity. This indicates that under proper scenario, optimal utilization of crowdsourcing is possible only by virtue of managing the number of subjects involved in a certain experiment or real-life situation.

The stated conclusion holds even in the presence of defection, which is very realistic. Our evolution was tuned towards optimizing the fitness of individual players, rather than the group’s capacity, done by gradually eliminating less fit players. Note however, that under these circumstances, it still holds that the group of critical size performs best inboth average fitnessand maximal capacity. That is to say, for the critical size, the system simultaneously reaches both individual and collective optimum, i.e., they do not necessarily exclude each other, as it is usually the case in the game-theoretic models. So, while our results do confirm that free-riding is related to the defection phenomena known from game theory, we show that in the considered model of knowledge dynamics a certain amount of free-riding can play a particular role, as it opens the avenue for individually oriented subjects to gain unique knowledges. As already mentioned, free riding in the real world might ultimately prove advantageous for the society, since this unique knowledge can still at some point be shared with other and become part of the knowledge support provided for future generations⁴⁴. Not surprisingly, it is well known that several societies throughout history become powerful by benefiting from inventions coming from global civilization, while withholding their own knowledge and technology generated under these circumstances. Still, such societies in return do eventually provide benefits to the rest of the world via different means⁴⁵.

Limitations of the model involve the fact that players’ collaboration probabilities are not affected by outcomes in iterations and rounds. We considered more realistic to adopt an evolutionary model instead, where the group population is adjusted for better performance by searching for the optimal distribution of collaboration probabilities. Another limitation is related to not tuning our model to favor capacity instead of fitness. This would have a different outcome, not allowing for any individualism. However, given the ubiquity of free-riding in nature, we consider more instructive to include it in our model. Of course, there is always an open question of designing better and more realistic models, which with modern computing power is increasingly easy to simulate. As far as robustness to the choice of parameters goes, we have tried several choices of parameters defining our model (for example the span ofG_i) and have always recovered qualitatively the same results.

The most important direction of this field lies of course in testing the potentials of crowdsourcing via real experiments. This refers to social experiments with controlled rules and known participants, as well as to uncontrolled online experiments, where people are engaged spontaneously and without external organization¹⁷. Although several important results have been recently reported in this direction, a standard framework for undertaking such experiments has yet to be established. Finally, this related to extending the range of applicability of crowdsourcing, chiefly using Internet and tools such as Mechanical Turk⁴⁶, where the crucial issue revolves around findings ways to best stimulate and engage the participants to contribute ideas and thoughts. In conclusion, we hope that our results provide a new stepping stone for better understanding and eventual large-scale usage of crowdsourcing, which is to lead to better exploiting the full potential of our more than ever connected civilization.

How to cite this article: Guazzini, A.et al. Modeling crowdsourcing as collective problem solving.Sci. Rep.5, 16557; doi: 10.1038/srep16557 (2015).

• 12 September 2016

  A correction has been published and is appended to both the HTML and PDF versions of this paper. The error has not been fixed in the paper.

Work supported by the EU H2020-MSCA-ITN-2015 project COSMOS 642563, by the Creative Core FISNM-3330-13-500033, by the Slovenian Research Agency via program P1-0383 and project J1-5454, Commission (FP7-ICT-2013-10) Proposal No. 611299 SciCafe 2.0 and by project CLARA (CLoud plAtform and smart underground imaging for natural Risk Assessment), funded by the Italian Ministry of Education and Research (PON 2007-2013: Smart Cities and Communities and Social Innovation; Asse e Obiettivo: Asse II - Azione Integrata per la Società dell’Informazione; Ambito: Sicurezza del territorio).

Authors and Affiliations

1. Department of Science of Education and Psychology, University of Florence, Florence, Italy

   Andrea Guazzini, Camillo Donati & Annalisa Nardi

2. Center for the Study of Complex Dynamics, University of Florence, Florence, Italy

   Andrea Guazzini

3. Laboratory of Agent Based Social Simulation, Institute of Cognitive Science and Technology, National Research Council, Rome, Italy

   Daniele Vilone

4. Faculty of Information Studies in Novo mesto, Novo mesto, Slovenia

   Zoran Levnajić

5. Faculty of Computer and Information Science, University of Ljubljana, Ljubljana, Slovenia

   Zoran Levnajić

Contributions

A.G. and D.V. conceived the numerical experiments. C.D. and A.N. executed the numerical experiments. D.V. made the analytical calculations. A.G., D.V. and Z.L. analyzed the results and made figures. Z.L. wrote the manuscript, all authors reviewed the manuscript.

Competing interests

The authors declare no competing financial interests.

This work is licensed under a Creative Commons Attribution 4.0 International License. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in the credit line; if the material is not included under the Creative Commons license, users will need to obtain permission from the license holder to reproduce the material. To view a copy of this license, visithttp://creativecommons.org/licenses/by/4.0/

Reprints and permissions