"A Logical Calculus of the Ideas Immanent in Nervous Activity" is a 1943 article written byWarren McCulloch andWalter Pitts.^[1] The paper, published in the journalThe Bulletin of Mathematical Biophysics, proposed a mathematical model of the nervous system as a network of simple logical elements, later known as artificial neurons, orMcCulloch-Pitts neurons. These neurons receive inputs, perform a weighted sum, and fire an output signal based on a threshold function. By connecting these units in various configurations, McCulloch and Pitts demonstrated that their model could perform all logical functions.

It is a seminal work incognitive science,computational neuroscience,computer science, andartificial intelligence. It was a foundational result inautomata theory.John von Neumann cited it as a significant result.^[2]

The artificial neuron used in the original paper is slightly different from the modern version. They considered neural networks that operate in discrete steps of time${\displaystyle t=0,1,\dots }$.

The neural network contains a number of neurons. Let the state of a neuron${\displaystyle i}$ at time${\displaystyle t}$ be${\displaystyle N_i(t)}$. The state of a neuron can either be 0 or 1, standing for "not firing" and "firing". Each neuron also has a firing threshold${\displaystyle \theta }$, such that it fires if the total input exceeds the threshold.

Each neuron can connect to any other neuron (including itself) with positivesynapses (excitatory) or negative synapses (inhibitory). That is, each neuron can connect to another neuron with a weight${\displaystyle w}$ taking an integer value. A peripheral afferent is a neuron with no incoming synapses.

We can regard each neural network as adirected graph, with the nodes being the neurons, and the directed edges being the synapses. A neural network has a circle or a circuit if there exists a directed circle in the graph.

Let${\displaystyle w_{ij}(t)}$ be the connection weight from neuron${\displaystyle j}$ to neuron${\displaystyle i}$ at time${\displaystyle t}$, then its next state is$${\displaystyle N_i(t+1)=H\left(\sum _{j=1}^n{w}_{ij}(t)N_j(t)-{\theta }_i(t)\right),}$$where${\displaystyle H}$ is theHeaviside step function (outputting 1 if the input is greater than or equal to 0, and 0 otherwise).

The paper used, as alogical language for describing neural networks, "Language II" fromThe Logical Syntax of Language byRudolf Carnap with some notations taken fromPrincipia Mathematica byAlfred North Whitehead andBertrand Russell. Language II covers substantial parts of classical mathematics, includingreal analysis and portions of set theory.^[3]

To describe a neural network with peripheral afferents${\displaystyle N_1,N_2,\dots ,N_p}$ and non-peripheral afferents${\displaystyle N_{p+1},N_{p+2},\dots ,N_n}$ they considered logical predicate of form$${\displaystyle Pr(N_1,N_2,\dots ,N_p,t)}$$where${\displaystyle Pr}$ is afirst-order logicpredicate function (a function that outputs aboolean),${\displaystyle N_1,\dots ,N_p}$ are predicates that take${\displaystyle t}$ as an argument, and${\displaystyle t}$ is the onlyfree variable in the predicate. Intuitively speaking,${\displaystyle N_1,\dots ,N_p}$ specifies the binary input patterns going into the neural network over all time, and${\displaystyle Pr(N_1,N_2,\dots ,N_n,t)}$ is a function that takes some binary input patterns, and constructs an output binary pattern${\displaystyle Pr(N_1,N_2,\dots ,N_n,0),Pr(N_1,N_2,\dots ,N_n,1),\dots }$.

A logical sentence${\displaystyle Pr(N_1,N_2,\dots ,N_n,t)}$ is realized by a neural network iff there exists a time-delay${\displaystyle T\ge 0}$, a neuron${\displaystyle i}$ in the network, and an initial state for the non-peripheral neurons${\displaystyle N_{p+1}(0),\dots ,N_n(0)}$, such that for any time${\displaystyle t}$, the truth-value of the logical sentence is equal to the state of the neuron${\displaystyle i}$ at time${\displaystyle t+T}$. That is,$${\displaystyle \mathrm{\forall }t=0,1,2,\dots ,Pr(N_1,N_2,\dots ,N_p,t)=N_i(t+T)}$$

In the paper, they considered some alternative definitions of artificial neural networks, and have shown them to be equivalent, that is, neural networks under one definition realizes precisely the same logical sentences as neural networks under another definition.

They considered three forms of inhibition: relative inhibition, absolute inhibition, and extinction. The definition above is relative inhibition. By "absolute inhibition" they meant that if any negative synapse fires, then the neuron will not fire. By "extinction" they meant that if at time${\displaystyle t}$, any inhibitory synapse fires on a neuron${\displaystyle i}$, then${\displaystyle {\theta }_i(t+j)={\theta }_i(0)+b_j}$ for${\displaystyle j=1,2,3,\dots }$, until the next time an inhibitory synapse fires on${\displaystyle i}$. It is required that${\displaystyle b_j=0}$ for all large${\displaystyle j}$.

Theorem 4 and 5 state that these are equivalent.

They considered three forms of excitation: spatial summation, temporal summation, and facilitation. The definition above is spatial summation (which they pictured as having multiple synapses placed close together, so that the effect of their firing sums up). By "temporal summation" they meant that the total incoming signal is${\displaystyle \sum _{\tau =0}^T\sum _{j=1}^n{w}_{ij}(t)N_j(t-\tau )}$ for some${\displaystyle T\ge 1}$. By "facilitation" they meant the same as extinction, except that${\displaystyle b_j\le 0}$. Theorem 6 states that these are equivalent.

They considered neural networks that do not change, and those that change byHebbian learning. That is, they assume that at${\displaystyle t=0}$, some excitatory synaptic connections are not active. If at any${\displaystyle t}$, both${\displaystyle N_i(t)=1,N_j(t)=1}$, then any latent excitatory synapse between${\displaystyle i,j}$ becomes active. Theorem 7 states that these are equivalent.

They considered "temporal propositional expressions" (TPE), which arepropositional formulas with one free variable${\displaystyle t}$. For example,${\displaystyle N_1(t)\vee N_2(t)\wedge \mathrm{\neg }{N}_3(t)}$ is such an expression. Theorem 1 and 2 together showed that neural nets without circles are equivalent to TPE.

For neural nets with loops, they noted that "realizable${\displaystyle Pr}$ may involve reference to past events of an indefinite degree of remoteness". These then encodes for sentences like "There was some x such that x was a ψ" or${\displaystyle (\mathrm{\exists }x)(\psi x)}$. Theorems 8 to 10 showed that neural nets with loops can encode allfirst-order logic with equality and conversely, any looped neural networks is equivalent to a sentence in first-order logic with equality, thus showing that they are equivalent in logical expressiveness.^[4]

As a remark, they noted that a neural network, if furnished with a tape, scanners, and write-heads, is equivalent to aTuring machine, and conversely, every Turing machine is equivalent to some such neural network. Thus, these neural networks are equivalent toTuring computability,Church'slambda-definability, andKleene'sprimitive recursiveness.

The paper built upon several previous strands of work.^[5][6]

In the symbolic logic side, it built on the previous work by Carnap, Whitehead, and Russell. This was contributed by Walter Pitts, who had a strong proficiency with symbolic logic. Pitts provided mathematical and logical rigor to McCulloch’s vague ideas on psychons (atoms of psychological events) and circular causality.^[7]

In the neuroscience side, it built on previous work by the mathematical biology research group centered aroundNicolas Rashevsky, of which McCulloch was a member. The paper was published in theBulletin of Mathematical Biophysics, which was founded by Rashevsky in 1939. During the late 1930s, Rashevsky's research group was producing papers that had difficulty publishing in other journals at the time, so Rashevsky decided to found a new journal exclusively devoted to mathematical biophysics.^[8]

Also in the Rashevsky's group wasAlston Scott Householder, who in 1941 published an abstract model of the steady-state activity of biological neural networks. The model, in modern language, is an artificial neural network withReLU activation function.^[9] In a series of papers, Householder calculated the stable states of very simple networks: a chain, a circle, and abouquet. Walter Pitts' first two papers formulated a mathematical theory of learning and conditioning. The next three were mathematical developments of Householder’s model.^[10]

In 1938, at age 15, Pitts ran away from home in Detroit and arrived in theUniversity of Chicago. Later, he walked into Rudolf Carnap's office with Carnap's book filled with corrections and suggested improvements. He started studying under Carnap and attending classes during 1938--1943. He wrote several early papers on neuronal network modelling and regularly attended Rashevsky's seminars in theoretical biology. The seminar attendants included Gerhard von Bonin and Householder. In 1940, von Bonin introduced Lettvin to McCulloch. In 1942, both Lettvin and Pitts had moved in with McCulloch's home.^[11]

McCulloch had been interested in circular causality from studies withcausalgia after amputation,epileptic activity of surgically isolated brain, andLorente de Nò's research showingrecurrent neural networks are needed to explainvestibular nystagmus. He had difficulty with treating circular causality until Pitts demonstrated how it can be treated by the appropriate mathematical tools ofmodular arithmetics and symbolic logic.^[4][10]

Both authors' affiliation in the article was given as "University of Illinois, College of Medicine, Department of Psychiatry at the Illinois Neuropsychiatric Institute, University of Chicago, Chicago, U.S.A."

It was a foundational result inautomata theory.John von Neumann cited it as a significant result.^[2] This work led to work on neural networks and their link tofinite automata.^[12] Kleene introduced the term"regular" for "regular language" in a 1951 technical report, where Kleene proved thatregular languages are all that could be generated by neural networks, among other results. The term "regular" was meant to be suggestive of "regularly occurring events" that the neural net automaton must process and respond to.^[13]

Marvin Minsky was influenced by McCulloch, built an early example of neural networkSNARC (1951), and did a PhD thesis on neural networks (1954).^[14]

McCulloch was the chair to the tenMacy conferences (1946--1953) on "Circular Causal and Feedback Mechanisms in Biological and Social Systems". This was a key event in the beginning ofcybernetics, and what later became known ascognitive science. Pitts also attended the conferences.^[15]

In the 1943 paper, they described how memories can be formed by a neural network with loops in it, or alterable synapses, which are operating over time, and implementslogical universals -- "there exists" and "for all". This was generalized for spatial objects, such as geometric figures, in their 1947 paperHow we know universals.^[16] Norbert Wiener found this a significant evidence for a general method for how animals recognizing objects, by scanning a scene from multiple transformations and finding a canonical representation. He hypothesized that this "scanning" activity is clocked by thealpha wave, which he mistakenly thought was tightly regulated at 10 Hz (instead of the 8 -- 13 Hz as modern research shows).^[17]

McCulloch worked withManuel Blum in studying how a neural network can be "logically stable", that is, can implement a boolean function even if the activation thresholds of individual neurons are varied.^[18]: 64 They were inspired by the problem of how the brain can perform the same functions, such as breathing, under influence ofcaffeine oralcohol, which shifts the activation threshold over the entire brain.^[4]

• Artificial neural network
• Perceptron
• Connectionism
• Principia Mathematica
• History of artificial neural networks

• Machine learning
• Artificial neural networks
• History of artificial intelligence
• Computer science papers

• CS1 errors: ISBN date
• Articles with short description
• Short description is different from Wikidata