US20060184462A1 - Methods, architecture, and apparatus for implementing machine intelligence and hierarchical memory systems - Google Patents

Abstract

Sophisticated memory systems and intelligent machines may be constructed by creating an active memory system with a hierarchical architecture. Specifically, a system may comprise a plurality of individual cortical processing units arranged into a hierarchical structure. Each individual cortical processing unit receives a sequence of patterns as input. Each cortical processing unit processes the received input sequence of patterns using a memory containing previously encountered sequences with structure and outputs another pattern. As several input sequences are processed by a cortical processing unit, it will therefore generate a sequence of patterns on its output. The sequence of patterns on its output may be passed as an input to one or more cortical processing units in next higher layer of the hierarchy. A lowest layer of cortical processing units may receive sensory input from the outside world. The sensory input also comprises a sequence of patterns.

Description

FIELD OF THE INVENTION
• The present invention relates to the field of memory systems and machine intelligence. In particular the present invention discloses methods, apparatuses, and architecture for implementing memory systems that exhibit machine intelligence.
BACKGROUND OF THE INVENTION
• The field of Artificial Intelligence (AI) has existed for over fifty years. Many useful programs have been created from artificial intelligence research such as expert systems, skilled game playing programs, and neural network based pattern matching systems. Many of the programs can accomplish feats that no human could possibly match due to the significant computational power of modern computer systems. However, no computer program has ever shown the type of understanding exhibited by the brain of even a young child.
• There have been two main schools of artificial intelligence research: classic artificial intelligence research and neural network research. These two main schools of artificial intelligence research differ in how the problem of attempting to create machine intelligence is addressed. The main difference between the two schools is in how the two schools of artificial intelligence research are related to known information about the human brain.
• Classic artificial intelligence proponents make no attempt to examine or replicate the manner in which the human brain operates. Proponents of classic artificial intelligence attempt to create programs that mimic basic human behaviors or problem solving in a manner that does not incorporate any fundamental understanding of how human brains actually work. People that followed the classic artificial intelligence research school of thought felt that they should not be limited by the particular solution discovered by nature. This school of thought has some resonance since we created flying machines that do not operate the way birds fly and we created fast land travel machines that do not operate the way a cheetah runs.
• To create classic artificial intelligence, a programmer examines the problem to be solved or the human behavior to be mimicked and then determines an algorithmic solution to the problem. The programmer then codes the algorithmic solution in computer software. Examples of classic artificial intelligence programs include chess playing programs and expert system programs. These programs use an algorithm solution and a set of rules created by a human expert in order to solve complex problems, respectively. However, these programs generally have no ability to learn. These programs can only handle the single problem that was addressed. Nor can such artificial intelligence programs generalize upon the knowledge incorporated into such programs in order to address completely novel input data.
• Neural network proponents have attempted to create limited intelligent systems by replicating the operation of interconnected neural cells. There is a large body of knowledge that describes how individual neural cells (neurons) operate and how connected neurons interact. Neural network proponents have built systems, known as “neural networks”, based upon this knowledge about neuron operation. Thus, neural network systems operate in a manner similar to a set of interconnected neurons. Neural network researchers are therefore often referred to as ‘connectionists.’ Interneuron connection strengths are known as synaptic weights and are used to store the learned knowledge.
• Before being used, a neural network must first be trained with a set of training information. The training information consists of input vectors with associated output vectors that are deemed to be the correct output for the associated input vector. During the training, the connections between the various simulated neurons in the neural network are adjusted such that the input vectors generate the associated output vectors (or a close approximation).
• Once trained, a neural network is used by presenting a novel input vector to the neural network such that an output vector is generated. With a proper neural network design and adequate training data, the neural network should generate the appropriate output vector for the given input vector. Neural networks have been proven to be useful in some limited applications.
• Although there have been some limited successes with neural networks, most neural network systems are relatively primitive. Most neural network systems are simply a three layer structure with a set of input nodes, a set of middle nodes (also known as the ‘hidden nodes’), and a set of output nodes. Although neural network systems are able to ‘learn’ in a very simple sense and exhibit a limited ability to generalize, there is clearly no real understanding of the world. Neural network systems merely create an internal function that best maps the training input vectors to the associated training output vectors. Thus, a neural network is only able to generalize in a limited sense by applying the internal function to the novel input vectors.
• To really advance the field of artificial intelligence, a new paradigm for artificial intelligence would be desirable. The classic artificial intelligence approach has probably failed since we do not fully understand the essence of intelligence. And without understanding the essence of intelligence, how can one be expected to encode intelligence in a computer program? The neural network approach has provided very limited results since neural networks generally emulate only relatively few interconnected neurons and does so in a manner that ignores most of the complex anatomy of the brain. Since current estimates postulate that the neocortex of the human brain contains approximately thirty billion neurons, such simple neural networks will never provide the real intelligence exhibited by the human brain. Thus, to advance the state of artificial intelligence it would be desirable to embark on a new approach that avoids the problems of the current main approaches.
SUMMARY OF THE INVENTION
• The present invention proposes creating sophisticated memory systems and intelligent machines by constructing a memory system with a hierarchical architecture. Specifically, a system may comprise a plurality of individual cortical processing units arranged into a hierarchical structure. Each individual cortical processing unit receives a sequence of patterns as input. Each cortical processing unit processes the received input sequence of patterns and outputs another pattern. As several input sequences are processed by a cortical processing unit, it will therefore generate a sequence of patterns on its output. The sequence of patterns on its output may be passed as an input to one or more cortical processing units in next higher layer of the hierarchy. A lowest layer of cortical processing units may receive sensory input from the outside world. The sensory input also comprises a sequence of patterns.
• Each individual cortical processing unit includes a memory that is used to help process the sequence of patterns received on the input. The memory may comprise previously encountered sequences of patterns that contain structure. An input sequence of patterns may be deemed to contain structure if that sequence of patterns has been encountered more than once. An individual cortical processing unit can use the stored sequences of patterns containing structure in order to later ‘recognize’ such structured sequences in the input stream.
• When a previously encountered sequence of patterns is recognized by a cortical processing unit, the cortical processing unit can report such recognition to higher layer cortical processing units. The output to the higher layer cortical processing units may comprise an identifier of the recognized pattern sequence. Over time, the output stream containing identifiers of recognized sequences comprises a sequence of patterns itself. That output stream of patterns may comprise an input that will be processed in a similar manner by the cortical processing unit in the next higher cortical processing unit layer. The outputs of more than one lower layer cortical processing unit may be joined together such that the joined outputs form a sequence of patterns that may be processed.
• A cortical processing unit may also use the memorized sequence of patterns in order to make predictions about the incoming sequences in order to resolve ambiguous information. Specifically, when a cortical processing unit determines that it is currently receiving a known sequence, the cortical processing unit may predict the next pattern to be received in the sequence. The processing unit may use the prediction to interpret the next pattern received. For example, if the input stream into a cortical processing unit is ambiguous, then that cortical processing unit may use the predicted pattern to help identify the next input pattern.
• The cortical processing unit may further provide the predicted next pattern down to the next lower cortical processing units. The lower cortical processing units may interpret this as a prediction of the next sequence that the lower unit will encounter (and hence output to the higher layer). Such prediction information may comprise information that helps the lower cortical processing unit layers correctly identify the next sequence.
• The prediction information may also be considered as useful predictions of what will occur in the outside world. And if an incoming pattern significantly differs from a predicted pattern or an incoming sequence of patterns significantly differs from the predicted sequence of patterns, then there is a prediction failure. The occurrence of a prediction failure may be transmitted to a cortical processing unit in a higher layer. A cortical processing unit in a higher layer may be able to resolve the error. A detected error may be resolved in a number of different ways. One way to resolve the error is to adjust the cortical hierarchy's model of the world by storing a new sequence of patterns.
• Other objects, features, and advantages of present invention will be apparent from the company drawings and from the following detailed description.
BRIEF DESCRIPTION OF THE DRAWINGS
• The objects, features, and advantages of the present invention will be apparent to one skilled in the art in view of the following detailed description in which:
• FIG. 1 illustrates a conceptual diagram of the visual processing portion of the brain concerned with recognizing forms and identifying objects.
• FIG. 2A illustrates a revised hierarchical diagram for the human visual processing system ofFIG. 1.
• FIG. 2B illustrates an alternate hierarchical diagram for the human visual processing system ofFIG. 1.
• FIG. 2C illustrates the alternate hierarchical diagram ofFIG. 2A with a local prediction feedback.
• FIG. 2D illustrates the alternate hierarchical diagram ofFIG. 2A with a prediction passed from a higher layer to a lower layer.
• FIG. 3A illustrates a grid of pixels in the limited visual world considered by a prototype program machine intelligence program.
• FIG. 3B illustrates the pixel grid ofFIG. 3A wherein the pixel grid has been divided into a set of sixty-four pixel patches of four by four pixels that will each be examined by an individual processing unit.
• FIG. 3C illustrates the pixel grid ofFIG. 3B wherein the sixty-four pixel patches have been grouped together in groups of four to form sixteen meta-patches that are examined by the sixteen processing units of the second processing unit layer.
• FIG. 4 illustrates one possible example of the hierarchy of the cortical processing units for the thirty-two by thirty-two grid of pixels illustrated inFIGS. 3A to3C.
• FIG. 5A illustrates the hierarchy of the cortical processing units to process information from thirty-two by thirty-two grid of pixels illustrated inFIG. 3A in a prototype program.
• FIG. 5B illustrates the hierarchy ofFIG. 5A with feedback information that provides context.
• FIG. 6 illustrates an alternate view of the cortical processing unit hierarchy ofFIG. 5A wherein the cortical processing units are each labeled with the addressing system set forth with reference toFIGS. 3B and 3C.
• FIG. 7 illustrates a Bayes belief net for the cortical processing unit hierarchy ofFIG. 5A.
• FIG. 8 illustrates ninety training shapes that were used to train the prototype program.
• FIG. 9 illustrates eight different possible sequences of four consecutive patterns.
• FIG. 10 illustrates some result data from three different line drawing tests using a prototype program.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
• Methods, architecture, and apparatus for implementing machine intelligence are disclosed. In the following description, for purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that these specific details are not required in order to practice the present invention. For example, the present invention has been described with reference to Bayesian networks for generating predictions. However, other types of systems may be used to analyze the learned information in order to generate predictions. The numerous teachings of the present invention are set forth with reference to a simple image recognition system that uses image information as sensory input. However, the principles of the present invention can be applied to any type of environment with any type of sensory input as long as there is detectable consistency within the environment.
Theory of Operation of the Human Brain
• The human brain consists of billions of interconnected nerve cells (neurons). Like a computer system, the neurons of the human brain are connected to both input units and output units. The input units consist of the senses connected to our brain such as hearing, sight, smell, touch, and taste. The output units consist of controlled muscles that allow us to perform complex actions with our bodies and allow us to communicate by speaking.
• But beyond that one similarity, computers and human brains are extremely different. A computer has a central processor that sequentially executes a set of instructions (with associated parameter information) retrieved from a flatly addressed memory system. In a human brain, there is no central processor. Instead, there are billions of neurons interconnected in some parallel manner wherein all the neurons operate concurrently. Nor is there a large flatly addressed memory system with perfect recall in a human brain. Instead, the human brain has billions of synapses that form connections between neurons. Each synapse in the network of synapses may be strengthened or weakened thus acting as a ‘memory’ in the form of modified operation of that synapse.
• As set forth in the background, independent efforts at creating intelligent machines have yielded many useful computer programs. However, nothing close to real human (or other mammal) intelligence has ever been created. No one has yet figured out how to independently program a real intelligent machine. Thus, in order to build an intelligent machine, it may be wise to determine the manner in which the human brain works in order to take advantage of the operational methods incorporated in that amazing product of millions of years of evolution by natural selection.
• The human brain has been studied by scientists for over a hundred years. This extensive research has lead to many independent discoveries that have revealed a large number of insights into the operation of the human brain. However, the overall operation of the human brain still largely remains a mystery. The teachings of the present invention combine a number of the discoveries about the human brain in order to create an overall theory on how the human brain operates. The operational theories mainly focus on the brain's neocortex since only mammals, the class of animals generally regarded to be the most intelligent, have a neocortex. A full description of the overall theory of operation of the brain's neocortex can be found in the book “On Intelligence” by Jeff Hawkins, Times Books, which is hereby incorporated by reference in its entirety.
• With an overall theory on how the neocortex operates, these teachings can then be implemented with a machine in order to create artificial intelligence. For example, the methods of the present invention may be implemented with software on a computer system. However, the teachings can be implemented in many other formats such as directly in an integrated circuit.
• Sensory Input Comprises Sequences of Patterns
• The human brain is protected within the confines of the strong bone that consists of the human skull. There is no direct contact between the human brain and the outside world. To learn about the outside world, the human brain has millions of connections to sensory nerve cells located all over the human body. Millions of nerve cells on the surface of the human body provide the input of touch. Millions of rods and cones on the retina of the human eye detect light and transmit information about the detected light along the optic nerve to the brain. Thousands of sensor cells in the human ear detect different sound frequencies and transmit the detected sound information along the auditory nerve to the brain. All of this sensory information is received along different nerve pathways to the brain. The brain must then interpret all of this information to form an understanding of the outside world.
• All of the different sensory input signals that are transmitted to the brain have at least two dimensions: a spatial dimension and a time dimension. The spatial dimension constitutes a collection of related sensory signals that arrive at the brain at the same time. The collective activities on related nerve fibers that are received by the brain at the same time are deemed to be a “pattern” of information in the spatial dimension. Over time, different patterns are received by the brain from each collection of related nerve fibers. The consecutive patterns received by the brain along related nerve fibers constitute a ‘sequence of patterns’ wherein consecutive patterns are closely related in time.
• An example can easily be provided with reference to the human sense of vision. All of the input signals received by the brain from the optic nerve at a particular instant in time represent a pattern of visual information in the spatial dimension. Over time, different visual information enters our eyes as we move our eyes (in movements known as ‘saccades’), move our head, or move our entire body. The constantly changing visual information received by our eyes is continually transmitted to the brain along the optic nerve. The constantly changing visual information creates different patterns on the optic nerve. The continuous stream of patterns transmitted on the optic nerve is thus a sequence of patterns of visual information.
• The pattern in a spatial dimension and a sequence of patterns in the time dimension model applies to all of our different senses. With regard to hearing, different hairs in our ears are activated by different frequencies of sound. The collection of all of the different frequency signals transmitted from the ear to the brain along the auditory nerve represents a pattern of auditory information. Over time, we hear different sounds such that sequence of auditory patterns is transmitted along the auditory nerve to the brain. Our sense of touch is actually much more detailed than simply touch pressure. Touch information has a wide variety of spatial dimension aspects including pressure, temperature, pain, and location. All of this touch information that the brain receives concurrently is a pattern of touch information in the spatial dimension. Over time, the body receives a sequence of patterns of touch information.
• Much of the artificial intelligence research using neural networks concerns the use of neural networks attempting to learn pattern information such that the neural network can later recognize similar patterns within novel input data. However, most of the neural network based pattern recognition research has been performed using spatial patterns with no consideration of time. For example, during a learning phase, a neural network may be presented with a collection of static images, one at a time, such as an apple, a boat, a tree, a banana, etc. The neural network is ‘taught’ what each presented pattern represents in order to train the neural network on different object patterns for object recognition. Later, novel image patterns are presented to the neural network such that the neural network attempts to identify an object in the novel image. There has been some success with such pattern recognition neural networks, but the success has been limited.
• Human brains do not receive information in such a static manner wherein a set of completely different images are presented with no respect to time. Instead, human brains receive information as a continuous sequence of patterns from the different sensory organs. For example, our eyes receive an ever-changing sequence of images that are transmitted to the brain as a sequence of patterns on the optic nerve. Our brains learn to group different sequential images of the same object together by observing continuous motions in this world.
• The sensory information in the sequence of patterns is generally highly correlated over time. For example, when a person looks out a window, much of the visual information (such as buildings, tree trunks, the ground, etc.) remains static. Some of the visual information (such as leaves blowing in the wind, a bird flying in the sky, a squirrel running across the grass, etc.) changes over time. But even the changing information is highly correlated in time since the leaves blowing in the wind do not move significantly, the bird travels in a straight line, and the squirrel moves around only on the ground in the same area.
• The correlation of the information in the visual stream is very important for learning. For example, a baby looking out the window of the previous example will learn over time that trees and buildings do not move whereas birds and squirrels do move. Thus, receiving a continual sequence of visual patterns provides a large amount of useful information for learning. Therefore, the system of the present invention proposes that the consideration of time such that a sequence of patterns is examined to be an important aspect of in the ability to learn. Thus, in many embodiments, the system of the present invention uses sequences of patterns generated over time as the stream of source information to learn from.
• The Neocortex in the Brain Uses a Single ‘Cortical Algorithm’
• The human brain is comprised of many different parts including the cerebellum, the basal ganglia, the hypothalamus, the brain stem and the neocortex. However, it is generally agreed upon that the main area of the human brain responsible for abstract thought and substantial learning capability is the neocortex. Only the animals that exhibit the best capacity for learning (mammals) have a neocortex structure in their brains.
• Different regions of the neocortex are generally used to address different problems presented to the neocortex. One region of the neocortex is used to process grammar; other regions are used for processing sounds; multiple regions are dedicated to processing visual information; and so on. Over time, scientists have attempted to map out the different functions performed by the different regions of the neocortex. The current brain function mappings of neocortex functions are relatively complex. And it has been found that the functional mappings of the neocortex only tend to vary by limited amounts from person to person.
• The relatively consistent mapping of neocortex functions has lead many brain researchers to assume that each area of the brain processes the information that it receives and solves problems in different ways. However, in 1978, a neuroscientist named Vernon Mountcastle made a novel assertion. He noted that the entire neocortex had a remarkably uniform appearance and structure. The regions of neocortex that handle auditory input look the same as the regions that handle touch, which look like the regions that control muscles, which look like the regions that handle vision, which look like the regions that generate language, and so on. Mountcastle asserted that since all these different regions of the neocortex appeared uniform, then perhaps all of these different regions of the neocortex performed the same basic operation. He asserted that there is likely to be a single ‘cortical algorithm’ that is used throughout the neocortex to handle all of the different sensory information. Assuming that this is true, then the different regions of the cortex that perform specific functions become specialized mainly due to the particular pattern information received by that cortical region and not because of any significantly different neural structure in that cortical region.
• A number of subsequent scientific studies support Mountcastle's assertion. One of the strongest pieces of evidence is the “plasticity” of the brain. As set forth above, there are many different regions of the brain that seem to be dedicated to performing very specific functions. However, it has been learned that different parts of the brain can easily adapt to handle a different function that is normally handled by another region of the neocortex. The ability to adapt operation in order to handle different circumstances is referred to as the brain's plasticity.
• In one set of experiments that supports Mountcastle's uniform cortical algorithm assertion, scientists have “rewired” the output from one sensory organ on a baby animal to connect to a different region of the of the animal's neocortex. For example, the optic nerve of a baby ferret has been rewired such that the information from the optic nerve flows into an area of the ferret's neocortex that is normally used to process sound information. When the ferret with the rewired brain grows up, the ferret is able to see. The area of the ferret's brain that would normally be used to process sound information was able to adapt to processing visual information. Such experiments show that the mammal neocortex is very ‘plastic’ in the sense that just about any region in the mammal neocortex can adjust itself to handle whatever type of input information flows into that region.
• The same plasticity has been shown to exist in human brains. For example, if the region of the neocortex that is usually dedicated to processing grammar (known as Broca's area) is damaged in the brain of a young child that has not yet learned to speak, then the child's damaged brain will adjust itself accordingly in order to adapt. Specifically, the child's brain will tend to use a different region of the brain for processing grammar (such as the same area on the other side) such that the child with a damaged brain will easily learn to speak. Many other examples of such human brain plasticity can be cited. Adults that are born deaf will use areas that are normally used to process sounds for processing visual information. Blind people will often use the region of the rear region of the brain normally used to process vision in order to learn to read Braille.
• Since different regions of the neocortex can be used to handle any different problem, then there must be a single ‘cortical algorithm’ that is used to handle every different problem presented to the brain. This is just what Mountcastle proposed. Although Mountcastle's proposal may seem relatively simple, his discovery is actually quite profound in its implications. Specifically, if a single cortical algorithm that is used throughout the entire human neocortex can be deciphered properly, then that cortical algorithm can be reproduced in a machine to create machine intelligence. Within the context of a machine, the cortical algorithm can be used to process many different types of information streams as long as each information stream is presented as a sequence of patterns. There for, a single type of machine can be used to solve problems in vision, language, audition, and robotics.
• The Human Neocortex is Organized Hierarchically
• Over the one hundred years of human brain research, many different aspects of the cortex of the human brain have been carefully examined. However, one of the most meticulously studied aspects of the human cortex is the processing of visual information.FIG. 1 illustrates a simplified conceptual diagram that is commonly used by brain researchers to describe the visual processing portion of the human cortex concerned with recognizing forms and identifying objects in the visual information stream.
• At the bottom ofFIG. 1, raw visual information (the sequence of visual patterns) enters theV1 region110. TheV1 region110 performs the initial processing of visual pattern information received from the optic nerve. The cells in theV1 region110 are divided into cell groups wherein each cell group analyzes the information from a small area of the retina. In this document, all the neural cells in a cell group that process the same sequence of input patterns or an artificial device that emulates a cell group that processing the same sequence of patterns will be referred to as a “cortical processing unit” or a “processing unit.” For example, a cell group that processes the information contained in sequence of input patterns directly from a small area of the retina may be referred to as a cortical processing unit.
• The information generated by the processing units in theV1 region110 is then passed to the V2 region120. The V2 region120 performs processing on the information received from theV1 region110 and then generates processed output information that is passed to the V4 region130. Similarly, the V4 region130 processes the information received from the V2 region120 and passes processed information to IT region140. Finally, the IT region140 receives information generated by the V4 region130. In the IT region140, the brain is known have been able to discern objects in the person's visual field. For example, certain cells will be activated if a particular face is visible anywhere in the person's visual field.
• As illustrated inFIG. 1, raw visual sensory information is received at the bottom and then progressively processed by a series of visual processing regions. At the top region, the visual processing system is known to recognize and identify objects. Thus, the visual processing portion of brain concerned with recognizing forms illustrated inFIG. 1 has a vertical hierarchy of successive processing until a complex result (an identified object) is achieved at the top.
• Laboratory tests indicate that the cells in the lower layers of the hierarchy (such as the V1 region110) are switching (activating and deactivating) very rapidly. This is understandable since our eyes are constantly moving such that different visual information is being presented to theV1 region110 of the cortex all the time. TheV1 region110 is also the largest cortex region of the four cortex regions conceptual illustrated inFIG. 1. Furthermore, experiments have shown that each cell group in theV1 region110 is only receptive to a very small area of the retina. Thus, in theV1 region110 there are many individual small cell groups responsible for processing information from each area of the retina.
• As one ascends the layers of hierarchy illustrated inFIG. 1, the activity of the cells at each higher hierarchical layer become more stable than the lower hierarchical layers. Thus, as one proceeds up the hierarchy, cells respond more and more to objects in the world as opposed to simple visual constructs. Furthermore, the receptive visual fields of the cell groups in each layer become larger as one ascends the hierarchy. That is, cell groups in thelow V1 region110 only process information from a small area of the retina whereas the IT region140 processes information from the entire visual field. Thus, each cell group in a layer above the first layer (the V1 region140) processes information from more than one cell group from the layer below. This can be seen since a face can appear in any region of a person's visual field and the person still recognizes it as a face such that the IT region140 must process the entire visual field.
• As one ascends the hierarchy, the cell groups in each ascending layer process a larger and larger visual field such that each cell group in regions above the first layer (V1 region110) processes the output of many cell groups from regions in the lower layers. At the top of the visual form recognition hierarchy inFIG. 1, the IT region140 processes the entire visual field since objects can be recognized in any part of the visual field.
• This information leads one to revise the simple block diagram ofFIG. 1 into a block diagram having an ascending hierarchy that generally has a large number of cortical smaller processing units at the bottom and generally has fewer but larger cortical processing units on each successive hierarchical layer. This architecture allows for distributed and relatively simple processing at the bottom but concentrated and more complex processing at the top.
• As set forth in the previous section, there is a single cortical algorithm that is used throughout the neocortex of the human brain. Furthermore, using the teachings in the first section on input, each cortical processing unit receives sequences of patterns that must be processed. Combining these teachings, one may deduce that each layer of the hierarchy must comprise cortical processing units that receive input in the form of a sequence of patterns but output a sequence of patterns since those output patterns will become the inputs to the cortical processing units in the next higher layer.
• Combining the information in the previous paragraphs leads one to revise the simple hierarchy ofFIG. 1 into an ascending hierarchy of cortical processing units wherein each cortical processing unit inputs and outputs sequences of patterns.FIG. 2A illustrates a block diagram one embodiment of a revised hierarchical diagram for the human visual processing system wherein each rectangular block represents a cortical processing unit. As set forth earlier, the hierarchical organization may not always consist of progressively fewer cortical processing units on each ascending layer. For example,FIG. 2B illustrates an embodiment of a cortical processing unit hierarchy wherein the third layer from the bottom has more cortical processing units than the second layer from the bottom.
• FIG. 2A illustrates a very simplified version of the proposed hierarchical architecture of the visual processing portion of the human neocortex concerned with recognizing forms and identifying objects. In the illustration ofFIG. 2A, each layer of the hierarchy comprises one or more cortical processing units wherein each cortical processing unit is represented by a rectangular box. Note thatFIG. 2A is just provided for conceptual explanation purposes. In reality, there are certainly many more layers and there are certainly many more processing units on each layer. With the proper determination of the single cortical algorithm that drives each processing unit, an intelligent machine may be constructed using the hierarchical architecture illustrated inFIG. 2A in order to perform visual object recognition.
• The Cortical Algorithm Stores Structure in Memory for Later Recognition
• Referring toFIG. 2A, we know that the sensory input information received at the bottom of the hierarchy consists of sequences of patterns from the sensory organs. Following the presumption that there is a single ‘cortical algorithm’ that is used throughout the brain then each cortical algorithm processing unit would likely also output patterns (that, over time, are sequences of patterns) such that those patterns may be used as the inputs for the next higher layer of cortical processing units in the hierarchy. Thus, the challenge is to determine what algorithm is used internally in each cortical processing unit to process the incoming sequence of patterns and generate an outgoing sequence of patterns.
• It is well known that the brain has a large distributed memory system. Following the single cortical algorithm model, then each cortical processing must include memory for storing information. Since each cortical processing unit receives an input sequence of patterns, has a memory, and outputs information; then that memory must be used by the cortical processing unit to help process the input sequence of patterns in order to create the output sequence of patterns.
• One of the great mysteries of the human brain is how it creates invariant representations of objects in the world. An invariant representation is an internal representation of an object (a physical object, a word, a song, etc.) in the brain that allows the brain to later recognize the same object in just about any different external condition. For example, a brain's invariant representation of a particular person allows the brain to later identify that person in any lighting condition, from any viewing angle, and from any distance provided that there is enough information in the sensory input stream to identify that person.
• The system of the present invention proposes that an invariant representation of an object can be constructed by using a hierarchy of cortical processing units. The invariant representation is formed by identifying invariant structure within the constantly changing sequence of patterns and storing that invariant structure so that it may later be recognized. Every cortical processing unit performs this process of identifying, storing, and later recognizing invariant structure at every level in the hierarchy. Thus, the overall invariant representation of an object comprises all of the invariant structures associated with that object that have been stored in a set of hierarchically arranged cortical processing units. Note that an invariant representation includes invariant structures at many different levels. All of these different levels invariant structures (including the entire invariant representation structure) can be used within the invariant representations of other objects.
• The neocortex must create these invariant representations of objects using the neocortex's only source of outside information, the constantly changing sequence of patterns from the various sensory organs. Thus, the neocortex must create an invariant representation of an object by identifying invariant structures within the constantly changing sequence of patterns. The memory in each cortical processing unit must be used to store invariant structure that has been located in the incoming sequences of patterns. In this manner, the stored invariant structures may later be recognized in the incoming sequence of patterns in order to recognize objects that include that invariant structure.
• But what exactly is an invariant structure in a sequence of patterns? A non-structured sequence of patterns is generally represented as a completely random stream of patterns. So a structured stream of patterns is not a random stream of patterns. But how can a cortical processing unit that has not had any learning identify a sequence of patterns that is not random? One possible method is to have the cortical processing unit look for a repeated patterns or repeated sequences of patterns. Repeated patterns and repeated sequences of patterns are unlikely to be random noise. Furthermore, the more times that a pattern or sequence of patterns repeats, the less likely that the repeated pattern or the repeated sequence of patterns is random. Thus, repeated patterns and repeated sequences of patterns are likely to be part of an invariant structure. A number of other methods may also be used to identify invariant structures with the stream of patterns. For example, ordered progressive pattern changes such as 1, 2, 3, 4, and so on are also mostly likely to be invariant structure.
• In one embodiment, each cortical processing unit may search for repeated patterns or repeated sequences of patterns in the input stream in order to identify invariant structures contained within the input stream. The cortical processing unit may then store those repeated patterns and repeated sequences of patterns in memory for later recognition.
• In order to limit the amount of memory used, a cortical processing unit may only memorize the most frequently repeated patterns or sequences of patterns. This limited number of repeated patterns or sequences of patterns stored in memory may be considered as a set of ‘categories’ that are used to classify incoming patterns or sequences of patterns. Later, when a previously seen and memorized pattern or sequences of pattern (a learned invariant structure) is encountered again, the cortical processing unit can inform the next higher cortical layer about the identified invariant structure found in the input stream.
• Upon recognition, a cortical processing unit may pass up a sequence identifier (or name) for the recognized sequence of patterns to the next higher layer in the hierarchy. As long as the incoming sequence of patterns continues to match the memorized sequence of patterns, the cortical processing unit will continue to output the same sequence identifier for that recognized sequence of patterns. Since the same sequence identifier is being output during the entire pattern sequence, one can see that higher layers will have processing units that switch less frequently than lower processing units.
• Note that if a cortical processing unit fails to recognize a pattern or sequence of patterns, the cortical processing unit may output an identifier to the cortical processing unit in the next higher layer that indicates that nothing was recognized in the current input stream. Alternatively, the cortical processing unit may output an identifier that indicates that there was an error between a predicted pattern and the actual pattern received.
• The sequence identifier itself is a pattern on the cortical processing unit's output. And the output pattern (the sequence identifier) becomes the input to the next higher cortical processing unit. Over time, a cortical processing unit will output different sequence identifiers as different sequences are recognized such that a sequence of patterns (a sequence of sequence identifiers) is output from the cortical processing unit.
• The cortical processing unit in that next higher layer can then apply the same cortical algorithm to its input stream(s) from one or more processing units in lower layers of the hierarchy. Specifically, the cortical processing unit in that next higher layer can examine the input stream for repeated patterns or sequences of patterns and store those repeated patterns and/or repeated sequences of patterns. When the cortical processing unit recognizes a stored sequence of patterns in the input stream, that cortical processing unit can inform a cortical processing unit in the next higher layer with a sequence identifier for the recognized sequence. In this manner, the hierarchy of cortical processing units (as illustrated inFIG. 2A) is able to identify structures within structures.
• With a large number of individual cortical processing units organized in a hierarchical manner, very complex invariant structures may be memorized and later recognized. The collective memory of invariant structures learned by the hierarchy of cortical processing units comprises a model of the surrounding world.
• The hierarchical structure promotes efficient usage of memory since simple structures in lower hierarchical layers can be combined in many different manners to create more complex structures in higher layers. For example, low level cortical processing units may recognize simple invariant structures such as lines of different orientations (vertical, horizontal, 45 degree, etc.). Then higher level cortical processing units may recognize more complex structures by combining together different permutations of the line orientations in the lower-level structures. For example, a higher level cortical processing unit may recognize a rectangle as a first pair of parallel lines that are coupled at each end by a second pair of parallel lines with an orientation that is orthogonal to the first pair of lines. A square could be recognized as first pair of parallel lines that are coupled at each end by a second pair of parallel lines having an orientation that is orthogonal to the first pair of lines and a length equal to the first pair of parallel lines. Note that the same low level structures are used in both higher level object definitions. The rectangle structure itself may be primitive structure used to identify more complex objects such as a house, a billboard, a television, etc.
• As one proceeds up the hierarchy of cortical processing units illustrated inFIG. 2A, the cortical processing units in each successive layer become more stable since the identified sequence of patterns will change less frequently. Eventually, at a high level, the information becomes very stable. For example, in a high layer such as IT region in the human neocortex's system for identifying visual objects, a consistent visual pattern such as a human face can be consistently recognized even when it moves about the person's field of view thus creating very different visual input information that causes rapid switching in lower regions such as the V1 region. Specifically, studies have shown that certain cells in the high levels such as the IT region remain activated as long as there is a human face within the field of view even when the face moves about the field of view.
• The Cortical Algorithm Makes Predictions Using its Stored Memories
• A core theory about the nature of intelligence set forth by this document is that intelligence is measured by the ability to make useful predictions. Intelligence allows humans (and other mammals) to use accumulated knowledge about the world in order to make predictions about the world. Simply identifying invariant structure in the surrounding world alone is not very useful. However, using that accumulated knowledge about the world in order to make accurate predictions about that world is a very useful adaptation for an animal struggling to survive in the world. When an entity can make accurate predictions about an object or situation, then that entity has intelligence such that the entity can be said to ‘understand’ that object or situation.
• An animal that is able to identify a previously encountered situation and then make accurate predictions about that situation will be better able to survive and reproduce. For example, a gazelle that previously encountered a lion catching and eating another gazelle may learn from that encounter such that it can predict that hanging around lions may get oneself killed. Based upon that prediction, the gazelle may opt for behaving in a manner that prevents the undesirable prediction from coming true . . . Run! Similarly, an animal with the ability to predict may adapt its behavior to cause favorable predictions to come true. If an animal previously encountered a watering hole by the patch of five trees, then the animal may predict that going to those five trees would again provide a watering hole. Therefore, natural selection favored animals with a neocortex well suited for making accurate predictions.
• As set forth in the earlier sections, the neocortex likely uses a single cortical algorithm in all the cortical processing units arranged in a hierarchy to address many different problems. Thus, both high level cortical processing units and low level cortical processing units make predictions. Very high level cortical processing units may make sophisticated decisions such as those presented in the previous paragraph that helped an animal survive. However, even very low level cortical processing units constantly make very simple predictions. A person's neocortex constantly makes many of these low level predictions with out that person being aware of those predictions. Those low level predictions are generally only of interest to the surrounding low level cortical processing units. (But even low level predictions may escalate up the hierarchy if the prediction does not match a sensed reality.)
• For example, if you are sitting in a chair then the low level cortical processing units coupled to your sense of touch predict that pressure will continue to be felt by your body in contact with the chair as long as you remain in that chair. This prediction is constantly made but is generally of interest only to other low level cortical processing units such that you are not even aware of the pressure against your body unless you make a conscious effort to detect it. As long as the incoming reality matches with the prediction, there is understanding of the situation. (In this case, the understanding is at a very low level.)
• When a prediction fails to match sensed input, there is confusion such that information about the failed prediction moves up the hierarchy for additional consideration. Thus, if the chair you are sitting on suddenly breaks causing you to drop then the low level cortical processing units that were predicting continued pressure from body contact will now signal a failed prediction. The nearby higher cortical processing units will not be able to resolve these failed predictions such that the failed prediction rapidly escalates far up the cortical processing unit hierarchy such that you become aware that you are falling. Thus, many predictions are constantly being made at various low levels outside of our consciousness. However, even a failed low level prediction may escalate up the hierarchy such that we become aware of the problem if no cortical processing unit in the hierarchy is able to resolve the failed prediction
• One of the most common examples of failed low level prediction that escalates into our consciousness that almost everyone has experienced is that sudden feeling of panic that you feel when you are walking up stairs and your foot passes through the plane where you predicted the next step should be. Your brain notes that reality does match the prediction such that the problem rapidly escalates up the cortical hierarchy. Since no intermediate cortical processing unit is able to resolve this failed prediction, the failed prediction escalates all the way up into your consciousness in order to bring your attention to the problem.
• Our brain can adjust to mismatches between sensory input and predicted reality in many different ways. If the sensory perception is deemed to be accurate but the prediction is not then the error indicates that the brain's model of the world is not accurate. The brain may adjust its model of reality by learning. Thus, in the preceding example, you may learn that the set of stairs has fewer steps than you previously expected.
• Assuming that cortical processing units make predictions then one must determine how these cortical processing units in the neocortex can make these predictions. As set forth in the previous section, each cortical processing unit stores sequences of patterns in memory (Ideally, sequences of pattern that represent structure in the world) for later recognition. If the incoming patterns match a particular memorized sequence, then the cortical processing unit ‘recognizes’ that particular sequence and informs the coupled processing units in higher layers that a known sequence is being observed. However, this sequence recognition can also be used to make a prediction. Specifically, if a cortical processing unit recognizes that it is currently in a known sequence of patterns, then that cortical processing unit may determine the next pattern that the processing unit expects to receive by identifying the next pattern in the recognized sequence of patterns. This next expected pattern constitutes a prediction by the cortical processing unit. Such predictions occur all the time in our brains at every level of the cortical processing unit hierarchy.
• The next pattern prediction made by a cortical processing unit may be used as an input to that processing unit to be processed. For example,FIG. 2C illustrates the cortical hierarchy ofFIG. 2A whereincortical processing unit231 has made a prediction of the next expected pattern (represented by the dashed line) and that prediction that is passed back intocortical processing unit231 as input information. There are various different ways of accomplishing this result. In one embodiment, the predicted pattern value may be delayed before being introduced into the input stream. In the book “On Intelligence”, it was suggested that the thalamus may be used to perform this function in the human brain.
• There are a number of uses for such predictions. One simple example is that the cortical processing unit may compare its prediction of the next pattern to be received from lower processing units with the actual pattern received. In such a manner, the cortical processing unit may be able to use the pattern prediction to resolve ambiguous information. For example, if the next pattern is very close to the expected pattern, then the cortical processing unit may deem that the next pattern was really meant to be the predicted pattern but it may have been distorted by noise. If the cortical unit presumes that the predicted pattern was really what was meant, then the cortical processing unit will deem that it is still within the recognized sequence. Thus, a pattern sequence that is very similar to a predicted pattern sequence may effectively be changed into that predicted sequence. Thus, a cortical processing unit may create a perceived reality by combining an actual real but ambiguous input (a closely matching pattern) with a known model of reality (the expected pattern) to create a perception of reality. If the expected pattern is significantly different than the expected pattern, then the cortical unit may try to recognize a different known sequence of patterns, signal an error to higher units, signal the error to lower units, or act in another manner.
• One should note that this use of prediction information to resolve ambiguous input information is similar to the operation of existing auto-associative memories. An auto-associative memory is a memory that produces a full clean version of known pattern or sequence of patterns when give a noisy or partial version of that known pattern or sequence of patterns. Thus, in an auto-associative memory, a group of similar input patterns or sequences of patterns will all retrieve and output the same output pattern or sequence of patterns. This is a form of generalization that humans can do effortlessly. However, most auto-associative memory systems will fail if the input pattern is shifted, rotated, scaled, or otherwise distorted in a significant manner.
• Human memories are often linked together by their own content like auto associative memories. For example, if you wish to recall information about a house that you lived in long ago, you may wish to view a photograph of that house to ‘prime’ your memory (the photograph being a partial input of the full house memory). Upon seeing the photograph of the house, a flood of memories about that house (the full clean version of the house memory) may be produced. Thus, memories of the house were addressed using the house itself, hence the term auto-associative memory.
• Auto-associative memories can be used to store sequences of patterns. To store a sequence of patterns in an auto-associative memory, one can associate the current input pattern with the previous input pattern, then associate the next input pattern with the current input pattern, and so on. Then, the output of the auto-associative memory is coupled to the input of the auto-associative memory. In this manner, when the auto-associative memory is primed with a spatial pattern that is associated with the first pattern in the sequence, the entire sequence will be recalled.
• Auto-associative memories may be used to construct primitive versions of the cortical hierarchy of the present invention. For example, auto-associative memory devices may be arranged in a hierarchical arrangement wherein the outputs from low-level auto associative memories are coupled to the inputs of higher auto-associative memories with some additional circuitry. In such a system, the auto-associative memories store, recognize, and recall the pattern sequences mentioned earlier. The additional circuitry is needed to transform the output sequences into names of output sequences that will be provided to the next higher level of auto-associative memories.
• Referring back to next pattern prediction made by a cortical processing unit, the next pattern prediction may also be passed between different cortical processing units.FIG. 2A illustrates the different hierarchical layers with information flowing both upwards and information flowing downwards. As set forth earlier, each individual cortical processing unit attempts to identify invariant structure in its input stream. When a processing unit recognizes a memorized sequence of pattern, information about that recognition (an identifier for the recognized sequence) is passed upward to the next higher cortical unit layer(s). Furthermore, information about failed predictions may flow upward. The downward information flow may include next pattern predictions made by higher cortical processing units in the hierarchy to lower higher cortical processing units.
• FIG. 2D explicitly illustrates a conceptual example of such a downward flow of prediction information. Specifically,cortical processing unit231 has made a prediction (represented by a dashed line) that is passed to the two cortical processing units (221 and222) in a lower cortical layer that are coupled tocortical processing unit231.
• Each cortical processing unit memorizes sequences of patterns that the cortical processing unit receives on its input. Those inputs are actually identifiers of recognized sequences from the cortical processing units in the lower layer. When an upper cortical processing unit recognizes that it is currently within a memorized sequence of patterns then the cortical processing unit may predict the next pattern in the sequence it is currently receiving. If that predicted pattern is passed to the cortical processing unit in the hierarchical layer below, that predicted pattern represents the next sequence of patterns that is expected from the lower cortical processing unit. Thus, the lower cortical processing unit receives a prediction about the next expected sequence of patterns it will likely encounter.
• Such a prediction about the next expected sequence may help a cortical processing unit resolve ambiguity on a sequence of patterns level. A cortical processing unit may not be able to match a new incoming sequence of patterns to any of the stored sequences of patterns. Or worse, the incoming sequence of patterns may closely resemble two different stored sequences of patterns. To determine how to properly categorize such an ambiguous current incoming sequence of patterns, the cortical processing unit may consider the prediction information received from a higher cortical processing unit. Thus, if a cortical processing unit cannot determine if a current sequence of patterns matches memorized sequence A or memorized sequence B and the higher level cortical processing unit predicts that it will receive sequence B, then the cortical processing unit may use that prediction from the higher cortical processing unit to resolve such ambiguous cases and thus select sequence B.
• This complex prediction system that creates predictions both within individual cortical processing units and between coupled cortical processing units creates a sophisticated prediction system that operates throughout the entire hierarchy in a coordinated manner. Specifically, every cortical processing unit in the hierarchy may use its own memory to predict the next pattern to be received in the currently recognized sequence of patterns. And every cortical processing unit in the hierarchy may pass that predicted next pattern down to the lower connected cortical processing units. A cortical processing unit may use a prediction received from a higher cortical processing unit to predict the next sequence of patterns that will be received.
• As long as each pattern or pattern sequence prediction matches the received input, things run smoothly. And predictions may be used to resolve ambiguous patterns or pattern sequences. But when a prediction is significantly wrong, an error signal or the unexpected input is propagated up the cortical hierarchy. If a cortical processing unit higher up in the hierarchy is able to resolve the problem, then correction information will flow from that cortical processing unit back down the hierarchy. If a problem propagates all the way to the ‘top’ of a hierarchy, then some new learning may need to occur in order to address the novel situation in the future.
• As set forth in an earlier section of this document, the collective memory of invariant structures learned by the hierarchy of cortical processing units comprises an invariant representative model of the surrounding world. By combining incoming sequences of patterns from sensory organs along with sequence and pattern predictions that are constantly made by all the cortical processing units using the invariant representative model of the surrounding world stored in the memory, our neocortex generates a full perception of reality. Thus, our understanding of ‘reality’ is more than simply what we can detect with our senses at any particular moment. Instead, our perception of reality is created with both sensed information (from our sensory organs) and information recalled from memory (our model of the world). (In fact, the majority of our perception of reality may come from memory.)
• This combination of sensed information with memorized information is very useful since our individual sensory organs alone do not provide enough information to completely understand the surrounding world. Only the combination of sensory information with information recalled from memory such that accurate predictions can be made leads to a real understanding of the surrounding world. The more invariant structure that has been deduced from the surrounding world and stored into memory, the more accurate predictions can be made about that surrounding world. And consequently, the more accurate predictions can be made about that surrounding world, the better understanding of the outside world.
• There are numerous examples that illustrate how our neocortex combines sensory input with our model of the world in memory to create a full perception of reality. For example, it is well known that people do not always hear every word that is spoken by another person in a typical conversation. However, our ability to predict the proper word that would be appropriate at the time of an unheard word allows us to understand what the other person said without hearing every word. This only occurs when a person is sufficiently fluent in a language. Before becoming fluent, a person learning a language will often need a speaker to slow his rate of speaking and annunciate every word. A similar type of prediction occurs when reading difficult to read handwriting. Specifically, our learned knowledge of sentence structure, grammar, the material being discussed, and other memorized information allow us to decipher handwritten words that could not be deciphered if presented alone.
• A very interesting example of combining sensory information with memorized information occurs with the blind spot in our eyes. Our eyes have a ‘blind spot’ where the optic nerve couples the rear of the eyeball. The blind spot exists because there are no light detecting cells on the retina where the optic couples to the eyeball. This blind spot only fully exists when one eye is closed since normally the other eye covers the field of vision containing the eye's blind spot. But even when a person has only one eye open, that person does not notice a gaping hole in his visual field. In fact, a person must make a concerted effort to even become aware the blind spot when looking with only one eye. How can this be possible?
• The reason that the blind spot is not easy to detect is that our brain combines the sensed visual information with predicted visual information such that the hole from the blind spot is ‘filled in’ to create the full visual perception. Specifically, the brain fills in the hole with information from the surrounding visual scene and makes intelligent predictions to fill in the hole. If the blind spot hole is within the pattern of a carpet, then that carpet pattern will be used to fill the hole. If a straight line passes through the blind spot, then a straight line segment will be filled into the blind spot in order to complete the straight line. Since our brains are very good at making predictions about our surroundings due to years of learning, the predicted information appears to be just as real as the actual sensed information. Thus, you really cannot always believe what you “see”.
• Cortical Algorithm Makes Predictions from Invariant Representations and Input Data
• Feedback or information flowing down the cortical hierarchy serves a number of functions. As set forth earlier, the feedback information can be used to resolve ambiguous input data. But the feedback data is also used for other very important functions.
• The feedback information may be required for learning. As every cortical processing unit and region of neocortex learns sequences of patterns, that cortical processing unit or region of neocortex learns the sequences in a particular context that is provided by the next higher cortical processing unit or region of neocortex. The context specifies a state of the higher cortical processing unit or region of neocortex. The context may be specified as the current element of a sequence being experienced by that higher cortical processing unit or region of neocortex. This context feedback information provides a means for the cortical processing unit or region of neocortex to properly group its input patterns together.
• Another very important use of feedback information from higher layers is to provide the ability to perform specific predictions using invariant representations. Predictions based on invariant representations occur when a high level pattern moves down the hierarchy towards the input. Such predictions occur in two different manners. A first manner occurs when the system predicts something that has not yet happened. For example, our neocortex allows us to predict (and hear) the next note of a song before it has happened. The other invariant representation type of prediction occurs when a system fills in missing information in something that is happening now. For example, our neocortex fills in missing or occluded parts of a visual image as previously described with reference to the blind spot.
• In both of these scenarios the system has a challenge in making the desired specific prediction. The challenge presented is that a pattern high up in the cortical hierarchy is in an invariant form. The invariant form lacks the details to generate a prediction that is precise and relevant to the current situation. But ideally, the system should generate as precise of a prediction as is possible. Thus, the invariant representations must be combined with some specific feed forward information to make specific predictions.
• A couple of examples will illustrate the concept. Imagine that you are expecting to see a dog (perhaps you heard a dog bark). In the high-level IT region in the neocortex's visual processing region, the neocortex knows that it should be representing “dog”. However, without any additional current information, the IT region of neocortex cannot predict what type of dog you will see, where the dog will be, and in what position the dog will be seen. However, if you are already seeing part of a dog (such as a foot or tail), the neocortex can use this partial feed forward information to help predict where the rest of the dog will be and what the dog will look like. Similarly if you are expecting to hear a particular note interval in a song, such as an octave step up, there is no way to determine what will be the precise next note without additional information. However, if your neocortex is able to combine this “invariant” octave step up prediction along with the last note that was heard then the neocortex will be able to precisely predict the next note that will be heard.
• As set forth above, invariant representations must be combined with some feed forward information in order to generate specific predictions. The general method to do this is as follows. As feed forward inputs arrive at a cortical processing unit or a region in a memory hierarchy from the units below, those feed forward inputs may be combined to form a spatial pattern. As part of the learning process, this spatial pattern must be stored as one type of example of whatever higher level context information is being fed down to this cortical processing unit or region. As different feed forward inputs (combined into spatial patterns) arrive, the feed forward inputs (spatial patterns) are stored as part of different higher-level contexts that are provided at that time. For example, a cortical unit or neocortex region may experience the following learning: pattern A (feed forward input pattern) is a dog (context), pattern B is a dog, pattern C is a cat and pattern D is a cat. Those A, B, C, and D patterns are the components of invariant representations that are learned for future reference.
• The stored patterns are later used during recognition. For example, when feedback information to a cortical unit or neocortex region specifies an expectation of seeing a dog, then the cortical units or neocortex region partially invokes all the stored patterns that were stored in the context of “dog”. In the example from the previous paragraph, the cortical units or neocortex region will partially invoke patterns A and B since those patterns were stored in the context of “dog”. To complete the prediction process, the partially invoked invariant representation must be combined with current information. Thus, partial feed forward input to the cortical units or neocortex region will preferentially match pattern A or pattern B such that the combination of the feed forward input and its matching pattern results in a precise prediction. The prediction is a specific prediction (pattern A or pattern B) created by combining the feed forward input to the known examples of the “dog” context.
• The book “On Intelligence” sets forth the details of one mechanism that may be used within the human neocortex. This mechanism uses sparse distributed spatial patterns. In the described mechanism,layer2 cells in the six layer columns of neocortex are a superset of all relevant invariant representation patterns. Feedback information from above activates an invariant representation inlayer2 and the activatedlayer2 cells signallayer6 cells in the same column. Feed forward input information from below representing the current situation is provided tolayer6 of the columns relevant to the feed forward information. The intersection of feedback createdlayer2 cell activity that signalslayer6 and feed forward activity inlayer6 determines the specific prediction.
• There are many other methods of performing the same function in other systems. The requirements of constructing a system are relatively simple. First, the system must have multiple different representations of multiple different objects (the different contexts). Each of the multiple different representations must be associated with a specific type of feed forward (or input) information such that the feed forward information will partially activate the representation. Feed back information that specifies a particular context (object) must be able to partially activate the different representations associated with that context (object). Any representation that is partially activated by feedback information and partially activated by feed forward information is considered fully activated. In such a system, a specific prediction can be made by specifying a particular context with feedback information (to partially activate a relevant set of representations) and specifying current feed forward input information (to partially activate representations related to the feed forward information) such that a fully activated representation is generated as a prediction. The prediction may then flow down the hierarchy.
• Initially, this may seem to be a very impractical method of operation since there are a nearly infinite number of possible ways objects can appear (different representations). If one attempted to store a different pattern representing each possible input, this would require an impossibly large amount of memory. But as set forth in the previous sections, a hierarchical memory organization is very efficient at representing very large numbers of different things such that this apparent limitation can be overcome. In essence the lower regions of the memory hierarchy form representations of lower level primitives that are shared among many higher-level object representations. This dramatically reduces the amount of memory required to represent a very large number of different items.
An Intelligent Machine Architecture
• To construct a sophisticated memory systems or an intelligent machine, the present invention proposes using aspects of the architecture and the operation theory of the human brain set forth in the previous sections as a blueprint for creating sophisticated memory systems and intelligent machines. By attempting to emulate important aspects deduced from the sophisticated intelligence system honed by evolution (the human neocortex), significant advances in the field of artificial intelligence may be achieved. This approach avoids the pitfalls of the current main two approaches of artificial intelligence.
• The subsections of the previous section set forth the main principles that have been derived from the mammal neocortex. Specifically:
• 1. Sensory Input Comprises Sequences of Patterns
• 2. The Neocortex Uses a Single ‘Cortical Algorithm’ for many different problems
• 3. The Neocortex Is Organized in a Hierarchical manner
• 4. The Cortical Algorithm Stores Structure in Memory for Later Recognition
• 5. The Cortical Algorithm Makes Predictions Using Its Stored Memories
• 6. Cortical Algorithm Makes Predictions from Invariant Representations and Input Data
• Using these six principles, various different sophisticated memory systems and ultimately an intelligent machine may be constructed. Not all of these different principles must be used in every embodiment that uses the teachings of the present invention. Many useful systems may be created by combining together only a few of these teachings.
• An intelligent system may comprise a plurality of individual cortical processing units arranged into a hierarchical structure. Each individual cortical processing unit receives a sequence of patterns as input. Each cortical processing unit processes the received input sequence of patterns and outputs another pattern. As several input sequences are processed by a cortical processing unit, it will generate a processed sequence of patterns on its output that will be passed to cortical processing units in higher layers.
• Each individual cortical processing unit includes a memory that is used to help process the input sequence of patterns. The memory may comprise previously encountered sequences of patterns that contain invariant structure. An input sequence of patterns may be deemed to contain structure if that sequence of patterns is repeated. An individual cortical processing unit can use the stored sequences of patterns in order to ‘recognize’ such sequences in the input stream.
• When a previously encountered sequence of patterns is recognized by a cortical processing unit, the cortical processing unit can report such recognition to higher layer cortical processing units. The output to the higher layer may comprise an identifier of the recognized sequence of patterns in the incoming sequence of patterns. The output stream containing identifiers of recognized sequences comprises a sequence of patterns itself. That output stream of patterns may comprise the input to a cortical processing unit in the next higher cortical processing unit layer.
• A cortical processing unit may also use the memorized sequence of patterns in order to make predictions about the incoming sequences in order to resolve ambiguous information. Specifically, when a cortical processing unit determines that it is currently receiving a known sequence, the cortical processing unit may predict the next pattern to be received in the sequence. The processing unit may use the prediction to interpret the next pattern received. For example, if the input stream into a cortical processing unit is ambiguous, then that cortical processing unit may use the predicted pattern to help identify the next input pattern.
• The cortical processing unit may further provide the predicted next pattern down to the next lower cortical processing units. The lower cortical processing units may interpret this as a prediction of the next sequence that it will output. Such prediction information may comprise information that helps the lower cortical processing unit layers correctly identify the next sequence.
• The prediction information may also be considered as useful predictions of what will occur in the outside world. And if an incoming pattern significantly differs from a predicted pattern or an incoming sequence of patterns significantly differs from the predicted sequence of patterns, then there is a prediction failure. The occurrence of a prediction failure may be transmitted to a cortical processing unit in a higher layer. A cortical processing unit in a higher layer may be able to resolve the error. A detected error may be resolved in a number of different ways. One way to resolve the error is to adjust the cortical hierarchy's model of the world by storing a new sequence of patterns.
One Visual Pattern Recognition Implementation
• The previous sections have described a system that has a hierarchy that stores sequences and makes predictions of future events. However, like the human visual system, the system can be used to solve spatial recognition problems. This section presents one example embodiment of a prototype program that uses the teachings of the previous sections to recognize visual object patterns by first learning about the object patterns. As one skilled in the art of computer knows, there are many different ways to implement the methods and architecture presented in the previous sections of this document. However, the principles of operation and the architecture are set forth with sufficient detail such that many different systems that incorporate the teachings of this document may be created.
• The prototype pattern recognition program is constructed of a hierarchy of cortical processing units that analyzes input training information consisting of line drawings in order to identify structures within that training information. The prototype program is also informed about the names of objects in some of the training information such that the prototype program can associate particular learned structures with particular object names. After the prototype program completes its learning phase, the prototype program can then accurately recognize objects in novel input data using the learned information.
• In this example prototype pattern recognition program, the hierarchical processing units of the prototype program receive pattern information that is presented as a continuous stream in the form of animated movies. Thus, sequences of patterns are presented to and learned by the prototype program. The prototype program then uses the learned information to create probabilistic models about the outside world. The probabilistic models are then used to recognize objects in novel pattern data using Bayesian belief propagation as is well-known in the art. Explicit details on Bayesian belief propagation can be found in the book “Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference” by Judea Pearl” from Morgan Kaufman Publishers, Inc. However, other implementations that incorporate the teachings of the present invention may process the learned information in different manners and may use other methods of inferring a final result.
• The Visual Pattern Recognition Problem Addressed
• The visual pattern recognition problem addressed by the prototype program implements the hierarchical architecture of the present invention with computer software. To keep the size of the software, the amount of memory required, and processing requirements of the prototype program under control and to quickly create a functioning prototype, some simplifications were made from an ideal implementation of the present invention.
• Furthermore, the visual pattern world of the prototype program was limited to a small pixel grid. Specifically,FIG. 3A illustrates the thirty-two by thirty-two grid of black or white pixels that make up the limited visual world of the prototype program. However, the thirty-two by thirty-two grid of black or white pixels presents ample room to create a variety of line drawings and closely related versions of those line drawings that should be identified as belonging to the same group. Examples of some of the line drawings created in the thirty-two by thirty-two black and white pixel grid are presented inFIG. 8.
• The Sensory Input Used for the Visual Pattern Recognition Problem
• The hierarchical arrangement of cortical processing units in the prototype program analyzes the entire thirty-two by thirty-two grid of pixels first during a learning phase and then later during a pattern recognition phase. In an example embodiment of the prototype program, the lowest level of cortical processing units each process an input sequence of patterns from a four by four patch of pixels.FIG. 3B illustrates the thirty-two by thirty-two grid of pixels divided into a set of four by four patches of pixels. Each individual four by four patch of pixels is analyzed by its own first level cortical processing unit. There are sixty-four individual four by four pixel patches illustrated inFIG. 3B such that the first cortical processing unit layer of the hierarchy will have sixty-four cortical processing units.
• Patch310 illustrates an example four by four patch of pixels taken from the full thirty-two by thirty-two grid of pixels. Each patch of pixels can be addressed using the row and column numbers illustrated on grid ofFIG. 3B. A full address should also include an identifier of the layer of the cortical processing unit hierarchy layer. In this example, the lowest hierarchical layer will be deemed layer “X”. Thus, the address for the example patch could be given as X17 wherein the first character specifies the cortical unit layer, the second character specifies the row, and the third character specifies the column.
• The Cortical Processing Unit Hierarchy
• In the next higher cortical unit layer, each cortical processing unit was assigned to examine the sequence of patterns created by four adjacent cortical processing units. Specifically,FIG. 3C illustrates sixteen groups of four by four pixel patches on the grid that are examined by the sixteen cortical processing units of the second cortical layer. As with the lower layer X, the sixteen patches ofFIG. 3C (and their respective cortical processing units) can be addressed using a row and column index. Thus, if the second layer is designated as layer Y then thecorner patch320 can be addressed as Y03 wherein Y is the layer, 0 is the row, and 3 is the column.
• Successive cortical processing unit layers would be implemented in a similar manner.FIG. 4 illustrates one possible example of the hierarchy of the cortical processing units for the thirty-two by thirty-two grid of pixels illustrated inFIGS. 3A to3C. In the example of hierarchy ofFIG. 4, a third layer has four cortical processing units wherein each processing unit handles data from four cortical processing units in the second layer. A fourth cortical processing unit layer consists of two cortical processing units wherein each considers the outputs from two cortical processing units in the third layer. Finally, fifth and top cortical processing unit layer processes the output from the two cortical processing units of the fourth layer.
• Note that the cortical processing units in the successively higher layers of the hierarchy inFIG. 4 process information from more than one cortical processing unit in the layer below. Thus, the receptive field of successively higher cortical processing units grows larger as one ascends the hierarchy of cortical processing units. This is designed to be similar to the visual system of the human neocortex that operates in a similar manner.
• Note thatFIG. 4 presents only one arbitrary selected hierarchical arrangement for the pattern recognition problem. Many other hierarchical arrangements could be created. For example, the processing units in the lowest layer of the hierarchy ofFIG. 4 each consider an input consisting of sixteen pixels in a four by four pixel patch. But many other divisions of sensory input could be used such as having the cortical units on the lowest layer only consider input from four pixels in two by two pixel patches. The same principle applies to all of the different cortical processing unit layers in the hierarchy. For example, the cortical processing units on thethird layer430 could consider the output from five or six cortical processing units in the second layer. In another embodiment, the cortical processing units on thethird layer430 may each only consider the output from four processing units in the lower second layer.
• In the prototype program, a smaller cortical processing unit hierarchy was constructed.FIG. 5A illustrates the cortical hierarchy architecture used to construct the prototype program. As illustrated inFIG. 5A, the lowest cortical processing unit layer, theX layer510, has sixty-four cortical processing units for processing the sensory input from the thirty-two by thirty-two pixel grid in individual four by four pixel patches as set forth withFIG. 3B. The second cortical processing unit layer, theY layer520, has sixteen cortical processing units wherein each cortical processing unit processes the output from four cortical processing units in theX layer510 as set forth withFIG. 3C. Finally, in the prototype program hierarchy ofFIG. 5A, a single cortical processing unit comprised the top layer, named theZ layer530. The single processing unit on theZ layer530 processes the output from all sixteen cortical processing units on theY layer520.
• FIG. 6 illustrates an alternate view of the cortical processing unit hierarchy ofFIG. 5A. In the alternate view ofFIG. 6, each of the cortical processing units is labeled with an address using the addressing system set forth with reference toFIGS. 3B and 3C.
• The Visual Patterns for Training
• To train the prototype program, a set of line drawing shapes were presented to the prototype program.FIG. 8 illustrates an example set of ninety training shapes that were used to train the prototype program. The line drawing shapes ofFIG. 8 were presented to the prototype program in the form of animated movies that had the line drawing training shape moving about in the thirty-two by thirty-two pixel world. This training allows the prototype program to learn about various geometric shapes inFIG. 8 that make up the world of the prototype program and the movements of those geometric shapes in its limited visual world.
• Unsupervised Training
• During initial training, the cortical processing units of the prototype program each memorize the most frequently occurring sequences of patterns that are encountered. For example, the cortical processing units in thelowest X layer510 of the prototype program memorizes the most frequently occurring sequences of patterns that are encountered on their local four by four pixel patch. In one embodiment of the prototype program, the cortical processing units in thelowest X layer510 memorize sequences of four consecutive patterns. Similar pattern sequence learning also occurs at the Y layer and the Z layer. This unsupervised learning allows the prototype program to learn the various primitive structures that make up its world.
• The unsupervised training was initially performed with animated movies of the specific objects that were to later be recognized. For example, the unsupervised training may have used the letter line drawings inFIG. 8. However, another unsupervised training session used a set of shapes that did not include any of the shapes that the system would later attempt to recognize. It was discovered that as long as the line drawings used in training included the same low-level primitive structures that exist in the line drawings to be recognized (such as the letters) then this unsupervised training without the letters worked fine. The low-level primitive structures in the letters include structures such as vertical lines, horizontal lines, corners, boxes, and parallel lines. Thus, it was immediately apparent that the hierarchical arrangement learned about low level primitive structures that consistently existed in its world. The prototype program then combined the low-level structures in various manners to create representations of many different higher level objects.
• FIG. 9 illustrates eight different possible sequences of four consecutive four by four pixel patterns that could be seen by the cortical processing units of the lowest layer of the cortical processing unit hierarchy (X layer510). The four horizontal sequences of patterns S₁, S₂, S₃, and S₄represent four pattern sequences commonly encountered during the animated movies of the unsupervised training. S₁and S₂depict a vertical line moving right and a horizontal line moving upward, respectively. S₃and S₄depict the motion of a lower left corner and an upper-right corner, respectively. The four vertical sequences of patterns S₅, S₆, S₇, and S₈depict very random pattern sequences that never (or almost never) occur. Thus, frequently occurring (repeated) sequences of patterns such as those of S₁, S₂, S₃, and S₄and other frequently occurring sequences were memorized by thelowest X layer510 cortical processing units as sequences that contain invariant structure.
• In one embodiment, only the thirteen most frequently occurring sequences of patterns were stored in the memory of cortical processing units in theX layer510. Each of the memorized frequently occurring sequence of patterns is given a unique identifier. In one embodiment, each of the different memorized sequences were very different such that theX layer510 cortical processing units could immediately recognize which memorized sequence was being experienced. (The sequences had no or almost no shared patterns.)
• The identifiers of recognized sequences of patterns are passed up to the next layer of cortical processing units, theY layer520 ofFIG. 5A. As illustrated in the particular hierarchical arrangement ofFIG. 5A, each cortical processing unit in theY layer520 receives information from four cortical processing units in thelower X layer510. The four sequence identifiers from the four X layer units are combined to form a spatial pattern received by the cortical processing unit in theY layer520. In one embodiment, the sequence identifiers from the four X layer processing units are simply concatenated together to form a unique pattern that is presented to the cortical processing unit in theY layer520. Note that the Y layer cortical processing units understand higher level concepts since Y layer processing units combine together sequence information from four individual X layer processing units.
• As previously set forth, one embodiment memorized thirteen different commonly encountered sequences in theX layer510. With sequence identifier information from four different X layer cortical processing units wherein each layer may be in one of thirteen different sequences; theY layer520 processing units may experience 13⁴=28561 possible different input patterns. However, only seven hundred and forty-four (744) different input patterns were actually experienced. Thus, as predicted, only a small percentage of the possible input patterns into a cortical processing are ever experienced by the cortical processing unit.
• The cortical processing units in theY layer520 may perform the same learning operation as the cortical processing units in theX layer510 as set forth above. Specifically, each cortical processing unit in theY layer520 identifies and then memorizes commonly experienced sequences of patterns on its input stream. The Y layer processing units would later attempt to recognize those memorized sequences in their input streams. The Y layer processing units could then report recognized sequences of patterns to the next higher cortical processing unit layer, theZ layer530. TheZ layer530 receives sequence identifiers from all sixteen different processing units in theY layer520. The sixteen sequence identifiers from theY layer520 may be combined to form a spatial pattern received by theZ layer530.
• In one particular embodiment, the prototype program also collects a set of statistics about the patterns and pattern sequences that are detected during the unsupervised learning phase. The collected statistics may include how many times a particular pattern or sequence is encountered and how many times a particular input value is encountered. As set forth in the in the earlier section on making specific predictions from invariant representations, one very important statistic to keep track of is the specific context when a particular pattern or sequence encountered. In one embodiment, the particular higher-layer pattern may be the context for the lower layer patterns and sequences. It is during this training phase that the system constructs a Bayes network such that Bayesian inference using Bayesian belief propagation may be used to perform pattern recognition. Details on Bayesian inference will be set forth in greater detail in a later section.
• Supervised Training
• After the unsupervised training, the prototype program has collected sequences of patterns and developed higher level object concepts from the lower layer sequences. Since the prototype program already understands object concepts at the top level, the training from layer Y to layer Z can proceed in a supervised manner. The supervised training links specific object names to object concepts that have been learned at the high Z layer. In this manner, the prototype program may later infer the specific object names when giving novel input data.
• The supervised training may be performed by presenting an object name to thetop Z layer530 while presenting an animated movie of the particular object shape. The prototype program thus links the high level concept at the Z layer (the current sequence with the object name currently being presented.
• For example, during supervised training with the third geometric shape from the left on the top row ofFIG. 8, the memorized sequences recognized at the Z layer are associated with the name “cat”. Similarly, during training with the sixth geometric shape from the left on the top row, the memorized sequences recognized at the Z layer are associated with the name “dog”. And so on with all the different geometric shapes during training.
• As with the unsupervised training phase, the prototype program also collects a set of statistics about the detected patterns and pattern sequences during the supervised learning phase. These statistics will be used to build probability tables in order to perform Bayesian inference as will be described in a later section.
• Pattern Recognition using Bayesian Inference
• The learning pattern recognition prototype program uses Bayesian inference techniques in order to generate very accurate object name using pattern recognition. In the prototype program, the objective is to have recognized sequence information at thetop Z layer530 associated with the correct object name. Ideally, even novel input patterns should output recognized Z layer sequences (and the object name associated with the recognized Z layer sequence) due to generalization of the knowledge acquired by the prototype program.
• Bayes' rule allows for a system to determine the probability of a particular result R outcome based upon available evidence E. To compute this probability, one needs to have observed a fair amount of information about the different possible results R and the different possible evidence E. The fundamental rule of Bayesian learning is that
  P(R|E)=P(E|R)P(R)/P(E)
  This states that the probability of a particular-result R when given certain evidence E is equal to the probability that the particular evidence E given the result R times the prior probability of encountering the result R divided by the probability of encountering the evidence E.
• To perform the desired pattern recognition with the prototype program, the prototype program must attempt to determine the output sequence Z at the top of the hierarchy (and thus the associated object name) when given particular input information at the bottom (X layer510) of the hierarchy. Since this is a bottom to top determination, it is technically referred to as an inference. The input information is supplied in the form of a static thirty-two by thirty-two pixel drawing that is provided into the lowest layer, theX layer510. Thelowest X layer510 processes the thirty-two by thirty-two pixel drawing to determine the mostly likely sequence at the X layer. If given time, the X layer could also receive and consider real pattern sequence information to generate more accurate results.
• To make the desired determination of the Z sequence, it would be desirable to know the probability of different possible output Z_jsequences when given the specific input sequence X_i. This probability is expressed as P(Z|X). Using Bayes' rule, we know that:
  P(Z_j|X_i)=P(X_i|Z_j)P(Z_j)/P(X_i)
  Therefore, to determine the probability of a particular Z sequence when given the input information of X, the prototype program must know:
• P(X_i|Z_j) which is the probability that the specific X_iinput is causing sequence Z_j
• P(Z_j) which is the probability encountering of the particular Z_jsequence; and
• P(X_i) which is the probability of encountering the specific input sequence X_i
• The information to determine all three of these needed probability values was collected during the learning phase.
• It should be noted that the probability of a specific X sequence input value given a particular Z sequence, P(X|Z), is not directly learned and placed into a probability table. Instead, the problem is broken down into a Bayes belief net. Thus, different probability tables are created during learning such that those probability tables can then be combined with a Bayes rule network to obtain the desired probability value.FIG. 7 illustrates the Bayes belief net for the pattern recognition system of the prototype program.
• As illustrated in the Bayes belief network ofFIG. 7, a Y sequence value can be selected using a probability of the Y sequence value given specific X sequence inputs [P(Y|X)]. Subsequently, the Z sequence value can be selected using a probability of the Z value given selected Y sequence inputs [P(Z|Y)]. These probabilities may be computed using Bayes' rule. In order to perform this, all of the individual Y layer processing units can observe the probability of a particular Y sequence value given a specific Z sequence value, P(Y|Z). Similarly, all of the individual X layer processing units can observe the probability of a particular X sequence value given the specific Y sequence value, P(X|Y).
• To store the needed probability values, one must have contextual feedback that is presented from higher cortical layers to lower cortical layers during training.FIG. 5B illustrates examples of the contextual feedback that may be presented from higher cortical layers to lower cortical layers in order to allow the needed probability tables to be created. For each cortical processing unit in the Y layer, the current Z sequence context information is provided. This is illustrated inFIG. 5B as the current Z sequence value being fed back to Y layer processing units. (Note that the context feed back is only illustrated for the two left-most Y layer processing units but the current Z sequence is provided to all of the sixteen Y layer processing units.) This Z sequence contextual feedback allows each cortical processing unit in the Y layer to create a probability table that specifies the probabilities of all the different Y sequences when given a specific Z sequence, P(Y|Z). The sixteen probability tables in the sixteen processing units in the Y layer may appear as follows:$\begin{array}{c}\begin{array}{ccccc}{y}_1& y_2& {\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00007.png,US20060184462A1-20060817-D00008.png"\; state="\{\{state\}\}">y</figure-callout>}}_3& \cdots & y_m\end{array}\\ \begin{array}{c}{z}_1\\ z_2\\ {\mathrm{<figure-callout\; label="z"\; filenames="US20060184462A1-20060817-D00007.png,US20060184462A1-20060817-D00008.png"\; state="\{\{state\}\}">z</figure-callout>}}_3\\ \cdots \\ z_n\end{array}\left[\begin{array}{ccccc}P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00011.png"\; state="\{\{state\}\}">y</figure-callout>}}_1\text{❘}{z}_1\right)& P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00007.png,US20060184462A1-20060817-D00008.png"\; state="\{\{state\}\}">y</figure-callout>}}_2\text{❘}{z}_1\right)& P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00007.png,US20060184462A1-20060817-D00008.png"\; state="\{\{state\}\}">y</figure-callout>}}_3\text{❘}{z}_1\right)& \cdots & P\left(y_m\text{❘}{z}_1\right)\\ P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00011.png"\; state="\{\{state\}\}">y</figure-callout>}}_1\text{❘}{z}_2\right)& P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00007.png,US20060184462A1-20060817-D00008.png"\; state="\{\{state\}\}">y</figure-callout>}}_2\text{❘}{z}_2\right)& P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00007.png,US20060184462A1-20060817-D00008.png"\; state="\{\{state\}\}">y</figure-callout>}}_3\text{❘}{z}_2\right)& \cdots & P\left(y_m\text{❘}{z}_2\right)\\ P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00011.png"\; state="\{\{state\}\}">y</figure-callout>}}_1\text{❘}{z}_3\right)& P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00007.png,US20060184462A1-20060817-D00008.png"\; state="\{\{state\}\}">y</figure-callout>}}_2\text{❘}{z}_3\right)& P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00007.png,US20060184462A1-20060817-D00008.png"\; state="\{\{state\}\}">y</figure-callout>}}_3\text{❘}{z}_3\right)& \cdots & P\left(y_m\text{❘}{z}_3\right)\\ \cdots & \cdots & \cdots & \cdots & \cdots \\ P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00011.png"\; state="\{\{state\}\}">y</figure-callout>}}_1\text{❘}{z}_n\right)& P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00007.png,US20060184462A1-20060817-D00008.png"\; state="\{\{state\}\}">y</figure-callout>}}_2\text{❘}{z}_n\right)& P\left({\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00007.png,US20060184462A1-20060817-D00008.png"\; state="\{\{state\}\}">y</figure-callout>}}_3\text{❘}{z}_n\right)& \cdots & P\left(y_m\text{❘}{z}_n\right)\end{array}\right]\end{array}$
• The same type of feedback is provided to the next lower layers as well. Specifically, for each cortical processing unit in the X layer, the current Y sequence context information is provided from its associated Y layer processing unit. This is illustrated inFIG. 5B as the current Y sequence value from Y layer processing units Y00 being fed back to two associated lower X layer processing units. (Note that only two feed back paths are illustrated from a Y layer processing unit to two X layer processing units. However all sixteen of the individual Y layer processing units would feedback their current Y sequence to their four associated X layer units.) This Y sequence contextual feedback allows each cortical processing unit in the X layer to create a probability table that specifies the probabilities of all the different X sequences when given a specific Y sequence, P(X|Y). These sixty-four probability tables in the various X layer processing units may appear as follows:$\begin{array}{c}\begin{array}{ccccc}{x}_1& x_2& x_3& \cdots & x_m\end{array}\\ \begin{array}{c}{y}_1\\ y_2\\ {\mathrm{<figure-callout\; label="y"\; filenames="US20060184462A1-20060817-D00007.png,US20060184462A1-20060817-D00008.png"\; state="\{\{state\}\}">y</figure-callout>}}_3\\ \cdots \\ y_n\end{array}\left[\begin{array}{ccccc}P\left(x_1\text{❘}{y}_1\right)& P\left(x_2\text{❘}{y}_1\right)& P\left(x_3\text{❘}{y}_1\right)& \cdots & P\left(x_m\text{❘}{y}_1\right)\\ P\left(x_1\text{❘}{y}_2\right)& P\left(x_2\text{❘}{y}_2\right)& P\left(x_3\text{❘}{y}_2\right)& \cdots & P\left(x_m\text{❘}{y}_2\right)\\ P\left(x_1\text{❘}{y}_3\right)& P\left(x_2\text{❘}{y}_3\right)& P\left(x_3\text{❘}{y}_3\right)& \cdots & P\left(x_m\text{❘}{y}_3\right)\\ \cdots & \cdots & \cdots & \cdots & \cdots \\ P\left(x_1\text{❘}{y}_n\right)& P\left(x_2\text{❘}{y}_n\right)& P\left(x_3\text{❘}{y}_n\right)& \cdots & P\left(x_m\text{❘}{y}_n\right)\end{array}\right]\end{array}$
  Overall Operation of the Prototype Program
• The prototype program is first trained during a training phase using shapes such as the shapes illustrated inFIG. 8. During this training period, each of the cortical processing units will extract structure observed in its world in the form of commonly repeated sequences of patterns. Such training may take place unsupervised. Furthermore, such training doesn't even require the specific objects that will later need to be recognized. The training may be occur with any shapes that have the same types of primitive structures that exist in the objects to be recognized. During the unsupervised training, each of the cortical processing units in the prototype program creates memories of the invariant structures that it detects during its exploration of its world.
• Later, supervised training is performed. During the supervised training, an object name associated with a particular object is presented to the prototype program in addition to an animated movie of the object that is being using for training. During such supervised training, the prototype program learns to associate the object name with the sequences on the Z layer.
• In an operation phase, novel data with an object image is presented to the prototype program. When the novel object data is presented, the prototype program extracts the structure from the object image in the form of stored structure recognized in the X, Y, and Z layers. The recognized structure is used to identify associated sets of probability tables. The identified probability tables are then used to calculate the probabilities of each object name being the correct name of the viewed object. Finally, the object name with the highest probability is inferred to be the name of the object in the novel image.
• The prototype pattern recognition program has provided highly accurate results. Furthermore, the prototype program has exhibited a strong ability to generalize.FIG. 10 illustrates some result data from three different line drawing tests. Column A presents the original training line drawing. Columns B and C illustrate human made line drawings that were successfully identified as belonging to the same class as the object in the Column A. Columns B and C illustrate human made line drawings that were successfully identified as belonging to the same class as the object in the Column A. Column D contains human made line drawings that were not successfully identified as belonging to the same class as the object in the Column A. Column E contains the line drawing class that program believed the line drawing of column D belonged within.
• The foregoing has described methods, architecture, and apparatus for implementing machine intelligence are disclosed. It is contemplated that changes and modifications may be made by one of ordinary skill in the art, to the materials and arrangements of elements of the present invention without departing from the scope of the invention.

Claims (43)

1. A memory system, said memory system comprising:

a hierarchy comprising a plurality of cortical processing units, each of said cortical processing units comprising

a first output, said first output for outputting a first set of pattern information,

a first input for receiving a second set of pattern information, said first input coupled to a sensory unit or to a first output from another cortical processing unit,

a memory, said memory storing information about patterns that may appear in said second set of pattern information on said first input, and

a processing method, said processing method comparing said second set of pattern information on said first input with said information about patterns stored in said memory.

2. The memory system as claimed inclaim 1, said memory system further comprising:

sensory units, said sensory units generating a third set of pattern information.

3. The memory system as claimed inclaim 2 wherein a lowest layer of said hierarchy of cortical processing units is coupled to said sensory units.

4. The memory system as claimed inclaim 1 wherein said pattern information comprises a sequence of patterns.

5. The memory system as claimed inclaim 1 wherein said memory containing information about recognized patterns comprises a set of sequences of patterns that have been repeated on said input.

6. The memory system as claimed inclaim 1, said memory system further comprising:

a second input, said second input for receiving a prediction from a cortical processing unit in a higher layer of said hierarchy.

7. The memory system as claimed inclaim 6 wherein said prediction comprises information about patterns in said memory.

8. The memory system as claimed inclaim 6 wherein said prediction comprises an identifier of a next sequence patterns expected.

9. The memory system as claimed inclaim 1, said memory system further comprising:

a second output, said second output comprising prediction information.

10. The memory system as claimed inclaim 9 wherein said second output is coupled to a cortical processing unit in a lower layer of said hierarchy.

11. (canceled)

12. A system, comprising:

a first level of modules, the modules each arranged to store a sequence of patterns; and

a second level of at least one module, the at least one module arranged to store a sequence of patterns, wherein the first level of modules and the second level of at least one module form at least part of a hierarchical network structure,

wherein, at least partially dependent on the sequences of patterns stored in the first level of modules and in the second level of at least one module, an object causing at least one pattern is determinable by information passing through the hierarchical network structure.

13. The system ofclaim 12, wherein the stored sequence in each module in the first level was determinable over time, and wherein the stored sequence in the at least one module in the second level was determinable over time.

14. The system ofclaim 12, wherein the object is determinable at least partially dependent on inference passing from a module in the first level to the at least one module in the second level.

15. The system ofclaim 14, wherein the inference passing is at least partially dependent on a probability distribution generated in the module in the first level.

16. The system ofclaim 12, wherein the object is determinable at least partially dependent on prediction passing from the at least one module in the second level to a module in the first level.

17. The system ofclaim 12, wherein sequences stored in the modules in the first level and the at least one module in the second level each form at least a portion of an invariant representation of the object.

18. The system ofclaim 12, wherein the hierarchical network structure comprises a Bayesian network structure.

19. The system ofclaim 12, wherein any of the modules are at least one of at least partially represented in software and at least partially implemented in hardware.

20. A method, comprising:

accessing a first module having a stored sequence of patterns;

accessing a second module having a stored sequence of patterns, wherein the first module and the second module form at least part of a hierarchical network; and

determining an object causing a pattern, the determining comprising at least one of:

passing first information from the first module to the second module dependent on the sequence stored in the first module, wherein the first information is indicative of at least one possible cause of the pattern, and

passing second information from the second module to the first module dependent on the sequence stored in the second module, wherein the second information is indicative of an expectation of a next pattern caused by the object.

21. The method ofclaim 20, further comprising:

learning and storing the sequence of patterns in the first module; and

learning and storing the sequence of patterns in the second module.

22. The method ofclaim 21, wherein any of the learning occurs over time.

23. The method ofclaim 20, wherein the stored sequence of patterns in the first module and the stored sequence of patterns in the second module each form at least part of an invariant representation of the object.

24. The method ofclaim 20, wherein the first module and the second module form part of a hierarchy of modules each arranged to perform a function substantially identical to the first module and the second module.

25. The method ofclaim 20, wherein the hierarchical network comprises a Bayesian network.

26. The method ofclaim 20, the determining further comprising:

passing third information from the second module to the first module, wherein the third information is arranged to modify a probability distribution in the first module.

27. A method, comprising:

inputting a first sensed pattern to a first processing unit;

inputting a second sensed pattern to a second processing unit;

in response to inputting the first sensed pattern, determining a first distribution having at least one possible cause of the first sensed pattern dependent on at least one sequence of patterns stored in the first processing unit;

in response to inputting the second sensed pattern, determining a second distribution having at least one possible cause of the second sensed pattern dependent on at least one sequence of patterns stored in the second processing unit;

passing the first distribution and the second distribution to a third processing unit; and

determining a cause of the first sensed pattern and the second sensed pattern at least partially dependent on at least one operation performed on the first distribution and the second distribution by the third processing unit.

28. The method ofclaim 27, wherein at least one of the first processing unit, the second processing unit, and the third processing unit is at least one of at least partially implemented in hardware and at least partially represented in software.

29. The method ofclaim 27, further comprising:

determining a third distribution having at least one possible cause of the first sensed pattern and the second sensed pattern dependent on at least one sequence of patterns stored in the third processing unit;

passing the third distribution to the first processing unit and the second processing unit; and

modifying the first distribution and the second distribution dependent on the third distribution.

30. The method ofclaim 27, further comprising:

inputting a first plurality of sequences of sensed patterns to the first processing unit;

inputting a second plurality of sequences of sensed patterns to the second processing unit;

the third processing unit determining at least one coincidence between the first plurality of sequences and the second plurality of sequences;

storing the at least one sequence of patterns in the first processing unit dependent on the at least one coincidence; and

storing the at least one sequence of patterns in the second processing unit dependent on the at least one coincidence.

31. A system, comprising:

a first processing unit arranged to input a sensed pattern and further arranged to generate an inference as to at least one possible cause of the sensed pattern dependent on a plurality of sequences stored in the first processing unit, wherein the inference comprises a probability distribution; and

a second processing unit arranged to, dependent on the inference, generate a prediction as to a next inputted sensed pattern dependent on a plurality of sequences stored in the second processing unit,

wherein a subsequent inference by the first processing unit is dependent on the prediction.

32. The system ofclaim 31, wherein the prediction is arranged to serve as a context for the subsequent inference.

33. The system ofclaim 31, wherein at least one of the first processing unit and the second processing unit is at least one of at least partially implemented in hardware and at least partially represented in software.

34. An apparatus, comprising:

a representation of a lower level of processing units, wherein the lower level processing units are arranged to output information regarding possible causes of sequences of sensed input patterns dependent on stored sequences previously learned by the lower level processing units; and

a representation of a higher level of at least one processing unit, wherein the at least one higher level processing unit is arranged to input the output information from the lower level processing units and output information regarding at least one possible cause of the sequences dependent on stored sequences previously learned by the higher level processing unit,

wherein the lower level of processing units and the higher level of at least one processing unit collectively represent at least part of a hierarchical network.

35. The apparatus ofclaim 34, wherein the output information from the lower level processing units is dependent on output information from the at least one higher level processing unit.

36. The apparatus ofclaim 34, wherein the hierarchical network comprises a Bayesian network.

37. The apparatus ofclaim 34, wherein at least one of the representation of the lower level of processing units and the representation of the higher level of at least one processing unit is in at least one of hardware and software.

38. A method, comprising:

storing sequences associated with an invariant structure of an object in a hierarchy of modules forming a hierarchical network;

receiving a sensed pattern;

first comparing the sensed pattern with a first part of the stored sequences;

determining at least one possible cause of the sensed pattern at least partially dependent on the comparing;

second comparing information associated with the at least one possible cause with a second part of the stored sequences, the second part representing a larger part of the invariant structure than the first part; and

determining the object as a cause of the sensed pattern at least partially dependent on the second comparing.

39. The method ofclaim 38, wherein the hierarchical network comprises a Bayesian network.

40. A system, comprising:

a hierarchical network of processing units, the hierarchical network having:

a first level of processing units each arranged to store at least one sequence of patterns associated with an object, and

a second level of at least one processing unit arranged to store at least one sequence of patterns associated with the object,

wherein a cause of input data received by the hierarchical network is determinable at least partially dependent on belief propagation through the hierarchical network, the belief propagation being at least partially dependent on the sequences stored in the first level of processing units and the second level of at least one processing unit.

41. The system ofclaim 40, wherein the hierarchical network comprises a Bayesian network.

42. The system ofclaim 40, wherein at least a portion of the hierarchical network is at least one of implemented in hardware and represented in software.

43. The system ofclaim 40, wherein the cause comprises the object.

Images