US20090141007A1 - Dynamic calibration of physiologically driven image triage systems - Google Patents

Abstract

A system and method of dynamically calibrating an image triage system. An image that may include one or more target entities is into a plurality of individual non-calibration image chips. Each non-calibration image chip is successively displayed to a user for a presentation time period. A calibration image chip that includes a synthetic target entity is selectively displayed, for the presentation time period, between the successive display of two non-calibration image chips. Calibration data are collected from the user at least while each calibration image chip is being displayed, and the image triage system is dynamically calibrated using the calibration data.

Description

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT
• This invention was made with Government support under contract HM1582-05-C-0046 awarded by the Defense Advanced Research Projects Agency (DARPA). The Government has certain rights in this invention.
TECHNICAL FIELD
• The present invention generally relates to a system and method for efficiently conducting image triage and, more particularly, to a system and method for dynamically calibrating neurally-driven high speed image triage systems.
BACKGROUND
• Analysts in various professions may, at times, be called upon to search relatively large collections of imagery to identify, if present, various types of relevant information (referred to herein as “a target entity” or “target entities”) in the collection of imagery. For example, medical analysts sometimes diagnose a physical impairment by searching complex imagery collections to identify one or more target entities therein that may be the cause of the physical impairment. Moreover, intelligence analysts may be called upon to search relatively complex imagery collections to identify target entities therein that may relate to various types of intelligence gathering activities.
• Advancements in both image collection and storage technology presently allow for the relatively low-cost storage of large volumes of high-quality imagery. However, the cost of searching through large sets of imagery for target entities can often be substantial. Indeed, in many professions, such as intelligence gathering, effective searching may rely on the expertise of highly skilled analysts, who typically search through relatively large sequences of images in a relatively slow manner. Presently, the number of skilled analysts available to search the amount of imagery that is stored, or can potentially be stored, is in many instances insufficient.
• In response to the foregoing, there has relatively recently been a focus on developing various systems and methods for triaging imagery. One of the methods that has shown promise combines electroencephalography (EEG) technology and rapid serial visualization presentation (RSVP). Various implementations of this combination have been researched and developed. For example, researchers have experimented with a system in which users are presented, using the RSVP paradigm, a sequence of images, some of which may include particular types of target entities. During the RSVP presentation, EEG data and/or physical response data are collected from the users. A trainable classifier processes the collected EEG data and/or physical response data to assign probabilities to each image. The probabilities are representative of the likelihood an image includes a target.
• Although useful in sorting a sequence of images, the above described system and method, as well as other systems and methods that employ these same technologies, do suffer certain drawbacks. For example, prior to the performance phase, in which images are searched for target entities, present systems and methods typically implement a calibration phase. During the calibration phase, images with known target entities are displayed to a user, and these data are used to train (or calibrate) the classifier. Present systems and methods thus rely on the calibration data collected during the calibration phase, even though signal characteristics during the performance phase may have changed since completion of the calibration phase. In particular, the characteristics of both the neural signals and/or the physical response signals change over time. As a result, the classifier may not be as accurate during later portions of the performance phase, which may lead to degraded target detection performance.
• Hence, there is a need for an efficient and effective system and method for increasing the likelihood of target identification in images after an initial calibration phase and throughout a performance phase. The present invention addresses at least this need.
BRIEF SUMMARY
• In one embodiment, and by way of example only, a method of dynamically calibrating an image triage system includes dividing an image that may include one or more target entities into a plurality of individual non-calibration image chips. Each non-calibration image chip is successively displayed to a user for a presentation time period. A calibration image chip that includes a synthetic target entity is selectively displayed, for the presentation time period, between the successive display of two non-calibration image chips. Calibration data are collected from the user at least while each calibration image chip is being displayed. The image triage system is dynamically calibrated using the calibration data.
• In yet another exemplary embodiment, a dynamically calibrated image triage system that is used to triage an image that may include one or more target entities includes a display, a data collector, and a processor. The display device is operable to receive display commands and, in response thereto, to display an image. The data collector is configured to at least selectively collect data from a user. The processor is coupled to receive the collected data from the data collector. The processor is further coupled to the display device and is configured to selectively retrieve an image, divide the image into a plurality of individual non-calibration image chips, successively command the display device to display each non-calibration image chip to a user for a presentation time period, selectively command the display device to display, for the presentation time period, a calibration image chip between the successive display of two non-calibration image chips, and dynamically calibrate the image triage system based at least in part on the data collected from the user at least while the calibration image chip is displayed.
• Furthermore, other desirable features and characteristics of the image triage system and method will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background.
BRIEF DESCRIPTION OF THE DRAWINGS
• The present invention will hereinafter be described in conjunction with the following drawing figures, wherein like numerals denote like elements, and wherein:
• FIG. 1 depicts a functional block diagram of an exemplary image triaging system;
• FIG. 2 depicts an exemplary process, in flowchart form, that may be implemented by the image triaging system ofFIG. 1;
• FIG. 3 depicts how an image may be divided into individual image chips, in accordance with a particular embodiment of the present invention;
• FIG. 4 depicts a particular rapid serial visualization presentation paradigm that may be implemented in accordance with an embodiment of the present invention; and
• FIG. 5 depicts an exemplary calibration image.
DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT
• The following detailed description of the invention is merely exemplary in nature and is not intended to limit the invention or the application and uses of the invention. Furthermore, there is no intention to be bound by any theory presented in the preceding background or the following detailed description.
• Turning first toFIG. 1, a functional block diagram of an exemplary system100 that may be used to triage images is depicted. The depicted system100 includes adisplay device102, adata collector104, and aprocessor106. AsFIG. 1 further depicts, in some embodiments the system100 may additionally include auser interface108, animage database110, and one or moreuser state monitors112. Thedisplay device102 is in operable communication with theprocessor106 and, in response to display commands received therefrom, displays one or more images to auser101. It will be appreciated that thedisplay device102 may be any one of numerous known displays suitable for rendering graphic, icon, and/or textual images in a format viewable by theuser101. Non-limiting examples of such displays include various cathode ray tube (CRT) displays, and various flat panel displays such as, for example, various types of LCD (liquid crystal display) and TFT (thin film transistor) displays. The display may additionally be based on a panel mounted display, a head up display (HUD) projection, or any known technology.
• Thedata collector104 in the depicted embodiment is a neurophysiological data collector that is configured to be disposed on, or otherwise coupled to, theuser101, and is operable to selectively collect neurophysiological data from theuser101. Preferably, and as depicted inFIG. 1, theneurological data collector104 is implemented as an electroencephalogram (EEG) system, and most preferably as amulti-channel EEG cap114, and appropriate EEG signal sampling andprocessing circuitry116. It will be appreciated that the number of EEG channels may vary. Moreover, the EEG signal sampling andprocessing circuitry116 may be implemented using any one of numerous known suitable circuits and devices including, for example, one or more analog-to-digital converters (ADC), one or more amplifiers, and one or more filters. No matter the particular number of EEG channels and the particular type of EEG signal sampling andprocessing circuitry116 that is used, it is in operable communication with, and is configured to supply the collected EEG data to, theprocessor106. As will be described in more detail further below, the EEG signal sampling andprocessing circuitry116 is further configured to receive trigger signals from theprocessor106, and to record the receipt of these trigger signals concurrently with the EEG signals.
• Theuser interface108 is in operable communication with theprocessor106 and is configured to receive input from theuser101 and, in response to the user input, supply various signals to theprocessor106. Theuser interface108 may be any one, or combination, of various known user interface devices including, but not limited to, a cursor control device (CCD), such as a mouse, a trackball, or joystick, and/or a keyboard, one or more buttons, switches, or knobs. In the depicted embodiment, theuser interface102 includes aCCD118 and akeyboard122. Theuser101 may use theCCD118 to, among other things, move a cursor symbol on thedisplay device102, and may use thekeyboard122 to, among other things, input various data. As will be described further below, theuser101 may additionally use either theCCD118 orkeyboard122 to selectively supply physical response data, the purpose of which are also described further below.
• Theimage database110 preferably has various types of imagery collections stored therein. The imagery collection types may vary, and may include, for example, various types of static imagery and various types of video imagery. It will additionally be appreciated that, although theimage database110 is, for clarity and convenience, shown as being stored separate from theprocessor106, all or portions of thisdatabase110 could be loaded into the on-board RAM105, or integrally formed as part of theprocessor106, and/orRAM105, and/orROM107. Theimage database110, or the image data forming portions thereof, could also be part of one or more non-illustrated devices or systems that are physically separate from the depicted system100.
• The one or more user state monitors112, if included, are operable to selectively collect various data associated with theuser101. The one or more user state monitors112 may include at least aneye tracker124, ahead tracker126, and one or more EOG (electrooculogram)sensors128. Theeye tracker124, if included, is configured to detect the movement of one or both of the user's pupils. Thehead tracker126, if included, is configured to detect the movement and/or orientation of the user's head. TheEOG sensors128, if included, are used to detect eye blinks and various eye movements of theuser101. Although any one of numerous devices may be used to implement theeye tracker124 andhead tracker126, in the depicted embodiment one or more appropriately mounted and located video devices, in conjunction with appropriate processing software components are used to implement these functions. Though not explicitly depicted inFIG. 1, appropriate signal sampling and processing circuitry, if needed or desired, may be coupled between theeye tracker124 and/or thehead tracker126 and theprocessor106. Moreover, the same or similar signal sampling andprocessing circuitry116 that is used with theEEG cap114 may additionally be used to supply appropriate EOG signals to theprocessor106. It will be appreciated that, at least in some embodiments, the system100 may be implemented without one or all of the user state monitors112. No matter which, if any, of the user state monitors112 that are included in the system100, each supplies appropriate user state data to theprocessor106.
• Theprocessor106 may include one or more microprocessors, each of which may be any one of numerous known general-purpose microprocessors or application specific processors that operate in response to program instructions. In the depicted embodiment, theprocessor106 includes on-board RAM (random access memory)105, and on-board ROM (read only memory)107. The program instructions that control theprocessor106 may be stored in either or both theRAM105 and theROM107. For example, the operating system software may be stored in theROM107, whereas various operating mode software routines and various operational parameters may be stored in theRAM105. It will be appreciated that this is merely exemplary of one scheme for storing operating system software and software routines, and that various other storage schemes may be implemented. It will also be appreciated that theprocessor106 may be implemented using various other circuits, not just one or more programmable processors. For example, digital logic circuits and analog signal processing circuits could also be used.
• No matter its specific configuration and implementation, theprocessor106 is in operable communication with thedisplay device102, theneurophysiological data collector104, theuser interface108, and theimage database110 via, for example, one or more communication buses orcables136. Theprocessor106 is configured to selectively retrieve one or more images from theimage database110 for selective display on the display device. In particular, and as will be described in more detail further below, theprocessor106 may divide a retrieved image up into several smaller discrete sub-images, referred to herein as image chips, for display on thedisplay device102. Theprocessor106 is additionally coupled to receive neurophysiological data from theneurophysiological data collector104, and may additionally receive physical response data from theuser interface108. Theprocessor106, based at least in part on one or more of these data, assigns probabilities to the image chips of a retrieved image. The assigned probabilities are representative of the likelihood that the image chips include a target entity.
• Although theprocessor106 may implement various techniques to assign the probabilities to displayed image chips, preferably theprocessor106 implements one or more trainable classifiers associated with theuser101. A trainable classifier, as is generally known, may be trained (or calibrated) during a calibration phase. As will be described in more detail further below, the trainable classifiers implemented by the depictedprocessor106 may also be dynamically calibrated during a subsequent performance phase. In the depicted embodiment, theprocessor106 implements two trainable classifiers—aneurophysiological classifier142 and aphysical response classifier144. It will be appreciated that any one of numerous known trainable classifiers may be used. However, in a particular preferred embodiment, a support vector machine (SVM) is used to implement eachclassifier142,144. It will be appreciated that any one of numerous types of SVMs may be used to implement the system100, but in a particular preferred embodiment non-linear SVMs with a radial basis function kernel are used.
• No matter the particular specie or sub-specie of classifiers that are used, eachclassifier142,144, during the performance phase, may determine the probability that an image chip includes a target entity. The outputs from the twoclassifiers142,144, at least in the depicted embodiment, may then be combined using a weighted combination of eachclassifier142,144 to generate combined values. For example, in a particular preferred embodiment the outputs of theneurophysiological classifier142 are weighted twice as high as the outputs of thephysical response classifier144. It will be appreciated that the specific and relative weighting of the classifier outputs may vary, and that weighting the outputs of theneurophysiological classifier142 twice as high as the outputs of thephysical response classifier144 is merely exemplary. Nonetheless, in the depicted embodiment, the combined values are scaled to provide values between 0.0 and 1.0 for each image chip, which are representative of the probability that each image chip includes a target entity.
• It was additionally noted above that theprocessor106, at least in some embodiments, may also receive user state data from the one or more user state monitors112. In such embodiments, theprocessor106 appropriately processes the user state data and the neurophysiological data to determine whether one or more of these data, either alone or in combination, indicate theuser101 is in a state that could adversely compromise the effectiveness of the image triage processing. It is noted that, based on this determination, theprocessor106 may generate one or more user alerts and/or vary the pace of one or more portions of the below-described image triage processing.
• As alluded to previously, during a calibration phase various calibration images are displayed to auser101. The calibration images may be stored in theimage database110, in theRAM105 orROM107, or in another non-depicted storage device. As the calibration images are being displayed, neurophysiological data, physical response data, or both, are collected from theuser101 and supplied to theprocessor106 to calibrate the image triage system100, and more specifically theclassifiers142,144. Thereafter, during a performance phase, non-calibration image chips are successively displayed to theuser101, and neurophysiological data, physical response data, or both, are collected from theuser101 and supplied to theprocessor106. Theprocessor106, based at least in part on one or more of these data, assigns probabilities to each non-calibration image chip. These assigned probabilities are representative of the likelihood that the non-calibration image chips include a target entity. As was also noted above, theprocessor106 may dynamically calibrate the image triage system100 during the performance phase. The overall process200 by which theprocessor106 implements these functions is depicted in flowchart form inFIG. 2, and with reference thereto will now be described in more detail. Before doing so, however, it is noted that the depicted process200 is merely exemplary of any one of numerous ways of depicting and implementing the overall process to be described. Moreover, before the process200 is initiated, it is noted that, if neurophysioligical data are collected, at least theneurophysiological data collector104 has preferably been properly applied to theuser101, and appropriately configured to collect neurophysiological data. If included, the one or more user monitors112 have also preferably been applied to theuser101, and appropriately configured to collect user state data. With this background in mind, it is additionally noted that the numerical parenthetical references in the following description refer to like steps in the flowchart depicted inFIG. 2.
• Turning now to the description of the process200, it is seen that the system100 is first calibrated for theuser101 by initiating a calibration phase (202). As noted above, during the calibration phase (202) various calibration images are displayed to theuser101 and, as the calibration images are displayed, neurophysiological data, physical response data, or both, are collected from theuser101 and supplied to theprocessor106. These data are then used to calibrate the image triage system100, and more specifically theclassifiers142,144. After the calibration phase (202) is complete, a performance phase may then be initiated (204). It will be appreciated that that other physiological data types, in addition to or instead of neurophysiological data and/or physical response data, could also be used.
• During the performance phase (204), an image is retrieved from theimage database110, and is divided into a plurality of non-calibration image chips (206). More specifically, and with reference toFIG. 3, the retrievedimage300, which in the depicted embodiment is a simplified representation of a broad area image of a port region, is divided into N-number of discrete non-calibration image chips302 (e.g.,302-1,302-2,302-3, . . .302-N). It will be appreciated that the number of non-calibration image chips302 that a retrievedimage300 may be divided into may vary, and may depend, for example, on the size and/or resolution of the retrievedimage300. In the embodiment depicted inFIG. 3, the retrievedimage300 is divided into783 non-calibration image chips (i.e., N=783).
• Returning once again toFIG. 2, after theimage300 has been divided into the plurality of non-calibration image chips302, the non-calibration image chips302 are individually and successively displayed, on thedisplay device102, to the user101 (208). In particular, the non-calibration image chips302 are preferably presented using a rapid serial visualization presentation (RSVP) technique. Thus, eachnon-calibration image chip302 is individually displayed, preferably at the same location on thedisplay device102, for a presentation time period, preferably in a predetermined sequence, and preferably at substantially equivalent luminance levels. It will be appreciated that the presentation time period may vary, and may be selected by theuser101.
• AsFIG. 2 further depicts, as the non-calibration image chips302 are being successively displayed during the performance phase, one or more calibration image chips are also selectively displayed to the user101 (210). More specifically, and asFIG. 4 depicts more clearly, acalibration image chip402 is selectively inserted into theRSVP stream404 of the non-calibration image chips302, and thus selectively displayed between the successive display of two non-calibration image chips302. Preferably, whenever acalibration image chip402 is displayed, it is displayed for the same presentation time period as each of the non-calibration image chips302. AlthoughFIG. 4 depicts a plurality ofcalibration image chips402 being displayed during the performance phase, it will be appreciated that in some instances only a singlecalibration image chip402 may be displayed during the performance phase. If more than onecalibration image402 is displayed during the performance phase, then thecalibration image chip402 may be randomly displayed between the display of two successive non-calibration image chips302, or at a set periodicity. It will additionally be appreciated that the samecalibration image chip402 may be displayed each time, or a plurality ofcalibration image chips402 may be displayed either randomly or in a predetermined order.
• Before proceeding further, and with quick reference toFIG. 5, it is noted that acalibration image chip402 is an image that includes a knowntarget entity404, and is thus referred to herein as a synthetic target entity. Thecalibration image chips402 that are displayed during the performance phase may be one or more of the same images that are used during the calibration phase (202), or thecalibration image chips402 may be wholly different.
• Returning once again toFIG. 2, it is seen that while the non-calibration andcalibration image chips302,402 are being displayed to theuser101, user data such as, neurophysiological data, physical response data, or both, are collected from the user101 (212). As was noted above, in some embodiments, user state data may additionally be collected via theuser interface108 and the one or more state monitors112, respectively. As was also previously noted, if neurophysiological data are collected, these data are preferably EEG data collected via themulti-channel EEG cap114. It will be appreciated that, if collected, either theCCD118 or thekeyboard122 may be used to collect the physical response data. In particular, theuser101 will hit either a predetermined button on theCCD118 or a predetermined key on thekeyboard122 each time theuser101 believes a displayedimage chip302,402 includes a target entity, or at least a portion of a target entity. In the depicted embodiment, theimage300 includes five target entities that, for simplicity of illustration, are labeled T₁through T₅onFIG. 3. It will be appreciated that in an actual physical implementation, theimage300 may include any number of target entities, which may be, for example, various types of land vehicles, seagoing vessels, special use land masses, weapons sites, or military bases, just to name a few examples.
• As previously noted, while the data are being collected, theprocessor106 supplies image triggers, or brief pulses, to theneurophysiological data collector104. The image triggers are supplied each time a non-calibration orcalibration image chip302,402 is displayed. During subsequent processing, which is described further below, a segment of neurophysiological data and a segment physical response data are extracted around each image trigger. These segments, referred to as epochs, contain neuophysiological data and physical response data from a predetermined time before an image trigger to a predetermined time after the image trigger. It will be appreciated that the predetermined time period before and after each image trigger, and concomitantly the total length of each epoch of data, may vary.
• While the non-calibration andcalibration image chips302,402 are being displayed theprocessor106 also determines whether the collected data are non-calibration data or calibration data (214). As used herein, non-calibration data are data associated with anon-calibration image chip302, and calibration data are data associated with acalibration image chip402. Each time calibration data are collected, these data are used to dynamically calibrate the system100 (215). In the depicted embodiment, dynamic calibration means updating the calibration of theclassifiers142,144. The dynamically updatedclassifiers142,144 process the non-calibration data, as described above, to assign a probability to each non-calibration image chip302 (216). The probability that is assigned to eachnon-calibration image chip302 is representative of the likelihood that thenon-calibration image chip302 includes a target entity.
• While at least one exemplary embodiment has been presented in the foregoing detailed description of the invention, it should be appreciated that a vast number of variations exist. It should also be appreciated that the exemplary embodiment or exemplary embodiments are only examples, and are not intended to limit the scope, applicability, or configuration of the invention in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing an exemplary embodiment of the invention. It being understood that various changes may be made in the function and arrangement of elements described in an exemplary embodiment without departing from the scope of the invention as set forth in the appended claims.

Claims (17)

1. A method of dynamically calibrating an image triage system an image, comprising the steps of:

dividing an image that may include one or more target entities into a plurality of individual non-calibration image chips;

successively displaying each non-calibration image chip to a user for a presentation time period;

selectively displaying, for the calibration time period, a calibration image chip between the successive display of two non-calibration image chips, the calibration image chip including a synthetic target entity;

collecting calibration data from the user at least while each calibration image chip is being displayed; and

dynamically calibrating the image triage system using the calibration data.

2. The method ofclaim 1, further comprising:

collecting non-calibration data from the user at least while each non-calibration image chip is being displayed; and

for each non-calibration image chip, assigning a probability that the non-calibration image chip at least includes a target entity, based at least in part on the collected non-calibration data.

3. The method ofclaim 2, further comprising:

collecting the non-calibration data and the calibration data from the user from a predetermined time period before a non-calibration image chip and a calibration image chip, respectively, are displayed to a predetermined time period after non-calibration image chip and a calibration image chip, respectively, are displayed.

4. The method ofclaim 2, wherein the collected calibration data and the collected non-calibration data are neurophysiological data.

5. The method ofclaim 2, wherein the collected calibration data and the collected non-calibration data are physical response data.

6. The method ofclaim 2, wherein the collected calibration data and the collected non-calibration data are neurophysiological data and physical response data.

7. The method ofclaim 1, further comprising:

displaying a plurality of calibration image chips to the user during a calibration phase, the calibration phase occurring prior to the step of successively displaying each individual image chip.

8. The method ofclaim 7, further comprising:

collecting calibration phase data from the user during the calibration phase; and

calibrating the image triage system using the calibration phase data.

9. The method ofclaim 1, wherein the non-calibration image chips and the calibration image chips are displayed to the user in accordance with a rapid serial visualization (RSVP) paradigm.

10. The method ofclaim 1, further comprising:

monitoring one or more states of the user; and

supplying one or more alerts to the user based on the one or more states of the user,

wherein the monitored states of the user include one or more of user attention lapses, eye activity, and head movements.

11. A system for conducting image triage of an image that may include one or more target entities, comprising:

a display device operable to receive display commands and, in response thereto, to display an image;

a data collector configured to at least selectively collect data from a user;

a processor coupled to receive the collected data from the data collector, the processor further coupled to the display device and configured to:

selectively retrieve an image,

divide the image into a plurality of individual non-calibration image chips,

successively command the display device to display each non-calibration image chip to a user for a presentation time period,

selectively command the display device to display, for the presentation time period, a calibration image chip between the successive display of two non-calibration image chips, the calibration image chip including a synthetic target entity, and

dynamically calibrate the image triage system based at least in part on the data collected from the user at least while the calibration image chip is displayed.

12. The system ofclaim 11, wherein the processor is further configured to assign a probability to each displayed non-calibration image chip based at least in part on the data collected at least when each non-calibration image chip is displayed, each assigned probability representative of a likelihood that the non-calibration image chip at least includes a target entity.

13. The system ofclaim 12, wherein the collected data are neurophysiological data.

14. The system ofclaim 12, wherein the collected data are physical response data.

15. The system ofclaim 12, wherein the collected are neurophysiological data and physical response data.

16. The system ofclaim 11, wherein the processor is further configured to command the display to display the non-calibration image chips and the calibration image chips in accordance with a rapid serial visualization (RSVP) paradigm.

17. The system ofclaim 11, further comprising:

one or more user state monitors configured to monitoring sensor one or more states of the user and supply user state data representative thereof,

wherein:

the processor is further configured to receive the user state data, and to determine the user is in a state that could adversely compromise probability assignment effectiveness, and selectively generate one or more alerts based on the one or more states of the user, and

the states of the user include one or more of user attention lapses, eye activity, and head movements.

Images