Entity Type	Tab Title	Description

Person	Affiliations	Describe affiliations of the person
		to companies, organizations,
		governments, agencies, etc
	Statements	Report statements made by the
		person on any topic
	Actions	Describe the actions of the person
	Related People	List acquaintances of the person
	Locations	List places & locations visited
		by the person
	Elections	Describe election campaign
		of the person
	Involvment in Events	Describe events in which the
		person is involved
ORG & GPE	Actions	Describe actions of the
		organization or of official
		representatives
	Related Orgs	Describe related organizations,
		such as subsidiaries.
	Associated People	List people associated with the
		ORG/GPE
	Statements	Report statements released by the
		organization or made by
		representatives

These snippets are selected by a collection of Information Gathering Modules (IGMs) specified in a configuration file. A typical IGM is based upon a machine learning model, further described below. Each IGM also associates a relevance score with each snippet.

To assemble the tab content, the snippets selected and scored by the IGMs are analyzed by appropriate Information Display Modules (IDMs), specified in a configuration file. IDMs group snippets with identical information for a tab into the same equivalence class. IDMs associate a score to each equivalence class, and sort the classes according to the score.

To visualize each equivalence class, IDMs produce a title, which is a short representation of the information it conveys, and select a representative snippet. They highlight the portions of the representative snippet that contain the information of interest to the tab, and create links to pages of other entities mentioned in the snippets. Additional sentences in the equivalence class are shown by clicking a link marked “Additional Supporting Results . . . ”. Since news agencies often reuse the same sentences over time, such sentences are available by clicking “Other Identical Results”.

IDMs create the data used to produce a visual summary of the content in the selected tab, shown in the rightmost frame of the top half of the GUI. For the Related People tab depicted inFIG. 6, this visualization is a network of relationships. For other tabs, it is a cloud of the content words in the tab.

The interface is not only useful for an analyst tracking an entity in the news, but also for financial analysts following news about a company, or web users getting daily updates of the news. The redundancy detection and systematic organization of information makes the content easy to digest.

In a news browsing application, entities can be highlighted in articles, as depicted inFIG. 7(a), and those entities for which an EP exists (i.e., there are relevant snippets for at least one tab) are hyperlinked to the EP. Users can also arrive at the EP by viewing a searchable list of entities in alphabetic order, or by frequency in the news as depicted inFIG. 7(b).

FIG. 8 shows an overview of an exemplary embodiment ofprogram storage device600 wherein instruction code contained therein for an IE, IGM and IDM are depicted.Processor700 executes the instruction code stored inprogram storage device600.

A crawler as previously described above can periodically download new content from a set of English text and Arabic text and video sites indocuments610. Audio from video sources can be segmented into chunks of 2-minute intervals and then transcribed. Arabic can be translated into English using a state-of-the-art machine translation system. Table 2 lists the average number of documents from each modality-language pair on a daily basis.

TABLE 2

Number of articles downloaded by the
crawler daily in different modalities.

	Source	# docs

	En-Text	1317
	Ar-Text	813
	Ar-Video	843

Subsequent components in the pipeline work on English text documents, and the framework can be easily extended to any language for which translation and transcription systems exist.

Each newtextual document610 may be analyzed by theIE pipeline620. The first step after tokenization is parsing, followed by mention detection. Within each document, mentions are clustered by a within-document co-reference-resolution algorithm. Thus, in the appropriate context, “Washington” and “White House” are grouped under the same entity (the USA), and “Leon Edward Panetta” and “Leon Panetta” under the same person (Secretary of Defense). Nominal and pronominal mentions are also added to the clusters. A cross-document co-reference system then links the entity clusters across documents. This is done by linking each cluster to the knowledge base (KB) used in the Text Analysis Conference (TAC) Entity Linking task, which is derived from a subset of the Wikipedia Internet encyclopedia. If a match in the KB is found, the cluster is assigned the KB ID of the match, which allows for the cross-referencing of entities across documents. Besides exact match with titles in the KB, the cross-document co-reference system uses soft match features and context information to match against spelling variations and alternate names. The system also disambiguates between entities with identical names. The next IE component extracts relations between the entities in the document, such as employed by, son of, etc. The mention detection, co-reference and relation extraction modules are trained on an internally annotated set of 1301 documents labeled according to the Knowledge from Language Understanding and Extraction (KLUE) 2 ontology. On a development set of 33 documents, these components achieve an F1 of 71.6%, 83.7% and 65% respectively. The entity linking component is unsupervised and achieves an accuracy of 73% on the TAC-2009 person queries.

Annotated documents are then analyzed by theIGMs630 andIDMs640 described above. In its basic form, an IGM takes as input a sentence and an entity, and extracts specific information about that entity from the sentence. For example, a specific IGM may detect whether a family relation of a given person is mentioned in the input sentence. A partial list of IGMs and the description of the extracted content is shown in Table 1. The output of the IGMs is then analyzed by IDMs, which assemble the content of the GUI tabs. These tabs either correspond to a question template from a pilot program or are derived from the above-mentioned relations. For each entity, IDMs selectively choose annotations produced by IGMs, group them into equivalence classes, rank the equivalence classes to prioritize the information displayed to the user, and assemble the content of the tab. The IGMs and IDMs are described in still further detail below.

IGMs extract specific information pertaining to a given entity from a specific sentence in two stages: First, they detect whether the snippet contains relevant information. Then they identify information nuggets.

Snippet relevance detection relies on statistical classifiers, trained on three corpora produced as part of the pilot program: i) data provided by Linguistic Data Consortium (LDC) to the pilot program teams during the early years of the program; ii) data provided by BAE Systems; and iii) internally annotated data. The data consist of queries and snippets with binary relevance annotation. The LDC and internally annotated data were specifically developed for training and testing purpose, while the BAE data also include queries from yearly evaluations, the answers provided by the teams that participated to the evaluations, and the official judgments of the answers. The statistical models are maximum entropy classifiers or averaged perceptrons chosen based on empirical performance. They use a broad array of features including lexical, structural, syntactic, dependency, and semantic features. Table 3 summarizes the performance of the models used on the year-4 unsequestered queries, run against an internally generated development set. The “TN” column denotes a template number.

TABLE 3

Performance of the IGM models

	Template	TN	P	R	F

Templates for Person Entities

Information	T3	75.60	90.07	82.20
Actions	T13	50.00	18.33	26.83
Whereabouts	T17	86.11	43.66	57.94
Election Campaign	T21	78.72	26.81	40.00

Templates for ORG/GPE Entities

Information	T4	71.50	90.79	80.00
Actions	T14	45.83	29.73	36.07
Arrests of Members	T15	75.51	74.00	74.75
Location of Representative	T18	36.36	44.94	40.20

IGMs analyze snippets selected by the template models and extract the information used by the IDMs to assemble and visualize the results. This step is called “Information Nugget Extraction”, where an information nugget is an atomic answer to a specific question. Extracted nuggets include the focus of the answer (e.g., the location visited by a person), the supporting text (a subset of the snippet), a summary of the answer (taken from the snippet or automatically generated). Different modules extract specific types of nuggets. These modules can be simple rule-based systems or full statistical models. Each tab uses a different set of nugget extractors, which can be easily assembled and configured to produce customized versions of the system.

IDMs use the information produced by IGMs to visualize the results. This involves grouping results into non-redundant sets, sorting the sets, producing a brief description of each set, selecting a representative snippet for each set, highlighting the portions of the snippet that contain information pertaining to the specific tab, constructing navigation hyperlinks to other pages, and generating the data used to graphically represent the tab content.

IGMs produce results in a generic format that supports a well-defined Application Program Interface (API). IDMs query this API to retrieve selected IGM products. For each tab, a configuration file specifies which IGM products to use for redundancy detection. For example, the content of the Affiliations tab for persons (see Table 1) is constructed from automatic content extraction (ACE)-style relations. The configuration file instructs the IDM to use the relation type and the KB-ID of the affiliated entity for redundancy reduction. Thus, if a snippet states that Sam Palmisano was manager of “IBM”, and another that Sam Palmisano was manager of “International Business Machines”, and “IBM” and “International Business Machines” have the same KB-ID, then the snippets are marked as redundant for the purpose of the Affiliations tab.

Redundancy detection groups results into equivalence classes. Each class contains unique values of the IGM products specified in the configuration file. IDMs can further group classes into superclasses or split the equivalence classes according to the values of IGM products. For example, they can partition the equivalence classes according to the date of the document containing the information. The resulting groups of documents constitute the unit of display. IDMs assign a score to each of these groups, for example, using a function of the score of the individual snippets and of the number of results in the group or in the equivalence class. The groups are sorted by score, and the highest scoring snippet is selected as representative for the group. Each group is then visualized as a section in the tab, with a title that is constructed using selected IGM products. The score of the group is also optionally shown. The text of representative snippet containing the evidence for the relevant information is highlighted in yellow. The named mentions are linked to the corresponding page, if available, and links to different views of the document are provided.

Each tab is associated with a graphical representation that summarizes its content, and that is shown in the rightmost section of the top half of the GUI ofFIG. 6. This visualization is generated dynamically by invoking an application on a server when the tab is visualized.

Exemplary embodiments of the system can support three different visualizations: a word cloud, and two styles of graphs that show connections between entities. A configuration file instructs the IDMs on which IGM products contain the information to be shown in the graphical representation. This information is then formatted to comply with the API of the program that dynamically constructs the visualization.

The exemplary embodiments described above can utilize natural language processing methods well known in the art. A fundamental reference is the book “Foundations of Statistical Natural Language Processing” by Manning and Schutze, which covers the main techniques that form such methods. Constructing language models based on co-occurrences (n-gram models) is taught inChapter 6. Identifying the sense of words using their context, called word-sense disambiguation is taught inChapter 7. Recognizing the grammatical type of words in a sentence, called part-of-speech tagging, is taught inChapter 9. Recognizing the grammatical structure of a sentence, called parsing, is taught in Chapter 11. Automatically translating from a source language to a destination language is taught in Chapter 13. The main topics of Information Retrieval are taught in Chapter 15. Automatic methods for text categorization are taught in Chapter 16.

Given the significant proportion of new material on the Internet that is news that centers around people, organizations and geopolitical entities (GPEs), named entities form a key aspect of news documents and one is often interested in tracking stories about a person (e.g., Leon Panetta), an organization (e.g., Apple Inc.) or a GPE (e.g., the United States). Exemplary embodiments described above provide a system that automatically constructs summary pages for named entities from news data. The EP page describing an entity is organized into sections that answer specific questions about that entity, such as Biographical Information, Statements made, Acquaintances, Actions, and the like. Each section contains snippets of text that support the facts automatically extracted from the corpus. Redundancy detection yields a concise summary with only novel and useful snippets being presented in the default display. The system can be implemented using a variety of sources, and shows information extracted not only from English newswire text, but also from machine-translated text and automatically transcribed audio.

While publicly available news aggregators like Google News show the top entities in the news, clicking on these typically results in a keyword search (with, perhaps, some redundancy detection). On the other hand, the exemplary embodiments described above provide a system that organizes and summarizes the content in a systematic way that is useful to the user. The system is not limited to a bag-of-words search, but uses deeper NLP technology to detect mentions of named entities, to resolve co-reference (both within a document and across documents), and to mine relationships such as employed by, spouse of, subsidiary of, etc., from the text. The framework is highly scaleable and can generate a summary for every entity that appears in the news in real-time. The flexible architecture of the system allows it to be quickly adapted to domains other than news, such as collections of scientific papers where the entities of interest are authors, institution, and countries.

The methodologies of the exemplary embodiments of the present disclosure may be particularly well-suited for use in an electronic device or alternative system. Accordingly, exemplary embodiments may take the form of an embodiment combining software and hardware aspects that may all generally be referred to as a “processor”, “circuit,” “module” or “system.” Furthermore, exemplary implementations may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code stored thereon.

Any combination of one or more computer usable or computer readable medium(s) may be utilized. The computer-usable or computer-readable medium may be a computer readable storage medium. A computer readable storage medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fibre, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus or device.

Computer program code for carrying out operations of the exemplary embodiments may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk. C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Exemplary embodiments are described herein with reference to flowchart illustrations and/or block diagrams. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions.

The computer program instructions may be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

It is to be appreciated that the term “processor” as used herein is intended to include any processing device, such as, for example, one that includes a central processing unit (CPU) and/or other processing circuitry (e.g., digital signal processor (DSP), microprocessor, etc.). Additionally, it is to be understood that the term “processor” may refer to more than one processing device, and that various elements associated with a processing device may be shared by other processing devices. The term “memory” as used herein is intended to include memory and other computer-readable media associated with a processor or CPU, such as, for example, random access memory (RAM), read only memory (ROM), fixed storage media (e.g., a hard drive), removable storage media (e.g., a diskette), flash memory, etc. Furthermore, the term “I/O circuitry” as used herein is intended to include, for example, one or more input devices (e.g., keyboard, mouse, etc.) for entering data to the processor, and/or one or more output devices (e.g., printer, monitor, etc.) for presenting the results associated with the processor.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Although illustrative embodiments of the present disclosure have been described herein with reference to the accompanying drawings, it is to be understood that the present disclosure is not limited to those precise embodiments, and that various other changes and modifications may be made therein by one skilled in the art without departing from the scope of the appended claims.