US20230047675A1 - Relational presentation of communications and application for transaction analysis - Google Patents

Abstract

The system provides a method of sorting and presenting messages in a way so that the relationship in message threads can be easily observed and related messages can be identified. The system provides a way to view messages and map message threads and inboxes in two and three dimensions so that the content of messages can be easily reviewed and the relationship between messages can be seen and followed. The system is not limited to email messages but can present the relationship between multiple types of communications including emails, instant messages, texts, tweets, bulletin boards, wilds, blogs, voice conversations postings on social networks and other types of communications. In addition, the system allows for the inclusion of transactional information, including financial transactions, physical movement, asset deployment, or other acts or activities that may be related to, or independent of, the communications.

Description

APPLICATION FOR TRANSACTION ANALYSIS
• This patent application claims priority to U.S. Provisional Patent Application No. 61/225,539 filed on Jul. 14, 2009, and to U.S. Provisional Patent Application No. 61/262,456 filed on Nov. 13, 2009, both of which are incorporated by reference herein in their entireties.
BACKGROUND OF THE SYSTEMField of the Invention
• The invention relates generally to a system of sorting and presenting communications and for analyzing and tracking transactions.
Background of the Invention
• Email has become an indispensable communication tool. There are a number of email applications used for sending and receiving email. Examples include email clients such as Microsoft Exchange, IBM Lotus Notes, Microsoft Outlook, Eudora, Mozilla Thunderbird, and GnuMail, and browser based email systems such as YahooMail and GoogleMail.
• As email has supplemented and replaced prior communication techniques, such as posted mail, messenger delivered communication, and other physical based written communication techniques, its importance as evidence in transactions has grown. Particularly in litigation, due in part to the impact of Sarbanes-Oxley and other corporate governance requirements, the preservation and production of email is required in every jurisdiction.
• The Federal Rules of Civil Procedure have been expanded to cover electronically stored information (ESI) including emails. This type of information must be preserved and produced in a controversy as well as in the normal course of corporate governance. A difficulty in producing email is the need to produce all the email required and only the email that is required of a party to produce. Failure to produce all requested email can result in potential loss of rights and/or penalties from governing bodies such as the SEC. Producing more email than is required can breach confidentiality and put a litigant or company in the position of revealing data that would otherwise have remained confidential.
• In the past, the culling, sorting, inspection, and producing of all and only the correct emails has been time consuming and expensive. Prior art email systems are not designed with litigation production in mind.
• Another disadvantage of prior art systems is the difficulty in analyzing non-communication transactions that may be related to communications such as email, instant messaging, texting, blogs, wilds, bulletin boards, twittering, phone calls, postings on social networks such as Facebook, My Space, etc. or other types of communication.
SUMMARY OF THE SYSTEM
• The system provides a method of sorting and presenting communications such as email messages, bulletin board messages, SMS messages, Twitter messages, social network communications and other communications in a way so that the relationship in communication threads can be easily observed and related messages can be identified. The system provides a way to view messages and map message threads in two and three dimensions so that the content of messages can be easily reviewed and the relationship between messages and their senders/recipients can be seen and followed. The system allows a user to enter into a message thread at any point and to then produce a visualization of the related threads, messages, and attachments associated with each individual message. A unique tool allows the navigation of message databases with ease. The system is not limited to email messages but can present the relationship between multiple types of communications including emails, instant messages, texts, tweets, wikis, bulletin boards, blogs, voice conversations such as phone calls, postings on social networks such as Facebook, My Space, etc. and other types of communications. In addition, the system allows for the inclusion of transactional information, including financial transactions, physical movement, asset deployment, or other acts or activities that may be related to, or independent of, the communications.
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG.1A is a two dimensional view of the presentation of an email thread in one embodiment of the system.
• FIG.1B is a color version ofFIG.1A.
• FIG.2A is a three dimensional view containing several email threads, one of which is similar to the presentation ofFIG.1.
• FIG.2B is a color version ofFIG.2A
• FIG.3 is a diagram illustrating the thread analysis of the system.
• FIG.4 is a flow diagram illustrating the operation of an embodiment of the system.
• FIG.5 is a flow diagram illustrating the use of the system for analysis.
• FIG.6 is a block diagram of an embodiment of the system.
• FIG.7 is a block diagram illustrating a system for collecting data in an embodiment of the system.
• FIG.8 shows the clipboard of the system.
• FIG.9 illustrates a system for visualization in an embodiment of the system.
• FIG.10 is a 3D view with a message opened and a note attachment entry field opened.
• FIG.11 illustrates embodiments of the trident navigation tool in an embodiment of the system.
• FIG.12 illustrates the elements of email entries.
• FIG.13A is an example of medical tracking in an embodiment of the system.
• FIG.13B is a color version ofFIG.13B.
• FIG.14 is a flow diagram illustrating operation of analysis in an embodiment of the system.
• FIG.15 illustrates the display of analysis data in an embodiment of the system.
• FIG.16 illustrates a second state of analysis data in an embodiment of the system.
• FIG.17 is a detailed view of the transactions layer ofFIG.16.
• FIG.18 illustrates document tracking in an embodiment of the system.
• FIG.19 is an example computer system in one embodiment.
DETAILED DESCRIPTION OF THE SYSTEM
• The present system provides a method of maintaining and displaying relationships between communications, and of viewing these communications. In one embodiment, the system is described in connection with email communications. However, the system has equal application to any type of communications such as instant messages, text messages, tweets, bulletin boards, blogs, wilds, voice communications, social networks, financial transactions, travel and geographic location, and the like.
• Referring to the example of emails, as emails are created, sent, and/or received, whether as sender, addressee, cc, or bcc, the system maintains information and linkages through the various email threads. When desired, a user may display a particular email message thread in a way that can graphically represent the history of the thread and the relationship of the timing and participants in the thread. The system may use color coding to represent various actions and relationships found in the thread. In one embodiment, the system can present a thread in a three dimensional presentation to more easily allow a user to see relationships. Although the example described below is directed to email, the system has equal application to bulletin boards, blogs, social networks, SMS message strings, document revision strings, software code revision strings, web content analysis strings (such as Wiki's), and other environments where there may be one or more threads of related content or other interrelationship of content. The system may be used with other forms of interrelationships. Any form of communication or transaction can be represented and analyzed by the system. Examples of parties can be a sender, recipient, bank account and the like. Examples of objects can be: money, e-mail, bulletin board postings, social network postings, transportation, and the like
• FIG.1 is a two dimensional analysis of a thread using an embodiment of the system. In one embodiment, all message threads begin with a root message that is the original message in the thread. In the embodiment ofFIG.1, the root email message is disposed in the lower left of the two dimensional presentation. (Note that the root email message can just as easily be disposed in some other location, such as the upper left of the two dimensional representation, and corresponding changes made in the ordering of the remaining messages in the thread.) The system illustrates a column for each sender/receiver participating in that email thread. In the example shown, there are five columns illustrated, but the system is not limited to this number.
• Column C1 is the sender of the root message. Columns C2-C5 are columns for each recipient, cc, bcc, or forwarded recipient of the root message and other messages in the thread. The image shown inFIG.1 is at a particular moment in time. Further activity in the thread would lead to additional messages and/or columns being displayed.
• In one embodiment of the system, a root message is illustrated in a particular color (e.g. dark red) so that it can be quickly found visually. In the embodiment ofFIG.1, other messages within the same thread that are sent by the sender of the root message appear in the same particular color (e.g. also dark red). To indicate that a recipient has received a copy of the email from the root sender, a rectangular box colored in a lighter shade (e.g. light red) of the particular color chosen for the root sender is placed in the column representing each recipient, cc, bcc, or forwarded recipient (for example see columns C2, C3, or C4 orrectangles107,108, or109). A “To” recipient (the named party in the “To:” box of an email) is shown in another color, a cc or bcc is shown in another color, a reply is shown in yet another color, and a forward operation in yet another color. If desired, these colors can be customized by the user as desired or may be limited to a default selection.
• In one embodiment, the system defines three types of participants, root, active, and passive participants. A root participant is the originator of the root email and is shown in dark red. Any sub-roots (first email in a sub-thread) originated by the root participant is also shown as dark red. To indicate that a recipient has received a copy of the email from the root sender, a rectangular box colored in a lighter shade (e.g. light red) of the particular color chosen for the root sender is placed in the column representing each recipient, whether they are a To, cc, bcc, or forwarded recipient. An active participant is any recipient that replies to a message and is shown in dark blue. To indicate that a recipient has received a copy of an email sent from an active participant, a rectangular box colored in a lighter shade (e.g. light blue) of the particular color chosen for active participants is placed in the column representing each recipient, whether they are a To, cc, bcc, or forwarded recipient. A passive participant is any recipient that does not respond to any message or create any message and is shown in light orange/tan.
• In the example ofFIG.1, the participants in column C2, C3, and C4 are all active participants as they have each originated at least one message in the thread, The user in column C5 is a passive participant in the example shown. This embodiment of the system shows messages as rows in the two dimensional image ofFIG.1. In other words, theroot message101 is on the bottom left and the recipients of that message are shown by lighter shadered rectangles107,108,109 on the same row to the right of the root message. Here the message was directly addressed (the “To:” line) to the users in columns C2-C4 and the user in column C5 was just a cc'd party. In this embodiment, a cc'd party has acc label102 indicated on the message instance.
• The original message resulted in a reply from the user in column C2, as shown by darkblue message103 on the second row from the bottom. This message was directed to the users in columns C1, C3, and C4, while the user in column C5 was not cc'd.
• The third row up illustrates a number of useful features of the system. The top message in104 is a reply by the root participant to the active participant in column C2'sreply message103. Looking to the right of the reply message (top message) in104 we see that it was addressed to the user in column C2 and cc'd to the user in column C3. But in addition to representing the reply message itself, the message can be expanded to show all of the previous messages in the thread in that single location.Message103 is shown just below the reply message (top message) in104 and theoriginal root message101 is shown just below that. The system contemplates where there are a larger number in columns and rows at any message location and allows the user to gather up all of the messages preceding any message simply by clicking on the special navigation button available on each message. An example of thenavigation tool100 is shown in full on the top right of the display but it is also available as part of each message header in the table.
• As can be seen inFIG.1, thenavigation tool110 only displays the possible actions in this embodiment, as the tool in column Cl lacks a “left” arrow button since there are no messages to the left of that column.
• The navigation button provides new navigational options for the user. A single click on the double up arrow on the navigation button contained in an email within the thread advances the user to the next email above (e.g. the email in the thread immediately following). A double click on the double up arrow moves one email down from the current email (e.g. next older). Clicking on the left arrow or right arrow allows the user to slide left or right as desired. The equal sign may be single clicked to expand or compress a chain. In one embodiment the equal sign is toggled so that alternate clicks expand and contract. In other embodiments, a single click expands and a double click contracts.
• The result is that in the table ofFIG.1, messages may appear in multiple locations depending on the actions of the navigation tool and desire of the user. This allows the user at any location to click for a presentation of the thread of messages that led to the message under review, even as those messages might appear elsewhere in the table.
• The system provides navigation to select the level of detail shown for email messages in the thread analysis GUI. The system's default view can be set to any of the following three options: (i) message header only (thin rectangle bar showing just the sender and date/time); (ii) message header including Subject header line; (iii) message header, Subject line, and first two lines of text of the message. For example, suppose the system's default view is set to option (i), so that each email is represented by its message header only. A single-click anywhere on the message header reveals the Subject header line. A single-click anywhere of the message header or subject line reveals the first two lines of the text body of the message. A single click anywhere on the message expands it further to show the full text of the email body. In other words, a single-click on any email within the thread expands that email message to the next level of detail. A double-click anywhere on an expanded email message contracts to the preceding level of detail. CTRL+single-click anywhere on a fully or partially expanded email contracts it all the way to the system's default starting level of detail (in the preceding example, it would contract all the way back to showing only the message header).
• Continuing with the example ofFIG.1, we see another darkred message105 immediately abovemessage104. This message is a second reply tomessage103 but with a different addressee, namely the user in column C4. The coloring and visual presentation of the system allows a view at a glance to see that there were two replies originating from a single message. In addition, date indicators are shown so that the timing of the messages can be seen or separated by day. Often a litigation discovery request will be date delimited, so the system allows the user to easily determine which messages fall into the appropriate time period.
• FIG.2 illustrates a three dimensional presentation of a plurality of threads in an embodiment of the system. The thread based onroot message101 is shown in the foreground ofFIG.2 moving from front to back. Users in columns C1-C5 are shown as separate slices along theparticipants axis200. Also shown in this view are otherrelated threads201 and202. The user is free to click on any thread or message in the visualization and to move between a two dimensional type view ofFIG.1 or the three dimensional view ofFIG.2.
• In one embodiment of the system, the user can select one or more threads or messages, both contiguous or non-contiguous, and perform a number of tasks on those selected messages, such as saving to a file, printing, sharing messages to other analysts or other desired activity.
• FIG.2 includes ahorizontal slider201 on the X axis that rotates the 3D cube about a vertical axis and avertical slider202 that rotates the cube about a horizontal axis. The origin (i.e. the starting message of a thread) may be in the top right corner (or any other corner or in fact in any selected position).
• FIG.2 includes thetrident tool203. Asearch box204 allows the user to search for words or phrases in the data being analyzed. The embodiment ofFIG.2 also includes aregion205 with some filtering tools such as advanced search, root participants, date, and keywords that allow filtering and customization of the view presented to the user. Theclipboard206 is shown as a tabbed space inFIG.2.
• The system can provide multiple views of the 3D cube and other actions in addition to rotation can be performed on the cube. The layers may be selectively removed and/or reordered. The system can “drill down” by removing the top layer and moving further into the screen. The time, thread, and/or participant axes can be scrolled as desired. There may also be an indication of color or icon that indicates that more information is available but not presented on the screen.
• FIG.3 illustrates a method of tracking messages and relationships in an embodiment of the system. Theroot message101 is defined as message1-1 (representingparticipant1 on the x-axis (participants) andmessage1 on the y-axis (relative time)). A two dimensional grid is populated based on the number of participants and can grow as the participants and messages increase.
• Each message is tagged with information representing the actions taken with that message. For example,message101 is shown with tags2-1,3-1,4-1, and5-1ccindicating that the message was directly addressed to the users in columns C2-C4 and cc'd to the user in column C5. Thereply message103 ofFIG.1 is tagged as2-2 inFIG.3. Each reply message also tags the messages that led to that message so that they can be expanded on demand.Message103 is tagged as2-2 because it is sent fromparticipant2 and is the second message in the chain.Messages104 and105 show what happens when a user sends two different replies to a single message.
• FIG.4 is a flow diagram illustrating the operation of an embodiment of the system. At step4011 the user initiates the analysis system. Atdecision block402, the system determines if it is a new analysis. If so, the system proceeds to step403. If not, meaning that it is a pre-existing analysis, the system proceeds to step406.
• If it is a new analysis atdecision block402, the system proceeds to step403 where the user is presented with a parameter input screen to help the user define and refine a search or analysis. The user can search for existing and saved reports or may search for specific email threads.
• Atstep404 the user is presented with search results and chooses one of the threads for analysis. Atstep405 the user can determine the scope, parameters, tags, and/or notes to be included in a report.
• Afterstep405, or pre-existing analysis was chosen atdecision block402, the system proceeds to step406. Atstep406, the user is presented with the main interface mode of the thread analysis system. The user can select threads from options provided in an interface. Atstep407 the user can elect to rotate the display along the X, Y and/or Z axis and can select certain emails to copy to a clipboard for later retrieval. This is helpful for responding to discovery requests as the user can see a graphic representation of a thread analysis and simply select responsive emails within that view, without the need to switch to another mode for collection.
• Atstep408, the user can zoom into a view of any part of the cube for more detailed analysis. Even in the zoomed mode, all tools and interfaces are still available to the user. Atstep409, the user is presented with a two dimensional view, as inFIG.1. Within this view,, the user can view individual emails atstep410 or attach notes to each thread and/or email atstep411. Atstep412, the user can select a new thread and/or view by using an options interface. Atstep413, the user can generate a report of all the emails or from previously tagged or clipboarded emails.
• Information and Transactional Analysis
• One feature of the system is the ability to maintain and display the relationships between communications activity and secondary information and/or other transactions (i.e. financial transactions, stock trades, travel, and the like) so that complex behavior patterns, including illicit or illegal activities, or any activities of interest, can be discovered and tracked.
• There are many circumstances where it is important to know certain information such as:
• 1. Understand which people knew about issues when the issues occurred.
• 2. Understand when those people learned what they knew
• 3. Understand how people disseminate the information they obtained. (Such as confidential information, privileged information or information that may have been “shaped” for specific purposes by the user.)
• 4. Understand what people did with the information they had (such as illicit maneuvers, financial gain, and/or criminal acts).
• FIG.5 is a flow diagram illustrating the steps that an analyst would take when tracking this information. Atstep501 directives and tasking is defined. This could involve indentifying individuals or entities and determining if they are participating in certain transactions. For example, the investigation could see if individual A is trading pharmaceutical stock. At step502 background research is performed to establish possible persons and entities to be analyzed and examined as well as understand additional context. Atstep503 the source of useful data is identified, such as email accounts, telephone accounts, instant message accounts, twitter accounts, bank accounts, stock accounts, social networks, etc. Atstep504 the analysis tools of the system are utilized. This means that the data from the identified sources is provided to the system for display. The display can be sorted, filtered, rotated, and manipulated to reveal patterns and connections that might otherwise be hidden. Atstep505 one or more reports can be generated that represent threads of communications and transactions that reveal a useful pattern. At step506 the data, reports, and analysis can be shared with others.
• The system allows patterns of communication to be revealed and associated with other behavior. There may be chains of behavior linked to email, text, instant messages and/or phone messages. The presence of a communication and a subsequent activity allows an inference that an actor in a transaction had knowledge or awareness of information prior to acting on the knowledge.
• The system not only allows presentation of the communications and transactions for analysis, but allows these to be shared with other analysts. The system in one embodiment uses preparation, message processing, and visualization to accomplish the analysis.
• The Preparation phase establishes “connectors” to the system. The system automatically connects to communication services such as Microsoft Exchange, voice message servers, and other communication systems. These connections can be established by direct loading onto an enterprise server or via a web-based system. Once connected, the system extracts and transforms data into a form suitable for use as well as loading the source data into a database for recall as needed.
• The Message Processing phase performs semantic and metadata processing, as well as thread reconstruction, which is described in more detail below. The system will also perform de-duplication, identify and account for common users, and handle multiple email, IM, social network, and other accounts.
• This visualization phase includes the visualization tools where the 2D and 3D representation of the data so that patterns can be divined. This phase also includes the filtering and sorting tools so that analysis can be enhanced. This information can be shared and reports can be authored.
• When displaying the data and showing the time-based relationship between events, the system can, in one embodiment, normalize the temporal dimension. In this manner, the sequential relationships are displayed more clearly than if the temporal dimension was not normalized. When there is a large time gap between events, a display without normalization could disseminate out more than would be comfortably visible on a typical display, masking or obscuring some of the relationships that need to be seen. There are situations when the actual temporal scale is desired, to show the real time lapse between cause and effect, but the system allows the user to choose between actual and normalized temporal displays.
• The 3D display of the data shows multidimensional space and allows the user to evaluate visually, using position and color, the richness of the communication. Multiple threads and multiple participants can be viewed in context with an infinite number of visual planes for analysis. The user of the system can determine which threads, which users, and can identify which planes (i.e. communication, social network, transaction) are to be included in the display. The 3D cube presentation can be zoomed, rotated, and explored in a variety of custom presentations. The zoom feature can be implemented via interface menu commands, or even the scroll wheel of a computer mouse (i.e. rolling forward to zoom in and rolling back to zoom out).
• The analyst tools in working with the system include the ability to tag and apply notes to any item or plane in the display. The user can copy collections of communications, threads, and transactions to a clipboard for later retrieval. The displays, annotations, clipboards and other data can be easily shared with other analysts, exported as reports to other programs such as spreadsheets or Microsoft Word documents, and can be statistically summarized. For example, words or sentences that repeat throughout selected threads can be identified and counted to show patterns, or communication patterns statistics can be analysed.
• Clipboard
• Theclipboard206 ofFIG.2 is shown in more detail inFIG.8. When expanded, theclipboard706 shows graphical representations of data copied to the clipboard. In the example ofFIG.8, there are three elements801-803 stored in the clipboard. Each element represents a root email of an email thread and may represent the entire string of communications and transactions associated with that root thread. Any message appearing in the 2D or 3D representation may be dragged to the clipboard. The clipboard is capable of holding a root email, a response email, a thread, a note, and any other object in the system, including transactions.
• FIG.10 illustrates the 3D cube with amessage1001 selected and opened to the user (via CLICK+Trident button equal sign). (It should be noted that although the description refers to “clicks” to activate navigation events, the system is not limited to clicks but may be enabled by any type of action or event without departing from the scope and spirit of the system). The message window may be closed using CLICK+Trident button equal sign or by clicking on the red circle with an “x” inside in the upper right corner of the message window. The text of the opened message can be navigated vertically using the scroll bar along the right side. The user has enabled a noteannotation entry field1002 so that the user can attach comments about the email for later use and analysis. The text of the note can be vertically navigated using the scroll bar along the right side of the note.
• When copied/pasted into a report document, or exported, the depiction can remain “live” via a web-based linkage back to the source system. In other words, for certain report formats, the reader of the report can click into the exported image area and begin to interact with the diagram as needed to rotate or otherwise view the data.
• Iconography
• FIG.12 illustrates the elements of email entries in an embodiment of the system.Message1201 is an example of a sender message. Themessage1201 includes the name ofsender1202, the date/time1203, and an indicator1204 when there is an attachment (this indication also includes a numeral representing the number of attachments). Thetrident navigation tool1205 is present on each message, along with the first twolines1206 of the message and theroot subject1207.Message1208 is an active participant's response added to an email chain.Message1210 is a “cc'd” received root email which in this example includes anindicator1209 that it is a “cc” copy versus the addressee.Message1211 is a cc recipient of the active participant's email response. These messages are color coded as noted above with respect toFIG.1.
• Description Of Trident Expand/Collapse Functionality
• The “Expanded email stack” means an email message cell that has been expanded to show all the emails up to that point in the thread; if the user has highlighted an email message within the thread, then SHFT+click on Trident button DOWN arrowheads will expand the “stack” of emails that represent the thread up until that point. The trident then would take the form of the left trident ofFIG.11
• SHFT+click on Trident button's DOWN arrowheads→expand the stack to show all the emails within the chain up until that point in the thread; the emails are shown in truncated form (sender, date and Subject Header only)
• SHFT+Trident button's DOWN arrowheads again→goes to the next level of detail for each message (sender, date, and Subject Header, add first2 lines of text of the body of each email message within the stack)
• SHFT+Trident button's DOWN arrowheads one more time goes to the next level of detail for each message (same as above, but add the full text body of each message within the stack)
• −SHFT+Trident button UP arrowheads: contracts the stack of email messages in the reverse order to what SHFT+double down arrowheads does.
• (a possible future modification: SHFT+Trident equal sign fully collapses or fully expands the stack immediately (don't have to expand/collapse the email messages in steps).
• In one embodiment, the trident tool can be adjusted to skip a certain number of threads or messages (e.g. 10 at a time) to speed up navigation through the data. This would be achieved using the CRTL+Trident UP or DOWN arrowheads and the trident would appear as on the right inFIG.11.
• Looking atFIG.1 a user can click on the right arrow on a Trident button inemail message103 or107 (within column C2). The corresponding action would be that the entire column to the right (C3) would slide “under” C2 and be hidden. To hide the next column (C4), you click on the right arrow again. CTRL+right arrow would expand all the hidden columns.
• The left arrow has similar functionality (but for columns to the left of the column you are currently interacting with).
• Thread Management
• After a data set has been provided to the system, such as by importing data from an email exchange, bulletin board database or bulletin board collection system, social network database, instant messaging database, etc, the system does the following to sort, identify, and create relationships.
• De-Duplication:
• Go through each message in the data set and examine the Message-Id: header. Import each message with a unique Message-Id: into the database. (Although the system is described in an example using emails, the system has equal application to other communication types such as those mentioned herein).
• Initial Threading:
• For each message in the database, create a thread with the thread name the same as the subject of the message.
• For each message in the database, examine its Message-ID: header and scan all other messages in the database for References: or In-Reply-To: headers. If a reference is found to the message-id and the messages are in different threads, join the threads by setting the thread id equal for all messages referencing the message-ID. More than one reference may be found to the message-id so more than one thread may need to be joined.
• Threads with a Missing Link (Split Thread):
• A split thread is when two messages in the same thread point to a third message that is missing from the database. It may be missing from the database because it was intentionally deleted or it may have been stored on a server that is not included in the dataset (messages on a partner or competitor's servers). This often occurs if a message is sent to a third party who discusses it internally and then replies.
• For each message in the database, examine its References: and In-Reply-To: headers. For each of the values in these headers that do not have a corresponding Message-Id: in the database, search the database again for different messages that have the same References: or In-Reply-To: id. Join threads in which the two messages refer to the same message but are in separate threads.
• After joining split threads it may be necessary to repeatstep2 of the initial threading process to join threads that could not be combined due to the missing message link.
• Inferred Thread Relationships:
• Some threads may still be broken after processing missing links. Remaining ways to combine threads are:
• Message content analysis
• Message subject collapsing
• Threads with the same subject and participants:
• In one embodiment, the system makes an assumption that a participant who sends two messages with the same subject within 24 hours is participating in the same thread. This may occur when someone makes an announcement to a large group, receives questions, and then issues a follow-up to the message to address those questions.
• For the purposes of this analysis, subjects are considered identical after any “Fwd:” or “Re:” prefixes are removed.
• Participation requirements should be tuned so that the most number of correct identifications occur with the least number of false identifications. Variables could include:
• Extending time criteria to two messages within one week
• Match messages with the same subject where participant matching can be sender or receiver
• Threads with the Same Content:
• Message textual analysis is complex. It compares all of the text in a message with all of the text in another message. A similarity measure is calculated, such as number of words (phrases, or sentences) matched divided by total number of words in the message. If the similarity measure exceeds a threshold, the messages are considered to be in the same thread.
• This method is interesting in that it may identify connections between very unrelated threads and help understand how information is being disseminated. For instance, if an executive sends a message to managers throughout the organization describing “the upcoming project Barnstorm” these messages could be linked to threads from employees who should not have received that information.
• At the same time, this method suffers from its inability to join messages that do not include the same language. This may happen if a manager sends a request to an employee asking for a timecard and the employee creates a new message with only an Excel document in it and does not include a copy of the manager's request.
• System Block Diagram
• FIG.6 is one embodiment of a block diagram of an embodiment of the system. The system obtains data from sources such asfile servers601,email servers602, andweb servers603. These sources providedata sources604 such as instant messages, email, common documents, version control repositories, internal wikis, social networks, etc. Aprocessing engine605 includes ametadata filter606,semantic processor607, andtext conversion engine606 to extract information and relationships from the source data and provides it torelational database609. This data is provided to visualization anddiscovery application610 that controls the display and representation of the information. This provides information to the case preparation application tool611 for report generation, sharing, etc.
• FIG.7 is a block diagram of an alternate embodiment of the system. The system operates in a plurality of stages.Stage701 is the communications stage.Stage702 is a metadata processing stage and is followed by asemantic processing stage703. The system includes arelationship building stage704 andindexing stage705.
• Thecommunication stage701 is where raw data is collected for importation to the rest of the system. The communication stage may take data from a plurality of sources, such asemail706 throughemail importer707, BBS information708 viabulletin board importer709,social media data710 viasocial media importer711, andmessage data712 throughinstant messenger importer713.
• Email data706 may be from any of one or more email system and may be in formats such as .pst files, Gmail files, .nsf (Lotus Notes) files, and the like. The email is imported with any attachments that may be associated with the emails. Bulletin board data708 may be messages or postings from web based bulletin boards and may come from linear or threaded (and sub-threaded) BBS sources (e.g. usenet) using bulletin board engines such as, for example, vBulletin. Bulletin board messages or postings are imported with any attachments or imbedded media such as video clips, photos, links, audio clips.
• Social media data710 may include Twitter postings, Facebook wall postings or any other message based communications on social media, such as comments to videos or blogs and associated comments. Social Media messages or postings are imported with any attachments or imbedded media such as video clips, photos, audio clips, links, etc.
• Messenger data712 includes AIM, SMS, Jabber, MSN, Blackberry Messenger, or any other instant message systems. Messenger messages are imported with any attachments or imbedded media such as video clips, photos, audio clips, links, etc.
• Communications from each of these sources may be processed as a batch, or in a continuous manner as data arrives from each source, such as live monitoring of communications. Communications may also arrive multiple times or out-of-order, as they are placed in order and de-duplicated by theindexing engine719.
• Thesemantic processing stage703 comprises anentity extractor715 and asimilarity engine716. Theentity extractor715 determines locations, names, places and other noun-phrase concepts which may be optionally guided by a user defined wordlist, related to each message or communication. Thesimilarity engine716 uses the concepts identified by the entity extractor to identify related documents. This process conducts a search for other documents that have locations, names, places, or other noun-phrase concepts similar to the document being ingested. If the number of matching concepts exceeds a threshold the documents will be identified as similar, for use in later processing steps.
• Therelationship builder stage704 includes aprofile builder717 and aconversation builder718. Theprofile builder717 identifies participants and any aliases, addresses, names, related accounts or user names, and other identifying information to identify a participant as well as possible regardless of the account that is being used. Theconversation builder718 analyzes documents to determine if they are part of a new or existing communications thread. The conversation builder uses the output of thesimilarity engine716 and the techniques described in86-106. This process incorporates a feedback loop so that each new document added to a thread helps to identify additional candidate documents. Theconversation builder718 is helpful when responding to discovery requests by allowing appropriate responses to be identified even if the communications do not share a common subject line, thread, or topic.
• Theindexing stage705 includes anindexing engine719 to eliminate duplicates and to annotate and index each message. The indexed and processed communications and messages are stored indatabase720 for later retrieval and display.
• Visualization
• FIG.9 is a block diagram illustrating an embodiment for generating display visualizations of data using the system. In this embodiment of the system there are a number of stage in the visualization process, includingQuery901, Refine902, Assemble903, Navigate904, and Visualize905.
• TheQuery stage901 comprises asearch engine906 to allow specific data to be pulled out of thedatabase720. The search defines certain parameters such as date range, participant, keyword, and other facets of the search. These other facets can include bank accounts, locations, travel, transfers, which message sources to use, and other filtering options. The analyst will construct a search query using these limiters and advance to the refinestage902. The search may be performed using any search capabilities offered by the search engine, such as those indicated above or semantic search, proximity search, lemmatization, Boolean operators, etc.
• The Refinestage902 comprises a RefineLimit Pivot capability907. ThePivot capability907 implements a pivot table which allows the analyst to select a new search based on additional concepts found in the current search. The Refine capability allows the analyst to assign each result a quality: find more like this result and show fewer like this result. The Limit capability allows the analyst to add additional constraints to the search to narrow the result set. The Refine Limit Pivot capability incorporates a feedback loop the search can be looped back multiple times as desired to refine the search and increase the confidence level of the analyst.
• The Assemblestage903 brings the data together in two ways so that the user is presented with all options upon actual display. The system uses agrid assembler908 to generate a 2D view of the data (as inFIG.1) and theuniverse assembler909 assembles a 3D view such as inFIG.2. The assembly typically uses a message at a starting origin point and builds the display from that point, determining position and color for each message that will be part of the visualization. In the grid assembler and universe assembler, a number of configurations for each axis are possible.
• For the participant axis, in one embodiment, participants are ordered based on number of messages sent. In another embodiment, participants are ordered based on their role in the organization (executive, manager, employee) or their business unit (engineering, quality control, manufacturing). Additional embodiments allow the user to specify the technique used to order participants.
• For the message axis, in one embodiment, messages are ordered based on time sent in a manner where the time between messages is preserved. In another embodiment, messages are ordered based on time sent where the time between messages is compressed. In another embodiment, messages are ordered based on a time scale that is non-aligned and resets back to 0 for each thread.
• TheNavigation stage904 links the messages in the display usingnavigation engine910 andcache911 so that the trident navigation tool can be used to move from message to message as described herein. TheVisualization stage905 presents any of the possible views, for example, the2D view912 or the3D view913.
• Medical Tracking
• The system can be used in one embodiment for medical tracking as shown in
• FIG.13. In one embodiment of the medical tracking system, each message includes at least certain information, including patient ID, participant or entity, the layer (e.g. communications, reports, prescriptions, etc.), message type (e.g. medication prescription), and the date and time. The system uses color coding or some other visual indicator to represent different types of messages in the medical tracking embodiment.
• For example, the system may use a dark color to represent an originating request, and a lighter color to represent a response to that request. In addition, a plurality of types of communications are defined by particular colors. This makes it possible to see at a glance what type of communication is represented by a message in a two or three dimensional representation of data. In the example of medical tracking, red and light red represent a GP's initiating communications and responses. Dark blue and light blue represent a specialists initiating communications and responses. Dark green and light green represent meds/prescription initiators and responses. Dark orange and light orange represent test/diagnostic requests and responses, and dark purple and light purple represent procedures/operations requests and responses.
• As can be seen inFIG.13, it is possible to overlay different types of data planes that each represent a different activity in the a patient's treatment. The messages can be time normalized so that the flow of activity can be seen very easily. Selecting or clicking on any individual message will bring up all documents associated with that communication or procedure.
• Financial Tracking/Intelligence
• The system enables analysts to correlate communications and activities from a plurality of participants to reveal patterns and causation of events. In some cases these events could be crimes that have been committed or are in the planning stage. By presenting and correlating data from multiple sources, an analyst can track activities, communications and transactions of multiple participants.
• In one embodiment, the system tracks a number of types of activity and can present each type in its own visual layer. In one embodiment, there may be a communications layer, a transactions layer, and a travel layer. This allows an analyst to see the relationship between communications, movement of funds, and location of participants.
• The system can be described in one embodiment in the flow diagram ofFIG.14. Atstep1401 the analyst determines the participants of the analysis. Atstep1402 the analyst identifies the communications accounts of each participant. This may include email, SMS messaging, social media, twitter, bulletin board postings, etc. Atstep1403 the system collects the communications from these sources and extracts metadata and other information using, for example, the system ofFIG.6.
• At step1404 the system builds a presentation layer for the communications of the participants. At step1405 the analyst identifies financial accounts associated with the participants. Atstep1406 the system retrieves the account information associated with the participants and at step1407 builds a presentation layer for the financial transactions. These transactions locations and bank accounts are based on other conversations these participants are having in other threads (with their financial backers, with the leader of a terrorist cell, with other parties, and so forth). The system software assembles this information to get a confidence interval, and correlates location and funds to the discussions. The system can import bank statements and location details from external databases/sources to improve that confidence.
• Atstep1408 the system collects travel and geographic information for each participant (along with time/date information) and atstep1409 the system builds a presentation layer for the travel/geographic information. Atstep1410 the system correlates and normalizes the layers so that they can be viewed in relationship to the overall activity of the participants. The timeline of the layers can be relative or normalized so that cause and effect can be shown more easily.
• The system identifies changes in a participant's location or bank account balance, and indicate those changes in a certain color. The system identifies specific locations or amounts (i.e. Yemen or amounts greater than $2 million) and indicate those in a certain color. The system allows the intelligence analyst to program in suspicious activity flags using Boolean logic and functions, like the expression evaluator in Excel. Alerts can notify the analyst whenever there's a correlation between databases, and indicate how closely things correlate.
• The following is an example to show some capabilities of the system.
• Abe is the ringleader and lives in the US.
• Bonnie is the money launderer and participates in the activity.
• Clyde is the financing source and lives in Japan.
• Darryl is the point man on the ground.
• Ed leads the terror cell and is being paid to set up and detonate explosives.
• The analyst has also identified certain bank accounts that are believed to be related to the group of participants.FIG.15 is a display that shows the state of thecommunications1501,transactions1502 andgeographic locations1503 of the participants.
• In the communication layer, each of the participants has their own column representing their communications. In this case, the communications have been filtered to show only those communications related to this particular plot. As noted previously, the messages may be color coded to show the nature of the message and the relationship between originator, recipient, responder, copied-to, etc. Thefinancial transactions1502 shows two accounts of interest. In this case, there is not direct correlation between the accounts. In other words, transactions are not made directly between the accounts, but may go through a plurality of middlemen. The system allows to show the cause and effect of withdrawals and deposits on the two accounts without the need to track the transaction through all the third parties.
• Thegeographic indicator1503 shows the location of a participant at the time of any particular message. By highlighting a message, the location of each participant (if known) at the time of that message is displayed in the geographic region. This display is dynamically changed as different messages are selected. The system also has the ability to “play” a scenario showing each message as it is created in time and showing the changes to thelocation map1503 andfinancial information1502 as the playback proceeds.
• FIG.16 illustrates the final state of thecommunications1601,transactions1602, andgeographic locations1603 of the participants in the plot. As can be seen, thecommunications layer1601 is populated with a number of messages, responses, and copies related to the participants. In addition, the transactions layer now shows three accounts of interest and different colors to represent the transactions. Finally, the location map shows the final locations of the participants.
• FIG.17 is a detailed view of thefinancial transactions layer1602. In the example shown, account B1 is a Turkish bank, account132 is a Swiss bank, and account B3 is a Mid-East bank. The system shows the transactions overlaying the communications layer so the relationship between the transactions and the communications can be seen more easily. The system also uses color or some other visual indicator to represent the state of the accounts. In this example, medium green represents an account, dark green represents the state when funds transfer in and light blue represents the state when funds transfer out.
• Again referring toFIG.17, account B1 begins with a zero balance while the account B2 has a balance of $100 million. In response to and correlated with a communication from one or more of the participants, a $10 million withdrawal is made from account B2. This is represented by the cut-out circle on the lower left of account B2 and a visible minus sign. This is an additional visual indicator representing withdrawal in addition to the change in color of account B2 after the transaction.
• Account131 then shows a new color (e.g. dark green) and a “plus” sign indicating a deposit, also of $10 million. Meanwhile, the status of account133 has not changed at this time and remains, for example, medium green. Subsequently, there is a withdrawal of $10 million for account B1 (shown by the circular cut-out and minus sign) and a corresponding deposit of $10 million in account B3, shown by color change (e.g. dark green) and the plus sign circle.
• If there are withdrawals from an account that has already had a prior withdrawal, in one embodiment the second withdrawal is shown by a second circular cut-out or indent with a minus sign. In another embodiment, the size of the circular cut-out is enlarged to indicate multiple withdrawals. In yet another embodiment, the system reverts the account to the neutral color (e.g. medium green) and then changes it to light green and shows the second withdrawal in the same manner as above, with a single cut-out or indicator.
• All of the features described previously apply to this application of the system as well. For example, the user can mouse over or right-click on the display to have messages appear in their full form, or to add or change the presence or status of participants.
• Document Tracking
• The system also capable of providing a display for document tracking.FIG.18 illustrates an embodiment of document tracking using the system. Three documents are represented by message icons1801,1802, and1803. The visualization layer uses color coding for different versions and revisions of documents. The user can. see who received the document, who revised it and when, latest and earlier versions, etc.
• Embodiment of Computer Execution Environment (Hardware)
• An embodiment of the system can be implemented as computer software in the form of computer readable program code executed in a general purpose computing environment such asenvironment1900 illustrated inFIG.19, or in the form of bytecode class files executable within a within a Java.™ or other virtual machine run time environment running in such an environment, or in the form of bytecodes running in such an environment, or in the form of bytecodes running on a processor (or devices enabled to process bytecodes) existing in a distributed environment (e.g., one or more processors on a network). Akeyboard1910 and mouse1911 are coupled to asystem bus1918. The keyboard and mouse are for introducing user input to the computer system and communicating that user input to central processing unit (CPU1913. Other suitable input devices may be used in addition to, or in place of, the mouse1911 andkeyboard1910, such as a tablet PC, iPad, touchscreen device, smartphone/PDA, etc. I/O (input/output) unit1919 coupled tobi-directional system bus1918 represents such I/O elements as a printer, A/V (audio/video) I/O, etc.
• Computer1901 may include acommunication interface1920 coupled tobus1918.Communication interface1920 provides a two-way data communication coupling via anetwork link1921 to alocal network1922. For example, ifcommunication interface1920 is an integrated services digital network (ISDN) card or a modem,communication interface1920 provides a data communication connection to the corresponding type of telephone line, which comprises part ofnetwork link1921. Ifcommunication interface1920 is a local area network (LAN) card,communication interface1920 provides a data communication connection vianetwork link1921 to a compatible LAN. Wireless links are also possible. In any such implementation,communication interface1920 sends and receives electrical, electromagnetic or optical signals which carry digital data streams representing various types of information.
• Network link1921 typically provides data communication through one or more networks to other data devices. For example,network link1921 may provide a connection throughlocal network1922 tolocal server computer1923 or to data equipment operated byISP1924.ISP1924 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet”1925Local network1922 and Internet1925 both use electrical, electromagnetic or optical signals which carry digital data streams. The signals through the various networks and the signals onnetwork link1921 and throughcommunication interface1920, which carry the digital data to and fromcomputer1900, are exemplary forms of carrier waves transporting the information.
• Processor1913 may reside wholly onclient computer1901 or wholly onserver1926 orprocessor1913 may have its computational power distributed betweencomputer1901 andserver1926.Server1926 symbolically is represented inFIG.19 as one unit, butserver1926 can also be distributed between multiple “tiers”. In one embodiment,server1926 comprises a middle and back tier where application logic executes in the middle tier and persistent data is obtained in the back tier. In the case whereprocessor1913 resides wholly onserver1926, the results of the computations performed byprocessor1913 are transmitted tocomputer1901 via Internet1925, Internet Service Provider (ISP)1924,local network1922 andcommunication interface1920. In this way,computer1901 is able to display the results of the computation to a user in the form of output.
• Computer1901 includes avideo memory1914,main memory1915 andmass storage1912, all coupled tobi-directional system bus1918 along withkeyboard1910, mouse1911 andprocessor1913.
• As withprocessor1913, in various computing environments,main memory1915 andmass storage1912, can reside wholly onserver1926 orcomputer1901, or they may be distributed between the two. Examples of systems whereprocessor1913,main memory1915, andmass storage1912 are distributed betweencomputer1901 andserver1926 include thin-client computing architectures and other personal digital assistants, Internet ready cellular phones and other Internet computing devices, and in platform independent computing environments.
• Themass storage1912 may include both fixed and removable media, such as magnetic, optical or magnetic optical storage systems or any other available mass storage technology. The mass storage may be implemented as a RAID array or any other suitable storage means.Bus1918 may contain, for example, thirty-two address lines for addressingvideo memory1914 ormain memory1915. Thesystem bus1918 also includes, for example, a32-bit data bus for transferring data between and among the components, such asprocessor1913,main memory1915,video memory1914 andmass storage1912. Alternatively, multiplex data/address lines may be used instead of separate data and address lines.
• In one embodiment of the invention, theprocessor1913 is a microprocessor such as manufactured by Intel, AMD, Sun, etc. However, any other suitable microprocessor or microcomputer may be utilized.Main memory1915 is comprised of dynamic random access memory (DRAM).Video memory1914 is a dual-ported video random access memory. One port of thevideo memory1914 is coupled tovideo amplifier1916. Thevideo amplifier1916 is used to drive the cathode ray tube (CRT)raster monitor1917.Video amplifier1916 is well known in the art and may be implemented by any suitable apparatus. This circuitry converts pixel data stored invideo memory1914 to a raster signal suitable for use bymonitor1917.Monitor1917 is a type of monitor suitable for displaying graphic images, such as computer monitor, tablet PC, iPad, smartphone/PDA, touchscreen device, etc
• Computer1901 can send messages and receive data, including program code, through the network(s),network link1921, andcommunication interface1920. In the Internet example,remote server computer1926 might transmit a requested code for an application program through Internet1925,ISP1924,local network1922 andcommunication interface1920. The received code maybe executed byprocessor1913 as it is received, and/or stored inmass storage1912, or other non-volatile storage for later execution. In this manner,computer1900 may obtain application code in the form of a carrier wave. Alternatively,remote server computer1926 may executeapplications using processor1913, and utilizemass storage1912, and/orvideo memory1915. The results of the execution atserver1926 are then transmitted through Internet1925,ISP1924,local network1922 andcommunication interface1920. In this example,computer1901 performs only input and output functions.
• Application code may be embodied in any form of computer program product. A computer program product comprises a medium configured to store or transport computer readable code, or in which computer readable code may be embedded. Some examples of computer program products are CD-ROM disks, ROM cards, floppy disks, magnetic tapes, computer hard drives, servers on a network, and carrier waves.
• The computer systems described above are for purposes of example only. An embodiment of the invention may be implemented in any type of computer system or programming or processing environment, such as tablet PC, iPad, touchscreen device, PDA, etc.
• Thus, a system for analyzing and displaying relationships between communications and transactions has been described.

Claims (21)

What is claimed is:

1. (canceled)

2. A method for presenting relational information, the method comprising:

extracting a plurality of entities from electronically stored information;

identifying a plurality of threads from the electronically stored information; and

determining a plurality of relationships among the plurality of entities according to the plurality of threads, wherein:

each of the plurality of threads is associated with a root originator,

each of the plurality of threads is associated with a first piece of data from the root originator,

each of the plurality of threads is associated with a second piece of data in response to the root originator,

the plurality of relationships comprises a plurality of relationship types.

3. The method ofclaim 2, wherein each of the plurality of threads comprises one or more messages from multiple types of communications.

4. The method ofclaim 2, wherein each of the plurality of threads comprises a financial information thread that is displayed independently from a communication thread.

5. The method ofclaim 2, wherein the method comprises using a feedback loop to identify a document related to a relationship of the plurality of relationships.

6. The method ofclaim 2, wherein the method comprises identifying a new message related to a relationship of the plurality of relationships.

7. The method ofclaim 2, wherein the method comprises displaying a graph of relationships of a selected relationship type.

8. The method ofclaim 2, wherein the method comprises overlaying a financial thread on a travel thread.

9. The method ofclaim 2, wherein the method comprises selectively normalizing a time history of the plurality of relationships.

10. The method ofclaim 2, wherein the plurality of entities are determined by a noun-phrase concept.

11. The method ofclaim 2, wherein the plurality of relationship types comprises a financial relationship.

12. A system for presenting relational information, the system comprising:

a memory operable to store information; and

a processor operable to:

extract a plurality of entities from electronically stored information;

identify a plurality of threads from the electronically stored information; and

determine a plurality of relationships among the plurality of entities according to the plurality of threads, wherein:

each of the plurality of threads is associated with a root originator,

each of the plurality of threads is associated with a first piece of data from the root originator,

each of the plurality of threads is associated with a second piece of data in response to the root originator,

the plurality of relationships comprises a plurality of relationship types.

13. The system ofclaim 12, wherein each of the plurality of threads comprises one or more messages from multiple types of communications.

14. The system ofclaim 12, wherein each of the plurality of threads comprises a financial information thread that is displayed independently from a communication thread.

15. The system ofclaim 12, wherein the processor is operable to use a feedback loop to identify a document related to a relationship of the plurality of relationships.

16. The system ofclaim 12, wherein the processor is operable to identify a new message related to a relationship of the plurality of relationships.

17. The system ofclaim 12, wherein the processor is operable to display a graph of relationships of a selected relationship type.

18. The system ofclaim 12, wherein the processor is operable to overlay a financial thread on a travel thread.

19. The system ofclaim 12, wherein the processor is operable to selectively normalize a time history of the plurality of relationships.

20. The system ofclaim 12, wherein the plurality of entities are determined by a noun-phrase concept.

21. The system ofclaim 12, wherein the plurality of relationship types comprises a financial relationship.

Images