Concept mining is an activity that results in the extraction ofconcepts fromartifacts. Solutions to the task typically involve aspects ofartificial intelligence andstatistics, such asdata mining andtext mining.[1][2] Because artifacts are typically a loosely structured sequence of words and other symbols (rather than concepts), the problem isnontrivial, but it can provide powerful insights into the meaning, provenance and similarity of documents.
Traditionally, the conversion of words to concepts has been performed using athesaurus,[3] and for computational techniques the tendency is to do the same. The thesauri used are either specially created for the task, or a pre-existing language model, usually related to Princeton'sWordNet.
The mappings of words to concepts[4] are oftenambiguous. Typically each word in a given language will relate to several possible concepts. Humans use context to disambiguate the various meanings of a given piece of text, where availablemachine translation systems cannot easily infer context.
For the purposes of concept mining, however, these ambiguities tend to be less important than they are with machine translation, for in large documents the ambiguities tend to even out, much as is the case with text mining.
There are many techniques fordisambiguation that may be used. Examples are linguistic analysis of the text and the use of word and concept association frequency information that may be inferred from large text corpora. Recently, techniques that base onsemantic similarity between the possible concepts and the context have appeared and gained interest in the scientific community.
One of the spin-offs of calculating document statistics in the concept domain, rather than the word domain, is that concepts form natural tree structures based onhypernymy andmeronymy. These structures can be used to generate simple tree membership statistics, that can be used to locate any document in aEuclidean concept space. If the size of a document is also considered as another dimension of this space then an extremely efficient indexing system can be created. This technique is currently in commercial use locating similar legal documents in a 2.5 million document corpus.
Standard numeric clustering techniques may be used in "concept space" as described above to locate and index documents by the inferred topic. These are numerically far more efficient than theirtext mining cousins, and tend to behave more intuitively, in that they map better to the similarity measures a human would generate.