Unstructured data (orunstructured information) is information that either does not have a pre-defineddata model or is not organized in a pre-defined manner. Unstructured information is typically text-heavy, but may contain data such as dates, numbers, and facts as well. This results in irregularities andambiguities that make it difficult to understand using traditional programs as compared to data stored in fielded form in databases orannotated (semantically tagged) in documents.
In 1998,Merrill Lynch said "unstructured data comprises the vast majority of data found in an organization, some estimates run as high as 80%."[1] It is unclear what the source of this number is, but nonetheless it is accepted by some.[2] Other sources have reported similar or higher percentages of unstructured data.[3][4][5]
As of 2012[update],IDC andDell EMC project that data will grow to 40zettabytes by 2020, resulting in a 50-fold growth from the beginning of 2010.[6] More recently, IDC andSeagate predict that the globaldatasphere will grow to 163 zettabytes by 2025[7] and majority of that will be unstructured. TheComputer World magazine states that unstructured information might account for more than 70–80% of all data in organizations.[1]
The earliest research intobusiness intelligence focused in on unstructured textual data, rather than numerical data.[8] As early as 1958,computer science researchers likeH.P. Luhn were particularly concerned with the extraction and classification of unstructured text.[8] However, only since the turn of the century has the technology caught up with the research interest. In 2004, theSAS Institute developed theSAS Text Miner, which usesSingular Value Decomposition (SVD) to reduce ahyper-dimensional textualspace into smaller dimensions for significantly more efficient machine-analysis.[9] The mathematical and technological advances sparked bymachine textual analysis prompted a number of businesses to research applications, leading to the development of fields likesentiment analysis,voice of the customer mining, and call center optimization.[10] The emergence ofBig Data in the late 2000s led to a heightened interest in the applications of unstructured data analytics in contemporary fields such aspredictive analytics androot cause analysis.[11]
The term is imprecise for several reasons:
Techniques such asdata mining,natural language processing (NLP), andtext analytics provide different methods tofind patterns in, or otherwise interpret, this information. Common techniques for structuring text usually involve manualtagging with metadata orpart-of-speech tagging for furthertext mining-based structuring. TheUnstructured Information Management Architecture (UIMA) standard provided a common framework for processing this information to extract meaning and create structured data about the information.
Software that creates machine-processable structure can utilize the linguistic, auditory, and visual structure that exist in all forms of human communication.[12] Algorithms can infer this inherent structure from text, for instance, by examining wordmorphology, sentence syntax, and other small- and large-scale patterns. Unstructured information can then be enriched and tagged to address ambiguities and relevancy-based techniques then used to facilitate search and discovery. Examples of "unstructured data" may include books, journals, documents,metadata,health records,audio,video,analog data, images, files, and unstructured text such as the body of ane-mail message,Web page, orword-processor document. While the main content being conveyed does not have a defined structure, it generally comes packaged in objects (e.g. in files or documents, ...) that themselves have structure and are thus a mix of structured and unstructured data, but collectively this is still referred to as "unstructured data".[13] For example, anHTML web page is tagged, but HTML mark-up typically serves solely for rendering. It does not capture the meaning or function of tagged elements in ways that support automated processing of the information content of the page.XHTML tagging does allow machine processing of elements, although it typically does not capture or convey the semantic meaning of tagged terms.
Since unstructured data commonly occurs inelectronic documents, the use of acontent ordocument management system which can categorize entire documents is often preferred over data transfer and manipulation from within the documents. Document management thus provides the means to convey structure ontodocument collections.
Search engines have become popular tools for indexing and searching through such data, especially text.
Specific computational workflows have been developed to impose structure upon the unstructured data contained within text documents. These workflows are generally designed to handle sets of thousands or even millions of documents, or far more than manual approaches to annotation may permit. Several of these approaches are based upon the concept ofonline analytical processing, or OLAP, and may be supported by data models such as text cubes.[14] Once document metadata is available through a data model, generating summaries of subsets of documents (i.e., cells within a text cube) may be performed with phrase-based approaches.[15]
Biomedical research generates one major source of unstructured data as researchers often publish their findings in scholarly journals. Though the language in these documents is challenging to derive structural elements from (e.g., due to the complicated technical vocabulary contained within and thedomain knowledge required to fully contextualize observations), the results of these activities may yield links between technical and medical studies[16] and clues regarding new disease therapies.[17] Recent efforts to enforce structure upon biomedical documents includeself-organizing map approaches for identifying topics among documents,[18] general-purposeunsupervised algorithms,[19] and an application of the CaseOLAP workflow[15] to determine associations between protein names andcardiovascular disease topics in the literature.[20] CaseOLAP defines phrase-category relationships in an accurate (identifies relationships), consistent (highly reproducible), and efficient manner. This platform offers enhanced accessibility and empowers the biomedical community with phrase-mining tools for widespread biomedical research applications.[20]
In Sweden (EU), pre 2018, some data privacy regulations did not apply if the data in question was confirmed as "unstructured".[21] This terminology, unstructured data, is rarely used in the EU afterGDPR came into force in 2018. GDPR does neither mention nor define "unstructured data". It does use the word "structured" as follows (without defining it);
GDPR Case-law on what defines a "filing system"; "the specific criterion and the specific form in which the set of personal data collected by each of the members who engage in preaching is actually structured is irrelevant, so long as that set of data makes it possible for the data relating to a specific person who has been contacted to beeasily retrieved, which is however for the referring court to ascertain in the light of all the circumstances of the case in the main proceedings." (CJEU,Todistajat v. Tietosuojavaltuutettu, Jehovan, Paragraph 61).
Ifpersonal data is easily retrieved - then it is a filing system and - then it is in scope for GDPR regardless of being "structured" or "unstructured". Most electronic systems today,[as of?] subject to access and applied software, can allow for easy retrieval of data.