Repository files navigation

We are thrilled to announce our newly launchedUnstructured API. While access to the hosted Unstructured API will remain free, API Keys are required to make requests. To prevent disruption, get yourshere now and start using it today! Check out thereadme here to get started making API calls.

We are releasing the beta version of our Chipper model to deliver superior performance when processing high-resolution, complex documents. To start using the Chipper model in your API request, you can utilize thehi_res strategy. Please refer to the documentationhere.

As the Chipper model is in beta version, we welcome feedback and suggestions. For those interested in testing the Chipper model, we encourage you to connect with us onSlack community.

This repo implements a pre-processing pipeline for the following documents. Currently, the pipeline is capable of recognizing the file type and choosing the relevant partition function to process the file.

Try our hosted API! It's freely available to use with any of the filetypes listed above. This is the easiest way to get started. If you'd like to host your own version of the API, jump down to theDeveloper Quickstart Guide.

 curl -X 'POST' \  'https://api.unstructured.io/general/v0/general' \  -H 'accept: application/json' \  -H 'Content-Type: multipart/form-data' \  -H 'unstructured-api-key: <YOUR API KEY>' \  -F 'files=@sample-docs/family-day.eml' \  | jq -C . | less -R

Four strategies are available for processing PDF/Images files:hi_res,fast,ocr_only andauto.fast is the defaultstrategy and works well for documents that do not have text embedded in images.

On the other hand,hi_res is the better choice for PDFs that may have text within embedded images, or for achieving greater precision ofelement types in the response JSON. Please be aware that, as of writing,hi_res requests may take 20 times longer to process compared to thefast option. See the example below for making ahi_res request.

 curl -X 'POST' \  'https://api.unstructured.io/general/v0/general' \  -H 'accept: application/json' \  -H 'Content-Type: multipart/form-data' \  -F 'files=@sample-docs/layout-parser-paper.pdf' \  -F 'strategy=hi_res' \  | jq -C . | less -R

Theocr_only strategy runs the document through Tesseract for OCR. Currently,hi_res has difficulty ordering elements for documents with multiple columns. If you have a document with multiple columns that do not have extractable text, we recommend using theocr_only strategy. Please be aware thatocr_only will fall back to another strategy if Tesseract is not available.

For the best of all worlds,auto will determine when a page can be extracted usingfast orocr_only mode, otherwise it will fall back tohi_res.

Thehi_res strategy supports different models, and the default isdetectron2onnx. You can also specifyhi_res_model_name parameter to runhi_res strategy with the chipper model while using the host API:

 curl -X 'POST' \  'https://api.unstructured.io/general/v0/general' \  -H 'accept: application/json' \  -H 'Content-Type: multipart/form-data' \  -F 'files=@sample-docs/layout-parser-paper.pdf' \  -F 'strategy=hi_res' \  -F 'hi_res_model_name=chipper'  \  | jq -C . | less -R

We also support models to be used locally, for example,yolox. Please refer to theusing-the-api-locally section for more information on how to use the local API.

Note: This kwarg will eventually be deprecated. Please uselanguages.You can also specify what languages to use for OCR with theocr_languages kwarg. See theTesseract documentation for a full list of languages and install instructions. OCR is only applied if the text is not already available in the PDF document.

curl -X 'POST' \  'https://api.unstructured.io/general/v0/general' \  -H 'accept: application/json' \  -H 'Content-Type: multipart/form-data' \  -F 'files=@sample-docs/english-and-korean.png' \  -F 'strategy=ocr_only' \  -F 'ocr_languages=eng'  \  -F 'ocr_languages=kor'  \  | jq -C . | less -R

You can also specify what languages to use for OCR with thelanguages kwarg. See theTesseract documentation for a full list of languages and install instructions. OCR is only applied if the text is not already available in the PDF document.

curl -X 'POST' \  'https://api.unstructured.io/general/v0/general' \  -H 'accept: application/json' \  -H 'Content-Type: multipart/form-data' \  -F 'files=@sample-docs/english-and-korean.png' \  -F 'strategy=ocr_only' \  -F 'languages=eng'  \  -F 'languages=kor'  \  | jq -C . | less -R

When elements are extracted from PDFs or images, it may be useful to get their bounding boxes as well. Set thecoordinates parameter totrue to add this field to the elements in the response.

 curl -X 'POST' \  'https://api.unstructured.io/general/v0/general' \  -H 'accept: application/json' \  -H 'Content-Type: multipart/form-data' \  -F 'files=@sample-docs/layout-parser-paper.pdf' \  -F 'coordinates=true' \  | jq -C . | less -R

Currently, we provide support for enabling and disabling table extraction for all file types. Set parameterskip_infer_table_types to specify the document types that you want to skip table extraction with. By default, we enable table extractionfor all file types (skip_infer_table_types=[]). Again, please note that table extraction only works withhi_res strategy. For example, if you want to skip table extraction for images, you can pass a list with matching image file types:

 curl -X 'POST' \  'https://api.unstructured.io/general/v0/general' \  -H 'accept: application/json' \  -H 'Content-Type: multipart/form-data' \  -F 'files=@sample-docs/layout-parser-paper-with-table.jpg' \  -F 'strategy=hi_res' \  -F 'skip_infer_table_types=["jpg"]' \  | jq -C . | less -R

You can specify the encoding to use to decode the text input. If no value is provided, utf-8 will be used.

curl -X 'POST' \ 'https://api.unstructured.io/general/v0/general' \ -H 'accept: application/json'  \ -H 'Content-Type: multipart/form-data' \ -F 'files=@sample-docs/fake-power-point.pptx' \ -F 'encoding=utf_8' \ | jq -C . | less -R

You can send gzipped file and api will un-gzip it.

curl -X 'POST' \ 'https://api.unstructured.io/general/v0/general' \ -H 'accept: application/json'  \ -H 'Content-Type: multipart/form-data' \ -F 'gz_uncompressed_content_type=application/pdf' \ -F 'files=@sample-docs/layout-parser-paper.pdf.gz'

If fieldgz_uncompressed_content_type is set, the API will use its value as content-type of all filesafter uncompressing the .gz files that are sent in single batch. If not set, the API will usevarious heuristics to detect the filetypes after uncompressing from .gz.

When processing XML documents, set thexml_keep_tags parameter totrue to retain the XML tags in the output. If not specified, it will simply extract the text from within the tags.

curl -X 'POST' \ 'https://api.unstructured.io/general/v0/general' \ -H 'accept: application/json'  \ -H 'Content-Type: multipart/form-data' \ -F 'files=@sample-docs/fake-xml.xml' \ -F 'xml_keep_tags=true' \ | jq -C . | less -R

For supported filetypes, set theinclude_page_breaks parameter totrue to includePageBreak elements in the output.

curl -X 'POST' \ 'https://api.unstructured.io/general/v0/general' \ -H 'accept: application/json'  \ -H 'Content-Type: multipart/form-data' \ -F 'files=@sample-docs/layout-parser-paper-fast.pdf' \ -F 'include_page_breaks=true' \ | jq -C . | less -R

By default, the element ID is a SHA-256 hash of the element text. This is to ensure thatthe ID is deterministic. One downside is that the ID is not guaranteed to be unique.Different elements with the same text will have the same ID, and there could also be hash collisions.To use UUIDs in the output instead, setunique_element_ids=true. Note: this means that the element IDswill be random, so with every partition of the same file, you will get different IDs.This can be helpful if you'd like to use the IDs as a primary key in a database, for example.

curl -X 'POST' \  'https://api.unstructured.io/general/v0/general' \ -H 'accept: application/json'  \ -H 'Content-Type: multipart/form-data' \ -F 'files=@sample-docs/layout-parser-paper-fast.pdf' \ -F 'unique_element_ids=true' \ | jq -C . | less -R

Use thechunking_strategy form-field to chunk text into larger or smaller elements. Defaults toNone which performs no chunking. The available chunking strategies arebasic andby_title.

Thebasic strategy combines whole consecutive document elements to maximally fill chunks ofmax_characters length. A single element that by itself exceedsmax_characters is divided into two or more chunks by text-splitting (on a word boundary).

Theby_title strategy has the same behaviors except document section boundaries are respected, meaning elements from two different sections never occur in the same chunk. ATitle (section heading) element introduces a new section, hence the name.

Additional Parameters (all optional):

`max_characters`  The hard maximum chunk size. No chunk will exceed this length. Defaults to 500.`new_after_n_chars`  A chunk of this length or greater is considered "full" and will not receive an additional element, even if it would fit within `max_characters`.  This "soft-maximum" defaults to `max_characters`.`overlap`  Specifies the length of a string ("tail") to be drawn from each chunk and prefixed to the  next chunk as a context-preserving mechanism. By default, this only applies to split-chunks  where an oversized element is divided into multiple chunks by text-splitting.`overlap_all`  Default: `False`. When `True`, apply overlap between "normal" chunks formed from whole  elements and not subject to text-splitting. Use this with caution as it entails a certain  level of "pollution" of otherwise clean semantic chunk boundaries.`combine_under_n_chars`  Combines elements (for example a series of titles) until a section reaches a  length of n characters. Defaults to 500. Only operative for the "by_title"  strategy.`multipage_sections`  If True, sections can span multiple pages. Defaults to True. Only operative for  the "by_title" strategy.

curl -X 'POST'  'https://api.unstructured.io/general/v0/general' \ -H 'accept: application/json'  \ -H 'Content-Type: multipart/form-data' \ -F 'files=@sample-docs/layout-parser-paper-fast.pdf' \ -F 'chunking_strategy=by_title' \ | jq -C . | less -R

• Usingpyenv to manage virtualenv's is recommended

  • Mac install instructions. Seehere for more detailed instructions.

    • brew install pyenv-virtualenv
    • pyenv install 3.12

  • Linux instructions are availablehere.

  • Create a virtualenv to work in and activate it, e.g. for one nameddocument-processing:

    pyenv virtualenv 3.12 unstructured-api
pyenv activate unstructured-api

See theUnstructured Quick Start for the many OS dependencies that are required, if the ability to process all file types is desired.

• Runmake install
• Start a local jupyter notebook server withmake run-jupyter
  OR
  just start the fast-API locally withmake run-web-app

After runningmake run-web-app (ormake docker-start-api to run in the container), you can now hit the API locally at port 8000. Thesample-docs directory has a number of example file types that are currently supported.

For example:

 curl -X 'POST' \  'http://localhost:8000/general/v0/general' \  -H 'accept: application/json' \  -H 'Content-Type: multipart/form-data' \  -F 'files=@sample-docs/family-day.eml' \  | jq -C . | less -R

The response will be a list of the extracted elements:

[  {    "element_id": "db1ca22813f01feda8759ff04a844e56",    "coordinates": null,    "text": "Hi All,",    "type": "UncategorizedText",    "metadata": {      "date": "2022-12-21T10:28:53-06:00",      "sent_from": [        "Mallori Harrell <mallori@unstructured.io>"      ],      "sent_to": [        "Mallori Harrell <mallori@unstructured.io>"      ],      "subject": "Family Day",      "filename": "family-day.eml"    }  },......

The output format can also be set totext/csv to get the data in csv format rather than json:

 curl -X 'POST' \  'http://localhost:8000/general/v0/general' \  -H 'accept: application/json' \  -H 'Content-Type: multipart/form-data' \  -F 'files=@sample-docs/family-day.eml' \  -F 'output_format="text/csv"'

The response will be a list of the extracted elements in csv format:

type,element_id,text,filename,sent_from,sent_to,subject,languages,filetypeUncategorizedText,db1ca22813f01feda8759ff04a844e56,"Hi All,",family-day.eml,['Mallori Harrell <mallori@unstructured.io>'],['Mallori Harrell <mallori@unstructured.io>'],Family Day,['eng'],message/rfc822NarrativeText,a663c393a5e143c01ef2bb5c98efa2c1,Get excited for our first annual family day! ,family-day.eml,['Mallori Harrell <mallori@unstructured.io>'],['Mallori Harrell <mallori@unstructured.io>'],Family Day,['eng'],message/rfc822NarrativeText,ce65ca3bef59957d3f1c2bab5725c82f,"There will be face painting, a petting zoo, funnel cake and more.",family-day.eml,['Mallori Harrell <mallori@unstructured.io>'],['Mallori Harrell <mallori@unstructured.io>'],Family Day,['eng'],message/rfc822NarrativeText,d7bcf988af9f06042d83e25c531e5744,Make sure to RSVP!,family-day.eml,['Mallori Harrell <mallori@unstructured.io>'],['Mallori Harrell <mallori@unstructured.io>'],Family Day,['eng'],message/rfc822Title,5550577db69c2c8aabcd90979698120a,Best.,family-day.eml,['Mallori Harrell <mallori@unstructured.io>'],['Mallori Harrell <mallori@unstructured.io>'],Family Day,['eng'],message/rfc822Title,ca1c571d993b6c1ed8ef56a06c16ba22,Mallori Harrell,family-day.eml,['Mallori Harrell <mallori@unstructured.io>'],['Mallori Harrell <mallori@unstructured.io>'],Family Day,['eng'],message/rfc822Title,d5b612de8cd918addd9569b0255b65b2,Unstructured Technologies,family-day.eml,['Mallori Harrell <mallori@unstructured.io>'],['Mallori Harrell <mallori@unstructured.io>'],Family Day,['eng'],message/rfc822Title,2e0b9e8ee04b9594a9c26d8535b818ff,Data Scientist,family-day.eml,['Mallori Harrell <mallori@unstructured.io>'],['Mallori Harrell <mallori@unstructured.io>'],Family Day,['eng'],message/rfc822

As mentioned above, processing a pdf usinghi_res is currently a slow operation. One workaround is to split the pdf into smaller files, process these asynchronously, and merge the results. You can enable parallel processing mode with the following env variables:

• UNSTRUCTURED_PARALLEL_MODE_ENABLED - set totrue to process individual pdf pages remotely, default isfalse.
• UNSTRUCTURED_PARALLEL_MODE_URL - the location to send pdf page asynchronously, no default setting at the moment.
• UNSTRUCTURED_PARALLEL_MODE_THREADS - the number of threads making requests at once, default is3.
• UNSTRUCTURED_PARALLEL_MODE_SPLIT_SIZE - the number of pages to be processed in one request, default is1.
• UNSTRUCTURED_PARALLEL_RETRY_ATTEMPTS - the number of retry attempts on a retryable error, default is2. (i.e. 3 attempts are made in total)

Due to the overhead associated with file splitting, parallel processing mode is only recommended for thehi_res strategy. Additionally users of the officialPython client can enable client-side splitting by settingsplit_pdf_page=True.

You may also set the optionalUNSTRUCTURED_API_KEY env variable to enable request validation for your self-hosted instance of Unstructured. If set, only requests including anunstructured-api-key header with the same value will be fulfilled. Otherwise, the server will return a 401 indicating that the request is unauthorized.

Some documents will use a lot of memory as they're being processed. To mitigate OOM errors, the server will return a 503 if the host's available memory drops below 2GB. This is configured with the environment variableUNSTRUCTURED_MEMORY_FREE_MINIMUM_MB, which defaults to 2048. You can lower this value to reduce these messages, that is, allow the server to use more memory. Otherwise, you can set to 0 to fully remove this check.

By default server will run for indefinitely. To change that theMAX_LIFETIME_SECONDS environmental variable can be set. If server is run with this variable set, it will enter a graceful shutdown period afterMAX_LIFETIME_SECONDS from its initialization. Graceful shutdown period lasts for up to 3600 seconds and during it:

• server denies any new requests - they're met with an empty response,
• server continues processing active requests and shuts down (ending graceful period) if all of them are processed.

After the graceful period is over if server is still running, it is shutdown forcefully, cancelling all active requests and sending empty responses to each of them.

Max lifetime requires gnutimeout to be installed, available by default on most linux systems. Downloadable on macOS as gtimeout with gnu coreutils.

The following instructions are intended to help you get up and running using Docker to interact withunstructured-api.Seehere if you don't already have docker installed on your machine.

NOTE: we build multi-platform images to support both x86_64 and Apple silicon hardware. Docker pull should download the corresponding image for your architecture, but you can specify with--platform (e.g. --platform linux/amd64) if needed.

We build Docker images for all pushes tomain. We tag each image with the corresponding short commit hash (e.g.fbc7a69) and the application version (e.g.0.5.5-dev1). We also tag the most recent image withlatest. To leverage this,docker pull from our image repository.

docker pull downloads.unstructured.io/unstructured-io/unstructured-api:latest

Once pulled, you can launch the container as a web app on localhost:8000.

docker run -p 8000:8000 -d --rm --name unstructured-api downloads.unstructured.io/unstructured-io/unstructured-api:latest

You can pass in a PORT variable to run the server on a different port in the container.

docker run -p 9500:9500 -d --rm --name unstructured-api -e PORT=9500 downloads.unstructured.io/unstructured-io/unstructured-api:latest

See oursecurity policy forinformation on how to report security vulnerabilities.

We’ve partnered with Scarf (https://scarf.sh) to collect anonymized user statistics to understand which features our community is using and how to prioritize product decision-making in the future. To learn more about how we collect and use this data, please read ourPrivacy Policy.