API Announcement!
While access to the hosted Unstructured API will remain free, API Keys will soon be required to make requests. To prevent any disruption, get yours here now and start using it today!
Checkout the readme here to get started making API calls. We'd love to hear your feedback, let us know how it goes in our community slack. And stay tuned for improvements to both quality and performance!
Open-Source Pre-Processing Tools for Unstructured Data
The `unstructured` library provides open-source components for pre-processing text documents such as **PDFs**, **HTML** and **Word** Documents. These components are packaged as *bricks* 🧱, which provide users the building blocks they need to build pipelines targeted at the documents they care about. Bricks in the library fall into three categories: - :jigsaw: ***Partitioning bricks*** that break raw documents down into standard, structured elements. - :broom: ***Cleaning bricks*** that remove unwanted text from documents, such as boilerplate and sentence fragments. - :performing_arts: ***Staging bricks*** that format data for downstream tasks, such as ML inference and data labeling. Unstructured also provides the capabilities from `unstructured` as an API. Checkout the [`unstructured-api` repo](https://github.com/Unstructured-IO/unstructured-api) to get started making API calls. You’ll also find instructions there about how to host your own version of the API. ## :eight_pointed_black_star: Quick Start Use the following instructions to get up and running with `unstructured` and test your installation. NOTE: We do not currently support python 3.11, please use an older version. - Install the Python SDK with `pip install "unstructured[local-inference]"` - If you do not need to process PDFs or images, you can run `pip install unstructured` - Install the following system dependencies if they are not already available on your system. Depending on what document types you're parsing, you may not need all of these. - `libmagic-dev` (filetype detection) - `poppler-utils` (images and PDFs) - `tesseract-ocr` (images and PDFs) - `libreoffice` (MS Office docs) - If you are parsing PDFs and want to use a model from the [layoutparser model zoo](https://github.com/Unstructured-IO/unstructured-inference#using-models-from-the-layoutparser-model-zoo), use the instructions [here](https://github.com/Unstructured-IO/unstructured-inference#detectron2). At this point, you should be able to run the following code: ```python from unstructured.partition.auto import partition elements = partition(filename="example-docs/eml/fake-email.eml") print("\n\n".join([str(el) for el in elements])) ``` The following table shows the document types the `unstructured` library currently supports. `partition` will recognize each of these document types and route the document to the appropriate partitioning function. If you already know your document type, you can use the partitioning function listed in the table directly. See our [documentation page](https://unstructured-io.github.io/unstructured/) for more details about the library. | Document Type | Partition Function | Strategies | Table Support | Options | | --- | --- | --- | --- | --- | | CSV Files (`.csv`) | `partition_csv` | N/A | Yes | None | | E-mails (`.eml`) | `partition_eml` | N/A | No | Encoding; Max Partition; Process Attachments | | E-mails (`.msg`) | `partition_msg` | N/A | No | Encoding; Max Partition; Process Attachments | | EPubs (`.epub`) | `partition_epub` | N/A | Yes | Include Page Breaks | | Excel Documents (`.xlsx`/`.xls`) | `partition_xlsx` | N/A | Yes | None | | HTML Pages (`.html`) | `partition_html` | N/A | No | Encoding; Include Page Breaks | | Images (`.png`/`.jpg`) | `partition_image` | `"auto"`, `"hi_res"`, `"ocr_only"` | Yes | Encoding; Include Page Breaks; Infer Table Structure; OCR Languages, Strategy | | Markdown (`.md`) | `partitin_md` | N/A | Yes | Include Page Breaks | | Org Mode (`.org`) | `partition_org` | N/A | Yes | Include Page Breaks | | Open Office Documents (`.odt`) | `partition_odt` | N/A | Yes | None | | PDFs (`.pdf`) | `partition_pdf` | `"auto"`, `"fast"`, `"hi_res"`, `"ocr_only"` | Yes | Encoding; Include Page Breaks; Infer Table Structure; Max Partition; OCR Languages, Strategy | | Plain Text (`.txt`) | `partition_text` | N/A | No | Encoding; Max Partition; Paragraph Grouper | | Power Points (`.ppt`) | `partition_ppt` | N/A | Yes | Include Page Breaks | | Power Points (`.pptx`) | `partition_pptx` | N/A | Yes | Include Page Breaks | | ReStructured Text (`.rst`) | `partition_rst` | N/A | Yes | Include Page Breaks | | Rich Text Files (`.rtf`) | `partition_rtf` | N/A | Yes | Include Page Breaks | | TSV Files (`.tsv`) | `partition_tsv` | N/A | Yes | None | | Word Documents (`.doc`) | `partition_doc` | N/A | Yes | None | | Word Documents (`.docx`) | `partition_docx` | N/A | Yes | None | | Word Documents (`.doc`) | `partition_doc` | N/A | Yes | Include Page Breaks | | Word Documents (`.docx`) | `partition_docx` | N/A | Yes | Include Page Breaks | | XML Documents (`.xml`) | `partition_xml` | N/A | No | Encoding; Max Partition; XML Keep Tags | ## :dizzy: Instructions for using the docker image The following instructions are intended to help you get up and running using Docker to interact with `unstructured`. See [here](https://docs.docker.com/get-docker/) if you don't already have docker installed on your machine. NOTE: we build multi-platform images to support both x86_64 and Apple silicon hardware. `docker pull` should download the corresponding image for your architecture, but you can specify with `--platform` (e.g. `--platform linux/amd64`) if needed. We build Docker images for all pushes to `main`. We tag each image with the corresponding short commit hash (e.g. `fbc7a69`) and the application version (e.g. `0.5.5-dev1`). We also tag the most recent image with `latest`. To leverage this, `docker pull` from our image repository. ```bash docker pull quay.io/unstructured-io/unstructured:latest ``` Once pulled, you can create a container from this image and shell to it. ```bash # create the container docker run -dt --name unstructured quay.io/unstructured-io/unstructured:latest # this will drop you into a bash shell where the Docker image is running docker exec -it unstructured bash ``` You can also build your own Docker image. If you only plan on parsing one type of data you can speed up building the image by commenting out some of the packages/requirements necessary for other data types. See Dockerfile to know which lines are necessary for your use case. ```bash make docker-build # this will drop you into a bash shell where the Docker image is running make docker-start-bash ``` Once in the running container, you can try things out directly in Python interpreter's interactive mode. ```bash # this will drop you into a python console so you can run the below partition functions python3 >>> from unstructured.partition.pdf import partition_pdf >>> elements = partition_pdf(filename="example-docs/layout-parser-paper-fast.pdf") >>> from unstructured.partition.text import partition_text >>> elements = partition_text(filename="example-docs/fake-text.txt") ``` ## :coffee: Installation Instructions for Local Development The following instructions are intended to help you get up and running with `unstructured` locally if you are planning to contribute to the project. * Using `pyenv` to manage virtualenv's is recommended but not necessary * Mac install instructions. See [here](https://github.com/Unstructured-IO/community#mac--homebrew) for more detailed instructions. * `brew install pyenv-virtualenv` * `pyenv install 3.8.17` * Linux instructions are available [here](https://github.com/Unstructured-IO/community#linux). * Create a virtualenv to work in and activate it, e.g. for one named `unstructured`: `pyenv virtualenv 3.8.17 unstructured`