Repository files navigation

Genalog is an open source, cross-platform python package forgenerating document images with synthetic noise that mimics scanned analog documents (thus the namegenalog). You can also add various text degradations to these images. The purpose of this tool is to provide a fast and efficient way to generate synthetic documents from text data by leveraging layout from templates that you create in simple HTML format.

This repo is now in maintenance mode withlimited support.

Genalog has various capabilities:

1. Flexible format Image Generation
2. Custom image degradation
3. Extract Text from Images using Cognitive Search Pipeline
4. Get OCR Performance Metrics

The aim of this project is to provide a complete solution for generating synthetic images from any text data rich in natural language and to imitate most of OCR noises founded in scanned text documents.

Please refer to ourGenalog documentation for more tutorials.

See theGenalog install guide for more details.

To install the latest release:

pip install genalog

We have a dependency onWeasyprint, which in turn has non-python dependencies includingPango,cairo andGDK-PixBuf that need to be installed separately.

So far,Pango,cairo andGDK-PixBuf libraries are available inUbuntu-18.04 and later by default.

If you are running on Windows, MacOS, or other Linux distributions, please seeinstallation instructions from WeasyPrint.

NOTE: If you encounter the errors likeno library called "libcairo-2" was found, this is probably due to the three extra dependencies missing.

The following is a summary of the common applications scenarios of Genalog. Please refer theJupyter notebook examples that make use of the core code base of Genalog and repository utilities.

If you are interested in a full document generation and degration pipeline, please see the following notebook:

Else we have in-depth walkthroughs of each of the module in Genalog.

We also provide notebooks for the complete end-to-end scenario of generating a synthetic dataset connecting all the components of genalog:

1. If you want to use the OCR Capabilities of Azure to Extract Text from the Images You'll require the following resources:

   1. Azure Cognitive Search ServiceQuickstart Guide Here
   2. Azure Blob StorageQuickstart Guide Here

   SeeAzure Docs for more information on Azure Cognitive Search.

Please seeRELEASE.md for more details on the release process.

We usetox to orchestrate most of the CI procedure. This will ensure the maximum environment parity between local dev boxes and remote CI pipelines.

1. git clone https://github.com/microsoft/genalog.git
2. pip install tox
3. To run static analysis:tox -e flake8
4. To run the test suites:tox -e -- -m "not azure"

genalog├────genalog│       ├─── generation                      # generate text images│       ├──── degradation                    # methods for image degradation│       ├──── ocr                            # running the Azure Search Pipeline│       └──── text                           # methods to Align OCR Output Text with ├────devops                                  # CI/CD pipelines├────docs                                    # containing online documentaions├────examples                                # example Jupyter Notebooks for Various ├────tests                                   # tests├────tox.ini                                 # CI orchestration and configurations├────README.md└────LICENSE

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft’s Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party’s policies.

This project has adopted theMicrosoft Open Source Code of Conduct. For more information see theCode of Conduct FAQ or contactopencode@microsoft.com with any additional questions or comments.

This project welcomes contributions and suggestions. Most contributions require you toagree to a Contributor License Agreement (CLA) declaring that you have the right to,and actually do, grant us the rights to use your contribution. For details, visithttps://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you needto provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow theinstructions provided by the bot. You will only need to do this once across all repositories using our CLA.

This project has adopted theMicrosoft Open Source Code of Conduct.For more information see theCode of Conduct FAQor contactopencode@microsoft.com with any additional questions or comments.

If you findgenalog helpful to your work, please consider citing our tool andpaper using the following BibTeX entry:

@article{  gupte2021genalog,  title={Lights, Camera, Action! A Framework to Improve NLP Accuracy over OCR documents},  author={Gupte, Amit and Romanov, Alexey and Mantravadi, Sahitya and Banda, Dalitso and Liu, Jianjie and Khan, Raza and Meenal, Lakshmanan Ramu and Han, Benjamin and Srinivasan, Soundar},  journal={Document Intelligence Workshop at KDD 2021},  year={2021}}

Genalog was originally developed by theMAIDAP team at Microsoft Cambridge NERD in association with the Text Analytics Team in Redmond.