Repository files navigation

🌐Web •📹Video

• Table of Contents
• 🔔News
• 🌟Overview
• 🚀Quick Start
  • Step1: Environment Setup
    • 🔩Manual Environment Configuration
    • 🐳Building With Docker Image
  • Step2: Start with Examples
    • 🖊️Start with YAML
    • 🖊️Start with Python
    • 🖊️Start with Web UI
• 🔍Further Usage
  • 💡Extraction Task Support
    • 1. Named Entity Recognition
    • 2. Relation Extraction
    • 3. Event Extraction
    • 4. Triple Extraction
      • Knowledge Graph
    • 5. Open Domain IE
  • 💡Data Source Support
  • 💡Extraction Model Support
  • 💡Extraction Method Support
  • 💡Knowledge Base Configuration
    • 1. Schema Repository
    • 2. Case Repository
• 🛠️Network Issue Solutions
• 🎉Contributors
• 🌻Acknowledgement

• [2025/02] We support the local deployment of theDeepSeek-R1 series in addition to the existing API service, as well as vllm acceleration for other LLMs.
• [2025/01]OneKE is accepted by WWW 2025 Demonstration Track 🎉🎉🎉.
• [2024/12] We open source theOneKE framework, supporting multi-agent knowledge extraction across various scenarios.
• [2024/04] We release a new bilingual (Chinese and English) schema-based information extraction model calledOneKE based on Chinese-Alpaca-2-13B.

OneKE is a flexible dockerized system for schema-guided knowledge extraction, capable of extracting information from the web and raw PDF books across multiple domains like science and news. It employs a collaborative multi-agent approach and includes a user-customizable knowledge base to enable tailored extraction. Embark on your information extraction journey with OneKE!

OneKE currently offers the following features:

• Various IE Tasks Support
• Various Data Sources Support
• Various LLMs Support
• Various Extraction Method Support
• User-Configurable Knowledge Base

We have developed a webpage demo for OneKE with Gradio, clickhere try information extraction in an intuitive way.

Note: The demo only displays OneKE's basic capabilities for efficiency. Consider the local deployment steps below for further features.

OneKE supports both manual and docker image environment configuration, choose your preferred method to build.

Conda virtual environments offer a light and flexible setup.

Prerequisites

• Anaconda Installation
• GPU support (recommended CUDA version: 12.4)

Configure Steps

1. Clone the repository:

git clone https://github.com/zjunlp/OneKE.git

2. Enter the working directory, and all subsequent commands should be executed in this directory.

cd OneKE

3. Create a virtual environment usingAnaconda.

conda create -n oneke python=3.9conda activate oneke

4. Install all required Python packages.

pip install -r requirements.txt# If you encounter network issues, consider setting up a domestic mirror for pip.

Docker image provides greater reliability and stability.

Prerequisites

• Docker Installation
• NVIDIA Container Toolkit
• GPU support (recommended CUDA version: 12.4)

Configure Steps

1. Clone the repository:

git clone https://github.com/zjunlp/OneKE.git

2. Pull the docker image from the mirror repository.

docker pull zjunlp/oneke:v4# If you encounter network issues, consider setting up domestic registry mirrors for docker.

3. Launch a container from the image.

docker run --gpus all \  -v ./OneKE:/app/OneKE \  -it oneke:v4 /bin/bash

If using locally deployed models, ensure the local model path is mapped to the container:

docker run --gpus all \  -v ./OneKE:/app/OneKE \  -v your_local_model_path:/app/model/your_model_name \  -it oneke:v4 /bin/bash

Map anynecessary local files to the container paths as shown above, and usecontainer paths in your code and execution.

Upon starting, the container will enter the/app/OneKE directory as its working directory. Just modify the code locally as needed, and the changes will sync to the container through mapping.

We offer three quick-start options. Choose your preferred method to swiftly explore OneKE with predefined examples.

Note:

• Ensure that your working directory is set to theOneKE folder, whether in a virtual environment or a docker container.
• Refer tohere to resolve thenetwork issues. If you have more questions, feel free to open an issue with us.

Step1: Prepare the configuration file

Several YAML configuration files are available in theexamples/config. These extraction scenarios cover different extraction data, methods, and models, allowing you to easily explore all the features of OneKE.

Web News Extraction:

Here is the example for the web news knowledge extraction scenario, with the source extraction text inHTML format:

# model configurationmodel:category:DeepSeek# model category, chosen from ChatGPT, DeepSeek, LLaMA, Qwen, ChatGLM, MiniCPM, OneKE.model_name_or_path:deepseek-chat# model name, chosen from deepseek-chat and deepseek-reasoner. Choose deepseek-chat to use DeepSeek-V3 or choose deepseek-reasoner to use DeepSeek-R1.api_key:your_api_key# your API key for the model with API service. No need for open-source models.base_url:https://api.deepseek.com# base URL for the API service. No need for open-source models.# extraction configurationextraction:task:Base# task type, chosen from Base, NER, RE, EE.instruction:Extract key information from the given text.# description for the task. No need for NER, RE, EE task.use_file:true# whether to use a file for the input text. Default set to false.file_path:./data/input_files/Tulsi_Gabbard_News.html# path to the input file. No need if use_file is set to false.output_schema:NewsReport# output schema for the extraction task. Selected the from schema repository.mode:customized# extraction mode, chosen from quick, detailed, customized. Default set to quick. See src/config.yaml for more details.update_case:false# whether to update the case repository. Default set to false.show_trajectory:false# whether to display the extracted intermediate steps

Book News Extraction:

Here is the example for the book news extraction scenario, with the source extraction text inPDF format:

model:# Recommend using ChatGPT or DeepSeek APIs for complex IE task.category:ChatGPT# model category, chosen from ChatGPT, DeepSeek, LLaMA, Qwen, ChatGLM, MiniCPM, OneKE.model_name_or_path:gpt-4o-mini# model name, chosen from the model list of the selected category.api_key:your_api_key# your API key for the model with API service. No need for open-source models.base_url:https://api.openai.com/v1# # base URL for the API service. No need for open-source models.extraction:task:Base# task type, chosen from Base, NER, RE, EE.instruction:Extract main characters and background setting from this chapter.# description for the task. No need for NER, RE, EE task.use_file:true# whether to use a file for the input text. Default set to false.file_path:./data/input_files/Harry_Potter_Chapter1.pdf#  # path to the input file. No need if use_file is set to false.mode:quick# extraction mode, chosen from quick, detailed, customized. Default set to quick. See src/config.yaml for more details.update_case:false# whether to update the case repository. Default set to false.show_trajectory:false# whether to display the extracted intermediate steps

Themodel section contains information about the extraction model, while theextraction section configures the settings for the extraction process.

You can choose an existing configuration file or customize the extraction settings as you wish. Note that when using an API service like ChatGPT and DeepSeek, pleaseset your API key.

Step2: Run the shell script

Specify the configuration file path and run the code to start the extraction process.

config_file=your_yaml_file_path# configuration file path, use the container path if inside a containerpython src/run.py --config$config_file# start extraction, executed in the OneKE directory

If you want to deploy the local models using vllm, run the following code:

config_file=your_yaml_file_path# REMEMBER to set vllm_serve to TRUE!python src/models/vllm_serve.py --config$config_file# deploy local model via vllm, executed in the OneKE directorypython src/run.py --config$config_file# start extraction, executed in the OneKE directory

Refer tohere to get an overview of the knowledge extraction results.

You can also try OneKE by directly running theexample.py file located in theexample directory. Specifically, execute the following commands:

python examples/example.py

This will complete a basic NER task, with the extraction results printed upon completion. You can further modify the code inexample.py to suit your extraction task setting or to access detailed extraction trajectory.

Named Entity Extraction:

Specifically, we present a NER case in theexample.py file:

importsyssys.path.append("./src")frommodelsimport*frompipelineimport*importjson# model configurationmodel=ChatGPT(model_name_or_path="gpt-4o-mini",api_key="your_api_key")pipeline=Pipeline(model)# extraction configurationTask="NER"Text="Finally , every other year , ELRA organizes a major conference LREC , the International Language Resources and Evaluation Conference."Constraint=nationality,countrycapital,placeofdeath,children,locationcontains,placeofbirth,placelived,administrativedivisionofcountry,countryofadministrativedivisions,company,neighborhoodof,companyfounders# get extraction resultresult,trajectory,frontend_schema,frontend_res=pipeline.get_extract_result(task=Task,text=Text,constraint=Constraint)print("Trajectory:",json.dumps(trajectory,indent=4))

First, select an appropriate extraction model, then complete the configuration of extraction parameters (such as extraction task, extraction text, etc.). Finally, call theget_extract_result function of thePipeline class to perform information extraction and obtain the final results.

Refer tohere to get an overview of the knowledge extraction results.

Note: Before starting with the web UI, make sure the packagegradio 4.44.0 is already installed in yourEnvironment.

Step1: Execute Command

Execute the following commands in theOneKE directory:

python src/webui.py

Step2: Open your Web Browser

The front-end is built with Gradio, and the default port of Gradio is 7860. Therefore, please enter the following URL in your browser's address bar to open the web interface:

 http://127.0.0.1:7860

Similarly, you can visually configure tasks and obtain results through the front-end interface.

1. 🎲 Quick Start with an Example 🎲: Quickly get a simple example to try OneKE.
2. Submit: After configuring your LLM, parameters, and tasks, click this button to run OneKE.
3. Clear: When a task is completed, click this button to restore the initial state.

You can try different types of information extraction tasks within the OneKE framework.

In subsequent code processing, we categorize tasks into four types:NER for Named Entity Recognition,RE for Relation Extraction,EE for Event Extraction,Triple for Triple Extraction, andBase for any other user-defined open-domain extraction tasks.

Named entity recognition seeks to locate and classify named entities mentioned in unstructured text into pre-defined entity types such as person names, organizations, locations, organizations, etc.

Refer to the case defined inexamples/config/NER.yaml as an example:

In this task setting,Text represents the text to be extracted, whileEntity Types denote the constraint on the types of entities to be extracted. Accordingly, we set thetext andconstraint attributes in the YAML file to their respective values.

Next, follow the steps below to complete the NER task:

• Complete./examples/config/NER.yaml:

  configure the necessary model and extraction settings.

• Run the shell script below:

  config_file=./examples/config/NER.yamlpython src/run.py --config$config_file

  ( Refer toissues for any network issues. )

The final extraction result should be:

Clickhere to obtain the raw results injson format.

Note: The actual extraction results may not exactly match this due to LLM randomness.

The result indicates that, given the text and entity type constraint, entities of typeconference have been extracted:ELRA,conference,International Language Resources and Evaluation Conference.

You can either specify entity type constraints or omit them. Without constraints, OneKE will extract all entities from the sentence.

Relationship extraction is the task of extracting semantic relations between entities from a unstructured text.

Refer to the case defined inexamples/config/RE.yaml as an example:

In this task setting,Text represents the text to be extracted, whileRelation Types denote the constraint on the types of relations of entities to be extracted. Accordingly, we set thetext andconstraint attributes in the YAML file to their respective values.

Next, follow the steps below to complete the RE task:

• Complete./examples/config/RE.yaml:configure the necessary model and extraction settings
• Run the shell script below:

  config_file=./examples/config/RE.yamlpython src/run.py --config$config_file

  ( Refer toissues for any network issues. )

The final extraction result should be:

Clickhere to obtain the raw results injson format.

Note: The actual extraction results may not exactly match this due to LLM randomness.

The result indicates that, the relationCountry-Capital is extracted from the given text based on the relation list, accompanied by the corresponding head entityGuinea and tail entityConakry, which denotes thatConakry is the capital of Guinea.

You can either specify relation type constraints or omit them. Without constraints, OneKE will extract all relation triples from the sentence.

Event extraction is the task to extract event type, event trigger words, and event arguments from a unstructed text, which is a more complex IE task compared to the first two.

Refer to the case defined inexamples/config/EE.yaml as an example:

The extraction text is:

UConn Health , an academic medical center , says in a media statement that it identified approximately 326,000 potentially impacted individuals whose personal information was contained in the compromised email accounts.

while the event type constraint is formatted as follows:

Each event type has its own corresponding event arguments.

Next, follow the steps below to complete the EE task:

• Complete./examples/config/EE.yaml:configure the necessary model and extraction settings
• Run the shell script below:

  config_file=./examples/config/EE.yamlpython src/run.py --config$config_file

  ( Refer toissues for any network issues. )

The final extraction result should be:

Clickhere to obtain the raw results injson format.

Note: The actual extraction results may not exactly match this due to LLM randomness.

The extraction results show that thedata breach event is identified using the triggercompromised, and the specific contents of different event arguments such ascompromised data andvictim have also been extracted.

You can either specify event constraints or omit them. Without constraints, OneKE will extract all events from the sentence.

Triple Extraction identifies subject-predicate-object triples in text. A triple is a fundamental data structure in information extraction, representing a piece of knowledge or a fact. Knowledge Graph (KG) can be quickly constructed after the Triple Extraction.

Here is an example:

The final extraction result should be:

Let's start in OneKE ~

The constraint can be customed as multiple styles, and it's formatted as follows:

• Defineentity types only:

  If you only need to specify the entity types, theconstraint should be a single list of strings representing the different entity types.

["Person","Place","Event","property"]

• Defineentity types andrelation types:

  If you need to specify both entity types and relation types, theconstraint should be a nested list. The first list contains the entity types, and the second list contains the relation types.

[["Person","Place","Event","property"], ["Interpersonal","Located","Ownership","Action"]]

• Definesubject entities types,relation types, andobject entities types:

  If you need to define the types of subject entities, relation types, and object entities, theconstraint should be a nested list. The first list contains the subject entity types, the second list contains the relation types, and the third list contains the object entity types.

[["Person"], ["Interpersonal","Ownership"], ["Person","property"]]

Next, follow the steps below to complete the Triple extraction task:

• Complete./examples/config/Triple2KG.yaml:

  configure the necessary model and extraction settings.

• Run the shell script below:

  config_file=./examples/config/Triple2KG.yamlpython src/run.py --config$config_file

  ( Refer toissues for any network issues. )

Here is anexample to start. And access a rawresults in JSON format here.

⚠️ Warning: If you do not intend to build a Knowledge Graph, make sure to remove or comment out the construct field in the yaml file. This will help avoid errors related to database connection issues.

✨ If you need to construct your Knowledge Graph (KG)with your Triple Extraction result, you can refer to thisexample for guidance. Mimic this example and add theconstruct field. Just update the field with your own database parameters.

construct:# (Optional) If you want to construct a Knowledge Graph, you need to set the construct field, or you must delete this field.database:Neo4j# your database type.url:neo4j://localhost:7687# your database URL，Neo4j's default port is 7687.username:your_username# your database username.password:"your_password"# your database password.

Once your database is set up, you can access your graph database through a browser. For Neo4j, the web interface connection URL is usually:

http://localhost:7474/browser

For additional information regarding the Neo4j database, please refer to it'sdocumentation.

⚠️ Warning Again: If you do not intend to build a Knowledge Graph, make sure to remove or comment out the construct field in the yaml file. This will help avoid errors related to database connection issues.

This type of task is represented asBase in the code, signifying any other user-defined open-domain extraction tasks.

We refer to theexample above for guidance.

In the context of customizedWeb News Extraction, we first set the extraction instruction toExtract key information from the given text, and provide the file path to extract content from the file. We specify the output schema from the schema repository as the predefinedNewsReport, and then proceed with the extraction.

Next, follow the steps below to complete this task:

• Complete./examples/config/NewsExtraction.yaml:configure the necessary model and extraction settings
• Run the shell script below:

  config_file=./examples/config/NewsExtraction.yamlpython src/run.py --config$config_file

  ( Refer toissues for any network issues. )

Here is an excerpt of the extracted content:

Clickhere to obtain the raw results injson format.

Note: The actual extraction results may not exactly match this due to LLM randomness.

In contrast to eariler tasks, theBase-Type Task requires you to provide an explicitInstruction that clearly defines your extraction task, while not allowing the setting ofconstraint values.

You can choose source texts of various lengths and forms for extraction.

In practice, you can use the YAML file configuration to handle different types of text input:

• Plain Text: Setuse_file tofalse and enter the text to be extracted in thetext field.For example:

  use_file:falsetext:Finally , every other year , ELRA organizes a major conference LREC , the International Language Resources and Evaluation Conference .

• File Content: Setuse_file totrue and specify the file path infile_path for the text to be extracted.For example:

  use_file:truefile_path:./data/input_files/Tulsi_Gabbard_News.html

You can choose from various open-source or proprietary model APIs to perform information extraction tasks.

Note: For complex IE tasks, we recommend using powerful models likeOpenAI's or orlarge-scale open-source LLMs.

Note: We recommend deploying the DeepSeek-R1 models with VLLM.

In practice, you can use the YAML file configuration to employ various LLMs:

• API Service: Set themodel_name_or_path to the available model name provided by the company, and enter yourapi_key as well as thebase_url.For exmaple:

  model:category:DeepSeek# model category, chosen from ChatGPT and  DeepSeekmodel_name_or_path:deepseek-chat# model name, chosen from deepseek-chat and deepseek-reasoner. Choose deepseek-chat to use DeepSeek-V3 or choose deepseek-reasoner to use DeepSeek-R1.api_key:your_api_key# your API key for the model with API service.base_url:https://api.deepseek.com# base URL for the API service. No need for open-source models.

• Local Deploy: Set themodel_name_or_path to either the model name on Hugging Face or the path to the local model. We support using eitherTransformer orvllm to access the models.
  • Transformer Example:

    model:category:LLaMA# model category, chosen from LLaMA, Qwen, ChatGLM, MiniCPM, OneKE.model_name_or_path:meta-llama/Meta-Llama-3-8B-Instruct# model name to download from huggingface or use the local model path.vllm_serve:false# whether to use the vllm. Default set to false.

    Note that the category of deployment modelmust be chosen from LLaMA, Qwen, ChatGLM, MiniCPM, OneKE.
  • VLLM Example:

    model:category:DeepSeek# model categorymodel_name_or_path:meta-llama/Meta-Llama-3-8B-Instruct# model name to download from huggingface or use the local model path.vllm_serve:true# whether to use the vllm. Default set to false.

    Note that theDeepSeek-R1 series models only supportVLLM deployment. Remember tostart the VLLM service before running the extraction task. The reference code is as follows:

    config_file=your_yaml_file_path# REMEMBER to set vllm_serve to TRUE!python src/models/vllm_serve.py --config$config_file# deploy local model via vllm, executed in the OneKE directory

    You can also run the commandvllm serve model_name_or_path directly to start the VLLM service. See theofficial documents for more details.

You can freely combine different extraction methods to complete the information extraction task.

The configuration for detail extraction methods and mode information can be found insrc/config.yaml. You can customize the extraction methods by modifying thecustomized within this file and set themode to customize in an external configuration file.

For example, first configure thesrc/config.yaml as follows:

# src/config.yamlcustomized:schema_agent:get_deduced_schemaextraction_agent:extract_information_directreflection_agent:reflect_with_case

Then, set themode of your custom extraction task inexamples/customized.yaml tocustomized:

# examples/customized.yamlmode:customized

This allows you to experience the customized extraction methods.

Tips:

• For longer text extraction tasks, we recommend using thedirect mode to avoid issues like attention dispersion and increased processing time.
• For shorter tasks requiring high accuracy, you can try thestandard mode to ensure precision.

You can view the predefined schemas within thesrc/modules/knowledge_base/schema_repository.py file. The Schema Repository is designed to be easily extendable. You just need to define your output schema in the form of a pydantic class following the format defined in the file, and it can be directly used in subsequent extractions.

For example, add a new schema in the schema repository:

# src/modules/knowledge_base/schema_repository.pyclassChemicalSubstance(BaseModel):name:str=Field(description="Name of the chemical substance")formula:str=Field(description="Molecular formula")appearance:str=Field(description="Physical appearance")uses:List[str]=Field(description="Primary uses")hazards:str=Field(description="Hazard classification")classChemicalList(BaseModel):chemicals:List[ChemicalSubstance]=Field(description="List of chemicals")

Then, set the method forschema_agent undercustomized toget_retrieved_schema insrc/config.yaml. Finally, set themode tocustomized in the external configuration file to enable custom schema extraction.

In this example, the extraction results will be a list ofchemical substances that strictly adhere to the defined schema, ensuring a high level of accuracy and flexibility in the extraction results.

Note that the names of newly created objectsshould not conflict with existing ones.

You can directly view the case storage in thesrc/modules/knowledge_base/case_repository.json file, but we do not recommend modifying it directly.

The Case Repository is automatically updated with each extraction process once settingupdate_repository toTrue in the configuration file.

When updating the Case Repository, you must provide external feedback to generate case information, either by including truth answer in the configuration file or during the extraction process.

Here is an example:

# examples/config/RE.yamltruth:{"relation_list": [{"head": "Guinea", "tail": "Conakry", "relation": "country capital"}]}# Truth data for the relationupdate_case:true

After extraction, OneKE compares results with the truth answer, generates analysis, and finally stores the case in the repository.

Here are some network issues you might encounter and the corresponding solutions.

• Pip Installation Failure: Use mirror websites, run the command aspip install -i [mirror-source] ....
• Docker Image Pull Failure: Configure the docker daemon to add repository mirrors.
• Nltk Download Failure: Manually download thenltk package and place it in the proper directory.
• Model Dowload Failure: Use theHugging Face Mirror site orModelScope to download model, and specify the local path to the model when using it.

  Note: We useall-MiniLM-L6-v2 model by default for case matching, so it needs to be downloaded during execution. If network issues occur, manually download the model, and update theembedding_model to its local path in thesrc/config.yaml file.

Ningyu Zhang,Haofen Wang, Yujie Luo, Xiangyuan Ru, Kangwei Liu, Lin Yuan, Mengshu Sun, Lei Liang, Zhiqiang Zhang, Jun Zhou, Lanning Wei, Da Zheng, Huajun Chen.

We deeply appreciate the collaborative efforts of everyone involved. We will continue to enhance and maintain this repository over the long term. If you encounter any issues, feel free to submit them to us!

We referenceitext2kg to aid in building the schema repository and utilize tools fromLangChain for file parsing. The experimental datasets we use are curated from theIEPile repository. We appreciate their valuable contributions!