Repository files navigation

Easy Data Generation For Large Language Model(abbreviated as EDG4LLM), A unified tool to generate fine-tuning datasets for LLMs, including questions, answers, and dialogues.

• Latest News
• Introduction
• Features
• Installation
• Quick Start
• Requirements
• Future Development Plans
• Acknowledgments
• License
• Contact me
• Star History

edg4llm is a Python library designed specifically for generating fine-tuning data using large language models. This tool aims to assist users in creating high-quality training datasets efficiently. At its current stage, it mainly supports text data generation. The generated data includes, but is not limited to:

• Question data
• Answer data
• Dialogue data

Withedg4llm, users can easily produce diverse datasets tailored to fine-tuning requirements, significantly enhancing the performance of large language models in specific tasks.

EDG4LLM is a unified tool designed to simplify and accelerate the creation of fine-tuning datasets for large language models. With a focus on usability, efficiency, and adaptability, it offers a range of features to meet diverse development needs while ensuring seamless integration and robust debugging support.

1. Simple to Use: Provides a straightforward interface that allows users to get started without complex configurations.
2. Lightweight: Minimal dependencies and low resource consumption make it efficient and easy to use.
3. Flexibility: Supports a variety of data formats and generation options, allowing customization to meet specific needs.
4. Compatibility: Seamlessly integrates with mainstream large language models and is suitable for various development scenarios.
5. Transparent Debugging: Provides clear and detailed log outputs, making it easy to debug and trace issues effectively.

To installedg4llm, simply run the following command in your terminal:

pip install edg4llm

• Supported Python Versions: Python 3.8 or higher is required for compatibility with this library. Ensure your environment meets this version requirement.

The current version of edg4llm supports the following large language model providers:

• InternLM

  • Developer: Developed by the Shanghai Artificial Intelligence Laboratory.
  • Advantages: InternLM is a series of open-source large language models that offer outstanding reasoning, long-text processing, and tool usage capabilities.

• ChatGLM

  • Developer: Jointly developed by Tsinghua University and Zhipu AI.
  • Advantages: ChatGLM is an open-source, bilingual dialog language model based on the General Language Model (GLM) architecture. It has been trained on a large corpus of Chinese and English text, making it highly effective for generating natural and contextually relevant responses.

• DeepSeek

  • Developer: Developed by the DeepSeek team.
  • Advantages: DeepSeek-V3 is a powerful and cost-effective open-source large language model. It offers top-tier performance, especially in tasks like language generation, question answering, and dialog systems.

• OpenAI ChatGPT

  • Developer: Developed by OpenAI.
  • Advantages: OpenAI's ChatGPT is a highly advanced language model known for its robust text generation capabilities. It has been trained on a vast amount of data, allowing it to generate high-quality and contextually relevant responses.

More providers will be added in future updates to extend compatibility and functionality.

To get started withedg4llm, follow the steps below. These examples demonstrate how to use the library to generate dialogue data based on a specific prompt.

If you want to usequestion mode, please assure that your user_prompt contains the following format:

[    {"question":"AAA"    }]

If you want to useanswer mode, please assure that your user_prompt contains the following format:

[    {"answer":"AAA"    }]

If you want to usedialogue mode, please assure that your user_prompt contains the following format:

[    {{"input":"AAA","output":"BBB"     }}]

1. Install theedg4llm package:

   pip install edg4llm

2. Ensure you have Python version 3.8 or higher.

3. Obtain the necessary API key and base URL for your chosen model provider (e.g., ChatGLM).

4. you can use the CLI to list the supported model_providers and model_names

usage: edg4llm-cli [-h] [--list-providers] [--list-models PROVIDER]View the list of supported models.options:  -h, --help            show this help message and exit  --list-providers      List all supported providers.  --list-models PROVIDER                        View the list of models for a specific provider.

# chatglm_demo.pyimportedg4llmprint(edg4llm.__version__)fromedg4llmimportEDG4LLMapi_key="xxx"base_url="https://open.bigmodel.cn/api/paas/v4/chat/completions"edg=EDG4LLM(model_provider='chatglm',model_name="glm-4-flash",base_url=base_url,api_key=api_key)# 设置测试数据system_prompt="""你是一个精通中国古代诗词的古文学大师"""user_prompt="""    目标: 1. 请生成过年为场景的连续多轮对话记录            2. 提出的问题要多样化。            3. 要符合人类的说话习惯。            4. 严格遵循规则: 请以如下格式返回生成的数据, 只返回JSON格式，json模板:                [                    {{                        "input":"AAA","output":"BBB"                    }}                ]                其中input字段表示一个人的话语, output字段表示专家的话语"""num_samples=1# 只生成一个对话样本# 调用 generate 方法生成对话data_dialogue=edg.generate(task_type="dialogue",system_prompt=system_prompt,user_prompt=user_prompt,num_samples=num_samples)

# chatglm_demo.pyimportedg4llmprint(edg4llm.__version__)fromedg4llmimportEDG4LLMapi_key="xxx"base_url="https://open.bigmodel.cn/api/paas/v4/chat/completions"edg=EDG4LLM(model_provider='chatglm',model_name="glm-4-flash",base_url=base_url,api_key=api_key)# Set the test datasystem_prompt="""You are a master of ancient Chinese literature, specializing in classical poetry."""user_prompt="""    Goal: 1. Please generate a multi-turn dialogue set in the context of celebrating the Lunar New Year.          2. The questions should be diverse.          3. The dialogue should align with natural human conversational habits.          4. Strictly follow this rule: Please return the generated data in the following format, only in JSON format. JSON template:                [                    {{                        "input":"AAA","output":"BBB"                    }}                ]                Where the input field represents a person's dialogue, and the output field represents the expert's response."""num_samples=1# Generate only one dialogue sample# Call the generate method to generate the dialoguedata_dialogue=edg.generate(task_type="dialogue",system_prompt=system_prompt,user_prompt=user_prompt,num_samples=num_samples)

1. Importing the Library: Import the edg4llm library and verify the version using print(edg4llm.version).

2. Initialization: Use EDG4LLM to initialize the library with the appropriate model provider, model name, base URL, and API key.

3. Prompts:

   • system_prompt defines the behavior or role of the assistant.
   • user_prompt provides specific instructions for generating data.

4. Data Generation: Use the generate method with the following parameters:

   • task_type: Defines the type of task.

     • dialogue: generate the question-answer pairs according to the prompt.
     • question: generate the question data according to the prompt.
     • answer: generate the answer data according to the questions and the prompt.

   • system_prompt and user_prompt: Provide context and task-specific instructions.

   • num_samples: Specifies how many samples to generate.

5. Output: The generated data is returned as a JSON object in the specified format.

This project hasminimal dependencies, requiring only the requests library. Make sure to have the following version installed:

• requests>=2.32.3

• Recording Introduction Video
• Support Gemini2
• Support local large language models
• Support other types of data, such as picture.
• preprocess the base_url and api_key

MIT License - SeeLICENSE for details.

Thank you for usingEDG4LLM! Your support and feedback are invaluable in making this project better.

If you encounter any issues, have suggestions, or simply want to share your thoughts, feel free to:

• Submit an Issue: Visit theIssues Page and describe the problem or suggestion.
• Email Me: You can also reach out directly via email atalannikos768@outlook.com. I'll do my best to respond promptly.

Your contributions and feedback are greatly appreciated. Thank you for helping improve this tool!