当前支持评估数据集

安装

pip install aigc_evals>=0.0.3# 源码安装git clone -b dev https://github.com/ssbuild/aigc_evals.gitpip install -e .

修改环境变量

auto_eval/config.py设置openai url等参数或者export OPENAI_API_KEY="your key"export OPENAI_API_BASE="http://192.168.2.180:8081/v1"

修改自定义评估模型

langchain/chat_model/chatglm2-6b-int4:  class: aigc_evals.completion_fns.langchain_llm:LangChainChatModelCompletionFn  args:    llm: ChatOpenAI    chat_model_kwargs:      model_name: chatglm2-6b-int4      model_kwargs: # langchain 未明确实现的参数        adapter_model: default        top_k: 1      max_retries: 10      request_timeout: 200      top_p: 1.0      temperature: 1.0      max_tokens: 2000

一键评估

cd auto_evalpython run_ceval.py

cd auto_evalpython run_cmmlu.py

wget https://people.eecs.berkeley.edu/~hendrycks/data.tar下载mmlu数据集cd auto_evalpython run_mmlu.py

cd auto_evalpython run_bleu.py

cd auto_evalpython run_rouge.py

cd auto_evalpython run_struct.py

ceval 评测结果

exec_aigc_evals 使用帮助

exec_aigc_evals --helpusage: exec_aigc_evals [-h] [--extra_eval_params EXTRA_EVAL_PARAMS] [--max_samples MAX_SAMPLES] [--cache CACHE]                       [--visible VISIBLE] [--seed SEED] [--user USER] [--record_path RECORD_PATH]                       [--log_to_file LOG_TO_FILE] [--registry_path REGISTRY_PATH] [--debug DEBUG]                       [--local-run LOCAL_RUN] [--http-run HTTP_RUN] [--http-run-url HTTP_RUN_URL]                       [--http-batch-size HTTP_BATCH_SIZE] [--http-fail-percent-threshold HTTP_FAIL_PERCENT_THRESHOLD]                       [--dry-run DRY_RUN] [--dry-run-logging DRY_RUN_LOGGING]                       completion_fn evalRun evals through the APIpositional arguments:  completion_fn         One or more CompletionFn URLs, separated by commas (,). A CompletionFn can either be the name                        of a model available in the OpenAI API or a key in the registry (see                        evals/registry/completion_fns).  eval                  Name of an eval. See registry.optional arguments:  -h, --help            show this help message and exit  --extra_eval_params EXTRA_EVAL_PARAMS  --max_samples MAX_SAMPLES  --cache CACHE  --visible VISIBLE  --seed SEED  --user USER  --record_path RECORD_PATH  --log_to_file LOG_TO_FILE                        Log to a file instead of stdout  --registry_path REGISTRY_PATH                        Path to the registry  --debug DEBUG  --local-run LOCAL_RUN                        Enable local mode for running evaluations. In this mode, the evaluation results are stored                        locally in a JSON file. This mode is enabled by default.  --http-run HTTP_RUN   Enable HTTP mode for running evaluations. In this mode, the evaluation results are sent to a                        specified URL rather than being stored locally or in Snowflake. This mode should be used in                        conjunction with the '--http-run-url' and '--http-batch-size' arguments.  --http-run-url HTTP_RUN_URL                        URL to send the evaluation results when in HTTP mode. This option should be used in                        conjunction with the '--http-run' flag.  --http-batch-size HTTP_BATCH_SIZE                        Number of events to send in each HTTP request when in HTTP mode. Default is 1, i.e., send                        events individually. Set to a larger number to send events in batches. This option should be                        used in conjunction with the '--http-run' flag.  --http-fail-percent-threshold HTTP_FAIL_PERCENT_THRESHOLD                        The acceptable percentage threshold of HTTP requests that can fail. Default is 5, meaning 5%                        of total HTTP requests can fail without causing any issues. If the failure rate goes beyond                        this threshold, suitable action should be taken or the process will be deemed as failing, but                        still stored locally.  --dry-run DRY_RUN  --dry-run-logging DRY_RUN_LOGGING

Licenses