Repository files navigation

AI Model Optimization Toolkit for the ONNX Runtime

Given a model and targeted hardware, Olive (abbreviation ofOnnxLIVE) composes the best suitable optimization techniques to output the most efficient ONNX model(s) for inferencing on the cloud or edge, while taking a set of constraints such as accuracy and latency into consideration.

• Reduce frustration of manual trial-and-error model optimization experimentation. Define your target and precision and let Olive automatically produce the best model for you.
• 40+ built-in model optimization components covering industry-leading techniques across model compression, optimization, finetuning, and compilation.
• Easy-to-use CLI for common model optimization tasks.
• Workflows to orchestrate model transformations and optimizations steps.
• Support for compiling LoRA adapters forMultiLoRA serving.
• Seamless integration withHugging Face andAzure AI.
• Built-incaching mechanism toimprove productivity.

Here are some recent videos, blog articles and labs that highlight Olive:

• [ Feb 2025 ]New Notebook available - Finetune and Optimize DeepSeek R1 with Olive 🐋
• [ Nov 2024 ]Democratizing AI Model optimization with the new Olive CLI
• [ Nov 2024 ]Unlocking NLP Potential: Fine-Tuning with Microsoft Olive (Ignite Pre-Day Lab PRE016)
• [ Nov 2024 ]Olive supports generating models for MultiLoRA serving on the ONNX Runtime
• [ Oct 2024 ]Windows Dev Chat: Optimizing models from Hugging Face for the ONNX Runtime (video)
• [ May 2024 ]AI Toolkit - VS Code Extension that uses Olive to fine tune models

For a full list of news and blogs, read thenews archive.

The following notebooks are available that demonstrate key optimization workflows with Olive and include the application code to inference the optimized models on the ONNX Runtime.

If you prefer using the command line directly instead of Jupyter notebooks, we've outlined the quickstart commands here.

We recommend installing Olive in avirtual environment or aconda environment.

pip install olive-ai[auto-opt]pip install transformers onnxruntime-genai

In this quickstart you'll be optimizingHuggingFaceTB/SmolLM2-135M-Instruct, which has many model files in the Hugging Face repo for different precisions that are not required by Olive. To minimize the download, cache the original Hugging Face model files (safetensors and configuration) in the main folder of the Hugging Face repo using:

huggingface-cli download HuggingFaceTB/SmolLM2-135M-Instruct*.json*.safetensors*.txt

Next, run the automatic optimization:

olive auto-opt \    --model_name_or_path HuggingFaceTB/SmolLM2-135M-Instruct \    --output_path models/smolm2 \    --device cpu \    --provider CPUExecutionProvider \    --use_ort_genai \    --precision int4 \    --log_level 1

olive auto-opt`--model_name_or_path HuggingFaceTB/SmolLM2-135M-Instruct`--output_path models/smolm2`--device cpu`--provider CPUExecutionProvider`--use_ort_genai`--precision int4`--log_level1

The automatic optimizer will:

1. Acquire the model from the local cache (note: if you skipped the model download step then the entire contents of the Hugging Face model repo will be downloaded).
2. Capture the ONNX Graph and store the weights in an ONNX data file.
3. Optimize the ONNX Graph.
4. Quantize the model toint4 using RTN method.

Olive can automatically optimize popular modelarchitectures like Llama, Phi, Qwen, Gemma, etc out-of-the-box -see detailed list here. Also, you can optimize other model architectures by providing details on the input/outputs of the model (io_config).

The ONNX Runtime (ORT) is a fast and light-weight cross-platform inference engine with bindings for popular programming language such as Python, C/C++, C#, Java, JavaScript, etc. ORT enables you to infuse AI models into your applications so that inference is handled on-device.

The following code creates a simple console-based chat interface that inferences your optimized model -select Python and/or C# to expand the code:

# app.pyimportonnxruntime_genaiasogmodel_folder="models/smolm2/model"# Load the base model and tokenizermodel=og.Model(model_folder)tokenizer=og.Tokenizer(model)tokenizer_stream=tokenizer.create_stream()# Set the max length to something sensible by default,# since otherwise it will be set to the entire context lengthsearch_options= {}search_options['max_length']=200chat_template="<|im_start|>user\n{input}<|im_end|>\n<|im_start|>assistant\n"# Keep asking for input prompts in a loopwhileTrue:text=input("Prompt (Use quit() to exit): ")ifnottext:print("Error, input cannot be empty")continueiftext=="quit()":break# Generate prompt (prompt template + input)prompt=f'{chat_template.format(input=text)}'# Encode the prompt using the tokenizerinput_tokens=tokenizer.encode(prompt)# Create params and generatorparams=og.GeneratorParams(model)params.set_search_options(**search_options)generator=og.Generator(model,params)# Append input tokens to the generatorgenerator.append_tokens(input_tokens)print("")print("Output: ",end='',flush=True)# Stream the outputtry:whilenotgenerator.is_done():generator.generate_next_token()new_token=generator.get_next_tokens()[0]print(tokenizer_stream.decode(new_token),end='',flush=True)exceptKeyboardInterrupt:print("  --control+c pressed, aborting generation--")print()print()delgenerator

mkdir ortappcd ortappdotnet new consoledotnet add package Microsoft.ML.OnnxRuntimeGenAI--version0.5.2

// Program.csusingMicrosoft.ML.OnnxRuntimeGenAI;internalclassProgram{privatestaticvoidMain(string[]args){stringmodelPath@"models/smolm2/model";Console.Write("Loading model from "+modelPath+"...");usingModelmodel=new(modelPath);Console.Write("Done\n");usingTokenizertokenizer=new(model);usingTokenizerStreamtokenizerStream=tokenizer.CreateStream();while(true){Console.Write("User:");stringprompt="<|im_start|>user\n"+Console.ReadLine()+"<|im_end|>\n<|im_start|>assistant\n";varsequences=tokenizer.Encode(prompt);usingGeneratorParamsgParams=newGeneratorParams(model);gParams.SetSearchOption("max_length",200);usingGeneratorgenerator=new(model,gParams);generator.AppendTokenSequences(sequences);Console.Out.Write("\nAI:");while(!generator.IsDone()){generator.GenerateNextToken();vartoken=generator.GetSequence(0)[^1]                Console.Out.Write(tokenizerStream.Decode(token));Console.Out.Flush();}Console.WriteLine();}}}

dotnet run

• Documentation
• Examples

• We welcome contributions! Please read thecontribution guidelines for more details on how to contribute to the Olive project.
• For feature requests or bug reports, file aGitHub Issue.
• For general discussion or questions, useGitHub Discussions.

Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under theMIT License.