Install ONNX Runtime generate() API

• Python package installation
  • CPU
  • DirectML
  • CUDA
    • CUDA 12
    • CUDA 11
• Nuget package installation
  • Pre-requisites
    • ONNX Runtime dependency
  • CPU
  • CUDA
  • DirectML

Python package installation

Note: only one of these sets of packages (CPU, DirectML, CUDA) should be installed in your environment.

CPU

pipinstallonnxruntime-genai

DirectML

pipinstallonnxruntime-genai-directml

CUDA

If you are installing the CUDA variant of onnxruntime-genai, the CUDA toolkit must be installed.

The CUDA toolkit can be downloaded from theCUDA Toolkit Archive.

Ensure that theCUDA_PATH environment variable is set to the location of your CUDA installation.

CUDA 12

pipinstallonnxruntime-genai-cuda

CUDA 11

To use this API with CUDA 11, you need tobuild and install from source.

Nuget package installation

Note: install only one of these packages (CPU, DirectML, CUDA) in your project.

Pre-requisites

ONNX Runtime dependency

ONNX Runtime generate() versions 0.3.0 and earlier came bundled with the core ONNX Runtime binaries. From version 0.4.0 onwards, the packages are separated to allow a more flexible developer experience.

CPU

dotnet add package Microsoft.ML.OnnxRuntimeGenAI

CUDA

Note: only CUDA 11 is supported for versions 0.3.0 and earlier, and only CUDA 12 is supported for versions 0.4.0 and later.

dotnet add package Microsoft.ML.OnnxRuntimeGenAI.Cuda

DirectML

dotnet add package Microsoft.ML.OnnxRuntimeGenAI.DirectML