Running Whisper inference using OpenVINO

This PR extendswhisper.cpp to run the Whisper Encoder onOpenVINO supported devices such as CPU, and Intel GPUs (integrated & discrete).

I've tested this on number of platforms, including

• NUC Intel(R) Core(TM) i7-6770HQ ('Skylake' Skull Canyon NUC) running Ubuntu 22.04
• Core(TM) i7-1185G7 ('Tiger Lake' laptop) running Windows 11 Pro
• Core(TM) i7-12700 ('Alder Lake' Beast Canyon NUC) with installed Intel(R) ARC(TM) A770 discrete graphics card, running Windows 11 Pro

For each platform, the performance of using OpenVINO-based encoder gives a great boost in performance over the default encoder -- even for CPU -- and the ability to easily offload to another OpenVINO-supported device by simply specifying a different string at runtime (e.g. "CPU" --> "GPU") is very convenient.

High-level description of changes

This introduction of OpenVINO Encode support is modeled very closely to how whisper.cpp uses CoreML (this should be pretty obvious in the change-set). If the project is built with OpenVINO support, an OpenVINO-specific encoder is pulled into the build and instantiated at application startup time.

Also similar to CoreML, the models required to be present to take advantage of the OpenVINO encoder can be generated using a new python script in 'models' directory.

Just to point out -- something that does differ between CoreML and the new OpenVINO integration is how/when support is enabled at runtime. CoreML is enabled within the call towhisper_init_*. For OpenVINO, because we want the ability to specify a device string ("CPU", "GPU", etc.), I exposed a new API that is dedicated to initializing OpenVINO, given a ctx:

(in whisper.h):

// Given a context, enable use of OpenVINO for encode inference.// openvino_model_path: Optional path to OpenVINO encoder IR model. If set to nullptr,//                      the path will be generated from the ggml model path that was passed//                      in to whisper_init_from_file. For example, if 'path_model' was//                      "/path/to/ggml-base.en.bin", then OpenVINO IR model path will be//                      assumed to be "/path/to/ggml-base.en-encoder-openvino.xml".// openvino_device: OpenVINO device to run inference on ("CPU", "GPU", etc.)// openvino_cache_dir: Optional cache directory that can speed up init time, especially for//                     GPU, by caching compiled 'blobs' there.//                     Set to nullptr if not used.// Returns 1 on success. If OpenVINO is not enabled in build, this// simply returns 0.WHISPER_APIintwhisper_ctx_init_openvino_encoder(structwhisper_context*ctx,constchar*openvino_model_path,constchar*openvino_device,constchar*openvino_cache_dir);

I'm happy to rework this if anyone has a better idea of how to enable OpenVINO support at init time.

main.cpp exposes a new parameter for user to set OpenVINO encode inference device (default is "CPU"):

...elseif (arg=="-oved"||arg=="--ov-e-device")    {params.openvino_encode_device=argv[++i]; }...

And the newwhisper_ctx_init_openvino_encoder API is called right after ctx creation:

// whisper initstructwhisper_context*ctx=whisper_init_from_file(params.model.c_str());if (ctx==nullptr) {fprintf(stderr,"error: failed to initialize whisper context\n");return3;    }// initialize openvino encoder. This has no effect on whisper.cpp builds that don't have OpenVINO configured.whisper_ctx_init_openvino_encoder(ctx,nullptr,params.openvino_encode_device.c_str(),nullptr);

How to generate models and enable OpenVINO for whisper.cpp builds

Here are the instructions for generating the OpenVINO models for use with OpenVINO-enabled builds of whisper.cpp:

• First, setup python virtual env. and install python dependencies. Python 3.10 is recommended.

  Windows:

  cd modelspython -m venv openvino_conv_envopenvino_conv_env\Scripts\activatepython -m pip install --upgrade pippip install -r openvino-conversion-requirements.txt

  Linux and macOS:

  cd modelspython3 -m venv openvino_conv_envsource openvino_conv_env/bin/activatepython -m pip install --upgrade pippip install -r openvino-conversion-requirements.txt

• Generate an OpenVINO encoder model. For example, to generate abase.en model, use:

  python convert-whisper-to-openvino.py --model base.en

  This will produce ggml-base.en-encoder-openvino.xml/.bin IR model files. It's recommended to relocate these to the same folder as ggml models, as that is the default location that the OpenVINO extension will search at runtime.

• Buildwhisper.cpp with OpenVINO support:

  Download OpenVINO package fromrelease page. The recommended version to use is2023.0.0.

  After downloading & extracting package onto your development system, set up required environment by sourcing setupvars script. For example:

  Linux:

  source /path/to/l_openvino_toolkit_ubuntu22_2023.0.0.10926.b4452d56304_x86_64/setupvars.sh

  Windows (cmd):

  C:\Path\To\w_openvino_toolkit_windows_2023.0.0.10926.b4452d56304_x86_64\setupvars.bat

  And then build the project using cmake:

  cd buildcmake -DWHISPER_OPENVINO=1 ..

• Run the examples as usual. For example:

  ./main -m models/ggml-base.en.bin -f samples/jfk.wav...whisper_ctx_init_openvino_encoder: loading OpenVINO model from'models/ggml-base.en-encoder-openvino.xml'whisper_ctx_init_openvino_encoder: first run on a device may take awhile ...whisper_openvino_init: path_model = models/ggml-base.en-encoder-openvino.xml, device = CPU, cache_dir = models/ggml-base.en-encoder-openvino-cachewhisper_ctx_init_openvino_encoder: OpenVINO model loadedsystem_info: n_threads = 4 / 8| AVX = 1| AVX2 = 1| AVX512 = 0| FMA = 1| NEON = 0| ARM_FMA = 0| F16C = 1| FP16_VA = 0| WASM_SIMD = 0| BLAS = 0| SSE3 = 1| VSX = 0| COREML = 0| OPENVINO = 1|...

  The first time run on an OpenVINO device is slow, since the OpenVINO framework will compile the IR (Intermediate Representation) model to a device-specific 'blob'. This device-specific blob will get
  cached for the next run.

You can use -oved [DEVICE] argument to main to specify OpenVINO device to offload encoder inference to. For example:

main -m ggml-base.bin -f gb1.wav -oved GPU