Repository files navigation

This repository contains the canonical information to use when converting IBM AI models to the GGUF format. It includes conversion scripts and testing requirements. Aspirationally, this repo. wil include and automated CI/CD process to convert, test and deploy models to the official IBM GGUF collection in Hugging Face.

Only a select set of IBM models will be converted to GGUF format based upon the following criteria:

• The IBM GGUF model needs to be referenced by an AI provider service (i.e., a local AI provider service) as a "supported" model.
• The GGUF model is referenced by a public blog, tutorial, demo, or other public use case.
  • Specifically, if the model is referenced in an IBMGranite Snack Cookbook

In addition, models should be canonically hosted in an official IBM's repository. Currently, this includes the following:

• Hugging Face
  • https://huggingface.co/ibm-granite

The following tables list the current target set of IBM models along with commentary on the rationale for inclusion:

See:https://huggingface.co/ibm-granite

These models are found in theGranite 3.0 Language Models collection and are designed to respond to general instructions and can be used to build AI assistants for multiple domains, including business applications.

where:

• Consensus default:
  • Comparable size to default models already referenced by multiple downstream providers and frameworks.
  • Size ideal for local CPU/GPU serving.
• Small form-factor:
  • Model size intended running locally on small form-factor devices such as watches and mobile devices.

RAG LoRA support

See granite3-dense Ollama model entry:

• https://ollama.com/library/granite3-dense
• and its various (tagged) quantizations:https://ollama.com/library/granite3-dense/tags

The GGUF format is defined in theGGUF specification. The specification describes the structure of the file, how it is encoded, and what information is included.

Currently, the primary means to convert from HF SafeTensors format to GGUF will be the canonical llama.cpp toolconvert-hf-to-gguf.py.

for example:

python llama.cpp/convert-hf-to-gguf.py ./models/modelrepo --outfile output_file.gguf --outtype q8_0

• https://github.com/ollama/ollama/blob/main/docs/import.md#quantizing-a-model

  $ ollama create --quantize q4_K_M mymodeltransferring model dataquantizing F16 model to Q4_K_Mcreating new layer sha256:735e246cc1abfd06e9cdcf95504d6789a6cd1ad7577108a70d9902fef503c1bdcreating new layer sha256:0853f0ad24e5865173bbf9ffcc7b0f5d56b66fd690ab1009867e45e7d2c4db0fwriting manifestsuccess

Note: The Ollama CLI tool only supports a subset of quantizations:- (rounding):q4_0,q4_1,q5_0,q5_1,q8_0- k-means:q3_K_S,q3_K_M,q3_K_L,q4_K_S,q4_K_M,q5_K_S,q5_K_M,q6_K

• https://huggingface.co/spaces/ggml-org/gguf-my-repo

Note:

• Access control to source repo. required???
• Similar to Ollama CLI, the web UI supports only a subset of quantizations.

As a baseline, each converted model MUST successfully be run in the following providers:

llama.cpp - As the core implementation of the GGUF format which is either a direct dependency or utilized as forked code in most all downstream GGUF providers, testing is essential. Specifically, testing to verify the model can be hosted using thellama-server service.-See the specific section onllama.cpp for more details on which version is considered "stable" and how the same version will be used in both conversion and testing.

Ollama - As a key model service provider supported by higher level frameworks and platforms (e.g.,AnythingLLM,LM Studio etc.), testing the ability topull andrun the model is essential.

Notes

• The official Ollama Docker imageollama/ollama is available on Docker Hub.
• Ollama does not yet support sharded GGUF models
  • "Ollama does not support this yet. Follow this issue for more info:ollama/ollama#5245"
  • e.g.,ollama pull hf.co/Qwen/Qwen2.5-14B-Instruct-GGUF

• GGUF format

  • Huggingface:GGUF - describes the format and some of the header structure.
  • llama.cpp:
    • GGUF Quantization types (ggml_ftype) -ggml/include/ggml.h
    • GGUF Quantization types (LlamaFileType) -gguf-py/gguf/constants.py

• GGUF Examples

  • llama.cpp/examples/quantize

• GGUF tools

  • GGUF-my-repo - Hugging Face space to build your own quants. without any setup.(ref. by llama.cpp example docs.)
  • CISCai/gguf-editor - batch conversion tool for HF model repos. GGUF models.

• llama.cpp Tutorials

  • How to convert any HuggingFace Model to gguf file format? - using thellama.cpp/convert-hf-to-gguf.py conversion script.

• Ollama tutorials

  • Importing a model - includes Safetensors, GGUF.
  • Use Ollama with any GGUF Model on Hugging Face Hub
  • Using Ollama models from Langchain - This example uses thegemma2 model supported by Ollama.

• Huggingface Granite

  • Bartowski:https://huggingface.co/bartowski?search_models=granite

Notes

• latest is relative to Ollama (proprietary) publishing and is not reflected in GGUF header.
• basename,finetune may be are different depending on person who created the GGUF even for the same company...
  • e.g., IBM Granite model "Granite 8b Code Instruct 128k" has afinetune name that does not match other IBM models (i.e.,code-instruct-128k).
• context_buffer (size) not mentioned infinetune for Ollamagranite-code models which have8k buffers, but is listed for128k buffers.
• Ollama modelinstructlab/granite-7b-lab is identical to thegranite-7b model.
• IQ2_XS quant. may have issues on Apple silicon
  • see commentary here:https://www.reddit.com/r/LocalLLaMA/comments/1ba55rj/overview_of_gguf_quantization_methods/

Note: "registries" are created using the domain name of the model repo. ref. during apull orrun command.

Notes

• downstream fine tunings or quants. lose identity (in the GGUF file) or drop (pedigree-related) fields or create new ones
  • general.name,general.basename,general.finetune, etc.
    • e.g.,general.name=liuhaotian is the name of the person who created the downstream GGUF(not the actual model name) (and it had nobasename, norfinetune)
  • .size_label did not match model declared size.
• when multiple GGUF models are in a repo. Ollama "grabs" the first one (alphanumerically)
  • e.g.,Qwen2.5-14B-Instruct repo.:https://huggingface.co/QuantFactory/Qwen2.5-14B-Instruct-GGUF/tree/main
    • has 14 quantizations... but itgrabbed the quant.Q2_K(least precise)