There isn't really a real writeup of all the mapping, but this should hopefully be a good central starting point if any maintainers needs some understanding of when each feature was added and the general specs of each. Updating it with more accurate information is greatly appreciated

This is not definitive, but is helpful when reading sourcecode or console output to understand what each means typically.

• <Encoding>_<Variants>
  • <Encoding> : This defines the most common encoding of individual weights in the model
    • Floating Point Formats:
      • BF16:16-bit bfloat16Google Brain truncated form of 32-bit IEEE 754 (1 sign bit, 8 exponent bits, 7 fractional bits)
      • F64:64-bit IEEE 754 floats per weight (1 sign bit, 11 exponent bits, 52 fractional bits)
      • F32:32-bit IEEE 754 floats per weight (1 sign bit, 8 exponent bits, 23 fractional bits)
      • F16:16-bit IEEE 754 floats per weight (1 sign bit, 5 exponent bits, 10 fractional bits)
    • Integer formats:
      • I<X>: X bits per weight, whereX could be4 (for 4 bits) or8 (for 8 bits) etc...
    • Quantized formats:
      • Q<X>: X bits per weight, whereX could be4 (for 4 bits) or8 (for 8 bits) etc...
      • KQ<X> (orQ<X>_K) : k-quant based models. X bits per weight, whereX could be4 (for 4 bits) or8 (for 8 bits) etc...
      • IQ<X>: i-quant based models. X bits per weight, whereX could be4 (for 4 bits) or8 (for 8 bits) etc...
  • <Variants>: This represents different strategies of packing quantized weights into a gguf file. This is because we may want a mix of different bit sizes for weights of varying importance, or we may be encoding a general offset to a block or super-block. This may be omitted if trivial or initial attempt, refer to encoding scheme name table for details.

• PR Filter of all Tensor Encoding Scheme Related Pull Requests
  • This is useful for locating Pull Requests where we merged in a new encoding scheme (or fix a bug with encoding etc...)

• All superblocks have fp16 scaling factor and contains up to 256 weights. Number of weights in a block must be divisible by 256. (To be confirmed)

You would find it all usually inggml-common.h where it typically be of this form

#defineQK4_0 32typedefstruct {ggml_halfd;// deltauint8_tqs[QK4_0 /2];// nibbles / quants}block_q4_0;static_assert(sizeof(block_q4_0)==sizeof(ggml_half)+QK4_0 /2,"wrong q4_0 block size/padding");

//// Super-block quantization structures//// 2-bit quantization// weight is represented as x = a * q + b// 16 blocks of 16 elements each// Effectively 2.625 bits per weighttypedefstruct {uint8_tscales[QK_K/16];// scales and mins, quantized with 4 bitsuint8_tqs[QK_K/4];// quantsunion {struct {ggml_halfd;// super-block scale for quantized scalesggml_halfdmin;// super-block scale for quantized mins        }GGML_COMMON_AGGR;ggml_half2dm;    };}block_q2_K;static_assert(sizeof(block_q2_K)==2*sizeof(ggml_half)+QK_K/16+QK_K/4,"wrong q2_K block size/padding");

This is as explained bycompilade inthis thread.

Regarding how to find the bit pattern structure of a packed tensor block in the gguf file... there isn't a consistent encoding scheme for each block as sometimes a single field in the structs stores multiple types of values, like inQ4_K whereblock_q4_K.scales stores 6-bit scales and mins in some pattern. The easiest way to understand what the bits mean is to have a look at the respectivedequantize_row function of each type.

The 12 bytes in Q4_K.scales are packed a bit like this, where the uppercased letters are bits for the scales and lowercased letters are the bits of the mins as seen below, which corresponds to this functionas shown here:

0: EEAAAAAA1: FFBBBBBB2: GGCCCCCC3: HHDDDDDD4: eeaaaaaa5: ffbbbbbb6: ggcccccc7: hhdddddd8: eeeeEEEE9: ffffFFFF10: ggggGGGG11: hhhhHHHH

Note that this is packing a 6bit scale and mins but split across multiple bytes. This use of byte offsets and bitwise operations is likely done to be more friendlier for SIMD processing. Ascompilade noted, he believes that the indexing is only done at the byte level, hence the packing and unpacking of the 6-bit values in this block will require bitwise operations. In his anecdotal experience he also noticed that when making the vec_dot of Q1_3, that shuffles are surprisingly as fast as additions in SIMD.