hello, i am doing what you suggested and i have results. I have 2 questions, when i want to run it without vector proccesor in qemu, what comand do i have to run? also , how can i check that those 2 runs are different and the one with vector proccesor working like i wanted to ? sorry, i am new to this

Hi, for running on CPU (scalar) provide the path to risc-v toolchain and then use qemu

make llama-cli CC="riscv64-unknown-linux-gnu-gcc -march=rv64gc -mabi=lp64d" CXX="riscv64-unknown-linux-gnu-g++ -march=rv64gc -mabi=lp64d"

qemu-riscv64 -L /path/to/sysroot/  -cpu rv64 ./llama-cli -m ./path/to/model.gguf -p "Anything" -n 100

You can set the seed to get the same results i.e; llama-cli -s (some_seed number) ...

More details:RVV article
Also, this is old, and many things have change, like main -> llama-cli etc.

Thank you

Hi, for running on CPU (scalar) provide the path to risc-v toolchain and then use qemu

make llama-cli CC="riscv64-unknown-linux-gnu-gcc -march=rv64gc -mabi=lp64d" CXX="riscv64-unknown-linux-gnu-g++ -march=rv64gc -mabi=lp64d"

qemu-riscv64 -L /path/to/sysroot/  -cpu rv64 ./llama-cli -m ./path/to/model.gguf -p "Anything" -n 100

You can set the seed to get the same results i.e; llama-cli -s (some_seed number) ...

More details:RVV article Also, this is old, and many things have change, like main -> llama-cli etc.

Thank you

yeah i read this article but when i do the make you provided , i get an error for "march=native" error and from what i searched on makefile, i have to do RISCV_CROSS_COMPILE=1 RISCV=1

Sorry yes, I just identified the makefile has been reorder and RISCV=1 is required in current version

After line 432 in makefile, update the flags for vector version with scalar, i.e

MK_CFLAGS += -march=rv64gc -mabi=lp64d
MK_CXXFLAGS += -march=rv64gc -mabi=lp64d

and then run qemu with,

make llama-cli RISCV=1 CC="riscv64-unknown-linux-gnu-gcc" CXX="riscv64-unknown-linux-gnu-g++"

okay okay, one last question, i use the same seed and i have results for both with vector and without, but the only difference in log print is the print time . with vector is 2-2.5X more than without. Is it correct?

Yes, on qemu the vector emulation time is much slower (the actual reason is not known to me, could be due to qemu has to additionally emulate vector processor with the scalar one or parallel processing issues, and also the log use the real-time for comparison), but this should not be the case with actual RISC-V vector board

hello again, i am running llama with vector extension on gem5 but since there isnt something on log to check if vector extension is enabled , how do i know ?

Hi, I’ve submitted a PR (#9442) which will print RISCV_VECT=1 on the terminal, if the vector processor is found. Also, I slightly changed Makefile so it no longer requires a flag for RISC-V vector boards—only RISCV_CROSS_COMPILATION=1 is needed for the emulator (i.e. QEMU).

The following is the output from the RISC-V BPI-F3 board with vector support,
...

...

...

...

Hello, i have a question, why the load time is less when rvv is enabled ? load time is the loading time of the model right ? how the rvv affects it ?

Hi, Actually that was due to the quantization happening while loading the weights into memory (Not all weights are quantized, some are done while loading them into memory; specifically check the function quantize_row_q8_0_reference in GGML quants.c file or quantize_row_q4_K), also enabling auto-vectorization can affect the load time.

okay but why the quantization affects the loadimg time ? can you explain me what changes with the rvv enabled ?

During model's initialization phase, there could be several reasons, such as auto-vectorization of memory operations by compiler, recalculation, decompression, and data alignment. However, during benchmarking, I noticed the most change due to the quantize_row function, since it was performing some computations before the weights were fully loaded into memory.

okay so when the rvv is enabled , whats happening with the quantize_row function and the load time is less ? also if the model is already quantized is there any change ?

when you enable the rvv , the quantize_row function that you mentioned that is perforforming computations doesnt execute ?
@Tameem-10xE

Sorry, I last worked on this project a year ago and did not go deeper into how it works except for what I could quantify. I might be mistaken or unclear about which functions are affecting the load time (also many things has been change). I think you should ask in the GitHub discussion or Discord if they have one. I also noticed a reduction in load time on x86 with vector (~5 times), and I think the most probable cause must be compiler auto-vectorization. If not, they may be able to provide the exact reason.

Sorry, I last worked on this project a year ago and did not go deeper into how it works except for what I could quantify. I might be mistaken or unclear about which functions are affecting the load time (also many things has been change). I think you should ask in the GitHub discussion or Discord if they have one. I also noticed a reduction in load time on x86 with vector (~5 times), and I think the most probable cause must be compiler auto-vectorization. If not, they may be able to provide the exact reason.

Okay, thank you very much. are the weights quantized differently in scalar than with rvv on ? I mean, does it make sense to compare the results of scalar with the rvv on or do they follow a different loading procedure ?

Welcome!
No, weights are quantized the same for scalar and RVV (lllama-qunatize script is independent of this), I think it does not matter much since loading is a one-time process, but the more important metric for comparison would be inference time i.e how much tokens are being generated with RVV on against the scalar time.

Hello again, i am simulating through gem5 a riscv environment and i am running llama-cli with a llama 4b model. When i change the vlen and elen of the rvv to greater than 256bit and 64 bit , the generated phrase i get , it doesnt make sense. Is the llama implementation working only on rvv with vlen=256 and elen=64 bit ?

Hi, What Output did you got?
Some functions were using dynamic vl i.e (size_t vl = __riscv_vsetvl_e8m1(qk/2);)
But to avoid duplicating instructions to load qh and smaller size arrays, I did limit the vl size to 256, but smaller VL should work without any issues.

For rvv with vlen=512 and elen=64 i got this

..., this could be a bug, I missed something or the second half of the register could be interfering with the output causing undefined or junk text. Sorry, due to other tasks, I will not be able to look into this for now, but meanwhile you can file an issue or ask further it about in RISC-V intrinsic repo.

Now that we build llama.cpp with cmake instead of make command , what file do i have to change to build it for riscv but without vector extension ? i changed the -march on makefile but it was built again with the vector extension

You can use this flag -DGGML_RVV=OFF i.e;

mkdir buildcd buildcmake -DGGML_RVV=OFF [... other flags ...]

..., this could be a bug, I missed something or the second half of the register could be interfering with the output causing undefined or junk text. Sorry, due to other tasks, I will not be able to look into this for now, but meanwhile you can file an issue or ask further it about in RISC-V intrinsic repo.

Hello again. What do you think i can change to fix this bug and the exe can run with higher vlen with correct results?

Usually, this could be due to the vl variable, outer or inner for loop (for (int i = 0; i < nb; i++) not completed or missed edge elements) or vsetvli instruction. Also, I am unsure about GEMM5 simulation, which could be different from the QEMU or RV board I tested on (with 256-bit VLEN). Sorry, I have this in my mind, but I will try to fix this next week.

Usually, this could be due to the vl variable, outer or inner for loop (for (int i = 0; i < nb; i++) not completed or missed edge elements) or vsetvli instruction. Also, I am unsure about GEMM5 simulation, which could be different from the QEMU or RV board I tested on (with 256-bit VLEN). Sorry, I have this in my mind, but I will try to fix this next week.

Any news ?

Really sorry for the late response. I was inactive last week.
Actually, I have only access to 256-bit RV Hardware, so my only option was to check it on an emulator (qemu), and it is working correctly for 512 VLEN with Q4_K. I also go through code but was unable to find any issue. It may be issue with GEMM5? But can't say for sure.

Really sorry for the late response. I was inactive last week. Actually, I have only access to 256-bit RV Hardware, so my only option was to check it on an emulator (qemu), and it is working correctly for 512 VLEN with Q4_K. I also go through code but was unable to find any issue. It may be issue with GEMM5? But can't say for sure.

Can you explain how do you run it on qemu ? I just run llama-cli for vlen=512 and i got this generated phrase with q4_k model
"" Anything�stagemblụ shelterirasichenForKeyronessageHinttexttåk Weltkrieg ""

Sorry, I plugged the wrong weights above (need to change my naming convention...).

Identified the issue in the function namedvoid ggml_vec_dot_q4_K_q8_K inggml-cpu-quant.c file. Unable to test this right now, but try to change this variable value ofsize_t vl=8 in this function to either16 or 32. (Only for 4-bit q4_k)

Sorry, I plugged the wrong weights above (need to change my naming convention...).

Identified the issue in the function namedvoid ggml_vec_dot_q4_K_q8_K inggml-cpu-quant.c file. Unable to test this right now, but try to change this variable value ofsize_t vl=8 in this function to either16 or 32. (Only for 4-bit q4_k)

I changed the vl to 16 and 32 and it still doesnt work correctly. The same happens for other models too. I tried a q2_k model for 512 and the generated phrase is not correct