- Notifications
You must be signed in to change notification settings - Fork328
Open
Description
Hi,
I have issue while using the GGUF version of models.
for example non-gguf model is working ;
SELECT pgml.transform ( task => '{ "model": "meta-llama/Meta-Llama-3.1-8B-Instruct" }'::jsonb, inputs => ARRAY[ $$A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: Please write an intro to a story about a woman living in New York. ASSISTANT: $$ ], args => '{ "max_new_tokens": 512 }'::JSONB );
but when i try GGUF versions i hate faced error ,
SELECT pgml.transform ( task => '{ "model_type": "gguf", "model": "hugging-quants/Llama-3.2-1B-Instruct-Q8_0-GGUF", "model_file":"llama-3.2-1b-instruct-q8_0.gguf" }'::jsonb, inputs => ARRAY['{"role":"user", "content": "What is the capital of France?"}'], args => '{ "max_new_tokens": 128 }'::jsonb );
ERROR: Traceback (most recent call last): File "transformers.py", line 489, in transform File "transformers.py", line 455, in create_pipeline File "transformers.py", line 217, in __init__ File "/var/lib/postgresql/.local/lib/python3.10/site-packages/ctransformers/hub.py", line 175, in from_pretrained llm = LLM( File "/var/lib/postgresql/.local/lib/python3.10/site-packages/ctransformers/llm.py", line 253, in __init__ raise RuntimeError( RuntimeError: Failed to create LLM 'gguf' from '/var/lib/postgresql/.cache/huggingface/hub/models--hugging-quants--Llama-3.2-1B-Instruct-Q8_0-GGUF/blobs/ba345c83bf5cc679c653b853c46517eea5a34f03ed2205449db77184d9ae62a9'. SQL state: XX000
I couldn't find how to solve this issue, I hope you can help me.