Arch-Function: Advanced Function Calling Models

📰 News & Updates

🚀 Current Model Collections

Collection 1: Base Function Calling Models

Collection 2: Chat-Optimized Models

Collection 3: Agentic Models

📚 1. Fine-tuning Arch-Function Models

1.1 Set up environment

pip install deepspeedpip install flash-attn --no-build-isolation

1.2 Prepare training data

[{"conversations": [{"from":"human","value":"user instruction"},{"from":"function_call","value":"tool arguments"},{"from":"observation","value":"tool result"},{"from":"gpt","value":"model response"}],"system":"system prompt (optional)","tools":"tool description (optional)"}]

"dataset_name": {"file_name":"data.json","formatting":"sharegpt","columns": {"messages":"conversations","system":"system","tools":"tools"}}

1.3 Training

CUDA_VISIBLE_DEVICES={YOUR_DEVICE_IDS} llamafactory-cli train {PATH_TO_YOUR_TRAINING_SCRIPT}

📚 2. Inference with Arch-Function Models

2.1 Set up environment

pip install transformers>=4.51.0

2.2 Inference

2.2.1 Create models and tokenizers

importjsonfromtypingimportAny,Dict,ListfromtransformersimportAutoModelForCausalLM,AutoTokenizer# Specify the desired model name heremodel_name="katanemo/Arch-Agent-7B"model=AutoModelForCausalLM.from_pretrained(model_name,device_map="auto",torch_dtype="auto",trust_remote_code=True)tokenizer=AutoTokenizer.from_pretrained(model_name)

2.2.2 Format prompts

# Please use the recommended prompt for each model.TASK_PROMPT= ("You are a helpful assistant designed to assist with the user query by making one or more function calls if needed.""\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\n""You are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{tool_text}""\n</tools>\n\nFor each function call, return a json object with function name and arguments within """"<tool_call></tool_call> XML tags:\n<tool_call>\n{{"name": <function-name>, """""""arguments": <args-json-object>}}\n</tool_call>""")# Define available toolstools= [    {"type":"function","function": {"name":"get_weather","description":"Get the current weather for a location","parameters": {"type":"object","properties": {"location": {"type":"str","description":"The city and state, e.g. San Francisco, New York",                    },"unit": {"type":"str","enum": ["celsius","fahrenheit"],"description":"The unit of temperature to return",                    },                },"required": ["location"],            },        },    }]# Helper function to create the system prompt for our modeldefformat_prompt(tools:List[Dict[str,Any]]):tool_text="\n".join(        [json.dumps(tool["function"],ensure_ascii=False)fortoolintools]    )returnTASK_PROMPT.format(tool_text=tool_text)system_prompt=format_prompt(tools)messages= [    {"role":"system","content":system_prompt},    {"role":"user","content":"What is the weather in Seattle?"},]

2.2.3 Run inference

#### 2.2.3 Run inferencemodel_inputs=tokenizer.apply_chat_template(messages,add_generation_prompt=True,return_tensors="pt",return_dict=True).to(model.device)generated_ids=model.generate(**model_inputs,max_new_tokens=32768)generated_ids= [output_ids[len(input_ids) :]forinput_ids,output_idsinzip(model_inputs.input_ids,generated_ids)]response=tokenizer.batch_decode(generated_ids,skip_special_tokens=True)[0]print(response)

📚 3. Deployment with Popular Hosting Frameworks

3.1 vLLM Deployment

3.1.1 Set up environment

# Install vLLMpip install vllm

3.1.2 Start vLLM server

vllm serve katanemo/Arch-Agent-7B \    --host 127.0.0.1 \    --port 8000 \    --tensor-parallel-size 1

3.1.3 Get responses

fromopenaiimportOpenAI# Point to the local serverclient=OpenAI(api_key="EMPTY",base_url="http://127.0.0.1:8000/v1",)# Send requests and get responses from the servercompletion=client.chat.completions.create(model="katanemo/Arch-Agent-7B",messages=[        {"role":"user","content":"Get the current temperature in San Francisco"}    ],temperature=0.01,max_tokens=1024)print(completion.choices[0].message.content)

3.2 ollama Deployment

3.2.1 Install ollama

pip install ollama

3.2.2 Start ollama server

ollama run hf.co/katanemo/Arch-Agent-7B.gguf

3.2.3 Get responses

fromollamaimportClient# Point to the local server. By default, it uses port 11434.client=Client(host="http://127.0.0.1:11434")# Send requests and get responses from the servercompletion=client.chat(model="hf.co/katanemo/Arch-Agent-1.5B.gguf",messages=[        {"role":"user","content":"Get the current temperature in San Francisco"}    ],options={"temperature":0.01,"num_ctx":1024})print(completion.message.content)

3.3 SGLang Deployment

3.3.1 Set up experiment

# Install SGLangpip install sglang[all]

3.3.2 Start SGLang server

python -m sglang.launch_server \    --model-path katanemo/Arch-Agent-7B \    --host 127.0.0.1 \    --port 8000 \    --tp 1 \    --trust-remote-code

3.3.3 Get responses

# Client code for vLLMfromopenaiimportOpenAI# Point to the local serverclient=OpenAI(api_key="EMPTY",base_url="http://127.0.0.1:8000/v1",)#completion=client.chat.completions.create(model="katanemo/Arch-Agent-7B",messages=[        {"role":"user","content":"Get the current temperature in San Francisco"}    ],temperature=0.01,max_tokens=1024)print(completion.choices[0].message.content)

🔬 Research & Development

📄 License

🤝 Contributing

📞 Support