OpenAI Chat Client#

Refer to thetrtllm-serve documentation for starting a server.

SourceNVIDIA/TensorRT-LLM.

 1 2fromopenaiimportOpenAI 3 4client=OpenAI( 5base_url="http://localhost:8000/v1", 6api_key="tensorrt_llm", 7) 8 9response=client.chat.completions.create(10model="TinyLlama-1.1B-Chat-v1.0",11messages=[{12"role":"system",13"content":"you are a helpful assistant"14},{15"role":"user",16"content":"Where is New York?"17}],18max_tokens=20,19)20print(response)