This guide assumes familiarity with the following concepts:

• Callbacks
• Custom callback handlers

If you use a syncCallbackHandler while using an async method to run your LLM / Chain / Tool / Agent, it will still work. However, under the hood, it will be called withrun_in_executor which can cause issues if yourCallbackHandler is not thread-safe.

If you're onpython<=3.10, you need to remember to propagateconfig orcallbacks when invoking otherrunnable from within aRunnableLambda,RunnableGenerator or@tool. If you do not do this,the callbacks will not be propagated to the child runnables being invoked.

import asyncio
from typingimport Any, Dict, List

from langchain_anthropicimport ChatAnthropic
from langchain_core.callbacksimport AsyncCallbackHandler, BaseCallbackHandler
from langchain_core.messagesimport HumanMessage
from langchain_core.outputsimport LLMResult


classMyCustomSyncHandler(BaseCallbackHandler):
defon_llm_new_token(self, token:str,**kwargs)->None:
print(f"Sync handler being called in a `thread_pool_executor`: token:{token}")


classMyCustomAsyncHandler(AsyncCallbackHandler):
"""Async callback handler that can be used to handle callbacks from langchain."""

asyncdefon_llm_start(
        self, serialized: Dict[str, Any], prompts: List[str],**kwargs: Any
)->None:
"""Run when chain starts running."""
print("zzzz....")
await asyncio.sleep(0.3)
        class_name= serialized["name"]
print("Hi! I just woke up. Your llm is starting")

asyncdefon_llm_end(self, response: LLMResult,**kwargs: Any)->None:
"""Run when chain ends running."""
print("zzzz....")
await asyncio.sleep(0.3)
print("Hi! I just woke up. Your llm is ending")


# To enable streaming, we pass in `streaming=True` to the ChatModel constructor
# Additionally, we pass in a list with our custom handler
chat= ChatAnthropic(
    model="claude-3-sonnet-20240229",
    max_tokens=25,
    streaming=True,
    callbacks=[MyCustomSyncHandler(), MyCustomAsyncHandler()],
)

await chat.agenerate([[HumanMessage(content="Tell me a joke")]])

zzzz....
Hi! I just woke up. Your llm is starting
Sync handler being called in a `thread_pool_executor`: token: Here
Sync handler being called in a `thread_pool_executor`: token: 's
Sync handler being called in a `thread_pool_executor`: token:  a
Sync handler being called in a `thread_pool_executor`: token:  little
Sync handler being called in a `thread_pool_executor`: token:  joke
Sync handler being called in a `thread_pool_executor`: token:  for
Sync handler being called in a `thread_pool_executor`: token:  you
Sync handler being called in a `thread_pool_executor`: token: :
Sync handler being called in a `thread_pool_executor`: token:

Why
Sync handler being called in a `thread_pool_executor`: token:  can
Sync handler being called in a `thread_pool_executor`: token: 't
Sync handler being called in a `thread_pool_executor`: token:  a
Sync handler being called in a `thread_pool_executor`: token:  bicycle
Sync handler being called in a `thread_pool_executor`: token:  stan
Sync handler being called in a `thread_pool_executor`: token: d up
Sync handler being called in a `thread_pool_executor`: token:  by
Sync handler being called in a `thread_pool_executor`: token:  itself
Sync handler being called in a `thread_pool_executor`: token: ?
Sync handler being called in a `thread_pool_executor`: token:  Because
Sync handler being called in a `thread_pool_executor`: token:  it
Sync handler being called in a `thread_pool_executor`: token: 's
Sync handler being called in a `thread_pool_executor`: token:  two
Sync handler being called in a `thread_pool_executor`: token: -
Sync handler being called in a `thread_pool_executor`: token: tire
zzzz....
Hi! I just woke up. Your llm is ending

LLMResult(generations=[[ChatGeneration(text="Here's a little joke for you:\n\nWhy can't a bicycle stand up by itself? Because it's two-tire", message=AIMessage(content="Here's a little joke for you:\n\nWhy can't a bicycle stand up by itself? Because it's two-tire", id='run-8afc89e8-02c0-4522-8480-d96977240bd4-0'))]], llm_output={}, run=[RunInfo(run_id=UUID('8afc89e8-02c0-4522-8480-d96977240bd4'))])