Description

This PR adds optional token tracking support to the openai_embed() function, enabling comprehensive monitoring of embedding token usage alongside existing LLM token tracking capabilities.

Related Issues

N/A - Enhancement to improve observability of API token usage for embeddings.

Changes Made

• Add optionaltoken_tracker parameter to openai_embed() function in lightrag/llm/openai.py
• Trackprompt_tokens andtotal_tokens for embedding API calls
• Update function docstring to document the newtoken_tracker parameter
• Enables monitoring of embedding token usage alongside LLM calls
• Maintains backward compatibility with existing code (parameter is optional)

Checklist

• Changes tested locally
• Code reviewed
• Documentation updated

Additional Notes

Implementation Details:

• Follows the same pattern used in openai_complete_if_cache() for consistency
• Token tracking only occurs whentoken_tracker is provided and response contains usage data
• No breaking changes - existing code continues to work without modifications

Usage Example:

fromlightrag.utilsimportTokenTrackerfromlightrag.llm.openaiimportopenai_embedtoken_tracker=TokenTracker()embeddings=awaitopenai_embed(texts=["example text"],token_tracker=token_tracker)print(token_tracker)# Shows token usage statistics