langchain-ai/langchain

`llm.ainvoke` is traced differently than `llm.invoke` (won't be traced)

Closed

#30,870 opened on Apr 16, 2025

 (8 comments) (1 reaction) (1 assignee)Python (22,617 forks)batch import
bugcoreexternalhelp wanted

Repository metrics

Stars
 (136,758 stars)
PR merge metrics
 (Avg merge 10d 2h) (288 merged PRs in 30d)

Description

Checked other resources

  • I added a very descriptive title to this issue.
  • I used the GitHub search to find a similar question and didn't find it.
  • I am sure that this is a bug in LangChain rather than my code.
  • The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).
  • I posted a self-contained, minimal, reproducible example. A maintainer can copy it and run it AS IS.

Example Code

We have a custom tracer implemented in python derived from AsyncBaseTracer. We noticed a diferent tracer behaviour when using llm.ainvoke vs llm.invoke

def some_node_in_graph():
    tools = [add]
    llm_with_tools = ChatAnthropic(api_key=get_anthropic_api_key(), model="claude-3-opus-20240229")().bind_tools(tools)
    messages = [
        SystemMessage(content="You are a helpfull asistant, please add two random numbers"),
        HumanMessage(content=state.message),
    ]

    response_message = llm_with_tools.invoke(messages) # this won't be traced
    response_message = await llm_with_tools.ainvoke(messages) # this will work with no problems

# tracer

class AsyncCustomTracer(AsyncBaseTracer):
    
   async def _start_trace(self, run: Run) -> None:
        await super()._start_trace(run)
        await self._persist_run(run) # our persist run will make an http post

    async def _end_trace(self, run: Run) -> None:
        await super()._end_trace(run)
        await self._persist_run(run)

Error Message and Stack Trace (if applicable)

No response

Description

I've narrowed it down to https://github.com/langchain-ai/langchain/blob/master/libs/core/langchain_core/callbacks/manager.py#L275

  • ainvoke will go to except NotImplementedError as e: if event_name == "on_chat_model_start" and will work perfectly with no issues
  • invoke will raise NotImplementedError and not be caught by this handler and will print an error log
Error in callback coroutine: NotImplementedError('Chat model tracing is not supported in for original format.')
Error in callback coroutine: TracerException('No indexed run ID 507c24d0-9f28-4737-aa66-551911a7b22c.')

System Info

Package Information
-------------------
> langchain_core: 0.3.51
> langchain: 0.3.23
> langchain_community: 0.3.21
> langsmith: 0.3.18
> langchain_anthropic: 0.3.10
> langchain_openai: 0.3.9
> langchain_text_splitters: 0.3.8
> langgraph_sdk: 0.1.59

Contributor guide