run-llama/LlamaIndexTS

add detail to image_url for openai chat completions

Open

#2,221 opened on Oct 20, 2025

 (3 comments) (0 reactions) (0 assignees)TypeScript (524 forks)auto 404
buggood first issuehelp wanted

Repository metrics

Stars
 (3,078 stars)
PR merge metrics
 (PR metrics pending)

Description

Describe the bug When sending messages containing image content to OpenAI's chat completions endpoint, the detail is ignored. OpenAI expects it to be inside the image_url object (see here), but llamaindex expects it to be next to the image_url: https://github.com/run-llama/LlamaIndexTS/blob/fc385dc16757a73dc19ca239ebbe1e6155ed4b4d/packages/core/src/llms/type.ts#L195-L199

and there's no translation happening when using llm.chat: https://github.com/run-llama/LlamaIndexTS/blob/fc385dc16757a73dc19ca239ebbe1e6155ed4b4d/packages/providers/openai/src/llm.ts#L248-L249

To Reproduce Code to reproduce the behavior:

import { OpenAI, OpenAIResponses } from "@llamaindex/openai";
import type { ChatMessage } from "llamaindex";

const followingTypes: ChatMessage = {
  role: "user",
  content: [
    {
      type: "image_url",
      detail: "high",
      image_url: { url: "data:image/jpeg;base64,aGVsbG8=" },
    },
  ],
};

const workaroundForChatCompletionsOnly: ChatMessage = {
  role: "user",
  content: [
    {
      type: "image_url",
      image_url: {
        url: "data:image/jpeg;base64,aGVsbG8=",
        // @ts-expect-error
        detail: "high",
      },
    },
  ],
};

const workaroundForBoth: ChatMessage = {
  role: "user",
  content: [
    {
      type: "image_url",
      detail: "high",
      image_url: {
        url: "data:image/jpeg;base64,aGVsbG8=",
        // @ts-expect-error
        detail: "high",
      },
    },
  ],
};

const messages = [
  followingTypes,
  workaroundForChatCompletionsOnly,
  workaroundForBoth,
];

const chat = OpenAI.toOpenAIMessage(messages);
const responses = new OpenAIResponses().toOpenAIResponseMessages(messages);

console.dir({ chat, responses }, { depth: null });

The workaroundForBoth ends up putting detail in two places for chat completions. I still need to confirm if this throws a 4xx error from OpenAI or if it's just ignored.

{
  "type": "image_url",
  "detail": "high",
  "image_url": {
    "url": "data:image/jpeg;base64,aGVsbG8=",
    "detail": "high"
  }
} 

Expected behavior We should be able to provide messages that adhere to llamaindex's ChatMessage type and still have the detail param make it into the correct spot for both Chat Completions and Responses APIs.

i.e. OpenAI.toOpenAIMessage([followingTypes]) should return:

[
  {
    "role": "user",
    "content": [
      {
        "type": "image_url",
        "image_url": {
          "url": "data:image/jpeg;base64,aGVsbG8=",
          "detail": "high"
        }
      }
    ]
  }
]

And new OpenAIResponses().toOpenAIResponseMessages([followingTypes]) should return:

[
  {
    "role": "user",
    "content": [
      {
        "type": "input_image",
        "image_url": "data:image/jpeg;base64,aGVsbG8=",
        "detail": "high"
      }
    ]
  }
]

Screenshots N/A

Desktop (please complete the following information):

  • OS: macOS
  • JS Runtime / Framework / Bundler (select all applicable)
  • Node.js
  • Deno
  • Bun
  • Next.js
  • ESBuild
  • Rollup
  • Webpack
  • Turbopack
  • Vite
  • Waku
  • Edge Runtime
  • AWS Lambda
  • Cloudflare Worker
  • Others (please elaborate on this)
  • Version: 1.2.22

Additional context N/A

Contributor guide