@regisss@Narsil please help review

@sywangyi Can you add a description of the issue and how this PR solves it please?

#3257 describe it, And I just give a WA as skipping applying chat template to VLM model like llava qwenvl

Gently pinging@Narsil for review

This is most likely a wrong fix. We cannot just skip using templates on some models because the answer is not to our liking.

We need to fix the template instead, no ? But overriding like this is most likely purely wrong.

VLM template input like,
conversation = [
{

  "role": "user",  "content": [      {"type": "text", "text": "What is shown in this image?"},      {"type": "image"},    ],},

]

which is different with llm.
{"role": "user", "content": "Who are you?"},

so we may need reconstruct the msg for vlm before apply chat template. also, there is model that does not have chat template. we need to handle it as well.