tensorrt_llm/inputs/multimodal.py (3)

482-484:Return type/docstring mismatch, long line (E501), and avoid unnecessary PIL conversions.

• find_mm_token_lengths now returns a dict but the annotation and comment still say List[int].
• Break the long f-string to satisfy E501.
• When items are tensors, compute H/W from shape instead of converting to PIL; validate non-empty video list.

-def find_mm_token_lengths(mm_data: Dict[str, Any],-                          input_processor: Any) -> List[int]:-    """Get multimodal token lengths from multimodal data items. """+def find_mm_token_lengths(mm_data: Dict[str, Any],+                          input_processor: Any) -> Dict[str, List[int]]:+    """Get per-modality multimodal token lengths from multimodal data items."""@@-        if not hasattr(input_processor, f"get_num_tokens_per_{modality}"):-            raise AttributeError(-                f"Input processor {type(input_processor).__name__} does not have 'get_num_tokens_per_{modality}' method required for multimodal hashing."-            )+        if not hasattr(input_processor, f"get_num_tokens_per_{modality}"):+            raise AttributeError(+                f"Input processor {type(input_processor).__name__} does not have "+                f\"get_num_tokens_per_{modality}\" required for multimodal hashing."+            )@@-            if modality == "image":-                if isinstance(item, torch.Tensor):-                    item = ToPILImage()(item)-                num_tokens = input_processor.get_num_tokens_per_image(-                    image_width=item.width,-                    image_height=item.height,-                )+            if modality == "image":+                if isinstance(item, torch.Tensor):+                    h, w = int(item.shape[-2]), int(item.shape[-1])+                else:+                    w, h = item.width, item.height+                num_tokens = input_processor.get_num_tokens_per_image(+                    image_width=w,+                    image_height=h,+                )                 modality_token_lengths.append(num_tokens)             elif modality == "video":-                assert isinstance(item, list), "Video must be a list of frames"-                if isinstance(item[0], torch.Tensor):-                    item = [ToPILImage()(frame) for frame in item]-                num_tokens = input_processor.get_num_tokens_per_video(-                    video_width=item[0].width,-                    video_height=item[0].height,-                    num_frames=len(item),-                )+                assert isinstance(item, list), "Video must be a list of frames"+                if not item:+                    raise ValueError("Video frame list is empty")+                if isinstance(item[0], torch.Tensor):+                    h, w = int(item[0].shape[-2]), int(item[0].shape[-1])+                else:+                    w, h = item[0].width, item[0].height+                num_tokens = input_processor.get_num_tokens_per_video(+                    video_width=w,+                    video_height=h,+                    num_frames=len(item),+                )                 modality_token_lengths.append(num_tokens)@@-    return num_mm_tokens  # flatten all mm instances to a single list+    return num_mm_tokens  # mapping: modality -> list of lengths

Also applies to: 493-496, 501-518, 521-521

---

423-430:Harden tensor serialization in serialize_item.

Calling .numpy() on non-CPU tensors raises; also ensure contiguity. Update the tensor branch accordingly.

defserialize_item(obj:object)->bytes:    ...ifisinstance(obj,torch.Tensor):t=obj.detach()ift.is_sparse:t=t.coalesce().to_dense()ift.device.type!="cpu":t=t.to("cpu")returnt.contiguous().numpy().tobytes()

---

482-484:Updatefind_mm_token_lengths signature and docstring

• Change the return annotation intensorrt_llm/inputs/multimodal.py from-> List[int] to-> Dict[str, List[int]] and revise its docstring to describe the mapping of modality names to token-length lists.
• Only one call site remains (intensorrt_llm/inputs/registry.py at lines 424–426), which already unpacks the dict vianext(iter(...)).

tensorrt_llm/inputs/registry.py (1)

311-331:Fix return types: methods promise Tuple[str, ...] but return generators.

Wrap the generator expressions with tuple().

     def get_registered_image_model_types(self) -> Tuple[str, ...]:-        return (+        return tuple(             model_type             for model_type in self._multimodal_placeholder_by_model_type             if "image" in self.             _multimodal_placeholder_by_model_type[model_type].placeholder_map)@@     def get_registered_video_model_types(self) -> Tuple[str, ...]:-        return (+        return tuple(             model_type             for model_type in self._multimodal_placeholder_by_model_type             if "video" in self.             _multimodal_placeholder_by_model_type[model_type].placeholder_map)@@     def get_registered_audio_model_types(self) -> Tuple[str, ...]:-        return (+        return tuple(             model_type             for model_type in self._multimodal_placeholder_by_model_type             if "audio" in self.             _multimodal_placeholder_by_model_type[model_type].placeholder_map)

tensorrt_llm/_torch/models/modeling_qwen2vl.py (6)

1-1:Add the required NVIDIA 2025 Apache-2.0 header block.

Per repository guidelines, prepend the license header to this Python source file.

Apply this diff:

+ # Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.+ # Licensed under the Apache License, Version 2.0 (the "License");+ # you may not use this file except in compliance with the License.+ # You may obtain a copy of the License at+ #     http://www.apache.org/licenses/LICENSE-2.0+ # Unless required by applicable law or agreed to in writing, software+ # distributed under the License is distributed on an "AS IS" BASIS,+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ # See the License for the specific language governing permissions and+ # limitations under the License.

---

224-226:Fix Python 3.8 typing: use typing.Dict/Any rather than PEP 585 builtins; also correctany toAny.

dict[str, any] is invalid on 3.8 andany is a function.

Apply:

-    def _preprocess(self, text: dict[str, any], mm_data: dict[str, any],+    def _preprocess(self, text: Dict[str, Any], mm_data: Dict[str, Any],                     mm_processor_kwargs: Dict[str, Any]):

---

252-258:Fix Python 3.8 typing in return annotation.

Replace PEP 585dict[...] withDict[...].

Apply:

-    def get_mrope_config(+    def get_mrope_config(             self,             input_ids: torch.IntTensor,             image_grid_thw: torch.LongTensor,             video_grid_thw: torch.LongTensor,             attention_mask: torch.Tensor,-            second_per_grid_ts: torch.Tensor = None) -> dict[str, torch.Tensor]:+            second_per_grid_ts: torch.Tensor = None) -> Dict[str, torch.Tensor]:

---

319-321:Fix Python 3.8 typing: useType[...] instead oftype[...].

Also importType from typing.

Apply:

-from typing import Any, Dict, List, Optional, Tuple, Union+from typing import Any, Dict, List, Optional, Tuple, Union, Type

-    def __init__(self, model_config: ModelConfig[PretrainedConfig],-                 model_class: type[PreTrainedModel]):+    def __init__(self, model_config: ModelConfig[PretrainedConfig],+                 model_class: Type[PreTrainedModel]):

---

123-131:Avoid shadowinginput_ids parameter; it leaks into later usage.

Inner loop reusesinput_ids, then Line 221 uses.device on the (now 1D) shadowed tensor. Rename local vars and reference the 2D tensor for device.

Apply:

-        for i, input_ids in enumerate(total_input_ids):-            input_ids = input_ids[attention_mask[i] == 1]+        for i, seq_input_ids in enumerate(total_input_ids):+            seq_input_ids = seq_input_ids[attention_mask[i] == 1]             image_nums, video_nums = 0, 0             vision_start_indices = torch.argwhere(-                input_ids == vision_start_token_id).squeeze(1)-            vision_tokens = input_ids[vision_start_indices + 1]+                seq_input_ids == vision_start_token_id).squeeze(1)+            vision_tokens = seq_input_ids[vision_start_indices + 1]             image_nums = (vision_tokens == image_token_id).sum()             video_nums = (vision_tokens == video_token_id).sum()-            input_tokens = input_ids.tolist()+            input_tokens = seq_input_ids.tolist()

-        mrope_position_deltas = torch.tensor(-            mrope_position_deltas, device=input_ids.device).unsqueeze(1)+        mrope_position_deltas = torch.tensor(+            mrope_position_deltas, device=total_input_ids.device).unsqueeze(1)

Also applies to: 220-221

---

246-250:Guard against missingvision_token_id; avoid AttributeError.

Use getattr and build the mask robustly.

Apply:

-        masks = (input_ids == self.model_config.image_token_id) | (-            input_ids == self.model_config.vision_token_id) | (-                input_ids == self.model_config.video_token_id)+        masks = (input_ids == self.model_config.image_token_id) | (+            input_ids == self.model_config.video_token_id)+        vision_token_id = getattr(self.model_config, "vision_token_id", None)+        if vision_token_id is not None:+            masks = masks | (input_ids == vision_token_id)