tensorrt_llm/_torch/models/modeling_phi4mm.py (1)

90-95:Add error handling for tensor-to-PIL conversion.

The conversion from normalized tensors to PIL images lacks error handling. Invalid tensor shapes or values could cause runtime errors.

         if images is not None:             if isinstance(images[0], torch.Tensor):                 # Convert normalized tensors (0-1) to PIL images (0-255).-                images = [-                    Image.fromarray((image.permute(1, 2, 0) * 255).to(-                        torch.uint8).cpu().numpy()) for image in images-                ]+                try:+                    converted_images = []+                    for image in images:+                        if image.dim() != 3 or image.shape[0] not in [1, 3, 4]:+                            raise ValueError(f"Invalid image tensor shape: {image.shape}")+                        img_array = (image.permute(1, 2, 0) * 255).clamp(0, 255).to(torch.uint8).cpu().numpy()+                        converted_images.append(Image.fromarray(img_array))+                    images = converted_images+                except Exception as e:+                    raise ValueError(f"Failed to convert tensor to PIL image: {e}")

tensorrt_llm/_torch/models/modeling_phi4mm.py (4)

3-3:Update or remove the TODO comment.

Since this PR reverts the aggregate mode support that was just implemented, this TODO comment about implementing aggregate mode is now outdated and misleading. Either update it to reflect the current state or remove it.

-# (todo) step 2: refactor the inference pipeline to use AGGREGATE mode (https://github.com/NVIDIA/TensorRT-LLM/pull/5522).+# (reverted) step 2: refactor the inference pipeline to use AGGREGATE mode - reverted in PR #6907

---

29-33:Document the workaround for transformers compatibility.

This custom PreTrainedModel class appears to be a workaround for a transformers version upgrade. Consider adding a more detailed docstring explaining why this is needed and when it can be removed.

 # Create a PreTrainedModel class for transformers=4.53.1 upgrade. # Core idea is to provide `prepare_inputs_for_generation` method from `GenerationMixin`. class NewPreTrainedModel(transformers.modeling_utils.PreTrainedModel,                          transformers.generation.GenerationMixin):+    """Temporary workaround for transformers 4.53.1 compatibility.++    This class bridges compatibility by combining PreTrainedModel with GenerationMixin+    to provide the prepare_inputs_for_generation method required by newer transformers versions.+    This can be removed once the minimum transformers version requirement is updated.+    """     pass

---

127-130:Verify mask application efficiency.

The multimodal token masking creates boolean masks for each token type and then combines them. For large sequences, this could be optimized.

-        image_token_mask = inputs['input_ids'] == _IMAGE_SPECIAL_TOKEN_ID-        audio_token_mask = inputs['input_ids'] == _AUDIO_SPECIAL_TOKEN_ID-        mm_token_mask = image_token_mask | audio_token_mask+        # Combine token ID checks in a single operation for better efficiency+        mm_token_mask = torch.isin(inputs['input_ids'], torch.tensor([_IMAGE_SPECIAL_TOKEN_ID, _AUDIO_SPECIAL_TOKEN_ID], device=inputs['input_ids'].device))

---

25-26:Document the special token IDs.

The special token IDs should have more detailed documentation explaining their purpose and relationship to the tokenizer vocabulary.

 # Special tokens-_IMAGE_SPECIAL_TOKEN_ID = 200010  # '<|endoftext10|>'-_AUDIO_SPECIAL_TOKEN_ID = 200011  # '<|endoftext11|>'+# Special tokens for multimodal inputs+# These IDs correspond to special tokens in the Phi4MM tokenizer vocabulary+_IMAGE_SPECIAL_TOKEN_ID = 200010  # '<|endoftext10|>' - Used to mark image positions in input+_AUDIO_SPECIAL_TOKEN_ID = 200011  # '<|endoftext11|>' - Used to mark audio positions in input

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these settings in your CodeRabbit configuration.

tensorrt_llm/_torch/models/modeling_phi4mm.py (1)

89-95:Potential IndexError when images is an empty list; also clamp before scaling.

images[0] will crash for[]. Clamp values to [0,1] before scaling to uint8.

-        if images is not None:-            if isinstance(images[0], torch.Tensor):+        if images:+            if isinstance(images[0], torch.Tensor):                 # Convert normalized tensors (0-1) to PIL images (0-255).-                images = [-                    Image.fromarray((image.permute(1, 2, 0) * 255).to(-                        torch.uint8).cpu().numpy()) for image in images-                ]+                images = [+                    Image.fromarray(+                        (image.permute(1, 2, 0).clamp(0, 1) * 255)+                        .to(torch.uint8)+                        .cpu()+                        .numpy()+                    )+                    for image in images+                ]

tensorrt_llm/_torch/models/modeling_phi4mm.py (3)

61-73:Global monkey-patching of transformers.PreTrainedModel is unsafe and not exception-safe.

This can cause race conditions and persistent global side-effects if an exception occurs during model loading.

Make the patching atomic and exception-safe with a lock and try/finally.

-        OldPreTrainedModel = transformers.modeling_utils.PreTrainedModel-        transformers.modeling_utils.PreTrainedModel = NewPreTrainedModel-        # TODO: Make separate Phi4VisionEncoder and Phi4AudioEncoder, and move them to LLM-side.-        ref_phi4mm_model = transformers.AutoModelForCausalLM.from_pretrained(-            model_path,-            trust_remote_code=True,-            # Flash_attn_2 only supports bf16 or fp16 and set in HF config.-            torch_dtype='auto',-            _attn_implementation='flash_attention_2',-        ).eval()-        transformers.modeling_utils.PreTrainedModel = OldPreTrainedModel+        _model_load_lock = threading.Lock()+        with _model_load_lock:+            OldPreTrainedModel = transformers.modeling_utils.PreTrainedModel+            try:+                transformers.modeling_utils.PreTrainedModel = NewPreTrainedModel+                # TODO: Make separate Phi4VisionEncoder and Phi4AudioEncoder, and move them to LLM-side.+                ref_phi4mm_model = transformers.AutoModelForCausalLM.from_pretrained(+                    model_path,+                    trust_remote_code=True,+                    # Flash_attn_2 only supports bf16 or fp16 and set in HF config.+                    torch_dtype='auto',+                    _attn_implementation='flash_attention_2',+                ).eval()+            finally:+                transformers.modeling_utils.PreTrainedModel = OldPreTrainedModel

Additionally, ensure the module imports threading:

# near other importsimportthreading

And validate expected attributes on the loaded model to fail fast:

         ref_phi4mm_model = transformers.AutoModelForCausalLM.from_pretrained(@@         ).eval()+        if not hasattr(ref_phi4mm_model.model, "embed_tokens_extend"):+            raise AttributeError("Loaded model is missing 'embed_tokens_extend'; ensure the Phi-4 MM variant is used.")

---

84-85:Remove unused variablemm_processor_kwargs.

This triggers Ruff F841 and is dead code.

-        text_prompt, mm_data, mm_processor_kwargs = inputs.get("prompt"), \-                        inputs.get("multi_modal_data", {}), inputs.get("mm_processor_kwargs", {})+        text_prompt = inputs.get("prompt")+        mm_data = inputs.get("multi_modal_data", {})

---

217-223:Validate multimodal data before access and surface clear errors.

Avoid AttributeError/KeyError by validating structure before indexing.

-        if len(multimodal_params) > 0:-            mm_embeds = [-                multimodal_param.multimodal_data["multimodal_embedding"]-                for multimodal_param in multimodal_params-            ]+        if len(multimodal_params) > 0:+            mm_embeds = []+            for idx, multimodal_param in enumerate(multimodal_params):+                if not hasattr(multimodal_param, 'multimodal_data') or not isinstance(multimodal_param.multimodal_data, dict):+                    raise ValueError(f"multimodal_param[{idx}] missing valid 'multimodal_data'")+                if "multimodal_embedding" not in multimodal_param.multimodal_data:+                    raise KeyError(f"multimodal_param[{idx}] missing 'multimodal_embedding'")+                mm_embeds.append(multimodal_param.multimodal_data["multimodal_embedding"])

tensorrt_llm/_torch/models/modeling_phi4mm.py (4)

98-101:Confirm AutoProcessor returns a .to()-capable object.

BatchFeature typically supports.to, but some processors can return Python lists for missing modalities. Add a defensive conversion for CPU fallback or missing keys.

If needed:

• Wrap withisinstance(inputs, transformers.feature_extraction_utils.BatchFeature) before calling.to.
• Alternatively, move tensors individually toself.device.

---

224-229:Possible dtype/device mismatch between input_ids and MM_TOKEN_IDS.

Ensureinput_ids dtype equalstorch.long and device matches the model’s device; otherwise== and indexing infuse_input_embeds can fail or trigger slow copies.

Consider normalizing upstream:

• Ensureinput_ids aretorch.long on the same device as embeddings before callingfuse_input_embeds.

---

24-26:Hardcoding special token IDs is brittle.

If upstream token IDs change, masking/fusion breaks. Prefer resolving from tokenizer/config with a runtime assert.

Example approach during InputProcessor init:

• Validate_IMAGE_SPECIAL_TOKEN_ID and_AUDIO_SPECIAL_TOKEN_ID equal tokenizer’s special token IDs (if available), otherwise warn or override via model config.

---

29-33:NewPreTrainedModel lacks documentation.

Add a short docstring explaining why this MRO shim is needed for the specific transformers version upgrade.

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these settings in your CodeRabbit configuration.