Repository files navigation

This is the repository of our accepted CVPR-2024paper forDEF-AI-MIA Workshop.

This code has been developed by adapting the GitHub repohttps://github.com/MedicineToken/Medical-SAM-Adapter fromJunde Wu (thanks a lot for your amazing paper ❤️) in order to optimize the network for brain glioma segmentation. Instructions to download the data, set the environment and train the architecture can be found in the documentINSTRUCTIONS.md.

We address in our study the primary challenge of adapting SAM for mp-MRI brain scans, which typically encompass multiple MRI modalities not fully utilized by standard three-channel vision models. We demonstrate that leveraging all available MRI modalities achieves superior performance compared to the standard mechanism of repeating a MRI scan to fit the input embedding. Furthermore, we incorporate Parameter Efficient Fine-Tuning (PEFT) through LoRA blocks to solve the lack of SAM's medical specific knowledge.

We propose to adapt the encoder by: 1) accounting for all the mp-MRI volumetric image modalities; and 2) specifically tuning of the encoder to retain the open-world segmentation capabilities of SAM.

We propose to modify the patch embedding layer, so that it accounts for the all the MRI modalities, allowing for a seamless integration of the information. Then, we employ LoRAs to tune Multi Layer Perceptron blocks (MLP) and Attention (Q,K,V embedding) layers of thetransformer blocks.

@INPROCEEDINGS{10678163,  author={Diana-Albelda, Cecilia and Alcover-Couso, Roberto and García-Martín, Álvaro and Bescos, Jesus},  booktitle={2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},   title={How SAM Perceives Different mp-MRI Brain Tumor Domains?},   year={2024},  volume={},  number={},  pages={4959-4970},  doi={10.1109/CVPRW63382.2024.00501}}