- Yaxi Chen ORCID:orcid.org/0009-0007-5906-899X13,
- Aleksandra Ivanova ORCID:orcid.org/0009-0000-4113-892814,
- Shaheer U. Saeed ORCID:orcid.org/0000-0002-5004-066315,
- Rikin Hargunani ORCID:orcid.org/0000-0002-0953-844316,
- Jie Huang ORCID:orcid.org/0000-0001-7951-221713,
- Chaozong Liu ORCID:orcid.org/0000-0002-9854-404314,16 &
- …
- Yipeng Hu ORCID:orcid.org/0000-0003-4902-048615
Part of the book series:Lecture Notes in Computer Science ((LNCS,volume 15249))
Included in the following conference series:
394Accesses
Abstract
The recently proposed Segment Anything Model (SAM) is a general tool for image segmentation, but it requires additional adaptation and careful fine-tuning for medical image segmentation, especially for small, irregularly-shaped, and boundary-ambiguous anatomical structures such as the knee cartilage that is of interest in this work. Repaired cartilage, after certain surgical procedures, exhibits imaging patterns unseen to pre-training, posing further challenges for using models like SAM with or without general-purpose fine-tuning. To address this, we propose a novel registration-based prompt engineering framework for medical image segmentation using SAM. This approach utilises established image registration algorithms to align the new image (to-be-segmented) and a small number of reference images, without requiring segmentation labels. The spatial transformations generated by registration align either the new image or pre-defined point-based prompts, before using them as input to SAM. This strategy, requiring as few as five reference images with defined point prompts, effectively prompts SAM for inference on new images, without needing any segmentation labels. Evaluation of MR images from patients who received cartilage stem cell therapy yielded Dice scores of 0.89, 0.87, 0.53, and 0.52 for segmenting femur, tibia, femoral- and tibial cartilages, respectively. This outperforms atlas-based label fusion and is comparable to supervised nnUNet, an upper-bound fair baseline in this application, both of which require full segmentation labels for reference samples. The codes are available at:https://github.com/chrissyinreallife/KneeSegmentWithSAM.git.
This is a preview of subscription content,log in via an institution to check access.
Access this chapter
Subscribe and save
- Get 10 units per month
- Download Article/Chapter or eBook
- 1 Unit = 1 Article or 1 Chapter
- Cancel anytime
Buy Now
- Chapter
- JPY 3498
- Price includes VAT (Japan)
- eBook
- JPY 7550
- Price includes VAT (Japan)
- Softcover Book
- JPY 9437
- Price includes VAT (Japan)
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alirr, O.I., Rahni, A.A.A., Golkar, E.: An automated liver tumour segmentation from abdominal ct scans for hepatic surgical planning. IJCARS13, 1169–1176 (2018)
Amyar, A., Modzelewski, R., Li, H., Ruan, S.: Multi-task deep learning based ct imaging analysis for covid-19 pneumonia: Classification and segmentation. COMPUT BIOL MED126, 104037 (2020)
Capobianco, E., Dominietto, M.: Assessment of brain cancer atlas maps with multimodal imaging features. J. Transl. Med.21(1), 1–11 (2023)
Chalcroft, L.F., Qu, J., Martin, S.A., Gayo, I.J., Minore, G.V., Singh, I.R., et al.: Development and evaluation of intraoperative ultrasound segmentation with negative image frames and multiple observer labels. In: MICCAI Workshop 2021. pp. 25–34. Springer (2021)
Chimutengwende-Gordon, M., Ahmad, M.A., Bentley, G., Brammah, J., Carrington, R., Miles, et al.: Stem cell transplantation for the treatment of osteochondral defects of the knee: Operative technique for a single-stage transplantation procedure using bone marrow-derived mesenchymal stem cells. The Knee28, 400–409 (2021)
Costea, M., Zlate, A., Durand, M., Baudier, T., Grégoire, V., Sarrut, D., et al.: Comparison of atlas-based and deep learning methods for organs at risk delineation on head-and-neck ct images using an automated treatment planning system. Radiotherapy and Oncology177, 61–70 (2022)
Czolbe, S., Arnavaz, K., Krause, O., Feragen, A.: Is segmentation uncertainty useful? In: IPMI 2021. pp. 715–726. Springer (2021)
Feyjie, A.R., Azad, R., Pedersoli, M., Kauffman, C., Ayed, I.B., Dolz, J.: Semi-supervised few-shot learning for medical image segmentation. arXiv preprintarXiv:2003.08462 (2020)
Haque, I.R.I., Neubert, J.: Deep learning approaches to biomedical image segmentation. IMU18, 100297 (2020)
Heimann, T., Morrison, B.J., Styner, M.A., Niethammer, M., Warfield, S.: Segmentation of knee images: a grand challenge. In: Proc. MICCAI Workshop on Medical Image Analysis for the Clinic. vol. 1. Beijing, China (2010)
Huang, J., Jiang, K., Zhang, J., Qiu, H., Lu, L., Lu, S., et al.: Learning to prompt segment anything models. arXiv preprintarXiv:2401.04651 (2024)
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods18(2), 203–211 (2021)
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., et al.: Segment anything. arXiv preprintarXiv:2304.02643 (2023)
Li, Y., Fu, Y., Gayo, I.J., Yang, Q., Min, Z., Saeed, S.U., et al.: Prototypical few-shot segmentation for cross-institution male pelvic structures with spatial registration. Med. Image Anal.90, 102935 (2023)
Litjens, G., Toth, R., Van De Ven, W., Hoeks, C., Kerkstra, S., Van Ginneken, B., et al.: Evaluation of prostate segmentation algorithms for mri: the promise12 challenge. Med. Image Anal.18(2), 359–373 (2014)
Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15(1), 654 (2024)
Malhotra, P., Gupta, S., Koundal, D., Zaguia, A., Enbeyle, W., et al.: Deep neural networks for medical image segmentation. J. Healthc. Eng.2022 (2022)
Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 3D vision 2016. pp. 565–571. Ieee (2016)
Modat, M., Ridgway, G.R., Taylor, Z.A., Lehmann, M., Barnes, J., Hawkes, D.J., et al.: Fast free-form deformation using graphics processing units. Comput Methods Programs Biomed98(3), 278–284 (2010)
Montaña-Brown, N., Ramalhinho, J., Allam, M., Davidson, B., Hu, Y., Clarkson, M.J.: Vessel segmentation for automatic registration of untracked laparoscopic ultrasound to ct of the liver. IJCARS16(7), 1151–1160 (2021)
Nguyen, N.T., Le, P.B.: Topological voting method for image segmentation. Journal of Imaging8(2), 16 (2022)
Rohlfing, T., Brandt, R., Menzel, R., Russakoff, D.B., Maurer Jr, C.R.: Quo vadis, atlas-based segmentation? In: Handbook of Biomedical Image Analysis: Volume III: Registration Models, pp. 435–486. Springer (2005)
Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI 2015. pp. 234–241. Springer (2015)
Rueckert, D., Sonoda, L.I., Hayes, C., Hill, D.L., Leach, M.O., Hawkes, D.J.: Nonrigid registration using free-form deformations: application to breast mr images. IEEE transactions on medical imaging18(8), 712–721 (1999)
Saeed, S.U., Fu, Y., Stavrinides, V., Baum, Z.M., Yang, Q., Rusu, M., , et al.: Image quality assessment for machine learning tasks using meta-reinforcement learning. Med. Image Anal.78, 102427 (2022)
Saeed, S.U., Yan, W., Fu, Y., Giganti, F., Yang, Q., Baum, Z., et al.: Image quality assessment by overlapping task-specific and task-agnostic measures: application to prostate multiparametric mr images for cancer segmentation. arXiv preprintarXiv:2202.09798 (2022)
Schreiner, M.M., Raudner, M., Marlovits, S., Bohndorf, K., Weber, M., Zalaudek, M., et al.: The mocart (magnetic resonance observation of cartilage repair tissue) 2.0 knee score and atlas. Cartilage13(1_suppl), 571S–587S (2021)
Shan, L., Zach, C., Charles, C., Niethammer, M.: Automatic atlas-based three-label cartilage segmentation from mr knee images. Med. Image Anal.18(7), 1233–1246 (2014)
Siddique, N., Paheding, S., Elkin, C.P., Devabhaktuni, V.: U-net and its variants for medical image segmentation: A review of theory and applications. Ieee Access9, 82031–82057 (2021)
Tang, H., Liu, X., Sun, S., Yan, X., Xie, X.: Recurrent mask refinement for few-shot medical image segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 3918–3928 (2021)
Thippeswamy, P.B., Nedunchelian, M., Rajasekaran, R.B., Riley, D., Khatkar, H., Rajasekaran, et al.: Updates in postoperative imaging modalities following musculoskeletal surgery. JCOT22, 101616 (2021)
Vorontsov, E., Tang, A., Pal, C., Kadoury, S.: Liver lesion segmentation informed by joint liver segmentation. In: ISBI 2018. pp. 1332–1335. IEEE (2018)
Wang, S., Yang, D.M., Rong, R., Zhan, X., Xiao, G.: Pathology image analysis using segmentation deep learning algorithms. AJP189(9), 1686–1698 (2019)
Wang, Y., Yao, Q., Kwok, J.T., Ni, L.M.: Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur)53(3), 1–34 (2020)
Acknowledgement
This work was supported in part by EPSRC [EP/T029404/1], Wellcome/EPSRC Centre for Interventional and Surgical Sciences [203145Z/16/Z] and the International Alliance for Cancer Early Detection, a partnership between Cancer Research UK [C73666/A31378], Canary Center at Stanford University, the University of Cambridge, OHSU Knight Cancer Institute, University College London and the University of Manchester.
Author information
Authors and Affiliations
Mechanical Engineering Department, University College London, London, UK
Yaxi Chen & Jie Huang
Institute of Orthopaedic and Musculoskeletal Science, University College London, London, UK
Aleksandra Ivanova & Chaozong Liu
Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional and Surgical Sciences, Department of Medical Physics and Biomedical Engineering, University College London, London, UK
Shaheer U. Saeed & Yipeng Hu
Royal National Orthopaedic Hospital, Stanmore, UK
Rikin Hargunani & Chaozong Liu
- Yaxi Chen
You can also search for this author inPubMed Google Scholar
- Aleksandra Ivanova
You can also search for this author inPubMed Google Scholar
- Shaheer U. Saeed
You can also search for this author inPubMed Google Scholar
- Rikin Hargunani
You can also search for this author inPubMed Google Scholar
- Jie Huang
You can also search for this author inPubMed Google Scholar
- Chaozong Liu
You can also search for this author inPubMed Google Scholar
- Yipeng Hu
You can also search for this author inPubMed Google Scholar
Corresponding author
Correspondence toYipeng Hu.
Editor information
Editors and Affiliations
King's College London, London, UK
Marc Modat
University of Sussex, Brighton, UK
Ivor Simpson
University of Ljubljana, Ljubljana, Slovenia
Žiga Špiclin
Erasmus MC, Rotterdam, The Netherlands
Wietske Bastiaansen
Radboud University Medical Center, Nijmegen, The Netherlands
Alessa Hering
Hong Kong University of Science and Technology, Hong Kong, China
Tony C. W. Mok
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, Y.et al. (2024). Segmentation by Registration-Enabled SAM Prompt Engineering Using Five Reference Images. In: Modat, M., Simpson, I., Špiclin, Ž., Bastiaansen, W., Hering, A., Mok, T.C.W. (eds) Biomedical Image Registration. WBIR 2024. Lecture Notes in Computer Science, vol 15249. Springer, Cham. https://doi.org/10.1007/978-3-031-73480-9_19
Download citation
Published:
Publisher Name:Springer, Cham
Print ISBN:978-3-031-73479-3
Online ISBN:978-3-031-73480-9
eBook Packages:Computer ScienceComputer Science (R0)
Share this paper
Anyone you share the following link with will be able to read this content:
Sorry, a shareable link is not currently available for this article.
Provided by the Springer Nature SharedIt content-sharing initiative