Movatterモバイル変換


[0]ホーム

URL:


Skip to main content

Advertisement

Springer Nature Link
Log in

Segmentation by Registration-Enabled SAM Prompt Engineering Using Five Reference Images

  • Conference paper
  • First Online:

Abstract

The recently proposed Segment Anything Model (SAM) is a general tool for image segmentation, but it requires additional adaptation and careful fine-tuning for medical image segmentation, especially for small, irregularly-shaped, and boundary-ambiguous anatomical structures such as the knee cartilage that is of interest in this work. Repaired cartilage, after certain surgical procedures, exhibits imaging patterns unseen to pre-training, posing further challenges for using models like SAM with or without general-purpose fine-tuning. To address this, we propose a novel registration-based prompt engineering framework for medical image segmentation using SAM. This approach utilises established image registration algorithms to align the new image (to-be-segmented) and a small number of reference images, without requiring segmentation labels. The spatial transformations generated by registration align either the new image or pre-defined point-based prompts, before using them as input to SAM. This strategy, requiring as few as five reference images with defined point prompts, effectively prompts SAM for inference on new images, without needing any segmentation labels. Evaluation of MR images from patients who received cartilage stem cell therapy yielded Dice scores of 0.89, 0.87, 0.53, and 0.52 for segmenting femur, tibia, femoral- and tibial cartilages, respectively. This outperforms atlas-based label fusion and is comparable to supervised nnUNet, an upper-bound fair baseline in this application, both of which require full segmentation labels for reference samples. The codes are available at:https://github.com/chrissyinreallife/KneeSegmentWithSAM.git.

This is a preview of subscription content,log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 7550
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 9437
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide -see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Similar content being viewed by others

References

  1. Alirr, O.I., Rahni, A.A.A., Golkar, E.: An automated liver tumour segmentation from abdominal ct scans for hepatic surgical planning. IJCARS13, 1169–1176 (2018)

    Google Scholar 

  2. Amyar, A., Modzelewski, R., Li, H., Ruan, S.: Multi-task deep learning based ct imaging analysis for covid-19 pneumonia: Classification and segmentation. COMPUT BIOL MED126, 104037 (2020)

    Article  Google Scholar 

  3. Capobianco, E., Dominietto, M.: Assessment of brain cancer atlas maps with multimodal imaging features. J. Transl. Med.21(1), 1–11 (2023)

    Article  Google Scholar 

  4. Chalcroft, L.F., Qu, J., Martin, S.A., Gayo, I.J., Minore, G.V., Singh, I.R., et al.: Development and evaluation of intraoperative ultrasound segmentation with negative image frames and multiple observer labels. In: MICCAI Workshop 2021. pp. 25–34. Springer (2021)

    Google Scholar 

  5. Chimutengwende-Gordon, M., Ahmad, M.A., Bentley, G., Brammah, J., Carrington, R., Miles, et al.: Stem cell transplantation for the treatment of osteochondral defects of the knee: Operative technique for a single-stage transplantation procedure using bone marrow-derived mesenchymal stem cells. The Knee28, 400–409 (2021)

    Google Scholar 

  6. Costea, M., Zlate, A., Durand, M., Baudier, T., Grégoire, V., Sarrut, D., et al.: Comparison of atlas-based and deep learning methods for organs at risk delineation on head-and-neck ct images using an automated treatment planning system. Radiotherapy and Oncology177, 61–70 (2022)

    Article  Google Scholar 

  7. Czolbe, S., Arnavaz, K., Krause, O., Feragen, A.: Is segmentation uncertainty useful? In: IPMI 2021. pp. 715–726. Springer (2021)

    Google Scholar 

  8. Feyjie, A.R., Azad, R., Pedersoli, M., Kauffman, C., Ayed, I.B., Dolz, J.: Semi-supervised few-shot learning for medical image segmentation. arXiv preprintarXiv:2003.08462 (2020)

  9. Haque, I.R.I., Neubert, J.: Deep learning approaches to biomedical image segmentation. IMU18, 100297 (2020)

    Google Scholar 

  10. Heimann, T., Morrison, B.J., Styner, M.A., Niethammer, M., Warfield, S.: Segmentation of knee images: a grand challenge. In: Proc. MICCAI Workshop on Medical Image Analysis for the Clinic. vol. 1. Beijing, China (2010)

    Google Scholar 

  11. Huang, J., Jiang, K., Zhang, J., Qiu, H., Lu, L., Lu, S., et al.: Learning to prompt segment anything models. arXiv preprintarXiv:2401.04651 (2024)

  12. Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods18(2), 203–211 (2021)

    Article  Google Scholar 

  13. Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., et al.: Segment anything. arXiv preprintarXiv:2304.02643 (2023)

  14. Li, Y., Fu, Y., Gayo, I.J., Yang, Q., Min, Z., Saeed, S.U., et al.: Prototypical few-shot segmentation for cross-institution male pelvic structures with spatial registration. Med. Image Anal.90, 102935 (2023)

    Article  Google Scholar 

  15. Litjens, G., Toth, R., Van De Ven, W., Hoeks, C., Kerkstra, S., Van Ginneken, B., et al.: Evaluation of prostate segmentation algorithms for mri: the promise12 challenge. Med. Image Anal.18(2), 359–373 (2014)

    Article  Google Scholar 

  16. Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications15(1),  654 (2024)

    Article  Google Scholar 

  17. Malhotra, P., Gupta, S., Koundal, D., Zaguia, A., Enbeyle, W., et al.: Deep neural networks for medical image segmentation. J. Healthc. Eng.2022 (2022)

    Google Scholar 

  18. Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 3D vision 2016. pp. 565–571. Ieee (2016)

    Google Scholar 

  19. Modat, M., Ridgway, G.R., Taylor, Z.A., Lehmann, M., Barnes, J., Hawkes, D.J., et al.: Fast free-form deformation using graphics processing units. Comput Methods Programs Biomed98(3), 278–284 (2010)

    Article  Google Scholar 

  20. Montaña-Brown, N., Ramalhinho, J., Allam, M., Davidson, B., Hu, Y., Clarkson, M.J.: Vessel segmentation for automatic registration of untracked laparoscopic ultrasound to ct of the liver. IJCARS16(7), 1151–1160 (2021)

    Google Scholar 

  21. Nguyen, N.T., Le, P.B.: Topological voting method for image segmentation. Journal of Imaging8(2),  16 (2022)

    Article  Google Scholar 

  22. Rohlfing, T., Brandt, R., Menzel, R., Russakoff, D.B., Maurer Jr, C.R.: Quo vadis, atlas-based segmentation? In: Handbook of Biomedical Image Analysis: Volume III: Registration Models, pp. 435–486. Springer (2005)

    Google Scholar 

  23. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI 2015. pp. 234–241. Springer (2015)

    Google Scholar 

  24. Rueckert, D., Sonoda, L.I., Hayes, C., Hill, D.L., Leach, M.O., Hawkes, D.J.: Nonrigid registration using free-form deformations: application to breast mr images. IEEE transactions on medical imaging18(8), 712–721 (1999)

    Article  Google Scholar 

  25. Saeed, S.U., Fu, Y., Stavrinides, V., Baum, Z.M., Yang, Q., Rusu, M., , et al.: Image quality assessment for machine learning tasks using meta-reinforcement learning. Med. Image Anal.78, 102427 (2022)

    Article  Google Scholar 

  26. Saeed, S.U., Yan, W., Fu, Y., Giganti, F., Yang, Q., Baum, Z., et al.: Image quality assessment by overlapping task-specific and task-agnostic measures: application to prostate multiparametric mr images for cancer segmentation. arXiv preprintarXiv:2202.09798 (2022)

  27. Schreiner, M.M., Raudner, M., Marlovits, S., Bohndorf, K., Weber, M., Zalaudek, M., et al.: The mocart (magnetic resonance observation of cartilage repair tissue) 2.0 knee score and atlas. Cartilage13(1_suppl), 571S–587S (2021)

    Google Scholar 

  28. Shan, L., Zach, C., Charles, C., Niethammer, M.: Automatic atlas-based three-label cartilage segmentation from mr knee images. Med. Image Anal.18(7), 1233–1246 (2014)

    Article  Google Scholar 

  29. Siddique, N., Paheding, S., Elkin, C.P., Devabhaktuni, V.: U-net and its variants for medical image segmentation: A review of theory and applications. Ieee Access9, 82031–82057 (2021)

    Article  Google Scholar 

  30. Tang, H., Liu, X., Sun, S., Yan, X., Xie, X.: Recurrent mask refinement for few-shot medical image segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 3918–3928 (2021)

    Google Scholar 

  31. Thippeswamy, P.B., Nedunchelian, M., Rajasekaran, R.B., Riley, D., Khatkar, H., Rajasekaran, et al.: Updates in postoperative imaging modalities following musculoskeletal surgery. JCOT22, 101616 (2021)

    Google Scholar 

  32. Vorontsov, E., Tang, A., Pal, C., Kadoury, S.: Liver lesion segmentation informed by joint liver segmentation. In: ISBI 2018. pp. 1332–1335. IEEE (2018)

    Google Scholar 

  33. Wang, S., Yang, D.M., Rong, R., Zhan, X., Xiao, G.: Pathology image analysis using segmentation deep learning algorithms. AJP189(9), 1686–1698 (2019)

    Google Scholar 

  34. Wang, Y., Yao, Q., Kwok, J.T., Ni, L.M.: Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur)53(3), 1–34 (2020)

    Article  Google Scholar 

Download references

Acknowledgement

This work was supported in part by EPSRC [EP/T029404/1], Wellcome/EPSRC Centre for Interventional and Surgical Sciences [203145Z/16/Z] and the International Alliance for Cancer Early Detection, a partnership between Cancer Research UK [C73666/A31378], Canary Center at Stanford University, the University of Cambridge, OHSU Knight Cancer Institute, University College London and the University of Manchester.

Author information

Authors and Affiliations

  1. Mechanical Engineering Department, University College London, London, UK

    Yaxi Chen & Jie Huang

  2. Institute of Orthopaedic and Musculoskeletal Science, University College London, London, UK

    Aleksandra Ivanova & Chaozong Liu

  3. Centre for Medical Image Computing, Wellcome/EPSRC Centre for Interventional and Surgical Sciences, Department of Medical Physics and Biomedical Engineering, University College London, London, UK

    Shaheer U. Saeed & Yipeng Hu

  4. Royal National Orthopaedic Hospital, Stanmore, UK

    Rikin Hargunani & Chaozong Liu

Authors
  1. Yaxi Chen

    You can also search for this author inPubMed Google Scholar

  2. Aleksandra Ivanova

    You can also search for this author inPubMed Google Scholar

  3. Shaheer U. Saeed

    You can also search for this author inPubMed Google Scholar

  4. Rikin Hargunani

    You can also search for this author inPubMed Google Scholar

  5. Jie Huang

    You can also search for this author inPubMed Google Scholar

  6. Chaozong Liu

    You can also search for this author inPubMed Google Scholar

  7. Yipeng Hu

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence toYipeng Hu.

Editor information

Editors and Affiliations

  1. King's College London, London, UK

    Marc Modat

  2. University of Sussex, Brighton, UK

    Ivor Simpson

  3. University of Ljubljana, Ljubljana, Slovenia

    Žiga Špiclin

  4. Erasmus MC, Rotterdam, The Netherlands

    Wietske Bastiaansen

  5. Radboud University Medical Center, Nijmegen, The Netherlands

    Alessa Hering

  6. Hong Kong University of Science and Technology, Hong Kong, China

    Tony C. W. Mok

Rights and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, Y.et al. (2024). Segmentation by Registration-Enabled SAM Prompt Engineering Using Five Reference Images. In: Modat, M., Simpson, I., Špiclin, Ž., Bastiaansen, W., Hering, A., Mok, T.C.W. (eds) Biomedical Image Registration. WBIR 2024. Lecture Notes in Computer Science, vol 15249. Springer, Cham. https://doi.org/10.1007/978-3-031-73480-9_19

Download citation

Publish with us

Societies and partnerships

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 7550
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 9437
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide -see info

Tax calculation will be finalised at checkout

Purchases are for personal use only


[8]ページ先頭

©2009-2025 Movatter.jp