Movatterモバイル変換

[0]ホーム
URL:

Skip to main content

Advertisement

Springer Nature Link
Log in

Adversarial Uni- and Multi-modal Stream Networks for Multimodal Image Registration

  • Conference paper
  • First Online:

Abstract

Deformable image registration between Computed Tomography (CT) images and Magnetic Resonance (MR) imaging is essential for many image-guided therapies. In this paper, we propose a novel translation-based unsupervised deformable image registration method. Distinct from other translation-based methods that attempt to convert the multimodal problem (e.g., CT-to-MR) into a unimodal problem (e.g., MR-to-MR) via image-to-image translation, our method leverages the deformation fields estimated from both: (i) the translated MR image and (ii) the original CT image in a dual-stream fashion, and automatically learns how to fuse them to achieve better registration performance. The multimodal registration network can be effectively trained by computationally efficient similarity metrics without any ground-truth deformation. Our method has been evaluated on two clinical datasets and demonstrates promising results compared to state-of-the-art traditional and learning-based methods.

This is a preview of subscription content,log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 11439
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 14299
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide -see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Similar content being viewed by others

References

  1. Armanious, K., Jiang, C., Abdulatif, S., Küstner, T., Gatidis, S., Yang, B.: Unsupervised medical image translation using cycle-MedGAN. In: 2019 27th European Signal Processing Conference (EUSIPCO), pp. 1–5. IEEE (2019)

    Google Scholar 

  2. Avants, B.B., Epstein, C.L., Grossman, M., Gee, J.C.: Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med. Image Anal.12(1), 26–41 (2008)

    Article  Google Scholar 

  3. Avants, B.B., Tustison, N.J., Song, G., Cook, P.A., Klein, A., Gee, J.C.: A reproducible evaluation of ants similarity metric performance in brain image registration. NeuroImage54, 2033–2044 (2011)

    Article  Google Scholar 

  4. Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: An unsupervised learning model for deformable medical image registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9252–9260 (2018)

    Google Scholar 

  5. Cao, X., et al.: Deformable image registration based on similarity-steered CNN regression. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 300–308. Springer, Cham (2017).https://doi.org/10.1007/978-3-319-66182-7_35

    Chapter  Google Scholar 

  6. Cao, X., Yang, J., Wang, L., Xue, Z., Wang, Q., Shen, D.: Deep learning based inter-modality image registration supervised by intra-modality similarity. In: Shi, Y., Suk, H.-I., Liu, M. (eds.) MLMI 2018. LNCS, vol. 11046, pp. 55–63. Springer, Cham (2018).https://doi.org/10.1007/978-3-030-00919-9_7

    Chapter  Google Scholar 

  7. Dice, L.R.: Measures of the amount of ecologic association between species. Ecology26(3), 297–302 (1945)

    Article  Google Scholar 

  8. Heinrich, M.P., et al.: Mind: modality independent neighbourhood descriptor for multi-modal deformable registration. Med. Image Anal.16(7), 1423–1435 (2012)

    Article  Google Scholar 

  9. Hiasa, Y., et al.: Cross-modality image synthesis from unpaired data using CycleGAN. In: Gooya, A., Goksel, O., Oguz, I., Burgos, N. (eds.) SASHIMI 2018. LNCS, vol. 11037, pp. 31–41. Springer, Cham (2018).https://doi.org/10.1007/978-3-030-00536-8_4

    Chapter  Google Scholar 

  10. Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369. IEEE (2010)

    Google Scholar 

  11. Hu, X., Kang, M., Huang, W., Scott, M.R., Wiest, R., Reyes, M.: Dual-stream pyramid registration network. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 382–390. Springer, Cham (2019).https://doi.org/10.1007/978-3-030-32245-8_43

    Chapter  Google Scholar 

  12. Hu, Y., et al.: Label-driven weakly-supervised learning for multimodal deformable image registration. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 1070–1074. IEEE (2018)

    Google Scholar 

  13. Hu, Y., et al.: Weakly-supervised convolutional neural networks for multimodal image registration. Med. Image Anal.49, 1–13 (2018)

    Article  Google Scholar 

  14. Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: Advances in Neural Information Processing Systems, pp. 2017–2025 (2015)

    Google Scholar 

  15. Kuang, D., Schmah, T.: FAIM – a ConvNet method for unsupervised 3D medical image registration. In: Suk, H.-I., Liu, M., Yan, P., Lian, C. (eds.) MLMI 2019. LNCS, vol. 11861, pp. 646–654. Springer, Cham (2019).https://doi.org/10.1007/978-3-030-32692-0_74

    Chapter  Google Scholar 

  16. Liu, C., Ma, L., Lu, Z., Jin, X., Xu, J.: Multimodal medical image registration via common representations learning and differentiable geometric constraints. Electron. Lett.55(6), 316–318 (2019)

    Article  Google Scholar 

  17. Mahapatra, D., Ge, Z., Sedai, S., Chakravorty, R.: Joint registration and segmentation of Xray images using generative adversarial networks. In: Shi, Y., Suk, H.-I., Liu, M. (eds.) MLMI 2018. LNCS, vol. 11046, pp. 73–80. Springer, Cham (2018).https://doi.org/10.1007/978-3-030-00919-9_9

    Chapter  Google Scholar 

  18. Qin, C., Shi, B., Liao, R., Mansi, T., Rueckert, D., Kamen, A.: Unsupervised deformable registration for multi-modal images via disentangled representations. In: Chung, A.C.S., Gee, J.C., Yushkevich, P.A., Bao, S. (eds.) IPMI 2019. LNCS, vol. 11492, pp. 249–261. Springer, Cham (2019).https://doi.org/10.1007/978-3-030-20351-1_19

    Chapter  Google Scholar 

  19. Sedghi, A., et al.: Semi-supervised image registration using deep learning. In: Medical Imaging 2019: Image-Guided Procedures, Robotic Interventions, and Modeling, vol. 10951, p. 109511G. International Society for Optics and Photonics (2019)

    Google Scholar 

  20. Tanner, C., Ozdemir, F., Profanter, R., Vishnevsky, V., Konukoglu, E., Goksel, O.: Generative adversarial networks for mr-ct deformable image registration. arXiv preprintarXiv:1807.07349 (2018)

  21. de Vos, B.D., Berendsen, F.F., Viergever, M.A., Sokooti, H., Staring, M., Išgum, I.: A deep learning framework for unsupervised affine and deformable image registration. Med. Image Anal.52, 128–143 (2019)

    Article  Google Scholar 

  22. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process.13(4), 600–612 (2004)

    Article  Google Scholar 

  23. Wei, D., et al.: Synthesis and inpainting-based MR-CT registration for image-guided thermal ablation of liver tumors. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11768, pp. 512–520. Springer, Cham (2019).https://doi.org/10.1007/978-3-030-32254-0_57

    Chapter  Google Scholar 

  24. West, J.B., et al.: Comparison and evaluation of retrospective intermodality image registration techniques. In: Medical Imaging 1996: Image Processing, vol. 2710, pp. 332–347. International Society for Optics and Photonics (1996)

    Google Scholar 

  25. Yang, H., et al.: Unpaired brain MR-to-CT synthesis using a structure-constrained CycleGAN. In: Stoyanov, D., et al. (eds.) DLMIA/ML-CDS -2018. LNCS, vol. 11045, pp. 174–182. Springer, Cham (2018).https://doi.org/10.1007/978-3-030-00889-5_20

    Chapter  Google Scholar 

  26. Zhao, S., Lau, T., Luo, J., Eric, I., Chang, C., Xu, Y.: Unsupervised 3D end-to-end medical image registration with volume tweening network. IEEE J. Biomed. Health Inform. (2019)

    Google Scholar 

  27. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Cyclegan (2017).https://github.com/xhujoy/CycleGAN-tensorflow

  28. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)

    Google Scholar 

Download references

Acknowledgement

This project was supported by the National Institutes of Health (Grant No. R01EB025964, R01DK119269, and P41EB015898) and the Overseas Cooperation Research Fund of Tsinghua Shenzhen International Graduate School (Grant No. HW2018008).

Author information

Authors and Affiliations

  1. Shenzhen International Graduate School, Tsinghua University, Shenzhen, China

    Zhe Xu, Jiangpeng Yan & Xiu Li

  2. Brigham and Women’s Hospital, Harvard Medical School, Boston, USA

    Zhe Xu, Jie Luo, Ritvik Pulya, William Wells III & Jayender Jagadeesan

  3. Graduate School of Frontier Sciences, The University of Tokyo, Tokyo, Japan

    Jie Luo

Authors
  1. Zhe Xu

    You can also search for this author inPubMed Google Scholar

  2. Jie Luo

    You can also search for this author inPubMed Google Scholar

  3. Jiangpeng Yan

    You can also search for this author inPubMed Google Scholar

  4. Ritvik Pulya

    You can also search for this author inPubMed Google Scholar

  5. Xiu Li

    You can also search for this author inPubMed Google Scholar

  6. William Wells III

    You can also search for this author inPubMed Google Scholar

  7. Jayender Jagadeesan

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence toJayender Jagadeesan.

Editor information

Editors and Affiliations

  1. University of Toronto, Toronto, ON, Canada

    Anne L. Martel

  2. The University of British Columbia, Vancouver, BC, Canada

    Purang Abolmaesumi

  3. University College London, London, UK

    Danail Stoyanov

  4. École Centrale de Nantes, Nantes, France

    Diana Mateus

  5. EURECOM, Biot, France

    Maria A. Zuluaga

  6. Chinese Academy of Sciences, Beijing, China

    S. Kevin Zhou

  7. Sorbonne University, Paris, France

    Daniel Racoceanu

  8. The Hebrew University of Jerusalem, Jerusalem, Israel

    Leo Joskowicz

Rights and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, Z.et al. (2020). Adversarial Uni- and Multi-modal Stream Networks for Multimodal Image Registration. In: Martel, A.L.,et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12263. Springer, Cham. https://doi.org/10.1007/978-3-030-59716-0_22

Download citation

Publish with us

Societies and partnerships

Access this chapter

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 11439
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 14299
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide -see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

[8]ページ先頭
©2009-2025 Movatter.jp